Bitswap retrieval

How to configure and use Bitswap retrievals in Boost

booster-bitswap is a binary that runs alongside the boostd process, to serve retrievals over the Bitswap protocol. This feature of boost provides a number of tools for managing a production grade Bitswap retrieval service for a Storage Provider's content.

There is currently no payment method in booster-bitswap. This endpoint is intended to serve free content.

Why enable retrievals via Bitswap?

Bitswap retrieval introduces interoperability between IPFS and Filecoin, as it enables clients to retrieve Filecoin data over IPFS. This expands the reach of the Filecoin network considerably, increasing the value proposition for users to store data on the Filecoin network. This benefits the whole community, including SPs. Users will be able to access data directly via IPFS, as well as benefit from retrieval markets (e.g. Saturn) and compute over data projects (e.g. Bacalhau).

booster-bitswap modes

There are two primary "modes" for exposing booster-bitswap to the internet.

1. In private mode the booster-bitswap peer ID is not publicly accessible to the internet. Instead, public Bitswap traffic goes to boostd itself, which then acts as a reverse proxy, forwarding that traffic on to booster-bitswap. This is similar to the way one might configure Nginx as a reverse proxy for an otherwise private web server. private mode is simpler to setup but may produce greater load on boostd as a protocol proxy.

2. In public mode the public internet firewall must be configured to forward traffic directly to the booster-bitswap instance. boostd is configured to announce the public address of booster-bitswap to the network indexer (the network indexer is the service that clients can query to discover where to retrieve content). This mode offers greater flexibility and performance. You can even setup booster-bitswap to run over a separate internet connection from boostd. However, it might require additional configuration and changes to your overall network infrastructure.

Bitswap architecture and possible configurations

Demo configuration

You can configure booster-bitswap in the demo mode and familiarise yourself with the configuration. Once you are confident and familiar with the options, please go ahead and configure booster-bitswap for production use.

The booster-bitswap binary is built and installed with boostd binary. If you are planning to run booster-bitswap on a different node, you can build and install the new binary. Otherwise, skip to step 3.

1. Clone the Boost repo and checkout the same release as your boostd version

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

2. Build and install the booster-bitswap binary:

make build
sudo make build

3. Initialize booster-bitswap:

booster-bitswap init

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Run booster-bitswap

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044"

6. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

7. Fetching over bitswap by running

booster-bitswap fetch /ip4/127.0.0.1/tcp/8888/p2p/{peerID} {rootCID} outfile.car

Where peerID is the peer id recorded when you ran booster-bitswap init and rootCID is the CID of a data CID known to be stored on your SP.

Setup booster-bitswap to serve retrievals

As described above, booster-bitswap can be configured to serve the retrievals in 2 modes. We recommend using public mode to avoid greater load on boostd as a protocol proxy.

Private Mode

The booster-bitswap binary is built and installed with boostd binary. If you are planning to run booster-bitswap on a different node, you can build and install the new binary. Otherwise, skip to step 3.

1. Clone the Boost repo and checkout the same release as your boostd version

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

2. Build and install the booster-bitswap binary:

make build
sudo make build

3. Initialize booster-bitswap:

booster-bitswap init

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Stop boostd and edit ~/.boost/config.toml to set the peer ID for bitswap

[Dealmaking]
  BitswapPeerID ="{peer id for booster bitswap you recorded earlier}"

6. Start boostd service again

7. Collect the token information for lotus-miner and lotus daemon API

export `lotus auth api-info --perm=admin`

export `lotus-miner auth api-info --perm=admin`

8. Run booster-bitswap

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --proxy={boostd multiaddress} --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO

You can get a boostd multiaddress by running boostd net listen and using any of the returned addresses

9. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

10. Try to fetch a payload CID over bitswap to verify your configuration

Public mode

The booster-bitswap binary is built and installed with boostd binary. If you are planning to run booster-bitswap on a different node, you can build and install the new binary. Otherwise, skip to step 3.

1. Clone the Boost repo and checkout the same release as your boostd version

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

2. Build and install the booster-bitswap binary:

make build
sudo make build

3. Initialize booster-bitswap:

booster-bitswap init

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Stop boostd and edit ~/.boost/config.toml to set the peer ID for bitswap

[DealMaking]
 BitswapPeerID ="{peer id for bosoter bitswap you recorded earlier}"
 BitswapPublicAddresses = ["/ip4/{booster-bitswap public IP}/tcp/{booster-bitswap public port}"]
 BitswapPrivKeyFile = "{path to libp2p private key file for booster bitswap}"

The libp2p private key file for booster-bitswap can generally be found at <booster-bitswap repo path>/libp2p.key

The reason boost needs to know the public multiaddresses and libp2p private key for booster-bitswap is so it can properly announce these records to the network indexer.

6. Start boostd service again

7. Collect the token information for lotus-miner and lotus daemon API

export `lotus auth api-info --perm=admin`

export `lotus-miner auth api-info --perm=admin`

8. Run booster-bitswap

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO

9. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

10. Try to fetch a payload CID over bitswap to verify your configuration

booster-bitswap configuration

booster-bitswap provides a number of performance and safety tools for managing a production grade bitswap server without overloading your infrastructure.

Bitswap server performance

Depending on your hardware you may wish to increase or decrease the default parameters for the bitswap server internals. In the following example we are increasing the worker count for various components up to 600. This will utilize more CPU and I/O, but improve the performance of retrievals. See the command line help docs for details on each parameter.

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" \
  --engine-blockstore-worker-count=600 \
  --engine-task-worker-count=600 \
  --max-outstanding-bytes-per-peer=33554432 \
  --target-message-size=1048576 \
  --task-worker-count=600

BadBits filtering

Booster-bitswap is automatically setup to deny all requests for CIDs that are on the BadBits Denylist. The default badbits list can be override or addition badbits list can be provided to the booster-bitswap instance.

To override the default badbits list

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --badbits-denylists <URL>

To provide additional badbits list

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --badbits-denylists https://badbits.dwebops.pub/denylist.json <URL1> <URL2>

Request filtering

booster-bitswap provides a number of controls for filtering requests and limiting resource usage. These are expressed in a JSON configuration file <booster-bitswap repo>/retrievalconfig.json

You can create a new retrievalconfig.json file if one does not exists

{
   "AllowDenyList": { // list of peers to either deny or allow (denying all others)
   	"Type": "allowlist",  // "allowlist" or "denylist"
   		"PeerIDs": [
   			"Qma9T5YraSnpRDZqRR4krcSJabThc8nwZuJV3LercPHufi",
   			"QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N"
   		]
   },
       "UnderMaintenance": false, // when set to true, denies all requests
       "StorageProviderLimits": {
           "Bitswap": {
                       "SimultaneousRequests": 100, // bitswap block requests served at the same time across peers
                       "SimultaneousRequestsPerPeer": 10, // bitswap block requests served at the same time for a single peer
   		"MaxBandwidth": "100mb" // human readable size metric, per second
   	}
   }
}

To make changes to the current configuration, you need to edit the retrievalconfig.json file and restart booster-bitswap for the changes to take affect. All configs are optional and absent parameters generally default to no filtering at all for the given parameter.

You can also configure booster-bitswap to fetch your retrieval config from a remote HTTP API, possibly one provided by a third party configuration tool like CIDGravity. To do this, start booster-bitswap with the --api-filter-endpoint {url} option where URL is the HTTP URL for an API serving the above JSON format. Optionally, add --api-filter-auth {authheader}, if you need to pass a value for the HTTP Authorization header with your API

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --api-filter-endpoint <URL> --api-filter-auth <OPTIONAL SCURITY HEADERS>

When you setup with an API endpoint, booster-bitswap will update its local configuration from the API every five minutes, so you won't have to restart booster-bitswap to make a change. Please, be aware that the remote config will overwrite, rather than merge, with the local config.

Bandwidth limiting

Limiting bandwidth within booster-bitswap will not provide the optimal user experience. Dependent on individual setup, setting up limitations within the software could have a larger impact on the storage provider operations. Therefore, we recommend storage providers to set up their own bandwidth limitations using existing tools.

There are multiple options to setup bandwidth limitating.

1. At the ISP level - dedicated bandwidth is provided to the node running booster-bitswap.

2. At the router level - we recommend configuring the bandwidth at the router level as it provides more flexibility and can be updated as needed. To configure the bandwidth on your router, please check with your manufacturer.

3. Limit the bandwidth using different tools available in Linux. Here are some of the examples of such tools. Please feel free to use any other tools not listed here and open a Github issue to add your example to this page.

TC

TC is used to configure Traffic Control in the Linux kernel. There are examples available online detailing how to configure rate limiting using TC.

You can use the below commands to run a very basic configuration.

sudo tc qdisc add dev <network interface> root handle 1: htb
sudo tc class add dev <network interface> parent 1: classid 1:20 htb rate 100mibit
sudo tc qdisc add dev <network interface> parent 1:20 handle 20: sfq perturb 10
sudo tc filter add dev <network interface> parent 1: protocol ip prio 1 basic match 'cmp(u16 at 0 layer transport eq 8888)' flowid 1:20

Trickle

Trickle is a portable lightweight user space bandwidth shaper, that either runs in collaborative mode (together with trickled) or in standalone mode. You can read more about rate limiting with trickle here. Here's a starting point for configuration in trickle to rate limit the booster-bitswap service.

[booster-bitswap]
Priority = <value>
Time-Smoothing = <value>
Length-Smoothing = <value>

Wondershaper

Another way of controlling network traffic is to limit bandwidth on individual network interface cards (NICs). Wondershaper is a small Bash script that uses the tc command-line utility in the background to let you regulate the amount of data flowing through a particular NIC. As you can imagine, while you can use wondershaper on a machine with a single NIC, its real advantage is on a machine with multiple NICs. Just like trickle, wondershaper is available in the official repositories of mainstream distributions. To limit network traffic with wondershaper, specify the NIC on which you wish to restrict traffic with the download and upload speed in kilobits per second.

For example,

wondershaper enp5s0 4096 1024