Boost is developing new market features on a regular basis as part of the overall market development. This section covers the experimental features released by boost along with details on how to use them.

It is not recommended to run experimental features in production environments. The features should be tested as per your requirements, and any issues or requests should be reported to the team via Github or Slack.

Once the new features have been tested and vetted, they will be released as part of a stable Boost release and all documentation concerning those features will be moved to an appropriate section of this site.

Current experimental features are listed below.

Boost is introducing a new feature that allows computing commP during the deal on a lotus-worker node.

This should reduce the overall resource utilisation on the Boost node.

In order to enable remote commP on a Boost node, update your config.toml:

[Dealmaking]
   RemoteCommp = true

Then restart the Boost node

By default the Boost daemon repository is located at ~/.boost

It contains the following files:

• api The local multi-address of Boost's libp2p API

• boost.db The sqlite database with all deal metadata

• boost.logs.db The sqlite database with the logs for deals

• config.toml The config file with all Boost's settings

• repo.lock A lock file created when Boost is running

• storage.json Deprecated (needed by legacy markets)

• token The token used when calling Boost's JSON RPC endpoints

It has the following directories:

• dagstore Contains indexes of CAR files stored with Boost

• datastore Contains metadata about deals for legacy markets

• deal-staging

A storage provider can run the lotus as a monolith process where everything is handled by a single lotus-miner process or separate the mining and market subsystems on different machines.

Boost supports migration from both monolith and a split-market miner. You can follow the below guides to migrate to boost.

Rollback

Boost is a tool for Storage Providers to manage data onboarding and retrieval on the Filecoin network. It replaces the go-fil-markets package in lotus with a standalone binary that runs alongside a Lotus daemon and Lotus miner.

Boost exposes libp2p interfaces for making storage and retrieval deals, a web interface for managing storage deals, and a GraphQL interface for accessing and updating real-time deal information.

The boostd executable runs as a daemon alongside a lotus node and lotus miner. This daemon replaces the current markets subsystem in the lotus miner. The boost daemon exposes a libp2p interface for storage and retrieval deals. It performs on-chain operations by making API calls to the lotus node. The daemon hands off downloaded data to the lotus miner for sealing via API calls to the lotus miner.

boostd has a web interface for fund management and deal monitoring. The web interface is a react app that consumes a graphql interface exposed by the daemon.

Backup

Boost now supports both online and offline backups. The backup command will output a backup directory containing the following files.

1. metadata

Using filters for fine-grained storage and retrieval deal acceptance

Your use case might demand very precise and dynamic control over a combination of deal parameters.

Lotus provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it:

Make storage deals with HTTP data transfer

Boost supports multiple options for data transfer when making storage deals, including HTTP. Clients can host their CAR file on an HTTP server, such as S3, and provide that URL when proposing the storage deal. Once accepted, Boost will automatically fetch the CAR file from the specified URL.

See for more details.

With the release of FVM, it is now possible for smart contracts to make deal proposals on-chain. This is made possible though the DealProposal FRC.

DataDAOs, as well as other clients who want to store data on Filecoin, can now deploy a smart contract on the FVM which adheres to the DealProposal FRC, and make deal proposals that are visible to every storage provider who monitors the chain.

How to enable FVM monitoring in order to process storage deal proposals published on-chain?

Boost already has support for the DealProposal FRC.

The code for FVM monitoring resides in the latest release of the Boost. It should be used with caution for production use. SPs must before proceeding to the next step.

To build for mainnet:

In order to enable DealProposal FRC, you have to edit your config.toml and enable contract deal monitoring. By default it is disabled. Here is an example configuration:

AllowlistContracts field could be left empty if you want to accept deals from any client. If you only want to accept deals from some clients, you can specify their contract addresses in the field.

From field should be set to your SP's FEVM address. Some clients may implement a whitelist which allows specific SPs to accept deal proposals from their contract. This field will help those clients identify your SP and match it to their whitelist.

How contract deals work in Boost

1. A contract publishes a DealProposalCreate event on the chain.

2. Boost monitors the chain for such events from all the clients by default. When such an event is detected, we go and fetch the data for the deal.

3. Deal is then run through the basic deal validation filters like clients has enough funds, SP has enough funds etc.

When designing the Local Index Directory we considered the needs of various Storage Providers (SPs) and the operational overhead LID would have on their systems. We built a solution for: - small- SPs - holding up to 1PiB), and - mid- and large- size SPs - holding anywhere from 1PiB, up to 100PiB data

Depending on underlying block size and data format, index size can vary in size. Typically block sizes are between 16KiB and 1MiB.

At the moment there are two implementations of LID: - a simple LevelDB implementation, for small SPs who want to keep all information in a single process database. - a scalable YugabyteDB implementation, for medium and large size SPs with tens of thousands of deals.

Index types

In order to support the described retrieval use cases, LID maintains the following indexes:

multihash → []piece cid

To look up which pieces contain a block

piece cid → sector information {sector ID, offset, size}

To look up which sector a piece is in

piece cid → map<mulithash → block offset / size>

To look up where in the piece a block is and the block’s size

1. If you haven't made any legacy deals with Boost:

   1. Stop boostd

   2. Run your lotus-miner markets service process as you previously did

2. If you have made new legacy deals with Boost, and want to migrate them back:

   1. Stop boostd

   2. Copy the dagstore directory from boost

1. You can report any issues or bugs here.

2. If you are having trouble, check the Troubleshooting page for common problems and solutions.

3. If you have a question, please join the and ask in #fil-help or #fil-lotus-help or #boost-help or start .

4. You can also about new feature and improvement ideas for the Boost.

Background

The monitoring stack we will use includes:

Considering that the Local Index Directory is a new feature, Storage Providers should initialise it after upgrading their Boost deployments.

There are two ways a Storage Provider can do that:

1. Migrate existing indices from the DAG store into LID: this solution assumes that the Storage Provider has been keeping an unsealed copy for every sector they prove on-chain, and has already indexed all their deal data into the DAG store. Typically index sizes for a given sector range between 100KiB up to 1GiB, depending on deal data and its blocks sizes. The DAG store keeps these indices in the repository directory of Boost under the ./dagstore/index and ./dagstore/datastore

Boost comes with a client executable, boost, that can be used to send a deal proposal to a Boost server.

The client is intentionally minimal meant for developer testing. It is not a full featured client and is not intended to be so. It does not require a daemon process, and can be pointed at any public Filecoin API for on-chain operations. This means that users of the client do not need to run a Filecoin node that syncs the chain.

Set the API endpoint environment variable

The hardware requirements for Boost are tied to the sealer part of the Lotus deployment it is attached to.

Depending on how much data you need to onboard, and how many deals you need to make with clients, hardware requirements in terms of CPU and Disk will vary.

General hardware requirements

Is there a way to stop boostd daemon? You can use the regular Unix OS signals

Is Boost compatible with the Lotus client? Can a client use lotus client deal to send a deal to Boost storage providers or do they have to use the boost client? Yes, boost should work with any client given it supports the storage market protocol / default standard of Filecoin network today.

Does Boost provide retrieval functionality? Yes, Boost provides 3 protocols for retrievals as of now. By default, Boost has Graphsync retrieval enabled. SPs can run Bitswap and HTTP retrievals by running booster-bitswap and booster-http respectively.

Does Boost client have retreival functionality? Yes, Boost client supports retrieval over graphsync protocol. But we highly recommend, using client for Filecoin/IPFS retrievals.

Can Boost make verified deals? Yes, payments for deals can be made either from a regular wallet, or from DataCap. Deals that are paid for with DataCap are called verified deals.

Can I run both Boost and markets at the same time? No, Boost replaces the legacy markets process. See

Background

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (cid).

Currently this task is performed by the DAG store component. The DAG store keeps its indexes on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools.

To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size.

Storing data on Filecoin

When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way cilents can later retrieve subsets of the original deal data, without retrieving the full deal data.

Retrieving data

When a client makes a request for data by cid, LID: - checks which piece the cid is in, and where in the piece the data is - checks which sector the piece is in, and where in the sector the piece is - reads the data from the sector

Use cases

The retrieval use cases that the Local Index Directory supports are:

Graphsync retrieval

Request one root cid with a selector, receive many blocks

LID is able to: - look up which piece contains the root cid - look up which sector contains the piece - for each block, get the offset into the piece for the block

Bitswap retrieval

Request one block at a time

LID is able to: - look up which piece contains the block - get the size of the block (Bitswap asks for the size before getting the block data) - look up which sector contains the piece - get the offset into the piece for the block

HTTP retrieval

Request a whole piece

LID is able to look up which sector contains the piece.

Request an individual block

LID is able to: - look up which piece contains the block - look up which sector contains the piece - get the offset into the piece for the block

Request a file by root cid

LID is able to: - look up which piece contains the block - look up which sector contains the piece - for each block, get the offset into the piece for the block

First, you need to initialise a new Boost client and also set the endpoint for a public Filecoin node. In this example we are using https://glif.io

export FULLNODE_API_INFO=https://api.node.glif.io

boost init

The init command will output your new wallet address, and warn you that the market actor is not initialised.

boost init

boost/init_cmd.go:53    default wallet set      {"wallet": "f3wfbcudimjcqtfztfhoskgls5gmkfx3kb2ubpycgo7a2ru77temduoj2ottwzlxbrbzm4jycrtu45deawbluq"}
boost/init_cmd.go:60    wallet balance  {"value": "0"}
boost/init_cmd.go:65    market actor is not initialised, you must add funds to it in order to send online deals

Then you need to send funds to the wallet, and add funds to the market actor (in the example below we are adding 1 FIL).

You can use the boostx utilities to add funds to the market actor:

You can confirm that the market actor has funds by running boost init again.

After that you need to generate a car file for data you want to store on Filecoin, and note down its payload-cid. We recommend using CLI to generate the car file.

Then you need to calculate the commp and piece size for the generated car file:

Place the generated car file on a public HTTP server, so that a storage provider can later fetch it.

Finally, trigger an online storage deal with a given storage provider:

Dependencies

Local Index Directory depends on a backend database to store various indices. Currently we support two implementations - YugabyteDB or LevelDB - depending on the size of deal data and indices a storage provider holds.

LevelDB is an open source on-disk key-value store, and can be used when indices fit on a single host.

YugabyteDB is an open source modern distributed database designed to run in any public, private, hybrid or multi-cloud environment.

Hardware requirements

YugabyteDB is designed to run on bare-metal machines, virtual machines (VMs), and containers. CPU and RAM

You should allocate adequate CPU and RAM. YugabyteDB has adequate defaults for running on a wide range of machines, and has been tested from 2 core to 64 core machines, and up to 200GB RAM.

Minimum requirement

Production requirement

Verify support for SSE2 and SSE4.2

YugabyteDB requires the SSE2 instruction set support, which was introduced into Intel chips with the Pentium 4 in 2001 and AMD processors in 2003. Most systems produced in the last several years are equipped with SSE2.

In addition, YugabyteDB requires SSE4.2.

To verify that your system supports SSE2, run the following command:

cat /proc/cpuinfo | grep sse2

To verify that your system supports SSE4.2, run the following command:

cat /proc/cpuinfo | grep sse4.2

Disks

We recommend a minimum of 1TiB or more allocated for YugabyteDB, depending on the amount of deal data you store and its average block size.

Boost stores metadata about deals in a sqlite database in the root directory of the Boost repo.

To open the database use a sqlite client:

sqlite3 boost.db

The database tables are

• Deals metadata about Boost storage deals (eg deal proposal) and their current state (eg checkpoint)

• FundsLogs log of each change in funds reserved for a deal

• FundsTagged how much FIL is tagged for deal collateral and publish message for a deal

• StorageLogs log of each change in storage reserved for a deal

• StorageTagged how much storage is tagged for a deal

Boost keeps a separate database just for deal logs, so as to make it easier to manage log data separately from deal metadata. The logs database is named boost.logs.db and it has a single table DealLogs that stores logs for each deal, indexed by uuid.

Migrations

Boost uses goose () tool and library for handling sqlite3 migrations.

goose can be installed following the instructions at

Migrations in Boost are stored in the /db/migrations directory.

Boost handles database migrations on start-up. If a user is running an older version of Boost, migrations up to the latest version are automatically applied on start-up.

Developers can use goose to inspect and apply migrations using the CLI:

Configuration

[Graphql]
  ListenAddress = "127.0.0.1"
  Port = 8080

[Monitoring]
  MpoolAlertEpochs = 30

By default, the web UI listens on the localhost interface on port 8080. We recommend keeping the UI listening on localhost or some internal IP within your private network to avoid accidentally exposing it to the internet.

You can access the web UI listening on the localhost interface on a remote server, you can open an SSH tunnel from your local machine:

Settings Page

Boost exposes a GraphQL API that is used by the Web UI to query and update information about Boost deals. The GraphQL API query endpoint is at http://localhost:8080/graphql/query

You can also run your own queries against the GraphQL API using CURL or a programming language that has a GraphQL client.

Boost has a built-in GraphQL explorer at http://localhost:8080/graphiql

You can test out queries, or explore the GraphQL API by clicking on the < Docs link at the top right of the page:

To run a graphql query with CURL:

curl -X POST
-H "Content-Type: application/json"
-d '{"query":"query { deals(offset: 5, limit: 10) { deals { ID CreatedAt PieceCid } } }"}'
http://localhost:8080/graphql/query | jq

This 1m video shows how to use these tools to build an run a GraphQL query against Boost:

Example Queries

1. Query failed deals

2. Cancel a deal where ab12345c-5678-90de-12f3-45a6b78cd9ef is the deal ID

If you are running a markets

Boost provides a capability to limit the number of simultaneous http transfer in progress to download the deal data from the clients.

This new configuration has been introduced in the ConfigVersion = 3 of the boost configuration file.

Configuration Variables

Initialization and Running

1. Make sure you have a Lotus node and miner running

2. Create and send funds to two new wallets on the lotus node to be used for Boost

Boost currently uses two wallets for storage deals:

• The publish storage deals wallet - This wallet pays the gas cost when Boost sends the PublishStorageDeals message.

• The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the StorageMarketActor.

3. Set the publish storage deals wallet as a control wallet.

4. Create and initialize the Boost repository

Boost keeps all data in a directory called the repository. By default the repository is at ~/.boost. To use a different location pass the --boost-repo parameter (must precede any particular command verb, e.g. boostd --boost-repo=/path init).

Export the environment variables needed for boostd init to connect to the lotus daemon and lotus miner.

Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the boost node to make JSON-RPC calls to the mining/sealing/proving node.

Run boostd init to create and initialize the repository:

• --api-sealer is the API info for the lotus-miner instance that does sealing

• --api-sector-index is the API info for the lotus-miner instance that provides storage

• --max-staging-deals-bytes

5. Update ulimit file descriptor limit if necessary. Boost deals will fail if the file descriptor limit for the process is not set high enough. This limit can be raised temporarily before starting the Boost process by running the command ulimit -n 1048576. We recommend setting it permanently by following the guide.

6. Make sure that the correct <PEER_ID> and <MULTIADDR> for your SP is set on chain, given that boost init generates a new identity. Use the following commands to update the values on chain:

7. Run the boostd service, which will start:

• libp2p listeners for storage and retrieval

• the JSON RPC API

• the graphql interface (used by the react front-end)

Web UI

1. Build the React frontend

1. Open the Web UI

Open http://localhost:8080 in your browser.

API Access

Boost API can be accessed by setting the environment variable BOOST_API_INFO same as LOTUS_MARKET_INFO.

You can also directly evaluate the boostd auth command with:

Boost introduced a new binary, booster-http, with release v1.2.0. This binary can be run alongside the boostd market process in order to serve retrievals over http.

Currently, there is no payment method or built-in security integrated in the new binary. It can be run with any stable release of boostd and can also be run on a separate machine from the boostd process.

Release v1.7.0-rc1 introduced support in booster-http for running an IPFS gateway, which enables Storage Providers to serve content to their users in multiple formats as described below and demonstrated using curl.

Retrieving a full Piece

When performing certain actions, such as replicating deals, it can be convenient to retrieve the entire Piece (with padding) to ensure commp integrity.

Retrieving a CAR file

To return the CAR file for a given CID, you can pass an Accept header with the application/vnd.ipld.car; format. This can be useful for retrieving the raw, unpadded data of a deal.

Retrieving specific files

For Storage Providers that have enabled serving raw files (disabled by default), users can retrieve specific files, such as images by their cid and path where applicable. See for a more in depth example.

Retrieving IPLD blocks

For advanced IPFS and IPLD use cases, you can now retrieve individual blocks by passing an Accept header with the application/vnd.ipld.raw; format

Local Setup

SPs should try a local setup and test their HTTP retrievals before proceeding to run booster-http in production.

To build and run booster-http :

1. Clone the boost repo and checkout the latest release

1. Build the new binary

1. Collect the token information for boost, lotus-miner and lotus daemon API

1. Start the booster-http server with the above details

Running Public Boost HTTP Retrieval

The booster-http server listens on localhost. To expose the server publically, SPs should run a reverse proxy such as to handle operational concerns like:

• SSL

• Authentication

• Load balancing

While booster-http may get more operational features over time, the intent is that providers who want to scale their HTTP operations will handle most of operational concerns via software in front of booster-http. You can setup a simple NGINX proxy using the in

Making HTTP Retrieval Discoverable

To enable public discovery of the Boost HTTP server, SPs should set the domain root in boostd's config.toml. Under the [DealMaking] section, set HTTPRetrievalMultiaddr to the public domain root in multi-address format.

Example config.toml section:

Clients can determine if an SP offers HTTP retrieval by running:

Clients can check the HTTP URL scheme version and supported queries

Clients can download a piece using the domain root configured by the SP:

The Boost source code repository is hosted at

Boost and Lotus compatibility Matrix

Announce over HTTP

IndexProvider.HttpPublisher.AnnounceOverHttp must be set to true to enable the http announcements. Once HTTP announcements are enabled, the local-index provider will continue to announce over libp2p gossipsub along with HTTP for the specific indexers.

The advertisements are send to the indexer nodes defined in DirectAnnounceURLs

Inspect

The new inspect page in the Boost UI helps with debugging retrieval problems. It allows the user to check the following using a payload CID or piece CID:

• Verify if the piece has been correctly added to the Piece Store

Storage providers might demand very precise and dynamic control over a combination of deal parameters.

Boost, similarly to Lotus, provides two IPC hooks allowing you to name a command to execute for every deal before the storage provider accepts it:

• Filter for storage deals.

• RetrievalFilter for retrieval deals.

The executed command receives a JSON representation of the deal parameters, as well as the current state of the sealing pipeline, on standard input, and upon completion, its exit code is interpreted as:

• 0: success, proceed with the deal.

• non-0: failure, reject the deal.

The most trivial filter rejecting any retrieval deal would be something like: RetrievalFilter = "/bin/false". /bin/false is binary that immediately exits with a code of 1.

lets the miner deny specific clients and only accept deals that are set to start relatively soon.

You can also use a third party content policy framework like bitscreen by Murmuration Labs, or :

Here is a sample JSON representation of the input sent to the deal filter:

Boost supports the same libp2p protocols as legacy markets, and adds new versions of the protocols used to propose a storage deal and to check the deal's status.

Propose Storage Deal Protocol

The client makes a deal proposal over v1.2.0 or v1.2.1 of the Propose Storage Deal Protocol: /fil/storage/mk/1.2.0 or /fil/storage/mk/1.2.1

It is a request / response protocol, where the request and response are CBOR-marshalled.

There are two new fields in the Request of v1.2.1 of the protocol, described in the table below.

Request

Response

Storage Deal Status Protocol

The client requests the status of a deal over v1.2.0 of the Storage Deal Status Protocol: /fil/storage/status/1.2.0

It is a request / response protocol, where the request and response are CBOR-marshalled.

Request

Response

Directory structure

By default, the dagstore root will be:

• $BOOST_PATH/dagstore

The directory structure is as follows:

1. index: holds the shard indices.

2. transients: holds temporary shard data (unsealed pieces) while they're being indexed.

3. datastore

First-time migration

When you first start your boost process without a dagstore repo, a migration process will register all shards for both legacy and Boost deals in lazy initialization mode. As deals come in, shards are fetched and initialized just in time to serve the retrieval.

• For legacy deals, you can monitor the progress of the migration in your log output, by grepping for the keyword migrator. Here's example output. Notice the first line, which specifies how many deals will be evaluated (this number includes failed deals that never went on chain, and therefore will not be migrated), and the last lines (which communicate that migration completed successfully):

• For Boost deals, you can do the same by grepping for the keyword boost-migrator.

Forcing bulk initialization

Forcing bulk initialization will become important in the near future, when miners begin publishing indices to the network to advertise content they have, and new retrieval features become available (e.g. automatic shard routing).

Initialization places IO workload on your storage system. You can stop/start this command at your wish/convenience as proving deadlines approach and elapse, to avoid IOPS starvation or competition with window PoSt.

To stop a bulk initialization(see the next paragraph), press Control-C. Shards being initialized at that time will continue in the background, but no more initializations will be performed. The next time you run the command, it will resume from where it left off.

You can force bulk initialization using the boostd dagstore initialize-all command. This command will force initialization of every shard that is still in ShardStateNew state for both legacy and Boost deals. To control the operation:

• You must set a concurrency level through the --concurrency=N flag.

  • A value of 0 will disable throttling and all shards will be initialized at once. ⚠️ Use with caution!

In our test environments, we found the migration to proceed at a rate of 400-500 shards/deals per second, on the following hardware specs: AMD Ryzen Threadripper 3970X, 256GB DDR4 3200 RAM, Samsung 970 EVO 2TB SSD, RTX3080 10GB GPU.

Configuration

The DAG store can be configured through the config.toml file of the node that runs the boost subsystem. Refer to the [DAGStore] section. Boost ships with sane defaults:

Automatic shard recovery on error

Shards can error for various reasons, e.g. if the storage system cannot serve the unsealed CAR for a deal/shard, if the shard index is accidentally deleted, etc.

Boost will automatically try to recover failed shards by triggering a recovery once.

You can view failed shards by using the boostd dagstore list-shards command, and optionally grepping for ShardStateErrored.

CLI commands

The boostd executable contains a dagstore command with several useful subcommands:

• boostd dagstore list-shards

• boostd dagstore initialize-shard <key>

• boostd dagstore initialize-all --concurrency=10

Refer to the --help texts for more information.

Dealmaking section

This section controls parameters for making storage and retrieval deals:

ExpectedSealDuration is an estimate of how long sealing will take and is used to reject deals whose start epoch might be earlier than the expected completion of sealing. It can be estimated by or by .

booster-bitswap is a binary that runs alongside the boostd process, to serve retrievals over the Bitswap protocol. This feature of boost provides a number of tools for managing a production grade Bitswap retrieval service for a Storage Provider's content.

Why enable retrievals via bitswap?

Bitswap retrieval introduces interoperability between IPFS and Filecoin, as it enables clients to retrieve Filecoin data over IPFS. This expands the reach of the Filecoin network considerably, increasing the value proposition for users to store data on the Filecoin network. This benefits the whole community, including SPs. Users will be able to access data directly via IPFS, as well as benefit from retrieval markets (e.g. Saturn) and compute over data projects (e.g. Bacalhau).

Booster-bitswap modes

There are two primary "modes" for exposing booster-bitswap to the internet.

1. In private mode the booster-bitswap peer ID is not publicly accessible to the internet. Instead, public Bitswap traffic goes to boostd itself, which then acts as a reverse proxy, forwarding that traffic on to booster-bitswap. This is similar to the way one might configure Nginx as a reverse proxy for an otherwise private web server. private mode is simpler to setup but may produce greater load on boostd as a protocol proxy.

Demo configuration

You can configure booster-bitswap in the demo mode and familiarise yourself with the configuration. Once you are confident and familiar with the options, please go ahead and configure booster-bitswap for .

1. Clone the the boost repo and checkout the latest stable release

2. Build the booster-bitswap binary:

3. Initialize booster-bitswap:

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Collect the boost API Info

6. Run booster-bitswap

7. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

8. Fetching over bitswap by running

Where peerID is the peer id recorded when you ran booster-bitswap init and rootCID is the CID of a data CID known to be stored on your SP.

Setup booster-bitswap To Serve Retrievals

As described above, booster-bitswap can be configured to serve the retrievals in 2 modes. We recommend using public mode to avoid greater load on boostd as a protocol proxy.

Private Mode

1. Clone the main branch from the boost repo

2. Build the booster-bitswap binary:

3. Initialize booster-bitswap:

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Stop boostd and edit ~/.boost/config.toml to set the peer ID for bitswap

6. Start boostd service again

7. Collect the boost API Info

8. Run booster-bitswap

9. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

10. Try to fetch a payload CID over bitswap to verify your configuration

Public Mode

1. Clone the release/booster-bitswap branch from the boost repo

2. Build the booster-bitswap binary:

3. Initialize booster-bitswap:

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Stop boostd and edit ~/.boost/config.toml to set the peer ID for bitswap

The reason boost needs to know the public multiaddresses and libp2p private key for booster-bitswap is so it can properly announce these records to the network indexer.

6. Start boostd service again

7. Collect the boost API Info

8. Run booster-bitswap

9. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

10. Try to fetch a payload CID over bitswap to verify your configuration

Booster-bitswap configuration

booster-bitswap provides a number of performance and safety tools for managing a production grade bitswap server without overloading your infrastructure.

Bitswap Server Performance

Depending on your hardware you may wish to increase or decrease the default parameters for the bitswap server internals. In the following example we are increasing the worker count for various components up to 600. This will utilize more CPU and I/O, but improve the performance of retrievals. See the command line help docs for details on each parameter.

BadBits filtering

Booster-bitswap is automatically setup to deny all requests for CIDs that are on the BadBits Denylist. The default badbits list can be override or addition badbits list can be provided to the booster-bitswap instance.

To override the default badbits list

To provide additional badbits list

Request Filtering

booster-bitswap provides a number of controls for filtering requests and limiting resource usage. These are expressed in a JSON configuration file <booster-bitswap repo>/retrievalconfig.json

To make changes to the current configuration, you need to edit the retrievalconfig.json file and restart booster-bitswap for the changes to take affect. All configs are optional and absent parameters generally default to no filtering at all for the given parameter.

You can also configure booster-bitswap to fetch your retrieval config from a remote HTTP API, possibly one provided by a third party configuration tool like . To do this, start booster-bitswap with the --api-filter-endpoint {url} option where URL is the HTTP URL for an API serving the above JSON format. Optionally, add --api-filter-auth {authheader}, if you need to pass a value for the HTTP Authorization header with your API

When you setup with an API endpoint, booster-bitswap will update its local configuration from the API every five minutes, so you won't have to restart booster-bitswap to make a change. Please, be aware that the remote config will overwrite, rather than merge, with the local config.

Bandwidth Limiting

Limiting bandwidth within booster-bitswap will not provide the optimal user experience. Dependent on individual setup, setting up limitations within the software could have a larger impact on the storage provider operations. Therefore, we recommend storage providers to set up their own bandwidth limitations using existing tools.

There are multiple options to setup bandwidth limitating.

1. At the ISP level - dedicated bandwidth is provided to the node running booster-bitswap.

2. At the router level - we recommend configuring the bandwidth at the router level as it provides more flexibility and can be updated as needed. To configure the bandwidth on your router, please check with your manufacturer.

3. Limit the bandwidth using different tools available in Linux. Here are some of the examples of such tools. Please feel free to use any other tools not listed here and open a Github issue to add your example to this page.

TC

is used to configure Traffic Control in the Linux kernel. There are examples available online detailing how to configure rate limiting using TC.

You can use the below commands to run a very basic configuration.

Trickle

is a portable lightweight user space bandwidth shaper, that either runs in collaborative mode (together with trickled) or in standalone mode. You can read more about rate limiting with trickle . Here's a starting point for configuration in trickle to rate limit the booster-bitswap service.

Wondershaper

Another way of controlling network traffic is to limit bandwidth on individual network interface cards (NICs). is a small Bash script that uses the tc command-line utility in the background to let you regulate the amount of data flowing through a particular NIC. As you can imagine, while you can use wondershaper on a machine with a single NIC, its real advantage is on a machine with multiple NICs. Just like trickle, wondershaper is available in the official repositories of mainstream distributions. To limit network traffic with wondershaper, specify the NIC on which you wish to restrict traffic with the download and upload speed in kilobits per second.

For example,

With the release v1.7.0-rc1 of booster-http, Storage Providers can now serve blocks and files directly over the HTTP protocol. booster-http now implements a IPFS HTTP gateway with a path resolution style. This will allow the clients to download individual IPFS blocks, car files and request uploaded files directly from their browser.

SPs can take advantage of the ecosystem of tools to manage HTTP traffic, like load balancers and reverse proxies.

Configuring what to serve

The booster-http service can be started with specific type of content on IPFS gateway API

This allows SPs to run multiple booster-http instances, each serving specific type of content like car files only or raw blocks only.

Enable serving files

In the curl request below we appended the query parameter format=raw to the URL to get the raw block data for the file.

But, if we try to open the file directly in a web browser, with no extra query parameters, we get an error message:

By default booster-http does not serve files in a format that can be read by a web browser. This is to protect Storage Providers from serving content that may be .

To enable serving files to web browsers, we must pass --serve-files=true to booster-http on startup. Once, booster-http is restarted with --serve-files=true, we can open the file directly from a web browser:

We can also browse all files in the CAR archive.

Protecting booster-http with NGINX

SPs must secure their booster-http before exposing it to the public. SPs can feel free to use any tool available to limit who can download files, the number of requests per second, and the download bandwidth each client can use per second.

Users can follow this example to use reverse proxy to secure their booster-http instance. In this section we’ve just scratched the surface of the ways in which nginx can set access limits, rate limits and bandwidth limits. In particular it’s possible to add limits by request token, or using JWT tokens. The examples in this section are adapted from which goes into more detail.

By default nginx puts configuration files into /etc/nginx

The default configuration file is /etc/nginx/sites-available/default

Setup server block

Setup nginx server listen on port 7575 and forward requests to booster-http on port 7777

The IPFS gateway serves files from /ipfs. So, we will add a server block for location /ipfs/

Limiting Access

Let’s limit access to the IPFS gateway using the standard .htaccess file. We need to set up an .htaccess file with a username and password. Create a user named alice

Include the .htaccess file in the /etc/nginx/sites-available/default

Now when we open any URL under the path /ipfs we will be presented with a Sign in dialog.

Rate Limiting

To prevent users from making too many requests per second, we should add rate limits.

1. Create a file with the rate limiting configuration at /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf

2. Add a request zone limit to the file of 1 request per second, per client IP

1. Include ipfs-gateway.conf in /etc/nginx/sites-available/default and set the response for too many requests to HTTP response code 429

1. Click the refresh button in your browser on any path under /ipfs more than once per second you will see a 429 error page

Bandwidth Limiting

It is also recommended to limit the amount of bandwidth that clients can take up when downloading data from booster-http. This ensures a fair bandwidth distribution to each client and prevents situations where one client ends up choking the booster-http instance.

1. Create a new .htaccess user called bob

1. Add a mapping from .htaccess username to bandwidth limit in /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf

1. Add the bandwidth limit to /etc/nginx/sites-available/default

1. To verify bandwidth limiting, use curl to download a file with user alice and then bob Note the difference in the Average Dload column (the average download speed).

Sample config file

Sealer

API

Libp2p

Storage