The Lotus stack is responsible for running a Filecoin full chain node, handling sector lifecycle, storing sectors, sector sealing, sector proving, handling wallets, and more. You can read about the Lotus components at https://docs.filecoin.io/storage-provider/architecture/lotus-components/

The Boost stack is responsible for handling storage and retrieval requests, transferring data, indexing data, announcing data to network indexers, and ensuring that clients can retrieve their data once it is stored on the SP's infrastructure.

By default the Boost daemon repository is located at ~/.boost

It contains the following files:

• api The local multi-address of Boost's libp2p API

• boost.db The sqlite database with all deal metadata

• boost.logs.db The sqlite database with the logs for deals

• config.toml The config file with all Boost's settings

• repo.lock A lock file created when Boost is running

• storage.json Deprecated (needed by legacy markets)

• token The token used when calling Boost's JSON RPC endpoints

It has the following directories:

• datastore Contains metadata about deals for legacy markets

• deal-staging The directory used by legacy markets for incoming data transfers

• incoming The directory used by Boost for incoming data transfers

• journal Contains journal events (used by legacy markets)

• keystore Contains the secret keys used by libp2p (eg the peer ID)

• kvlog Used by legacy markets datastore

Make storage deals with HTTP data transfer

Boost supports multiple options for data transfer when making storage deals, including HTTP. Clients can host their CAR file on an HTTP server, such as S3, and provide that URL when proposing the storage deal. Once accepted, Boost will automatically fetch the CAR file from the specified URL.

Web UI

Boost comes with a web interface that can be used to manage deals, watch disk usage, monitor funds, adjust settings and more.

A client for proposing deals that doesn't require a Lotus node

Boost comes with a client that can be used to make storage deals, and can be configured to point at a public Filecoin API endpoint. That means clients don't need to run a Filecoin node or sync from chain.

Boost stores metadata about deals in a SQLite database in the root directory of the Boost repository.

To open the database use a SQLite client:

sqlite3 boost.db

The database tables are

• Deals metadata about Boost storage deals (eg deal proposal) and their current state (e.g. checkpoint)

• FundsLogs log of each change in funds reserved for a deal

• FundsTagged how much FIL is tagged for deal collateral and publish message for a deal

• StorageLogs log of each change in storage reserved for a deal

• StorageTagged how much storage is tagged for a deal

Boost keeps a separate database just for deal logs, so as to make it easier to manage log data separately from deal metadata. The logs database is named boost.logs.db and it has a single table DealLogs that stores logs for each deal, indexed by UUID.

Migrations

Boost uses goose (https://pressly.github.io/goose/) tool and library for handling sqlite3 migrations.

goose can be installed following the instructions at https://pressly.github.io/goose/installation/

Migrations in Boost are stored in the /db/migrations directory.

Boost handles database migrations on start-up. If a user is running an older version of Boost, migrations up to the latest version are automatically applied on start-up.

Developers can use goose to inspect and apply migrations using the CLI:

➜  ~ goose
Usage: goose [OPTIONS] DRIVER DBSTRING COMMAND
...
Commands:
    up                   Migrate the DB to the most recent version available
    up-by-one            Migrate the DB up by 1
    up-to VERSION        Migrate the DB to a specific VERSION
    down                 Roll back the version by 1
    down-to VERSION      Roll back to a specific VERSION
    redo                 Re-run the latest migration
    reset                Roll back all migrations
    status               Dump the migration status for the current DB
    version              Print the current version of the database
    create NAME [sql|go] Creates new migration file with the current timestamp
    fix                  Apply sequential ordering to migrations

Boost is a tool for Storage Providers to manage data onboarding and retrieval on the Filecoin network.

Boost exposes libp2p interfaces for making storage and retrieval deals, a web interface for managing storage deals, and a GraphQL interface for accessing and updating real-time deal information.

Boost daemon exposes a GraphQL API that is used by the Web UI to query and update information about storage and retrieval deals.

You can test out queries, or explore the GraphQL API by clicking on the < Docs link at the top right of the page:

To run a GraphQL query with curl:

This 1m video shows how to use these tools to build an run a GraphQL query against Boost:

Example Queries

1. Query failed deals

2. Cancel a deal where ab12345c-5678-90de-12f3-45a6b78cd9ef is the deal ID

Go JSON-RPC client

To use the Boost Go client, the Go RPC-API library can be used to interact with the Boost API node.

1. Import the necessary Go module:

1. Create the following script:

1. Run go mod init to set up your go.mod file

2. You should now be able to interact with the Boost API.

Python JSON-RPC client

The JSON-RPC API can also be communicated with programmatically from other languages. Here is an example written in Python. Note that the method must be prefixed with Filecoin.

Groups

Uncategorized

Discover

Perms: read

Inputs: null

Response:

Auth

AuthNew

Perms: admin

Inputs:

Response: "Ynl0ZSBhcnJheQ=="

AuthVerify

Perms: read

Inputs:

Response:

Blockstore

BlockstoreGet

There are not yet any comments for this method.

Perms: read

Inputs:

Response: "Ynl0ZSBhcnJheQ=="

BlockstoreGetSize

Perms: read

Inputs:

Response: 123

BlockstoreHas

Perms: read

Inputs:

Response: true

Boost

BoostDeal

Perms: admin

Inputs:

Response:

BoostDealBySignedProposalCid

Perms: admin

Inputs:

Response:

BoostDirectDeal

Perms: admin

Inputs:

Response:

BoostDummyDeal

Perms: admin

Inputs:

Response:

BoostIndexerAnnounceAllDeals

There are not yet any comments for this method.

Perms: admin

Inputs: null

Response: {}

BoostIndexerAnnounceDeal

Perms: admin

Inputs:

Response: null

BoostIndexerAnnounceDealRemoved

Perms: admin

Inputs:

Response: null

BoostIndexerAnnounceLatest

Perms: admin

Inputs: null

Response: null

BoostIndexerAnnounceLatestHttp

Perms: admin

Inputs:

Response: null

BoostIndexerAnnounceLegacyDeal

Perms: admin

Inputs:

Response: null

BoostIndexerListMultihashes

Perms: admin

Inputs:

Response:

BoostLegacyDealByProposalCid

Perms: admin

Inputs:

Response:

BoostOfflineDealWithData

Perms: admin

Inputs:

Response:

Log

LogList

Perms: write

Inputs: null

Response:

LogSetLevel

Perms: write

Inputs:

Response: {}

Market

MarketGetAsk

Perms: read

Inputs: null

Response:

Net

ID

Perms: read

Inputs: null

Response: "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"

NetAddrsListen

Perms: read

Inputs: null

Response:

NetAgentVersion

Perms: read

Inputs:

Response: "string value"

NetAutoNatStatus

Perms: read

Inputs: null

Response:

NetBandwidthStats

Perms: read

Inputs: null

Response:

NetBandwidthStatsByPeer

Perms: read

Inputs: null

Response:

NetBandwidthStatsByProtocol

Perms: read

Inputs: null

Response:

NetBlockAdd

Perms: admin

Inputs:

Response: {}

NetBlockList

Perms: read

Inputs: null

Response:

NetBlockRemove

Perms: admin

Inputs:

Response: {}

NetConnect

Perms: write

Inputs:

Response: {}

NetConnectedness

Perms: read

Inputs:

Response: 1

NetDisconnect

Perms: write

Inputs:

Response: {}

NetFindPeer

Perms: read

Inputs:

Response:

NetLimit

Perms: read

Inputs:

Response:

NetPeerInfo

Perms: read

Inputs:

Response:

NetPeers

Perms: read

Inputs: null

Response:

NetPing

Perms: read

Inputs:

Response: 60000000000

NetProtectAdd

Perms: admin

Inputs:

Response: {}

NetProtectList

Perms: read

Inputs: null

Response:

NetProtectRemove

Perms: admin

Inputs:

Response: {}

NetPubsubScores

Perms: read

Inputs: null

Response:

NetSetLimit

Perms: admin

Inputs:

Response: {}

NetStat

Perms: read

Inputs:

Response:

Backup

OnlineBackup

There are not yet any comments for this method.

Perms: admin

Inputs:

Response: {}

PieceDirectory

PdBuildIndexForPieceCid

There are not yet any comments for this method.

Perms: admin

Inputs:

Response: {}

PdCleanup

Perms: admin

Inputs: null

Response: {}

PdRemoveDealForPiece

Perms: admin

Inputs:

Response: {}

In order to support the described retrieval use cases, LID maintains the following indexes:

multihash → []piece cid

To look up which pieces contain a block

piece cid → sector information {sector ID, offset, size}

To look up which sector a piece is in

piece cid → map<mulithash → block offset / size>

To look up where in the piece a block is and the block’s size

YugabyteDB is used to store the retrievals indexes. When a client makes a retrieval request, LID service is used to lookup the requested data within the Lotus miner. For more details see the architecture of LID.

Depending on the size of data your SP is holding, you can run YugabyteDB in a variety of ways. You can find more information about how to run YugabyteDB here.

For production use, YugabyteDB can be deployed as a highly available (HA) cluster. The cluster can be deployed on a Kubernetes instance or as a manual deployment. Please ensure to read all the pre-requisites before proceeding.

You can then start boostd-data on :8044 and connect it to the yugabyte with:

boostd-data run yugabyte --hosts 127.0.0.1 \
                         --connect-string="postgresql://postgres:postgres@127.0.0.1:5433" \
                         --addr 0.0.0.0:8044

YugabyteDB FAQ

1. What is YugabyteDB and why YugabyteDB? YugabyteDB is a high-performance distributed SQL database. Built using a unique combination of high-performance document store, per-shard distributed consensus replication and multi-shard ACID transactions (inspired by Google Spanner), YugabyteDB serves both scale-out RDBMS and internet-scale OLTP workloads with low query latency, extreme resilience against failures and global data distribution. We tested multiple open source databases for the Boost use case and found YugabyteDB to be well suited as it is highly performant and scales well horizonally.

2. How do I learn about YugabyteDB and what do I need to know about YugabyteDB as an SP? SPs should familiarize themselves with the “Deploy and Manage” section of the documentation along with the architecture before deploying YugabyteDB. YugabyteDB will also be utilized by Lotus V2 architecture as well. We plan to allow SPs to connect all of their Boost instances to a single LID (YugabyteDB) with Boost v2.1.0 release. This will also allow SPs to serve retrievals from any of their miners using a single booster-http or booster-bitswap process.

3. Which deployment should I choose? We recommend deploying YugabyteDB either locally on bare metal or using a managed deployment. Users can choose to deploy on the cloud if they can guarantee that scaling up the DB will not be impacted by the network bandwidth and infrastructure. It is recommended that YugabyteDB is highly available so that if one of the nodes is not available, your SP operations will not be impacted. You can find some of the example YugabyteDB deployment here. Please feel free to add your experience and deployment details to the discussion.

4. Can I convert YugabyteDB processes to systemd services? YugabyteDB does not ship as a systemd service by default. You will need to create a new service based on the commands you are running to start YT-Master and YT-Server processes. These commands can be customized based on user requirements and infrastructure.

5. Once YugabyteDB is deployed, do I need to perform any additional steps? Ideally, once the deployment is complete and can be reached over the network by boostd-data service, users do not need to perform any additional steps. If you wish to change the default username/password, you must also update the same on the --connect-string of boostd-data service.

YugabyteDB Maintenance and Upgrades

Boost is not currently tied to a specific version of YugabyteDB. We recommend setting up the latest stable version of the YugabyteDB when creating a new LID instance. Over the time, users can upgrade their YugabyteDB. When upgrading:

1. Boost-related processes must be stopped beforehand. If YugabyteDB is being utilised by other services apart from Boost then those services must be stopped as well.

2. Read through and follow the upgrade guide and reach out to YugabyteDB team on Slack if you have any questions.

As per the industry best practices, YugabyteDB should be regularly backed up for redundancy. Check out YugabyteDB docs if you need any help with management of the database.

YugabyteDB Troubleshooting

If you require help with YugabyteDB, we recommend checking out the troubleshooting guide. If the problems is not resolved then users should reach out to YugabyteDB team via Slack or Github for any support. You can also reach out to other users in #boost-help channel of filecoin slack to seek help from your peers.

YugabyteDB is designed to run on bare-metal machines, virtual machines (VMs), and containers. CPU and RAM

You should allocate adequate CPU and RAM. YugabyteDB has adequate defaults for running on a wide range of machines, and has been tested from 2 core to 64 core machines, and up to 200GB RAM.

Minimum requirement (Testing)

2 cores
2GB RAM

Production requirement

16+ cores
32GB+ RAM
Add more CPU (compared to adding more RAM) to improve performance.

Verify support for SSE2 and SSE4.2

YugabyteDB requires the SSE2 instruction set support, which was introduced into Intel chips with the Pentium 4 in 2001 and AMD processors in 2003. Most systems produced in the last several years are equipped with SSE2.

In addition, YugabyteDB requires SSE4.2.

To verify that your system supports SSE2, run the following command:

cat /proc/cpuinfo | grep sse2

To verify that your system supports SSE4.2, run the following command:

cat /proc/cpuinfo | grep sse4.2

Disks

SSDs (solid state disks) are required.

It is recommend that a minimum of 1TiB or more is allocated for YugabyteDB, depending on the amount of deal data you store and its average block size.

The hardware requirements for Boost are tied to the Lotus stack in a Filecoin SP deployment.

Depending on how much data you need to onboard, and how many deals you need to make with clients, hardware requirements in terms of CPU and Disk will vary.

General hardware requirements

CPU

A miner will need an 8+ core CPU.

We strongly recommend a CPU model with support for Intel SHA Extensions: AMD since Zen microarchitecture, or Intel since Ice Lake. Lack of SHA Extensions results in a very significant slow down.

The most significant computation that Boost has to do is the Piece CID calculation (also known as Piece Commitment or CommP). When Boost receives data from a client, it calculates the Merkle root out of the hashes of the Piece (padded .car file). The resulting root of the clean binary Merkle tree is the Piece CID.

RAM

128 GiB of RAM is needed at the very least.

Disk

Boost stores all data received from clients before Piece CID is calculated and compared against deal parameters received from clients. Next, deals are published on-chain, and Boost waits for a number of epoch confirmations before proceeding to pass data to the Lotus sealing subsystem. This means that depending on the throughput of your operation, you must have disk space for at least a few staged sectors.

For small deployments 100 GiB of disk are needed at the very least if we assume that Boost is to keep three 32 GiB sectors before passing them to the sealing subsystem.

We recommend using NVME disk for Boost. As Dagstore grows in size, the overall performance might slow down due to slow disk.

Background

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (cid).

Currently this task is performed by the DAG store component. The DAG store keeps its indices on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools.

To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it, the client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size.

At the moment there are two implementations of LID: - a simple LevelDB implementation, for small SPs who want to keep all information in a single process database. - a scalable YugabyteDB implementation, for medium and large size SPs with tens of thousands of deals.

When designing LID we considered the needs of various Storage Providers (SPs) and the operational overhead LID would have on their systems. We built a solution for: - small- SPs - holding up to 1PiB, and - mid- and large- size SPs - holding anywhere from 1PiB, up to 100PiB data

Depending on underlying block size and data format, index size can vary in size. Typically block sizes are between 16KiB and 1MiB.

Storing data on Filecoin

When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way clients can later retrieve subsets of the original deal data, without retrieving the full deal data.

Retrieving data

When a client makes a request for data by cid, LID: - checks which piece the cid is in, and where in the piece the data is - checks which sector the piece is in, and where in the sector the piece is - reads the data from the sector

Use cases

The retrieval use cases that the Local Index Directory supports are:

Graphsync retrieval

Request one root cid with a selector, receive many blocks

LID is able to: - look up which piece contains the root cid - look up which sector contains the piece - for each block, get the offset into the piece for the block

Bitswap retrieval

Request one block at a time

LID is able to: - look up which piece contains the block - get the size of the block (Bitswap asks for the size before getting the block data) - look up which sector contains the piece - get the offset into the piece for the block

HTTP retrieval

Request a whole piece

LID is able to look up which sector contains the piece.

Request an individual block

LID is able to: - look up which piece contains the block - look up which sector contains the piece - get the offset into the piece for the block

Request a file by root cid

LID is able to: - look up which piece contains the block - look up which sector contains the piece - for each block, get the offset into the piece for the block

Go to the following page for more information on booster-http:

Boost provides a capability to limit the number of simultaneous http transfer in progress to download the deal data from the clients.

This new configuration has been introduced in the ConfigVersion = 3 of the boost configuration file.

Configuration Variables

HTTP variables

Storage variables

How TransferLimiter works

The transferLimiter maintains a queue of transfers with a soft upper limit on the number of concurrent transfers.

To prevent slow or stalled transfers from blocking up the queue there are a couple of mitigations: The queue is ordered such that we

• start transferring data for the oldest deal first

• prefer to start transfers with peers that don't have any ongoing transfer

• once the soft limit is reached, don't allow any new transfers with peers that have existing stalled transfers

Note that peers are distinguished by their host (eg foo.bar:8080) not by libp2p peer ID. For example, if there is

• one active transfer with peer A

• one pending transfer (peer A)

• one pending transfer (peer B)

The algorithm will prefer to start a transfer with peer B than peer A. This helps to ensure that slow peers don't block the transfer queue.

The limit on the number of concurrent transfers is soft. Example: if there is a limit of 5 concurrent transfers and there are

• three active transfers

• two stalled transfers

then two more transfers are permitted to start (as long as they're not with one of the stalled peers)

Boost compatibility matrix

Building and Installing

Prerequisites

Linux / Ubuntu

macOS

Linux

Please ignore any output or onscreen instruction during the npm build unless there is an error.

macOS

Please ignore any output or onscreen instruction during the npm build unless there is an error.

Calibration Network

To build boost for calibnet, please complete the above pre-requisites and build using the following commands.

Using filters for fine-grained storage and retrieval deal acceptance

Your use case might demand very precise and dynamic control over a combination of deal parameters.

Lotus provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it:

• Filter for storage deals.

• RetrievalFilter for retrieval deals.

The executed command receives a JSON representation of the deal parameters on standard input, and upon completion, its exit code is interpreted as:

• 0: success, proceed with the deal.

• non-0: failure, reject the deal.

The most trivial filter rejecting any retrieval deal would be something like: RetrievalFilter = "/bin/false". /bin/false is binary that immediately exits with a code of 1.

This Perl script lets the miner deny specific clients and only accept deals that are set to start relatively soon.

You can also use a third party content policy framework like CIDgravity or bitscreen by Murmuration Labs:

# grab filter program
go get -u -v github.com/Murmuration-Labs/bitscreen

# add it to both filters
Filter = "/path/to/go/bin/bitscreen"
RetrievalFilter = "/path/to/go/bin/bitscreen"

Configuration

[Graphql]
  ListenAddress = "127.0.0.1"
  Port = 8080

[Monitoring]
  MpoolAlertEpochs = 30

By default, the web UI listens on the localhost interface on port 8080. We recommend keeping the UI listening on localhost or some internal IP within your private network to avoid accidentally exposing it to the internet.

You can access the web UI listening on the localhost interface on a remote server, you can open an SSH tunnel from your local machine:

ssh -L 8080:localhost:8080 myserver

Settings Page

Background

The monitoring stack we will use includes:

• Prometheus - collects metrics and powers dashboards in Grafana

• Tempo - collects traces and powers traces search in Grafana with Jaeger

• Grafana - provides visualization tools and dashboards for all metrics and traces

Lotus and Boost are already instrumented to produce traces and stats for Prometheus to collect.

The Boost team also packages a set of Grafana dashboards that are automatically provisioned as part of this setup.

Prerequisites

This setup has been tested on macOS and on Linux. We haven’t tested it on Windows, so YMMV.

All the monitoring stack containers run in Docker.

Steps

1. Install Docker

We have tested this setup with Docker 20.10.23 on macOS and Ubuntu.

1. DNS resolution for Prometheus

Depending on where your Filecoin processes (boostd, lotus, lotus-miner, booster-bitswap, etc.) are running, you need to confirm that they are reachable from Prometheus so that it can scrape their metrics.

By default the setup expects to find them within the same Docker network, so if you are running them elsewhere (i.e. on the `host` network), make sure to update the docker-compose file for the same.

1. Loki plugin for Docker

The loki plugin for docker is required to allow collecting logs from the services running on docker itself

1. Storage location

The default value is $HOME/.boost-monitoring

Please ensure that the storage directory exists and has permission 0775 or 0777 depending on your user.

1. Update metrics endpoints for Prometheus

In case, you are running all Boost services and monitoring stack on the same node, you can skip this step.

1. Start the monitoring stack using docker-compose

Verify Prometheus Targets

If you are running software firewall like `ufw`, you might need to modify your iptables and allow access from the Prometheus container / network to the Filecoin stack network, for example:

sudo docker network inspect monitoring # note the Subnet for the network sudo ufw allow from 172.18.0.0/16

Grafana dashboards

booster-http is a binary that can be run alongside the boostd process in order to serve retrievals over HTTP. This can be used to serve full pieces and also block data in the form of CAR files or raw block bytes. This can be further extended using a bifrost-gateway server if a Storage Provider wishes to also serve plain files and directories.

HTTP retrievals currently have no in-built payment or permissioning mechanism. Storage Providers may wish to front the booster-http server with a reverse proxy such as NGINX to handle operational concerns like SSL, authentication and load balancing.

Version History

• booster-http was introduced with Boost v1.2.0, able to serve full pieces using the /piece/ endpoint.

• In Boost v1.7.0 booster-http was extended to be able to serve CAR files, IPLD blocks and a full "trusted" gateway (files and directories).

• In Boost v2.1.0 booster-http was simplified to only serve "trustless" data: full pieces, CAR files and IPLD blocks; deferring optional "trusted" mode to bifrost-gateway for those Storage Providers who wish to offer this service to their users.

Trustless vs Trusted

"Trustless" refers to a mode in which the client does not need to trust a data provider in order to have confidence in the integrity and authenticity of the data they are receiving. Instead, trust is achieved by content-addressing, by having the server deliver to the client everything they require in order to verify the data they are receiving.

"Trusted" refers to a mode in which the client must trust the data provider to deliver the correct data. This is the standard mode that most web servers operate in, and is a common method of interacting with IPFS gateways (including bifrost-gateway). In this mode, the client must trust that the server is delivering the correct data, and that the server is not maliciously modifying the data in transit. Typically, the client does not receive enough information in order to verify the relationship between the content received and the content-address (CID) requested, hence the need to "trust" the server.

Trustless retrievals are always preferable where there does not exist a trust relationship between parties. By serving data only in a trustless mode, Storage Providers can avoid accusations of data alteration, and clients can be confident that they are receiving the data they requested. It also reduces the risk of having an HTTP endpoint as being flagged for serving illicit content. Learn more at Gateways and gatekeepers.

However, there are many circumstances in which a Storage Provider may wish to offer a trusted retrieval service, particularly due to convenience and interoperability with standard HTTP clients.

Piece retrievals

When performing certain actions, such as replicating deals, it can be convenient to retrieve the entire Piece (with padding) to ensure CommP integrity. When piece retrievals are enabled with booster-http with the --serve-pieces flag (enabled by default), the /piece/ endpoint is exposed.

curl 'http://{booster-http exposed url}/piece/bagaSomePieceCID' -o bagaSomePieceCID.piece

Payload retrievals (CAR and raw)

booster-http supports the IPFS Trustless Gateway specification for both CAR and raw IPLD data contained within deal payloads. When booster-http is run with the --serve-cars flag (enabled by default), the /ipfs/ endpoint is exposed.

To download a CAR file containing all the blocks linked under a given root CID, you can either pass an Accept: application/vnd.ipld.car header, or a ?format=car query parameter to the /ipfs/ endpoint. Likewise, to download raw single IPLD block bytes, you can either pass an Accept: application/vnd.ipld.raw header, or a ?format=raw query parameter to the /ipfs/ endpoint.

The /ipfs/ endpoint must be followed by a CID and one of the indicators above, for example:

# Fetch a complete DAG
curl 'http://{booster-http exposed url}/ipfs/bafySomePayloadCID?format=car' -o bafySomePayloadCID.car
# Fetch an individual block's bytes
curl 'http://{booster-http exposed url}/ipfs/bafySomeBlockCID?format=raw' -o bafySomeBlockCID.block

Alternatively, using Accept header:

# Fetch a complete DAG
curl -H 'Accept: application/vnd.ipld.car' 'http://{booster-http exposed url}/ipfs/bafySomePayloadCID' -o bafySomePayloadCID.car
# Fetch an individual block's bytes
curl -H 'Accept: application/vnd.ipld.car' 'http://{booster-http exposed url}/ipfs/bafySomeBlockCID' -o bafySomeBlockCID.block

It is also possible to use Lassie, a Trustless retrieval client, to perform and verify retrievals from booster-http:

lassie fetch --provider 'http://{booster-http exposed url}' bafySomePayloadCID

See the IPFS Trustless Gateway specification for more information on the ranges of queries possible with this protocol; most of which are implemented by Lassie as a client.

Trusted (plain file/directory) retrievals

For Storage Providers that wish to offer trusted access to payload data, in the same way that an IPFS Gateway bridges IPLD data to standard browser-accessible files and directories, bifrost-gateway can be used to translate booster-http's trustless data into trusted data. This is a separate process that must be configured to communicate with booster-http.

See Trusted HTTP Gateway Setup for full setup instructions.

curl 'http://{bifrost-gateway exposed url}/ipfs/{content ID}/{optional path to resource}' -o myimage.png

Local Setup

SPs should try a local setup and test their HTTP retrievals before proceeding to run booster-http in production.

The booster-http binary is built and installed with boostd binary. If you are planning to run booster-http on a different node, you can build and install the new binary. Otherwise, skip to step 3.

1. Clone the Boost repo and checkout the same release as your boostd version

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

1. Build and install the new binary

make build
sudo make build

1. Collect the token information for lotus-miner and lotus daemon API

export ENV_FULLNODE_API_INFO=`lotus auth api-info --perm=admin`
export FULLNODE_API_INFO=`echo $ENV_FULLNODE_API_INFO | awk '{split($0,a,"="); print a[2]}'`

export ENV_MINER_API_INFO=`lotus-miner auth api-info --perm=admin`
export MINER_API_INFO=`echo $ENV_MINER_API_INFO | awk '{split($0,a,"="); print a[2]}'`

1. Start the booster-http server with the above details

booster-http run --api-lid="ws://<boostd-data IP>:8044" --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO

Running Public Boost HTTP Retrieval

By default, the booster-http server listens to 0.0.0.0 port 7777. This can be changed with --address and --port; for example, you may wish to change the --address to 127.0.0.1 to restrict it to localhost-only.

When exposing booster-http to a public address for general retrievals, it is recommended that you run an HTTP reverse proxy such as NGINX to handle operational concerns such as:

• SSL

• Authentication and authorization

• Compression

• Load balancing

• Logging

booster-http can provide some of this functionality itself, but dedicated reverse proxy software is recommended for scalable production deployments. See HTTP Reverse Proxy Setup for more information and example configuration.

Making HTTP Retrieval Discoverable

To enable public discovery of the Boost HTTP server, Storage Providers should set the domain root in boostd's config.toml. Under the [DealMaking] section, set HTTPRetrievalMultiaddr to the public domain root in multi-address format.

Example config.toml section:

[DealMaking]
  HTTPRetrievalMultiaddr = "/dns/foo.com/tcp/443/https"

Clients can determine if a Storage Provider offers HTTP retrieval by running:

boost provider retrieval-transports <miner id>

Clients can also check the HTTP URL scheme version and supported queries

# Version
curl 'http://{bifrost-gateway exposed url}/info'

# Supported queries
curl 'http://{bifrost-gateway exposed url}/'

Storage Providers on Filecoin support various modes of retrievals for deal data that they have accepted from clients.

Reputation systems periodically crawl the network and record success rate for Storage Providers and determine whether they should make future FIL+ deals with them depending on how well they serve retrievals.

Boost v2.2.0 enables support for Direct Data Onbording (DDO) and no longer support legacy deals in any form. These changes resulted in some breaking changes in the configuration. Please follow the below steps to upgrade.

1. Please save you current configuration to a file with boostd config updated --diff

2. Note down the asking price setting from the boostd UI.

3. Pull the new stable release v2.2.0 or RC candidate v2.2.0-rcx

4. Rebuild the binaries with the new version

5. Stop booster-http, booster-bitswap, boostd and boostd-data in that order

6. Start boostd-data service first.

7. Start boostd after upgrade and shutdown after a successful start

8. Review the <boost repo>/config.toml for configuration parameters saved in step 1 and update them if they have been reset to default values.

9. Start boostd, booster-http and booster-bitswap services.

10. Restore the asking price setting from UI.

Start

1. Start YugabyteDB and wait for services to come online.

2. Start boostd-data process.

3. Start boostd process.

4. Start booster-http and booster-bitswap processes.

Stop

1. Stop booster-http and booster-bitswap processes.

2. Stop boostd process.

3. Stop boostd-data process.

4. Finally, you can stop YugabyteDB services.

Configuration options

booster-http is an independent process that can be run on the same machine as boostd or on a different machine. Multiple instances can also be run with different listen addresses if required.

   --pprof                                                  run pprof web server on localhost:6070 (default: false)
   --base-path value                                        the base path at which to run the web server
   --address value, --addr value                            the listen address for the web server (default: "0.0.0.0")
   --port value                                             the port the web server listens on (default: 7777)
   --api-lid value                                          the endpoint for the local index directory API, eg 'http://localhost:8042'
   --add-index-throttle value                               the maximum number of add index operations that can run in parallel (default: 4)
   --api-fullnode value                                     the endpoint for the full node API
   --api-storage value [ --api-storage value ]              the endpoint for the storage node API
   --serve-pieces                                           enables serving raw pieces (default: true)
   --serve-cars                                             serve CAR files with the Trustless IPFS Gateway API (default: true)
   --compression-level value                                compression level to use for responses, 0-9, 0 is no compression, 9 is maximum compression; set to 0 to disable if using a reverse proxy with compression (default: 1)
   --log-file value                                         path to file to append HTTP request and error logs to, defaults to stdout (-) (default: "-")
   --tracing                                                enables tracing of booster-http calls (default: false)
   --tracing-endpoint value                                 the endpoint for the tracing exporter (default: "http://tempo:14268/api/traces")
   --api-filter-endpoint value                              the endpoint to use for fetching a remote retrieval configuration for requests
   --api-filter-auth value                                  value to pass in the authorization header when sending a request to the API filter endpoint (e.g. 'Basic ~base64 encoded user/pass~'
   --badbits-denylists value [ --badbits-denylists value ]  the endpoints for fetching one or more custom BadBits list instead of the default one at https://badbits.dwebops.pub/denylist.json (default: "https://badbits.dwebops.pub/denylist.json")
   --help, -h                                               show help

booster-http, boostd and lotus

booster-http must have network, or localhost access to a full lotus node, a lotus-miner and a LID instance. The following options are required:

• --api-lid - the Local Index Directory (LID) service endpoint

• --api-fullnode - the lotus full node API endpoint, discoverable by running lotus auth api-info --perm=admin

• --api-storage - the lotus-miner API endpoint, discoverable by running lotus-miner auth api-info --perm=admin

Address and port

--address and --port configure the listen address and port of the booster-http server. By default HTTP server will listen on 0.0.0.0:7777. This can be set to other addresses and ports as required, e.g. 127.0.0.1 to serve localhost-only if running a reverse-proxy on the same server with a public listen address.

Pieces and CARs

--serve-pieces is enabled by default and allows retrieval of raw pieces on the /piece/ endpoint of your booster-http server. Requests for pieces require the full piece CID appended to /piece/.

Piece retrieval is typically performed to replicate deals, or by clients that are able to decode raw piece data.

--serve-cars is enabled by default and allows IPFS Trustless Gateway retrievals on the /ipfs/ endpoint. This is not a full "trusted" gateway, and requests must either ask for CARs containing one or more blocks from a root CID, or raw block bytes for a single CID. Requests can either pass an Accept: application/vnd.ipld.car header, or a ?format=car query parameter for CAR data. Or, to download raw single IPLD block bytes, either an Accept: application/vnd.ipld.raw header, or a ?format=raw query parameter.

A trustless retrieval client is recommended for performing and verifying retrievals from booster-http. See Lassie for more information. Providing Lassie with the --provider http://{booster-http exposed url} will perform verified, trustless retrievals to your booster-http instance.

BadBits denylist

booster-http (and booster-bitswap) automatically filter out known flagged content using the denylist maintained at https://badbits.dwebops.pub/denylist.json. Use one or more --badbits-denylists flags to point to a custom, valid BadBits denylist and override the default.

Compression

By default, booster-http will compress responses with gzip compression. --compression-level can be set between values of 0 and 9, where 0 is no compression and 9 is maximum compression. The default value is 1, which optimises for speed over compression ratio but this may be increased if required.

Compression can be disabled by setting --compression-level 0. If you are running a reverse proxy, such as NGINX, in front of booster-http that performs compression, you should disable compression in booster-http to avoid double-compression.

Logging

booster-http logs HTTP requests and errors to stdout by default. This can be overridden with --log-file to log to a file instead. The log file format is similar to typical NGINX or Apache log file formats and is suitable for ingestion into log aggregation tools such as Splunk or ELK. The format is as follows:

%s %s %s "%s" %d %d %d %s "%s" "%s"

Where the elements are:

1. RFC 3339 timestamp

2. Remote address

3. HTTP Method

4. Request path

5. Response status code

6. Response duration (in milliseconds)

7. Response size

8. Compression ratio (or - if no compression)

9. Remote user agent

10. Error (or "" if no error)

When using a reverse proxy, log output from the reverse proxy may be more suitable for storage and analysis than booster-http logging.

Trusted HTTP Gateway Setup

Where a Storage Provider wishes to serve plain files, including streaming video, audio and other media, a trusted HTTP gateway can be used to translate booster-http's trustless data into trusted data. This is a separate process that must be configured to communicate with booster-http.

Be aware that trusted HTTP responses typically do not count as "successful" retrievals by reputation systems and other retrieval checkers. It is also not possible to use trusted HTTP gateways to retrieve CAR files, which are required for verified retrievals by clients such as Lassie which is a recommended client for Filecoin downloads. When enabling trusted HTTP gateways, it is recommended to also enable the trustless CAR gateway to allow CAR retrievals; this includes via reverse proxies (see below).

bifrost-gateway

bifrost-gateway is an IPFS Trusted Gateway server that can be used to translate booster-http's trustless data into trusted data. bifrost-gateway is a separate process that must be configured to communicate with booster-http.

When running bifrost-gateway, two environment variables must be set:

• PROXY_GATEWAY_URL=http://{booster-http exposed url} to point to the booster-http address (without path)

• GRAPH_BACKEND=true to instruct bifrost-gateway to perform full CAR retrievals rather than individual IPLD block retrievals for efficiency

Additionally, --gateway-port may be used to override the default listen port of 8081.

PROXY_GATEWAY_URL=http://localhost:7777 GRAPH_BACKEND=true bifrost-gateway

curl --output /tmp/museum.jpg "http://localhost:8081/ipfs/bafybeidqindpi4ucx7kmrtnw3woc6jtl7bqvyiokrkpbbuy6gs6trn57tm/vincent/Vincent%20van%20Gogh_files/Caf%C3%A9tafel_met_absint_-_s0186V1962_-_Van_Gogh_Museum.jpg"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11830  100 11830    0     0   140k      0 --:--:-- --:--:-- --:--:--  175k

open /tmp/museum.jpg

A reverse proxy can be configured to talk to bifrost-gateway, but be aware that IPFS gateway's are typically exposed on the /ipfs/ endpoint, which is also the endpoint of the trustless gateway which is required for standard Filecoin retrievals (e.g. using Lassie). A reverse proxy combining both the booster-http trustless endpoint and a bifrost-gateway trusted endpoint must be configured to route /ipfs/ requests to booster-http where the Accept header contains application/vnd.ipld.car or application/vnd.ipld.raw, and /ipfs/ requests to bifrost-gateway where the Accept header contains anything else, such as text/html or */*. Alternatively, separate reverse proxies may be configured for both booster-http and bifrost-gateway.

Reverse Proxy Setup

Storage Providers should secure their booster-http before exposing it to the public. Storage Providers may use any tool available to limit who can download files, the number of requests per second, and the download bandwidth each client can use per second.

NGNIX is one such reverse proxy which may be used in front of a booster-http instance. This section provides only a basic coverage of the ways in which NGINX can set access limits, rate limits and bandwidth limits. In particular it’s possible to add limits by request token, or using JWT tokens. The examples in this section are adapted from Deploying NGINX as an API Gateway which goes into more detail.

By default NGINX puts configuration files into /etc/nginx

The default configuration file is /etc/nginx/sites-available/default

Setup server block

In this example, we are setting up an NGINX server listen on port 7575 and forward requests to booster-http on port 7777.

The IPFS Trustless Gateway serves content from /ipfs/ and the piece retrieval endpoint is /piece/. A location block that matches both of these paths will forward requests to booster-http.

# ipfs gateway config
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location ~* ^/(ipfs|piece)/ {
                proxy_pass http://127.0.0.1:7777;
        }
}

Alternatively, to only forward /ipfs/ requests to booster-http our location directive can be simplified:

        location /ipfs/ {
                proxy_pass http://127.0.0.1:7777;
        }

Limiting Access

We can limit access to the IPFS gateway using the standard .htaccess file. This file contains usernames and passwords. In this example we create a user named alice:

mkdir /etc/nginx/ipfs-gateway.conf.d
htpasswd -c /etc/nginx/ipfs-gateway.conf.d/.htpasswd alice
New password:
Re-type new password:
Adding password for user alice

Include the .htaccess file in the /etc/nginx/sites-available/default

 # ipfs gateway config
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location ~* ^/(ipfs|piece)/ {
                # htaccess authentication
                auth_basic "Restricted Server";
                auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd;
                proxy_pass http://127.0.0.1:7777;
        }
}

Now when we open any URL under the path /ipfs/ we will be presented with a Sign-in dialog.

Rate Limiting

To prevent users from making too many requests per second, we can add rate limits.

1. Create a file with the rate limiting configuration at /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf

2. Add a request zone limit to the file of 1 request per second, per client IP

limit_req_zone $binary_remote_addr zone=client_ip_10rs:1m rate=1r/s;

1. Include ipfs-gateway.conf in /etc/nginx/sites-available/default and set the response for too many requests to HTTP response code 429

include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf;
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location ~* ^/(ipfs|piece)/ {
                # htaccess authentication
                auth_basic "Restricted Server";
                auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd;

                limit_req zone=client_ip_10rs;
                limit_req_status 429;
                proxy_pass http://127.0.0.1:7777;
        }
}

1. Click the refresh button in your browser on any path under /ipfs/ more than once per second you will see a 429 error page.

Bandwidth Limiting

It is also recommended to limit the amount of bandwidth that clients can take up when downloading data from booster-http. This ensures a fair bandwidth distribution to each client and prevents situations where one client ends up choking the booster-http instance.

1. Create a new .htaccess user called bob

htpasswd /etc/nginx/ipfs-gateway.conf.d/.htpasswd bob

1. Add a mapping from .htaccess username to bandwidth limit in /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf

map $remote_user $bandwidth_limit {
    default  1k;
    "alice"  10k;
    "bob"    512k;
}

1. Add the bandwidth limit to /etc/nginx/sites-available/default

include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf;
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location ~* ^/(ipfs|piece)/ {
                # htaccess authentication
                auth_basic "Restricted Server";
                auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd;

                limit_rate $bandwidth_limit;

                limit_req zone=client_ip_10rs;
                limit_req_status 429;
                proxy_pass http://127.0.0.1:7777;
        }
}

1. To verify bandwidth limiting, use curl to download a file with user alice and then bob Note the difference in the Average Dload column (the average download speed).

Compression

We can configure NGINX's compression settings in its main http block, typically in the /etc/nginx/nginx.conf file.

http {
    ...
    gzip on;
    gzip_vary on;      # Add Vary: Accept-Encoding header
    gzip_proxied any;  # Compress data for all proxied requests
    ...
}

See the NGINX gzip module documentation for more information.

HTTPS

NGINX can be configured to serve HTTPS traffic. This is recommended for production deployments. See NGINX HTTPS configuration for more information.

Caching

NGINX can be configured to cache responses from booster-http. Since booster-http serves content addressed data that does not change, this is particularly well suited to caching in cases where certain content is frequently requested. booster-http sets a long Cache-Control header by default, so NGINX will cache responses for a long time by default.

See the NGINX proxy module documentation for more information on how to configure caching.

booster-bitswap is a binary that runs alongside the boostd process, to serve retrievals over the Bitswap protocol. This feature of boost provides a number of tools for managing a production grade Bitswap retrieval service for a Storage Provider's content.

Why enable retrievals via Bitswap?

Bitswap retrieval introduces interoperability between IPFS and Filecoin, as it enables clients to retrieve Filecoin data over IPFS. This expands the reach of the Filecoin network considerably, increasing the value proposition for users to store data on the Filecoin network. This benefits the whole community, including SPs. Users will be able to access data directly via IPFS, as well as benefit from retrieval markets (e.g. Saturn) and compute over data projects (e.g. Bacalhau).

booster-bitswap modes

There are two primary "modes" for exposing booster-bitswap to the internet.

1. In private mode the booster-bitswap peer ID is not publicly accessible to the internet. Instead, public Bitswap traffic goes to boostd itself, which then acts as a reverse proxy, forwarding that traffic on to booster-bitswap. This is similar to the way one might configure Nginx as a reverse proxy for an otherwise private web server. private mode is simpler to setup but may produce greater load on boostd as a protocol proxy.

2. In public mode the public internet firewall must be configured to forward traffic directly to the booster-bitswap instance. boostd is configured to announce the public address of booster-bitswap to the network indexer (the network indexer is the service that clients can query to discover where to retrieve content). This mode offers greater flexibility and performance. You can even setup booster-bitswap to run over a separate internet connection from boostd. However, it might require additional configuration and changes to your overall network infrastructure.

Demo configuration

You can configure booster-bitswap in the demo mode and familiarise yourself with the configuration. Once you are confident and familiar with the options, please go ahead and configure booster-bitswap for production use.

The booster-bitswap binary is built and installed with boostd binary. If you are planning to run booster-bitswap on a different node, you can build and install the new binary. Otherwise, skip to step 3.

1. Clone the Boost repo and checkout the same release as your boostd version

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

2. Build and install the booster-bitswap binary:

make build
sudo make build

3. Initialize booster-bitswap:

booster-bitswap init

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Run booster-bitswap

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044"

6. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

7. Fetching over bitswap by running

booster-bitswap fetch /ip4/127.0.0.1/tcp/8888/p2p/{peerID} {rootCID} outfile.car

Where peerID is the peer id recorded when you ran booster-bitswap init and rootCID is the CID of a data CID known to be stored on your SP.

Setup booster-bitswap to serve retrievals

As described above, booster-bitswap can be configured to serve the retrievals in 2 modes. We recommend using public mode to avoid greater load on boostd as a protocol proxy.

Private Mode

The booster-bitswap binary is built and installed with boostd binary. If you are planning to run booster-bitswap on a different node, you can build and install the new binary. Otherwise, skip to step 3.

1. Clone the Boost repo and checkout the same release as your boostd version

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

2. Build and install the booster-bitswap binary:

make build
sudo make build

3. Initialize booster-bitswap:

booster-bitswap init

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Stop boostd and edit ~/.boost/config.toml to set the peer ID for bitswap

[Dealmaking]
  BitswapPeerID ="{peer id for booster bitswap you recorded earlier}"

6. Start boostd service again

7. Collect the token information for lotus-miner and lotus daemon API

export `lotus auth api-info --perm=admin`

export `lotus-miner auth api-info --perm=admin`

8. Run booster-bitswap

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --proxy={boostd multiaddress} --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO

9. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

10. Try to fetch a payload CID over bitswap to verify your configuration

Public mode

The booster-bitswap binary is built and installed with boostd binary. If you are planning to run booster-bitswap on a different node, you can build and install the new binary. Otherwise, skip to step 3.

1. Clone the Boost repo and checkout the same release as your boostd version

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

2. Build and install the booster-bitswap binary:

make build
sudo make build

3. Initialize booster-bitswap:

booster-bitswap init

4. Record the peer ID output by booster-bitswap init -- we will need this peer id later

5. Stop boostd and edit ~/.boost/config.toml to set the peer ID for bitswap

[DealMaking]
 BitswapPeerID ="{peer id for bosoter bitswap you recorded earlier}"
 BitswapPublicAddresses = ["/ip4/{booster-bitswap public IP}/tcp/{booster-bitswap public port}"]
 BitswapPrivKeyFile = "{path to libp2p private key file for booster bitswap}"

The reason boost needs to know the public multiaddresses and libp2p private key for booster-bitswap is so it can properly announce these records to the network indexer.

6. Start boostd service again

7. Collect the token information for lotus-miner and lotus daemon API

export `lotus auth api-info --perm=admin`

export `lotus-miner auth api-info --perm=admin`

8. Run booster-bitswap

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO

9. By default, booster-bitswap runs on port 8888. You can use --port to override this behaviour

10. Try to fetch a payload CID over bitswap to verify your configuration

booster-bitswap configuration

booster-bitswap provides a number of performance and safety tools for managing a production grade bitswap server without overloading your infrastructure.

Bitswap server performance

Depending on your hardware you may wish to increase or decrease the default parameters for the bitswap server internals. In the following example we are increasing the worker count for various components up to 600. This will utilize more CPU and I/O, but improve the performance of retrievals. See the command line help docs for details on each parameter.

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" \
  --engine-blockstore-worker-count=600 \
  --engine-task-worker-count=600 \
  --max-outstanding-bytes-per-peer=33554432 \
  --target-message-size=1048576 \
  --task-worker-count=600

BadBits filtering

Booster-bitswap is automatically setup to deny all requests for CIDs that are on the BadBits Denylist. The default badbits list can be override or addition badbits list can be provided to the booster-bitswap instance.

To override the default badbits list

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --badbits-denylists <URL>

To provide additional badbits list

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --badbits-denylists https://badbits.dwebops.pub/denylist.json <URL1> <URL2>

Request filtering

booster-bitswap provides a number of controls for filtering requests and limiting resource usage. These are expressed in a JSON configuration file <booster-bitswap repo>/retrievalconfig.json

{
   "AllowDenyList": { // list of peers to either deny or allow (denying all others)
   	"Type": "allowlist",  // "allowlist" or "denylist"
   		"PeerIDs": [
   			"Qma9T5YraSnpRDZqRR4krcSJabThc8nwZuJV3LercPHufi",
   			"QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N"
   		]
   },
       "UnderMaintenance": false, // when set to true, denies all requests
       "StorageProviderLimits": {
           "Bitswap": {
                       "SimultaneousRequests": 100, // bitswap block requests served at the same time across peers
                       "SimultaneousRequestsPerPeer": 10, // bitswap block requests served at the same time for a single peer
   		"MaxBandwidth": "100mb" // human readable size metric, per second
   	}
   }
}

To make changes to the current configuration, you need to edit the retrievalconfig.json file and restart booster-bitswap for the changes to take affect. All configs are optional and absent parameters generally default to no filtering at all for the given parameter.

You can also configure booster-bitswap to fetch your retrieval config from a remote HTTP API, possibly one provided by a third party configuration tool like CIDGravity. To do this, start booster-bitswap with the --api-filter-endpoint {url} option where URL is the HTTP URL for an API serving the above JSON format. Optionally, add --api-filter-auth {authheader}, if you need to pass a value for the HTTP Authorization header with your API

booster-bitswap run --api-lid="ws://<boostd-data IP>:8044" --api-filter-endpoint <URL> --api-filter-auth <OPTIONAL SCURITY HEADERS>

When you setup with an API endpoint, booster-bitswap will update its local configuration from the API every five minutes, so you won't have to restart booster-bitswap to make a change. Please, be aware that the remote config will overwrite, rather than merge, with the local config.

Bandwidth limiting

Limiting bandwidth within booster-bitswap will not provide the optimal user experience. Dependent on individual setup, setting up limitations within the software could have a larger impact on the storage provider operations. Therefore, we recommend storage providers to set up their own bandwidth limitations using existing tools.

There are multiple options to setup bandwidth limitating.

1. At the ISP level - dedicated bandwidth is provided to the node running booster-bitswap.

2. At the router level - we recommend configuring the bandwidth at the router level as it provides more flexibility and can be updated as needed. To configure the bandwidth on your router, please check with your manufacturer.

3. Limit the bandwidth using different tools available in Linux. Here are some of the examples of such tools. Please feel free to use any other tools not listed here and open a Github issue to add your example to this page.

TC

TC is used to configure Traffic Control in the Linux kernel. There are examples available online detailing how to configure rate limiting using TC.

You can use the below commands to run a very basic configuration.

sudo tc qdisc add dev <network interface> root handle 1: htb
sudo tc class add dev <network interface> parent 1: classid 1:20 htb rate 100mibit
sudo tc qdisc add dev <network interface> parent 1:20 handle 20: sfq perturb 10
sudo tc filter add dev <network interface> parent 1: protocol ip prio 1 basic match 'cmp(u16 at 0 layer transport eq 8888)' flowid 1:20

Trickle

Trickle is a portable lightweight user space bandwidth shaper, that either runs in collaborative mode (together with trickled) or in standalone mode. You can read more about rate limiting with trickle here. Here's a starting point for configuration in trickle to rate limit the booster-bitswap service.

[booster-bitswap]
Priority = <value>
Time-Smoothing = <value>
Length-Smoothing = <value>

Wondershaper

Another way of controlling network traffic is to limit bandwidth on individual network interface cards (NICs). Wondershaper is a small Bash script that uses the tc command-line utility in the background to let you regulate the amount of data flowing through a particular NIC. As you can imagine, while you can use wondershaper on a machine with a single NIC, its real advantage is on a machine with multiple NICs. Just like trickle, wondershaper is available in the official repositories of mainstream distributions. To limit network traffic with wondershaper, specify the NIC on which you wish to restrict traffic with the download and upload speed in kilobits per second.

For example,

wondershaper enp5s0 4096 1024

Introduction

Boost v2 introduces the Local Index Directory as a replacement for the DAG store. It scales horizontally and provides a more intuitive experience for users, by surfacing problems in the UI and providing repair functionality.

Prerequisites

Install YugabyteDB

The Local Index Directory (LID) stores retrieval indices in YugabyteDB. Retrieval indices store the size and location of each block in the deal data.

It is recommend to run YugabyteDB on a dedicated host with SSD drives. Depending on how many blocks there are in the user data, the retrieval indices may require up to 2% of the size of the unsealed data. e.g. 1 TiB of unsealed user data may require a 20 GiB index.

YugabyteDB requires about the same amount of space as your DAG store requires today.

You can find more information about YugabyteDB in the Components section:

Instructions

Follow these instructions in order to setup a new Boost instance with boostd-data service:

1. Make sure you have a Lotus node and miner running

2. Create and send funds to two new wallets on the lotus node to be used for Boost. Boost currently uses two wallets for storage deals:

• The publish storage deals wallet - This wallet pays the gas cost when Boost sends the PublishStorageDeals message.

• The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the StorageMarketActor.

export PUBLISH_STORAGE_DEALS_WALLET=`lotus wallet new bls`
export COLLAT_WALLET=`lotus wallet new bls`
lotus send --from mywallet $PUBLISH_STORAGE_DEALS_WALLET 10
lotus send --from mywallet $COLLAT_WALLET 10

1. Set the publish storage deals wallet as a control wallet

lotus-miner actor control set --really-do-it $PUBLISH_STORAGE_DEALS_WALLET

1. Build the latest Boost V2 binaries

2. Run the boostd-data service

boostd-data is a data proxy service which abstracts the access to LID through an established API. It makes it easier to secure the underlying database and not expose it. boostd-data listens to a websocket interface, which is the entrypoint which should be exposed to boostd, andbooster-http

Start the boostd-data service with parameters to connect to YugabyteDB on its Cassandra and PostgreSQL interfaces:

./boostd-data run yugabyte \
  --hosts <yugabytedb-hosts> \
  --connect-string="postgresql://<username>:<password>@<yugabytedb>:5433" \
  --addr 0.0.0.0:8044

1. Create and initialize the Boost repository

Boost keeps all data in a directory called the repository. By default the repository is at ~/.boost. To use a different location pass the --boost-repo parameter (must precede any particular command verb, e.g. boostd --boost-repo=<PATH> init).

Export the environment variables needed for boostd init to connect to the lotus daemon and lotus miner.

export $(lotus auth api-info --perm=admin)
export $(lotus-miner auth api-info --perm=admin)

Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the boost node to make JSON-RPC calls to the mining/sealing/proving node.

export APISEALER=`echo $MINER_API_INFO`
export APISECTORINDEX=`echo $MINER_API_INFO`

Run boostd init to create and initialize the repository:

boostd --vv init \
       --api-sealer=$APISEALER \
       --api-sector-index=$APISECTORINDEX \
       --wallet-publish-storage-deals=$PUBLISH_STORAGE_DEALS_WALLET \
       --wallet-deal-collateral=$COLLAT_WALLET \
       --max-staging-deals-bytes=50000000000

• --api-sealer is the API info for the lotus-miner instance that does sealing

• --api-sector-index is the API info for the lotus-miner instance that provides storage

• --max-staging-deals-bytes is the maximum amount of storage to be used for downloaded files (once the limit is reached Boost will reject subsequent incoming deals)

1. Update ulimit file descriptor limit if necessary. Boost deals will fail if the file descriptor limit for the process is not set high enough. This limit can be raised temporarily before starting the Boost process by running the command ulimit -n 1048576. We recommend setting it permanently by following the Permanently Setting Your ULIMIT System Value guide.

2. Before you start your boostd, it is important that it is reachable from any peer in the Filecoin network. For this, you will need a stable public IP and edit your <boostd repo>/config.toml as follows:

   Copy

   [Libp2p]
     ListenAddresses = ["/ip4/0.0.0.0/tcp/24001"] # choose a fixed port
     AnnounceAddresses = ["/ip4/<YOUR_PUBLIC_IP_ADDRESS>/tcp/24001"] # important!

3. Add boostd-data details to the boostd config Configure boostd repository config (located at <boostd repo>/config.toml) to point to the exposed boostd-data service endpoint. Note that the connection must be configured to go over a websocket. For example:

   Copy

     [LocalIndexDirectory]
     ServiceApiInfo = "ws://<boostd-data>:8044"

4. Start the boostd process

   Copy

   boostd --vv run

5. Make sure that the correct peer id and multiaddr for your SP is set on chain, given that boost init generates a new identity. Use the following commands to update the values on chain:

lotus-miner actor set-addrs <MULTIADDR>
lotus-miner actor set-peer-id <PEER_ID>

Conclusion

At this stage you should have the latest version of Boost running with the Local Index Directory. Go to the Local Index Directory page and review the number sections:

Pieces

Pieces section shows counters for total pieces of user data that your SP is storing as well as whether you are keeping unsealed and indexed copies of them.

Flagged pieces

Flagged pieces are pieces that either lack an unsealed copy or are missing an index. For the sealed-only user data, you should make sure that you unseal individual sectors if you want this data to be retrievable.

Sealed only copies of data are not retrievable and are only being proven on-chain within the corresponding deadline / window. Typically sealed only data is considered as archival as it is not immediately retrievable. If the client requests it, the SP sealing pipeline must first unseal it, which typically takes 1-2 hours, and only then the data becomes available.

Flagged (unsealed) pieces is user data that your SP is hosting, which is not indexed.

Deal Sector Copies

Deal Sector Copies section displays counters of your sectors state - whether you keep unsealed copies for all sectors or not. Ideally the SP should keep unsealed copies for all data that should be immediately retrievable.

Sector Proving State

Sector Proving State section displays counters of your active and inactive sectors - active sectors are those that are actively proven on-chain, inactive sectors are those that you might have failed to publish a WindowPoSt for, or are expired or removed.

Introduction

Boost v2 introduces the Local Index Directory as a replacement for the DAG store. It scales horizontally and provides a more intuitive experience for users, by surfacing problems in the UI and providing repair functionality.

Architecture

When boost receives a storage deal, it creates an index of all the block locations in the deal data, and stores the index in LID.

When boostd / booster-http etc gets a request for a block it:

• gets the block sector and offset from the LID index

• requests the data at that sector and offset from the miner

A large miner with many incoming retrieval requests needs many boostd / booster-http / booster-bitswap processes to serve those requests. These processes need to look up block locations in a centralized index.

We tested several databases and found that YugabyteDB is best suited to the indexing workload because

• it performs well on off-the-shelf hardware

• it's easy to scale up by adding more machines

• it has great documentation

• once set up, it can be managed through a web UI

Connecting multiple boost instances to a single LID

It is possible to connect multiple boostd instances to a single LID instance. In this scenario, each boostd instance still stores data to a single miner. eg boostd A stores data to miner A, boostd B stores data to miner B. However each boostd instance saves retrieval indexes in a single, shared LID instance.

For retrieval, each boostd instance can query the shared LID instance (to find out which miner has the data) and retrieve data from any miner in the cluster.

booster-bitswap and booster-http can also be configured to query the shared LID instance, and retrieve data from any miner in the cluster.

Prerequisites

Install YugabyteDB

The Local Index Directory stores retrieval indices in a YugabyteDB database. Retrieval indices store the size and location of each block in the deal data.

We recommend running YugabyteDB on a dedicated machine with SSD drives. Depending on how many blocks there are in the user data, the retrieval indices may require up to 2% of the size of the unsealed data. e.g. 1 TiB of unsealed user data may require a 20 GiB index.

YugabyteDB should require about the same amount of space as your DAG store requires today.

You can find more information about YugabyteDB in the Components section:

Instructions

Follow these instructions in order to migrate your existing DAG store into the new Local Index Directory and upgrade from Boost v1 to Boost v2:

1. Clone the Boost repository to a temporary directory

Note: Don’t overwrite your existing boost instance at this stage

cd /tmp
git clone https://github.com/filecoin-project/boost.git boostv2
cd boostv2

2. Check out the Boost v2 release

git checkout v2.x.x

3. Build from source

make

4. Migrate dagstore indices

Depending on the amount of data your SP is storing, this step could take anywhere from a few minutes to a few hours. You can run it even while Boost v1 continues to run. The command can be stopped and restarted. It will continue from where it left off.

Run the migration with parameters to connect to YugabyteDB on its Cassandra and PostgreSQL interfaces:

./migrate-lid yugabyte \
  --hosts <yugabytedb-hosts> \
  --connect-string="postgresql://<username>:<password>@<yugabytedb>:5433" \
  dagstore

It will output a progress bar, and also a log file with detailed migration information at migrate-yugabyte.log

5. Run the boostd-data service

boostd-data is a data proxy service which abstracts the access to LID through an established interface. It makes it easier to secure the underlying database and not expose it. boostd-data listens to a websocket interface, which is the entrypoint which should be exposed to boostd, andbooster-http

Start the boostd-data service with parameters to connect to YugabyteDB on its Cassandra and PostgreSQL interfaces:

./boostd-data run yugabyte \
  --hosts <yugabytedb-hosts> \
  --connect-string="postgresql://<username>:<password>@<yugabytedb>:5433" \
  --addr 0.0.0.0:8044

6. Update boostd repository config

Configure boostd repository config (located at <boostd repo>/config.toml) to point to the exposed boostd-data service endpoint. Note that the connection must be configured to go over a websocket.

For example:

[LocalIndexDirectory]
  ServiceApiInfo = "ws://<boostd-data>:8044"

6.1 Add miners to boostd repository config

If you are deploying a single LID instance with multiple boost instances, you will also need to add to config the RPC endpoint for each miner in the cluster. This allows boostd to serve data for each miner over Graphsync.

[DealMaking]
  GraphsyncStorageAccessApiInfo = [
    # Make sure to include the miner that this boostd instance
    # stores data to, as well as the other miners.
    # Use `lotus-miner auth api-info` to get the RPC API connect string.
    "<auth token>:/ip4/<ip>/tcp/2345/http",
    "<auth token>:/ip4/<ip>/tcp/2345/http"
  ]

Make sure to test that this boostd instance can reach each miner by running

$ MINER_API_INFO=<auth token>:/ip4/<ip>/tcp/2345/http lotus-miner info

7. Install Boost v2

make install

Note that in v2 booster-http and booster-bitswap take slightly different parameters (see below).

8. Stop boostd, booster-http and booster-bitswap

You need to stop boostd before migrating piece info data.

9. Migrate piece info data (information about which sector each deal is stored in)

This should take no more than a few minutes.

./migrate-lid yugabyte \
  --hosts <yugabytedb-hosts> \
  --connect-string="postgresql://<username>:<password>@<yugabytedb>:5433" \
  pieceinfo

10. Start the upgraded versions of boostd, booster-http and booster-bitswap

Note that booster-http and booster-bitswap take slightly different parameters:

• --api-boost is removed

• There is a new parameter --api-lid that points to the boostd-data service (which hosts LID), e.g. --api-lid="ws://<boostd-data>:8044"

• If you are deploying a single LID instance with multiple booster-bitswap and booster-http instances, you should supply a --api-storage flag for each one

  • eg --api-storage=MINER_API_INFO_1 --api-storage=MINER_API_INFO_2

  • Make sure to test that this booster-http / booster-bitswap instance can reach each miner by running

    Copy

    $ MINER_API_INFO=MINER_API_INFO_1 lotus-miner info

11. Clean up the dagstore directory from boostd repo and the temporary boost github repo

Verify the setup

1. Test how long it takes to reindex a piece

time boostd lid gen-index <piece CID>

2. Perform a retrieval using Graphsync, Bitswap and HTTP

# boost retrieve --provider=<miner id> -o output.dat <cid>

# booster-bitswap fetch /ip4/127.0.0.1/tcp/8888/p2p/{peerID} {rootCID} outfile.car

# curl -H "Accept:application/vnd.ipld.car;" http://{SP's http retrieval URL}/ipfs/bafySomePayloadCID -o bafySomePayloadCID.car

Conclusion

At this stage you should have the latest version of Boost running with the Local Index Directory. Go to the Local Index Directory page and review the number sections:

Pieces

Pieces section shows counters for total pieces of user data that your SP is storing as well as whether you are keeping unsealed and indexed copies of them.

Flagged pieces

Flagged pieces are pieces that either lack an unsealed copy or are missing an index. For the sealed-only user data, you should make sure that you unseal individual sectors if you want this data to be retrievable.

Sealed only copies of data are not retrievable and are only being proven on-chain within the corresponding deadline / window. Typically sealed only data is considered as archival as it is not immediately retrievable. If the client requests it, the SP sealing pipeline must first unseal it, which typically takes 1-2 hours, and only then the data becomes available.

Flagged (unsealed) pieces is user data that your SP is hosting, which is not indexed.

Deal Sector Copies

Deal Sector Copies section displays counters of your sectors state - whether you keep unsealed copies for all sectors or not. Ideally the SP should keep unsealed copies for all data that should be immediately retrievable.

Sector Proving State

Sector Proving State section displays counters of your active and inactive sectors - active sectors are those that are actively proven on-chain, inactive sectors are those that you might have failed to publish a WindowPoSt for, or are expired or removed.

Fixing individual flagged unsealed pieces from the Boost Web UI is possible directly from the Web UI.

If the SP has a lot of flagged pieces, you can automate the re-indexing of pieces with the following commands:

Fetch the piececids of all the flagged unsealed pieces from LID

Trigger re-indexing of each piececid

Automatically re-index marked pieces only

This little script will fetch up to a thousand flagged pieces that are not processed yet, and will process them 4 at a time. Change the -P4 parameter to change the concurrency.

1. Pull the new stable release v2.1.0 or RC candidate v2.1.0-rcx

2. Rebuild the binaries with the new version

3. Stop booster-http, booster-bitswap, boostd and boostd-data in that order

4. Start boostd-data service first. If you are using systemd service files to manage the process, then please start it manually without using the systemd files.

1. Once boostd-data service starts, it will throw the below error and quit the process

1. If you do not see the above error and process does not exit, then you do not require a migration. At this point, please skip to step 9.

2. Run the one time migration.

1. Once the migration is finished (few seconds to 2 minutes), SPs using systemd to maintain boostd-data service should also update their connect-string in the systemd service files.

2. Start the boostd-data process (or service) and start your boostd instance. Go to the UI and confirm that you can see your minerID on the top left side of the page and the LID page is being populated correctly.

3. Start booster-http and booster-bitswap services.

Backup

Boost now supports both online and offline backups. The backup command will output a backup directory containing the following files.

1. metadata - contains backup of leveldb

2. boostd.db - backup of deals database

3. keystore - directory containing libp2p keys

4. token - API token

5. config - directory containing all config files and config.toml link

6. storage.json - file containing storage details

Backup does not back up deal logs and Local Index Directory.

Online backup

You can take an online backup with the below command

The online backup supports running only one instance at a time and you might see a locking error if another instance of backup is already running.

Offline backup

1. Shutdown boostd before taking a backup

2. Take a backup using the command line

Restore

1. Make sure that --boost-repo flag is set if you wish to restore to a custom location. Otherwise, it will be restored to ~/.boost directory

2. Restore the boost repo using the command line

Dealmaking section

This section controls parameters for making storage and retrieval deals:

StartEpochSealingBuffer allows lotus-miner to seal a sector before a certain epoch. For example: if the current epoch is 1000 and a deal within a sector must start on epoch 1500, then lotus-miner must wait until the current epoch is 1500 before it can start sealing that sector. However, if Boost sets StartEpochSealingBuffer to 500, the lotus-miner can start sealing the sector at epoch 1000.

If there are multiple deals in a sector, the deal with a start time closest to the current epoch is what StartEpochSealingBuffer will be based off. So, if the sector in our example has three deals that start on epoch 1000, 1200, and 1400, then lotus-miner will start sealing the sector at epoch 500.

Publishing several deals in one message

The PublishStorageDeals message can publish multiple deals in a single message. When a deal is ready to be published, Boost will wait up to PublishMsgPeriod for other deals to be ready before sending the PublishStorageDeals message.

However, once MaxDealsPerPublishMsg is ready, Boost will immediately publish all the deals.

For example, if PublishMsgPeriod is 1 hour:

• At 1:00 pm, deal 1 is ready to publish. Boost will wait until 2:00 pm for other deals to be ready before sending PublishStorageDeals.

• At 1:30 pm, Deal 2 is ready to publish

• At 1:45 pm, Deal 3 is ready to publish

• At 2:00pm, Boost publishes Deals 1, 2, and 3 in a single PublishStorageDeals message.

If MaxDealsPerPublishMsg is 2, then in the above example, when deal 2 is ready to be published at 1:30, Boost would immediately publish Deals 1 & 2 in a single PublishStorageDeals message. Deal 3 would be published in a subsequent PublishStorageDeals message.

Using filters for fine-grained storage and retrieval deal acceptance

Your use case might demand very precise and dynamic control over a combination of deal parameters.

Boost provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it:

• Filter for storage deals.

• RetrievalFilter for retrieval deals.