By default the Boost daemon repository is located at ~/.boost

It contains the following files:

• api The local multi-address of Boost's libp2p API

• boost.db The sqlite database with all deal metadata

• boost.logs.db The sqlite database with the logs for deals

• config.toml The config file with all Boost's settings

• repo.lock A lock file created when Boost is running

• storage.json Deprecated (needed by legacy markets)

• token The token used when calling Boost's JSON RPC endpoints

It has the following directories:

• dagstore Contains indexes of CAR files stored with Boost

• datastore Contains metadata about deals for legacy markets

• deal-staging The directory used by legacy markets for incoming data transfers

• incoming The directory used by Boost for incoming data transfers

• journal Contains journal events (used by legacy markets)

• keystore Contains the secret keys used by libp2p (eg the peer ID)

• kvlog Used by legacy markets datastore

The hardware requirements for Boost are tied to the sealer part of the Lotus deployment it is attached to.

Depending on how much data you need to onboard, and how many deals you need to make with clients, hardware requirements in terms of CPU and Disk will vary.

General hardware requirements

CPU

A miner will need an 8+ core CPU.

We strongly recommend a CPU model with support for Intel SHA Extensions: AMD since Zen microarchitecture, or Intel since Ice Lake. Lack of SHA Extensions results in a very significant slow down.

The most significant computation that Boost has to do is the Piece CID calculation (also known as Piece Commitment or CommP). When Boost receives data from a client, it calculates the Merkle root out of the hashes of the Piece (padded .car file). The resulting root of the clean binary Merkle tree is the Piece CID.

RAM

2 GiB of RAM are needed at the very least.

Disk

Boost stores all data received from clients before Piece CID is calculated and compared against deal parameters received from clients. Next, deals are published on-chain, and Boost waits for a number of epoch confirmations before proceeding to pass data to the Lotus sealing subsystem. This means that depending on the throughput of your operation, you must have disk space for at least a few staged sectors.

For small deployments 100 GiB of disk are needed at the very least if we assume that Boost is to keep three 32 GiB sectors before passing them to the sealing subsystem.

We recommend using NVME disk for Boost. As Dagstore grows in size, the overall performance might slow down due to slow disk.

Make storage deals with HTTP data transfer

Boost supports multiple options for data transfer when making storage deals, including HTTP. Clients can host their CAR file on an HTTP server, such as S3, and provide that URL when proposing the storage deal. Once accepted, Boost will automatically fetch the CAR file from the specified URL.

See As a client for more details.

Web UI

Boost comes with a web interface that can be used to manage deals, watch disk usage, monitor funds, adjust settings and more.

Backwards compatibility with go-fil-markets package

Boost supports the same endpoints as go-fil-markets package for making storage and retrieval deals, getting the storage and retrieval ask, and getting the status of ongoing deals. This ensures that clients running lotus can make deals with Storage Providers running boost.

A client for proposing deals that doesn't require a Lotus node

Boost comes with a client that can be used to make storage deals, and can be configured to point at a public Filecoin API endpoint. That means clients don't need to run a Filecoin node or sync from chain.

See As a client for details.

The Boost source code repository is hosted at github.com/filecoin-project/boost

Boost and Lotus compatibility Matrix

Building and installing

Prerequisites

Environment Variables in Boost

BOOST_CLIENT_REPO         - repo directory for Boost client
BOOSTER_BITSWAP_REP       - repo directory for Booster bitswap
BOOST_PATH                - boost repo path
FULLNODE_API_INFO         - Lotus daemon node API connection string

Linux / Ubuntu

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang build-essential hwloc libhwloc-dev wget -y

MacOS

brew install node@16
brew install bzr jq pkg-config hwloc coreutils

Linux

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1

git clone https://github.com/filecoin-project/boost
cd boost
git checkout <Stable tag or branch>
make clean build
sudo make install

Please ignore any output or onscreen instruction during the npm build unless there is an error.

MacOS

export LIBRARY_PATH=$LIBRARY_PATH:/opt/homebrew/lib
git clone https://github.com/filecoin-project/boost
cd boost
git checkout <Stable tag or branch>
make clean build
sudo make install

Please ignore any output or onscreen instruction during the npm build unless there is an error.

Calibration Network

To build boost for calibnet, please complete the above pre-requisites and build using the following commands.

git clone https://github.com/filecoin-project/boost
cd boost
git checkout <Stable tag or branch>
make clean calibnet

Upgrading Boost

Linux

1. Make sure that Boost daemon is not running. Run the below commands to upgrade the binary.

cd boost
git checkout main
git pull
git checkout <New release>
make clean build
sudo make install

2. Please ignore any onscreen instruction during the npm build unless there is an error.

3. Start the boost daemon.

MacOS

1. Make sure that Boost daemon is not running. Run the below commands to upgrade the binary.

export LIBRARY_PATH=$LIBRARY_PATH:/opt/homebrew/lib
cd boost
git checkout main
git pull
git checkout <New release>
make clean build
sudo make install

2. Please ignore any onscreen instruction during the npm build unless there is an error.

3. Start the boost daemon.

Boost stores metadata about deals in a sqlite database in the root directory of the Boost repo.

To open the database use a sqlite client:

sqlite3 boost.db

The database tables are

• Deals metadata about Boost storage deals (eg deal proposal) and their current state (eg checkpoint)

• FundsLogs log of each change in funds reserved for a deal

• FundsTagged how much FIL is tagged for deal collateral and publish message for a deal

• StorageLogs log of each change in storage reserved for a deal

• StorageTagged how much storage is tagged for a deal

Boost keeps a separate database just for deal logs, so as to make it easier to manage log data separately from deal metadata. The logs database is named boost.logs.db and it has a single table DealLogs that stores logs for each deal, indexed by uuid.

Migrations

Boost uses goose (https://pressly.github.io/goose/) tool and library for handling sqlite3 migrations.

goose can be installed following the instructions at https://pressly.github.io/goose/installation/

Migrations in Boost are stored in the /db/migrations directory.

Boost handles database migrations on start-up. If a user is running an older version of Boost, migrations up to the latest version are automatically applied on start-up.

Developers can use goose to inspect and apply migrations using the CLI:

➜  ~ goose
Usage: goose [OPTIONS] DRIVER DBSTRING COMMAND
...
Commands:
    up                   Migrate the DB to the most recent version available
    up-by-one            Migrate the DB up by 1
    up-to VERSION        Migrate the DB to a specific VERSION
    down                 Roll back the version by 1
    down-to VERSION      Roll back to a specific VERSION
    redo                 Re-run the latest migration
    reset                Roll back all migrations
    status               Dump the migration status for the current DB
    version              Print the current version of the database
    create NAME [sql|go] Creates new migration file with the current timestamp
    fix                  Apply sequential ordering to migrations

The boostd executable runs as a daemon alongside a lotus node and lotus miner. This daemon replaces the current markets subsystem in the lotus miner. The boost daemon exposes a libp2p interface for storage and retrieval deals. It performs on-chain operations by making API calls to the lotus node. The daemon hands off downloaded data to the lotus miner for sealing via API calls to the lotus miner.

boostd has a web interface for fund management and deal monitoring. The web interface is a react app that consumes a graphql interface exposed by the daemon.

Storage Deal Flow

The typical flow for a Storage Deal is:

1. The Client puts funds in escrow with the Storage Market Actor on chain.

2. The Client uploads a CAR file to a web server.

3. The Client sends a storage deal proposal to Boost with the URL of the CAR file.

4. Boost checks that the client has enough funds in escrow to pay for storing the file.

5. Boost accepts the storage deal proposal.

6. Boost downloads the CAR file from the web server.

7. Boost publishes the deal on chain.

8. The client checks that the deal was successfully published on chain.

Boost exposes a libp2p interface to listen for storage deal proposals from clients. This is similar to the libp2p interface exposed by the lotus market subsystem.

Boost communicates with the lotus node over its JSON-RPC API for on-chain operations like checking client funds and publishing the deal.

Once the deal has been published, Boost hands off the downloaded file to lotus-miner for sealing.

Boost is a tool for Storage Providers to manage data onboarding and retrieval on the Filecoin network. It replaces the go-fil-markets package in lotus with a standalone binary that runs alongside a Lotus daemon and Lotus miner.

Boost exposes libp2p interfaces for making storage and retrieval deals, a web interface for managing storage deals, and a GraphQL interface for accessing and updating real-time deal information.

Boost supports the same libp2p protocols as legacy markets, and adds new versions of the protocols used to propose a storage deal and to check the deal's status.

Propose Storage Deal Protocol

The client makes a deal proposal over v1.2.0 or v1.2.1 of the Propose Storage Deal Protocol: /fil/storage/mk/1.2.0 or /fil/storage/mk/1.2.1

It is a request / response protocol, where the request and response are CBOR-marshalled.

There are two new fields in the Request of v1.2.1 of the protocol, described in the table below.

Request

Response

Storage Deal Status Protocol

The client requests the status of a deal over v1.2.0 of the Storage Deal Status Protocol: /fil/storage/status/1.2.0

It is a request / response protocol, where the request and response are CBOR-marshalled.

Request

Response

Directory structure

By default, the dagstore root will be:

• $BOOST_PATH/dagstore

The directory structure is as follows:

1. index: holds the shard indices.

2. transients: holds temporary shard data (unsealed pieces) while they're being indexed.

3. datastore: records shard state and metadata so it can survive restarts.

4. .shard-registration-complete: marker file that signals that initial migration for legacy markets deals is complete.

5. .boost-shard-registration-complete: marker file that signals that initial migration for boost deals is complete.

First-time migration

When you first start your boost process without a dagstore repo, a migration process will register all shards for both legacy and Boost deals in lazy initialization mode. As deals come in, shards are fetched and initialized just in time to serve the retrieval.

• For legacy deals, you can monitor the progress of the migration in your log output, by grepping for the keyword migrator. Here's example output. Notice the first line, which specifies how many deals will be evaluated (this number includes failed deals that never went on chain, and therefore will not be migrated), and the last lines (which communicate that migration completed successfully):

• For Boost deals, you can do the same by grepping for the keyword boost-migrator.

Forcing bulk initialization

Forcing bulk initialization will become important in the near future, when miners begin publishing indices to the network to advertise content they have, and new retrieval features become available (e.g. automatic shard routing).

Initialization places IO workload on your storage system. You can stop/start this command at your wish/convenience as proving deadlines approach and elapse, to avoid IOPS starvation or competition with window PoSt.

To stop a bulk initialization(see the next paragraph), press Control-C. Shards being initialized at that time will continue in the background, but no more initializations will be performed. The next time you run the command, it will resume from where it left off.

You can force bulk initialization using the boostd dagstore initialize-all command. This command will force initialization of every shard that is still in ShardStateNew state for both legacy and Boost deals. To control the operation:

• You must set a concurrency level through the --concurrency=N flag.

  • A value of 0 will disable throttling and all shards will be initialized at once. ⚠️ Use with caution!

• By default, only unsealed pieces will be indexed to avoid forcing unsealing jobs. To index also sealed pieces, use the --include-sealed flag.

In our test environments, we found the migration to proceed at a rate of 400-500 shards/deals per second, on the following hardware specs: AMD Ryzen Threadripper 3970X, 256GB DDR4 3200 RAM, Samsung 970 EVO 2TB SSD, RTX3080 10GB GPU.

Configuration

The DAG store can be configured through the config.toml file of the node that runs the boost subsystem. Refer to the [DAGStore] section. Boost ships with sane defaults:

Automatic shard recovery on error

Shards can error for various reasons, e.g. if the storage system cannot serve the unsealed CAR for a deal/shard, if the shard index is accidentally deleted, etc.

Boost will automatically try to recover failed shards by triggering a recovery once.

You can view failed shards by using the boostd dagstore list-shards command, and optionally grepping for ShardStateErrored.

CLI commands

The boostd executable contains a dagstore command with several useful subcommands:

• boostd dagstore list-shards

• boostd dagstore initialize-shard <key>

• boostd dagstore initialize-all --concurrency=10

• boostd dagstore gc

Refer to the --help texts for more information.

1. If you haven't made any legacy deals with Boost:

   1. Stop boostd

   2. Run your lotus-miner markets service process as you previously did

2. If you have made new legacy deals with Boost, and want to migrate them back:

   1. Stop boostd

   2. Copy the dagstore directory from boost repository to markets repository.

   3. Export Boost deals datastore keys/values: lotus-shed market export-datastore --repo <repo> --backup-dir <backup-dir> Wrote backup file to <backup-dir>/markets.datastore.backup

   4. Import the exported deals datastore keys/values from boost to lotus markets: lotus-shed market import-datastore --repo <repo> --backup-path <backup-path> Completed importing from backup file <backup-path>

If you are running a markets service as a separate lotus-miner process:

1. Stop accepting incoming deals

2. Wait for incoming deals to complete

3. Shutdown the markets process

4. Backup the markets repository

5. Backup the markets datastore (in case you decide to roll back from Boost to Lotus) with:

6. Make sure you have a Lotus node and miner running

7. Create and send funds to two new wallets on the lotus node to be used for Boost

Boost currently uses two wallets for storage deals:

• The publish storage deals wallet - This wallet pays the gas cost when Boost sends the PublishStorageDeals message.

• The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the StorageMarketActor.

8. Boost keeps all data in a directory called the repository. By default the repository is at ~/.boost. To use a different location pass the --boost-repo parameter.

9. Export the environment variables needed for boostd migrate-markets to connect to the lotus daemon and lotus miner.

Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the boost node to make JSON-RPC calls to the mining/sealing/proving node.

10. Set the publish storage deals wallet as a control wallet.

11. Run boostd migrate-markets to initialize the repository and start the migration:

The migrate-markets command

• Initializes a Boost repository

• Migrates markets datastore keys to Boost

  • Storage and retrieval deal metadata

  • Storage and retrieval ask data

• Migrates markets libp2p keys to Boost

• Migrates markets config to Boost (libp2p endpoints, settings etc)

• Migrates the markets DAG store to Boost

12. Run the boostd service, which will start:

• libp2p listeners for storage and retrieval

• the JSON RPC API

• the graphql interface (used by the react front-end)

• the web server for the react front-end

Web UI

Open http://localhost:8080 in your browser.

API Access

Boost API can be accessed by setting the environment variable BOOST_API_INFO same as LOTUS_MARKET_INFO.

Prepare to migrate

1. Make sure you have a Lotus node and miner running

2. Create and send funds to two new wallets on the lotus node to be used for Boost

Boost currently uses two wallets for storage deals:

• The publish storage deals wallet - This wallet pays the gas cost when Boost sends the PublishStorageDeals message.

• The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the StorageMarketActor.

3. Set the publish storage deals wallet as a control wallet.

4. Set up environment variables needed for Boost migration

Export the environment variables needed for boostd migrate-monolith to connect to the lotus daemon and lotus miner.

Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the boost node to make JSON-RPC calls to the mining/sealing/proving node.

Shut down lotus-miner

1. Stop accepting incoming deals

2. Wait for incoming deals to complete

3. Shutdown the lotus-miner

4. Backup the lotus-miner repository

5. Backup the lotus-miner datastore (in case you decide to roll back from Boost to Lotus) with:

6. Set the environment variable LOTUS_FULLNODE_API to allow access to the lotus node API.

Migrate from the lotus-miner repo to the Boost repo

Run boostd migrate-monolith to create and initialize the boost repository:

The migrate-monolith command

• Initializes a Boost repository

• Migrates markets datastore keys to Boost

  • Storage and retrieval deal metadata

  • Storage and retrieval ask data

• Migrates markets libp2p keys to Boost

• Migrates markets config to Boost (libp2p endpoints, settings etc)

• Migrates the markets DAG store to Boost

Update the lotus-miner config

1. Backup lotus-miner's config.toml

2. Disable the markets subsystem in miner config:

Boost replaces the markets subsystem in the lotus-miner, so we need to disable the subsystem in config:

Under the [Subsystems] section set EnableMarkets = false

3. Change the miner's libp2p port

Boost replaces the markets subsystems, and listens on the same libp2p port, so we need to change the libp2p port that the miner is listening on.

Under the [Libp2p] section change the port in ListenAddresses

Restart lotus-miner

Start lotus-miner up again so that Boost can connect to the miner when it starts.

Run the boostd service

The boostd service will start:

• libp2p listeners for storage and retrieval

• the JSON RPC API

• the graphql interface (used by the react front-end)

• the web server for the react front-end

Web UI

Open http://localhost:8080 in your browser.

API Access

Boost API can be accessed by setting the environment variable BOOST_API_INFO same as LOTUS_MARKET_INFO.

Migrating Boost from one machine to another

Once the Boost has been split from the monolith miner, it can be moved to another physical or virtual machine by following the below steps.

2. Copy the boost repo from the original monolith miner machine to the new dedicated boost machine.

3. Set the environment variable LOTUS_FULLNODE_API to allow access to the lotus node API.

4. Open the required port on the firewall on the monolith miner machine to allow connection to lotus-miner API.

6. Start the boostd process.

Configuration

By default, the web UI listens on the localhost interface on port 8080. We recommend keeping the UI listening on localhost or some internal IP within your private network to avoid accidentally exposing it to the internet.

You can access the web UI listening on the localhost interface on a remote server, you can open an SSH tunnel from your local machine:

Settings Page

Sample config file

Sealer

API

Libp2p

Storage

Dealmaking

Dealmaking section handles deal making configuration explicitly for boost deal that uses the new /fil/storage/mk/1.2.0 protocol.

Wallets

LotusFees

DAGStore

IndexProvider

Advertising 128-bit long multihashes with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to 256MiB when full.

Boost is introducing a new feature that allows computing commP during the deal on a lotus-worker node.

This should reduce the overall resource utilisation on the Boost node.

In order to enable remote commP on a Boost node, update your config.toml:

Then restart the Boost node

Boost provides a capability to limit the number of simultaneous http transfer in progress to download the deal data from the clients.

This new configuration has been introduced in the ConfigVersion = 3 of the boost configuration file.

Configuration Variables

HTTP variables

Storage variables

How TransferLimiter works

The transferLimiter maintains a queue of transfers with a soft upper limit on the number of concurrent transfers.

To prevent slow or stalled transfers from blocking up the queue there are a couple of mitigations: The queue is ordered such that we

• start transferring data for the oldest deal first

• prefer to start transfers with peers that don't have any ongoing transfer

• once the soft limit is reached, don't allow any new transfers with peers that have existing stalled transfers

Note that peers are distinguished by their host (eg foo.bar:8080) not by libp2p peer ID. For example, if there is

• one active transfer with peer A

• one pending transfer (peer A)

• one pending transfer (peer B)

The algorithm will prefer to start a transfer with peer B than peer A. This helps to ensure that slow peers don't block the transfer queue.

The limit on the number of concurrent transfers is soft. Example: if there is a limit of 5 concurrent transfers and there are

• three active transfers

• two stalled transfers

then two more transfers are permitted to start (as long as they're not with one of the stalled peers)

Using filters for fine-grained storage and retrieval deal acceptance

Your use case might demand very precise and dynamic control over a combination of deal parameters.

Lotus provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it:

• Filter for storage deals.

• RetrievalFilter for retrieval deals.

The executed command receives a JSON representation of the deal parameters on standard input, and upon completion, its exit code is interpreted as:

• 0: success, proceed with the deal.

• non-0: failure, reject the deal.

The most trivial filter rejecting any retrieval deal would be something like: RetrievalFilter = "/bin/false". /bin/false is binary that immediately exits with a code of 1.

This Perl script lets the miner deny specific clients and only accept deals that are set to start relatively soon.

You can also use a third party content policy framework like CIDgravity or bitscreen by Murmuration Labs:

# grab filter program
go get -u -v github.com/Murmuration-Labs/bitscreen

# add it to both filters
Filter = "/path/to/go/bin/bitscreen"
RetrievalFilter = "/path/to/go/bin/bitscreen"

Boost comes with a client executable, boost, that can be used to send a deal proposal to a Boost server.

The client is intentionally minimal meant for developer testing. It is not a full featured client and is not intended to be so. It does not require a daemon process, and can be pointed at any public Filecoin API for on-chain operations. This means that users of the client do not need to run a Filecoin node that syncs the chain.

Set the API endpoint environment variable

export FULLNODE_API_INFO=<filecoin API endpoint>

Initialize the client

boost -vv init

The init command

• Creates a Boost client repository (at ~/.boost-client by default)

• Generates a libp2p peer ID key

• Generates a wallet for on-chain operations and outputs the wallet address

Add funds to the wallet and to the market actor

To make deals you will need to: a) add funds to the wallet b) add funds to the market actor for that wallet address

Make a storage deal

Currently, we don't distribute binaries, so you will have to build from source.

boost -vv deal --provider=<f00001> \
               --http-url=<https://myserver/my.car> \
               --commp=<commp> \
               --car-size=<car-size> \
               --piece-size=<piece-size> \
               --payload-cid=<payload-cid>

When a storage provider accepts the deal, you should see output of the command similar to:

sent deal proposal
  deal uuid: 9e68fb16-ff9a-488e-ad0a-1289b512d176
  storage provider: f0127896
  client wallet: f1sw5zjcyo4mff5cbvgsgmm8uoko6gcr4tptvtkhy
  payload cid: bafyaa6qshafcmalqudsbeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam
  url: https://webserver/file.car
  commp: baga6ea4seaqh5prrl6ykov4t64k6m5giijsc44dcxtdnzsp4izjakqhs7twauiq
  start epoch: 1700711
  end epoch: 2219111
  provider collateral: 358.687 μFIL

Check deal status

You can check the deal status with the following command:

boost deal-status --provider=<provider> --deal-uuid=<deal-uuid>

got deal status response
  deal uuid: 9e68fb16-ff8a-488e-ad0a-1289b512d176
  deal status: Transfer Queued
  deal label: bafyaa6qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam
  publish cid: <nil>
  chain deal id: 0

Backup

Boost now supports both online and offline backups. The backup command will output a backup directory containing the following files.

1. metadata - contains backup of leveldb

2. boostd.db - backup of deals database

3. keystore - directory containing libp2p keys

4. token - API token

5. config - directory containing all config files and config.toml link

6. storage.json - file containing storage details

Backup does not backs up the deal logs and dagstore.

Online backup

You can take an online backup with the below command

boostd backup <backup directory>

The online backup supports running only one instance at a time and you might see a locking error if another instance of backup is already running.

Offline backup

1. Shutdown boostd before taking a backup

2. Take a backup using the command line

boostd backup --offline <backup directory>

Restore

1. Make sure that --boost-repo flag is set if you wish to restore to a custom location. Otherwise, it will be restored to ~/.boost directory

2. Restore the boost repo using the command line

boostd restore <backup directory>

Dealmaking section

This section controls parameters for making storage and retrieval deals:

[LotusDealmaking]
  # When enabled, the miner can accept online deals
  #
  # type: bool
  # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINESTORAGEDEALS
  #ConsiderOnlineStorageDeals = true

  # When enabled, the miner can accept offline deals
  #
  # type: bool
  # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINESTORAGEDEALS
  #ConsiderOfflineStorageDeals = true

  # When enabled, the miner can accept retrieval deals
  #
  # type: bool
  # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINERETRIEVALDEALS
  #ConsiderOnlineRetrievalDeals = true

  # When enabled, the miner can accept offline retrieval deals
  #
  # type: bool
  # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINERETRIEVALDEALS
  #ConsiderOfflineRetrievalDeals = true

  # When enabled, the miner can accept verified deals
  #
  # type: bool
  # env var: LOTUS_LOTUSDEALMAKING_CONSIDERVERIFIEDSTORAGEDEALS
  #ConsiderVerifiedStorageDeals = true

  # When enabled, the miner can accept unverified deals
  #
  # type: bool
  # env var: LOTUS_LOTUSDEALMAKING_CONSIDERUNVERIFIEDSTORAGEDEALS
  #ConsiderUnverifiedStorageDeals = true

  # A list of Data CIDs to reject when making deals
  #
  # type: []cid.Cid
  # env var: LOTUS_LOTUSDEALMAKING_PIECECIDBLOCKLIST
  #PieceCidBlocklist = []

  # Maximum expected amount of time getting the deal into a sealed sector will take
  # This includes the time the deal will need to get transferred and published
  # before being assigned to a sector
  #
  # type: Duration
  # env var: LOTUS_LOTUSDEALMAKING_EXPECTEDSEALDURATION
  #ExpectedSealDuration = "24h0m0s"

  # Maximum amount of time proposed deal StartEpoch can be in future
  #
  # type: Duration
  # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSTARTDELAY
  #MaxDealStartDelay = "336h0m0s"

  # When a deal is ready to publish, the amount of time to wait for more
  # deals to be ready to publish before publishing them all as a batch
  #
  # type: Duration
  # env var: LOTUS_LOTUSDEALMAKING_PUBLISHMSGPERIOD
  # PublishMsgPeriod = "40m0s"

  # The maximum number of deals to include in a single PublishStorageDeals
  # message
  #
  # type: uint64
  # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSPERPUBLISHMSG
  #MaxDealsPerPublishMsg = 8

  # The maximum collateral that the provider will put up against a deal,
  # as a multiplier of the minimum collateral bound
  #
  # type: uint64
  # env var: LOTUS_LOTUSDEALMAKING_MAXPROVIDERCOLLATERALMULTIPLIER
  #MaxProviderCollateralMultiplier = 2

  # The maximum allowed disk usage size in bytes of staging deals not yet
  # passed to the sealing node by the markets service. 0 is unlimited.
  #
  # type: int64
  # env var: LOTUS_LOTUSDEALMAKING_MAXSTAGINGDEALSBYTES
  # MaxStagingDealsBytes = 100000000000

  # The maximum number of parallel online data transfers for storage deals
  #
  # type: uint64
  # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGE
  #SimultaneousTransfersForStorage = 20

  # The maximum number of simultaneous data transfers from any single client
  # for storage deals.
  # Unset by default (0), and values higher than SimultaneousTransfersForStorage
  # will have no effect; i.e. the total number of simultaneous data transfers
  # across all storage clients is bound by SimultaneousTransfersForStorage
  # regardless of this number.
  #
  # type: uint64
  # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGEPERCLIENT
  #SimultaneousTransfersForStoragePerClient = 0

  # The maximum number of parallel online data transfers for retrieval deals
  #
  # type: uint64
  # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORRETRIEVAL
  #SimultaneousTransfersForRetrieval = 20

  # Minimum start epoch buffer to give time for sealing of sector with deal.
  #
  # type: uint64
  # env var: LOTUS_LOTUSDEALMAKING_STARTEPOCHSEALINGBUFFER
  #StartEpochSealingBuffer = 480

  # A command used for fine-grained evaluation of storage deals
  # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details
  #
  # type: string
  # env var: LOTUS_LOTUSDEALMAKING_FILTER
  #Filter = ""

  # A command used for fine-grained evaluation of retrieval deals
  # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details
  #
  # type: string
  # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALFILTER
  #RetrievalFilter = ""

  [LotusDealmaking.RetrievalPricing]
    # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_STRATEGY
    #Strategy = "default"

    [LotusDealmaking.RetrievalPricing.Default]
      # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_DEFAULT_VERIFIEDDEALSFREETRANSFER
      #VerifiedDealsFreeTransfer = true

    [LotusDealmaking.RetrievalPricing.External]
      # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_EXTERNAL_PATH
      #Path = ""

ExpectedSealDuration is an estimate of how long sealing will take and is used to reject deals whose start epoch might be earlier than the expected completion of sealing. It can be estimated by benchmarking or by pledging a sector.

StartEpochSealingBuffer allows lotus-miner to seal a sector before a certain epoch. For example: if the current epoch is 1000 and a deal within a sector must start on epoch 1500, then lotus-miner must wait until the current epoch is 1500 before it can start sealing that sector. However, if Boost sets StartEpochSealingBuffer to 500, the lotus-miner can start sealing the sector at epoch 1000.

If there are multiple deals in a sector, the deal with a start time closest to the current epoch is what StartEpochSealingBuffer will be based off. So, if the sector in our example has three deals that start on epoch 1000, 1200, and 1400, then lotus-miner will start sealing the sector at epoch 500.

Publishing several deals in one message

The PublishStorageDeals message can publish multiple deals in a single message. When a deal is ready to be published, Boost will wait up to PublishMsgPeriod for other deals to be ready before sending the PublishStorageDeals message.

However, once MaxDealsPerPublishMsg is ready, Boost will immediately publish all the deals.

For example, if PublishMsgPeriod is 1 hour:

• At 1:00 pm, deal 1 is ready to publish. Boost will wait until 2:00 pm for other deals to be ready before sending PublishStorageDeals.

• At 1:30 pm, Deal 2 is ready to publish

• At 1:45 pm, Deal 3 is ready to publish

• At 2:00pm, Boost publishes Deals 1, 2, and 3 in a single PublishStorageDeals message.

If MaxDealsPerPublishMsg is 2, then in the above example, when deal 2 is ready to be published at 1:30, Boost would immediately publish Deals 1 & 2 in a single PublishStorageDeals message. Deal 3 would be published in a subsequent PublishStorageDeals message.

Using filters for fine-grained storage and retrieval deal acceptance

Your use case might demand very precise and dynamic control over a combination of deal parameters.

Boost provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it:

• Filter for storage deals.

• RetrievalFilter for retrieval deals.

The executed command receives a JSON representation of the deal parameters on standard input, and upon completion, its exit code is interpreted as:

• 0: success, proceed with the deal.

• non-0: failure, reject the deal.

The most trivial filter rejecting any retrieval deal would be something like: RetrievalFilter = "/bin/false". /bin/false is binary that immediately exits with a code of 1.

This Perl script lets the miner deny specific clients and only accept deals that are set to start relatively soon.

You can also use a third party content policy framework like CIDgravity or bitscreen by Murmuration Labs:

# grab filter program
go get -u -v github.com/Murmuration-Labs/bitscreen

# add it to both filters
Filter = "/path/to/go/bin/bitscreen"
RetrievalFilter = "/path/to/go/bin/bitscreen"

Initialization and Running

1. Make sure you have a Lotus node and miner running

2. Create and send funds to two new wallets on the lotus node to be used for Boost

Boost currently uses two wallets for storage deals:

• The publish storage deals wallet - This wallet pays the gas cost when Boost sends the PublishStorageDeals message.

• The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the StorageMarketActor.

export PUBLISH_STORAGE_DEALS_WALLET=`lotus wallet new bls`
export COLLAT_WALLET=`lotus wallet new bls`
lotus send --from mywallet $PUBLISH_STORAGE_DEALS_WALLET 10
lotus send --from mywallet $COLLAT_WALLET 10

3. Set the publish storage deals wallet as a control wallet.

lotus-miner actor control set --really-do-it $PUBLISH_STORAGE_DEALS_WALLET

4. Create and initialize the Boost repository

Boost keeps all data in a directory called the repository. By default the repository is at ~/.boost. To use a different location pass the --boost-repo parameter (must precede any particular command verb, e.g. boostd --boost-repo=/path init).

Export the environment variables needed for boostd init to connect to the lotus daemon and lotus miner.

export $(lotus auth api-info --perm=admin)
export $(lotus-miner auth api-info --perm=admin)

Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the boost node to make JSON-RPC calls to the mining/sealing/proving node.

export APISEALER=`echo $MINER_API_INFO`
export APISECTORINDEX=`echo $MINER_API_INFO`

Run boostd init to create and initialize the repository:

boostd --vv init \
       --api-sealer=$APISEALER \
       --api-sector-index=$APISECTORINDEX \
       --wallet-publish-storage-deals=$PUBLISH_STORAGE_DEALS_WALLET \
       --wallet-deal-collateral=$COLLAT_WALLET \
       --max-staging-deals-bytes=50000000000

• --api-sealer is the API info for the lotus-miner instance that does sealing

• --api-sector-index is the API info for the lotus-miner instance that provides storage

• --max-staging-deals-bytes is the maximum amount of storage to be used for downloaded files (once the limit is reached Boost will reject subsequent incoming deals)

5. Update ulimit file descriptor limit if necessary. Boost deals will fail if the file descriptor limit for the process is not set high enough. This limit can be raised temporarily before starting the Boost process by running the command ulimit -n 1048576. We recommend setting it permanently by following the Permanently Setting Your ULIMIT System Value guide.

6. Make sure that the correct <PEER_ID> and <MULTIADDR> for your SP is set on chain, given that boost init generates a new identity. Use the following commands to update the values on chain:

lotus-miner actor set-addrs <MULTIADDR>
lotus-miner actor set-peer-id <PEER_ID>

7. Run the boostd service, which will start:

• libp2p listeners for storage and retrieval

• the JSON RPC API

• the graphql interface (used by the react front-end)

• the web server for the react front-end

boostd --vv run

Web UI

1. Build the React frontend

cd react

# Download and install npm packages needed by the React frontend
npm install --legacy-peer-deps

# Build the optimized JavaScript and CSS in boost/react/build
npm run build

1. Open the Web UI

Open http://localhost:8080 in your browser.

ssh -L 8080:localhost:8080 myserver

API Access

Boost API can be accessed by setting the environment variable BOOST_API_INFO same as LOTUS_MARKET_INFO.

boostd auth api-info --perm=admin

export BOOST_API_INFO=<TOKEN>:<API Address>

You can also directly evaluate the boostd auth command with:

export $(boostd auth api-info --perm=admin)

Announce over HTTP

IndexProvider.HttpPublisher.AnnounceOverHttp must be set to true to enable the http announcements. Once HTTP announcements are enabled, the local-index provider will continue to announce over libp2p gossipsub along with HTTP for the specific indexers.

The advertisements are send to the indexer nodes defined in DirectAnnounceURLs. You can specify more than 1 URL to announce to multiple indexer nodes.

Once an IPNI node starts processing the advertisements, it will reach out to the Boost node to fetch the data. Thus, Boost node needs to specify a public IP and port which can be used by the indexer node to query for data.

[IndexProvider]
  # Enable set whether to enable indexing announcement to the network and expose endpoints that
  # allow indexer nodes to process announcements. Enabled by default.
  #
  # type: bool
  # env var: LOTUS_INDEXPROVIDER_ENABLE
  #Enable = true

  # EntriesCacheCapacity sets the maximum capacity to use for caching the indexing advertisement
  # entries. Defaults to 1024 if not specified. The cache is evicted using LRU policy. The
  # maximum storage used by the cache is a factor of EntriesCacheCapacity, EntriesChunkSize and
  # the length of multihashes being advertised. For example, advertising 128-bit long multihashes
  # with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to
  # 256MiB when full.
  #
  # type: int
  # env var: LOTUS_INDEXPROVIDER_ENTRIESCACHECAPACITY
  #EntriesCacheCapacity = 1024

  # EntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk.
  # Defaults to 16384 if not specified. Note that chunks are chained together for indexing
  # advertisements that include more multihashes than the configured EntriesChunkSize.
  #
  # type: int
  # env var: LOTUS_INDEXPROVIDER_ENTRIESCHUNKSIZE
  #EntriesChunkSize = 16384

  # TopicName sets the topic name on which the changes to the advertised content are announced.
  # If not explicitly specified, the topic name is automatically inferred from the network name
  # in following format: '/indexer/ingest/<network-name>'
  # Defaults to empty, which implies the topic name is inferred from network name.
  #
  # type: string
  # env var: LOTUS_INDEXPROVIDER_TOPICNAME
  #TopicName = ""

  # PurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine
  # starts. By default, the cache is rehydrated from previously cached entries stored in
  # datastore if any is present.
  #
  # type: bool
  # env var: LOTUS_INDEXPROVIDER_PURGECACHEONSTART
  #PurgeCacheOnStart = true

  [IndexProvider.Announce]
    # Make a direct announcement to a list of indexing nodes over http.
    # Note that announcements are already made over pubsub regardless
    # of this setting.
    #
    # type: bool
    # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_ANNOUNCEOVERHTTP
    #AnnounceOverHttp = true

    # The list of URLs of indexing nodes to announce to.
    #
    # type: []string
    # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_DIRECTANNOUNCEURLS
    #DirectAnnounceURLs = ["https://cid.contact/ingest/announce"]

  [IndexProvider.HttpPublisher]
    # If not enabled, requests are served over graphsync instead.
    #
    # type: bool
    # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_ENABLED
    #Enabled = true

    # Set the public hostname / IP for the index provider listener.
    # eg "82.129.73.111"
    # This is usually the same as the for the boost node.
    #
    # type: string
    # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PUBLICHOSTNAME
    #PublicHostname = "82.129.73.111"

    # Set the port on which to listen for index provider requests over HTTP.
    # Note that this port must be open on the firewall.
    #
    # type: int
    # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PORT
    #Port = 3401

First, you need to initialise a new Boost client and also set the endpoint for a public Filecoin node. In this example we are using https://glif.io

export FULLNODE_API_INFO=https://api.node.glif.io

boost init

The init command will output your new wallet address, and warn you that the market actor is not initialised.

boost init

boost/init_cmd.go:53    default wallet set      {"wallet": "f3wfbcudimjcqtfztfhoskgls5gmkfx3kb2ubpycgo7a2ru77temduoj2ottwzlxbrbzm4jycrtu45deawbluq"}
boost/init_cmd.go:60    wallet balance  {"value": "0"}
boost/init_cmd.go:65    market actor is not initialised, you must add funds to it in order to send online deals

Then you need to send funds to the wallet, and add funds to the market actor (in the example below we are adding 1 FIL).

You can use the boostx utilities to add funds to the market actor:

boostx market-add 1

You can confirm that the market actor has funds by running boost init again.

After that you need to generate a car file for data you want to store on Filecoin, and note down its payload-cid. We recommend using go-car CLI to generate the car file.

car create -f my-data.car --version 1 <my-data>
car root my-data.car

bafykbzacedzjq6jvlqnclrseq8pp5lypa6ozuqgug3wjie3orh67berkwv7e4

Then you need to calculate the commp and piece size for the generated car file:

boostx commp ./my-data.car

CommP CID:  baga6ea4seaqjaxked6ovoj5f3bdisfeuwtjhrzh3s34mg5cyzevgoebe7tdckdi
Piece size:  2097152
Car file size: 1101978 

Place the generated car file on a public HTTP server, so that a storage provider can later fetch it.

Finally, trigger an online storage deal with a given storage provider:

export FULLNODE_API_INFO=https://api.node.glif.io

boost deal --verified=false \
           --provider=f0026876 \
           --http-url=https://public-http-server.com/my-data.car \
           --commp=baga6ea4seaqjaxked6ovoj5f3bdisfeuwtjhrzh3s34mg5cyzevgoebe7tdckdi \
           --car-size=1101978 \
           --piece-size=2097152 \
           --payload-cid=bafykbzacedzjq6jvlqnclrseq8pp5lypa6ozuqgug3wjie3orh67berkwv7e4

Storage providers might demand very precise and dynamic control over a combination of deal parameters.

Boost, similarly to Lotus, provides two IPC hooks allowing you to name a command to execute for every deal before the storage provider accepts it:

• Filter for storage deals.

• RetrievalFilter for retrieval deals.

The executed command receives a JSON representation of the deal parameters, as well as the current state of the sealing pipeline, on standard input, and upon completion, its exit code is interpreted as:

• 0: success, proceed with the deal.

• non-0: failure, reject the deal.

The most trivial filter rejecting any retrieval deal would be something like: RetrievalFilter = "/bin/false". /bin/false is binary that immediately exits with a code of 1.

This Perl script lets the miner deny specific clients and only accept deals that are set to start relatively soon.

You can also use a third party content policy framework like bitscreen by Murmuration Labs, or CID gravity:

# grab filter program
go get -u -v github.com/Murmuration-Labs/bitscreen

# add it to both filters
Filter = "/path/to/go/bin/bitscreen"
RetrievalFilter = "/path/to/go/bin/bitscreen"

Here is a sample JSON representation of the input sent to the deal filter:

{
  "DealParams": {
    "DealUUID": "48c31c8c-dcc8-4372-a0ac-b5468eea555b",
    "IsOffline": false,
    "ClientDealProposal": {
      "Proposal": {
        "PieceCID": {
          "/": "baga6ea4seaqh5prrl6ykov4t64k6m6giijsc44dcxtdnzsp4izjakqhs7twauiq"
        },
        "PieceSize": 2147483648,
        "VerifiedDeal": false,
        "Client": "f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy",
        "Provider": "f0127896",
        "Label": "bafyaa7qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam",
        "StartEpoch": 1717840,
        "EndEpoch": 2236240,
        "StoragePricePerEpoch": "1",
        "ProviderCollateral": "363196619502649",
        "ClientCollateral": "0"
      },
      "ClientSignature": {
        "Type": 1,
        "Data": "SmgcBnQE+0ZIb4zAXw7TpxLliSaliShEvX9P4+uwvxBhRDlJD+F6N3NFoNrA2y5bTeWF5aWWuL93w+SSmXFkoAA="
      }
    },
    "DealDataRoot": {
      "/": "bafyaa7qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam"
    },
    "Transfer": {
      "Type": "http",
      "ClientID": "",
      "Params": "eyJVUkwiOiJodHRwczovL2FudG8uLXB1YmxpYy1idWNrZXQtYm9vc3QuczMuZXUtY2VudHJhbC0xLmFtYXpvbmF3cy5jb20vcmFuZGZpbGVfMkdCXzAuY2FyIiwiSGVhZGVycyI6bnVsbH0=",
      "Size": 2000177948
    }
  },
  "SealingPipelineState": {
    "SectorStates": {
      "Available": 684,
      "Proving": 307,
      "Removed": 82,
      "TerminateFailed": 1,
      "TerminateWait": 5
    },
    "Workers": null
  }
}

A storage provider can run the lotus as a monolith process where everything is handled by a single lotus-miner process or separate the mining and market subsystems on different machines.

Boost supports migration from both monolith and a split-market miner. You can follow the below guides to migrate to boost.

Rollback

Inspect

The new inspect page in the Boost UI helps with debugging retrieval problems. It allows the user to check the following using a payload CID or piece CID:

• Verify if the piece has been correctly added to the Piece Store

• Validate if the piece is indexed in the DAG store

• Check for an unsealed copy of the piece

• Verify that the payload CID -> piece CID index has been created correctly

Failed to connect to peer

If the client cannot connect to Boost running on a Storage provider, with an error similar to the following:

failed to connect to peer <peer id>: failed to dial <peer id>:
  * <multi-address> failed to negotiate security protocol:
    peer id mismatch: expected <peer id>,
    but remote key matches <different peer id>

The problem is that:

• The SP registered their peer id and address on chain.

eg "Register the peer id 123abcd at address ip4/123.456.12.345/tcp/1234"

lotus-miner actor set-peer-id 123abcd
lotus-miner actor set-addrs ip4/123.456.12.345/tcp/1234

• The SP changed their peer id locally but didn't update the peer id on chain.

• The client wants to make a storage deal with peer 123abcd. The client looks on chain for the address of peer 123abcd and sees peer 123abcd has registered an address ip4/123.456.12.345/tcp/1234.

• The client sends a deal proposal for peer 123abcd to the SP at address ip4/123.456.12.345/tcp/1234.

• The SP has changed their peer ID, so the SP responds to the deal proposal request with an error: peer id mismatch

To fix the problem, the SP should register the new peer id on chain:

lotus-miner actor set-peer-id <new peer id>

Update storage provider's on chain address

Clients would not be able to connect to Boost running on a Storage provider after an IP change. This happens as clients lookup the registered peer id and address on chain for a SP. When a SP changes their IP or address locally, they must update the same on chain.

The SP should register the new peer id on chain using the following lotus-miner command

lotus-miner actor set-addrs /ip4/<YOUR_PUBLIC_IP_ADDRESS_OF_BOOST_NODE>/tcp/<Boostd Port>

The on chain address change requires access to the worker key and thus the command lives in lotus-miner instead of Boost.

Error in lotus-miner info output

After migrating to Boost, following error is seen when running lotus-miner info :

ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

Problem:

lotus-miner is making a call on lotus-market process which has been replaced by Boost, but lotus-miner is not aware of the new market process.

Solution:

Export the MARKETS_API_INFO variable on your lotus-miner node.

export MARKETS_API_INFO=<Boost token:api>

Fix retrievals with error "failed to lookup index for mh"

The following error shows up when trying to retrieve the data from a storage provider.

ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid Qmf1ykhUo63qB5dJ8KRyeths9MZfyxpVdT5xwnmoLKefz7: getting pieces containing block Qmf1ykhUo63qB5dJ8KRyeths92mfyxpVdT5xi1moLKefz7: failed to lookup index for mh 1220f7ce2d20772b959c1071868e9495712f12785b1710ee88752af120dd49338190, err: datastore: key not found

The error indicates that dagstore does not have a corresponding index shard for the piece containing the requested data. When a retrieval is requested, the dagstore on storage provider side is queried and a reverse look up is used to determine the key(piece CID). This key is then used to query the piece store to find the sector containing the data and byte offset.

If for any reason the shard is not registered with the dagstore then reverse look up to find the piece CID fails and the above error is seen. The most widely know reason for not having the shard registered with dagstore is the below error.

2022-02-21T20:06:03.950+1100 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventFailed", "proposal CID": "bafyreihr743zllr2eckgfiweouiap7pgcjqa3mg3t75jjt7sfcpu", "state": "StorageDealError", "message": "error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1570875): failed to look up deal on chain: deal 3964985 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed"}

To fix the deals where retrievals are impacted by above error, user will need to register the shards manually with dagstore:

boostd dagstore register-shard <piece CID>

If you have multiple deals in such state then you will need to generate a list of registered pieces with piece store and then compare with the shards available in the dagstore to create a list of missing shards.

1. Create a list of all sectors on lotus-miner and redirect the output to a file. Copy the output file to boost node to be used by the below command.

lotus-miner sectors list | awk '{print $1 " " $2}' | grep -v ID > aclist.txt

2. Generate a list of shards to be registered

comm -13 <(for i in $(boostd pieces list-pieces); do sector_list=`boostd pieces piece-info $i | awk '{print $2}'| sed -ne '/SectorID/,$p' | grep -v SectorID`; for j in $sector_list; do grep -w $j aclist.txt > /dev/null; if [ $? -eq 0 ]; then break; else echo "$i"; fi; done; done) <(comm -13 <(boostd dagstore list-shards | awk '{print $1}' | sed 1d | sort) <(boostd pieces list-pieces | sort))

3. Register the shards with dagstore in an automated fashion.

for i in `cat <OUTPUT OF STEP 2 IN A FILE>` ; do boostd dagstore register-shard $i; done

Please note that each shard may take upto 3-5 minutes to get registered. So, the above command might take hours or days to complete depending upon the number of missing shards.

Boost is developing new market features on a regular basis as part of the overall market development. This section covers the experimental features released by boost along with details on how to use them.

It is not recommended to run experimental features in production environments. The features should be tested as per your requirements, and any issues or requests should be reported to the team via Github or Slack.

Once the new features have been tested and vetted, they will be released as part of a stable Boost release and all documentation concerning those features will be moved to an appropriate section of this site.

Current experimental features are listed below.

With the release of FVM, it is now possible for smart contracts to make deal proposals on-chain. This is made possible though the DealProposal FRC.

DataDAOs, as well as other clients who want to store data on Filecoin, can now deploy a smart contract on the FVM which adheres to the DealProposal FRC, and make deal proposals that are visible to every storage provider who monitors the chain.

How to enable FVM monitoring in order to process storage deal proposals published on-chain?

Boost already has support for the DealProposal FRC.

The code for FVM monitoring resides in the latest release of the Boost. It should be used with caution for production use. SPs must before proceeding to the next step.

To build for mainnet:

In order to enable DealProposal FRC, you have to edit your config.toml and enable contract deal monitoring. By default it is disabled. Here is an example configuration:

AllowlistContracts field could be left empty if you want to accept deals from any client. If you only want to accept deals from some clients, you can specify their contract addresses in the field.

From field should be set to your SP's FEVM address. Some clients may implement a whitelist which allows specific SPs to accept deal proposals from their contract. This field will help those clients identify your SP and match it to their whitelist.

How contract deals work in Boost

1. A contract publishes a DealProposalCreate event on the chain.

2. Boost monitors the chain for such events from all the clients by default. When such an event is detected, we go and fetch the data for the deal.

3. Deal is then run through the basic deal validation filters like clients has enough funds, SP has enough funds etc.

4. Once deal passes the validation, we create a new deal handler in Boost and pass this deal for execution like other Boost deals.

Boost introduced a new binary, booster-http, with release v1.2.0. This binary can be run alongside the boostd market process in order to serve retrievals over http.

Currently, there is no payment method or built-in security integrated in the new binary. It can be run with any stable release of boostd and can also be run on a separate machine from the boostd process.

Release v1.7.0-rc1 introduced support in booster-http for running an , which enables Storage Providers to serve content to their users in multiple formats as described below and demonstrated using curl.

Retrieving a full Piece

When performing certain actions, such as replicating deals, it can be convenient to retrieve the entire Piece (with padding) to ensure commp integrity.

Retrieving a CAR file

To return the CAR file for a given CID, you can pass an Accept header with the application/vnd.ipld.car; format. This can be useful for retrieving the raw, unpadded data of a deal.

Retrieving specific files

For Storage Providers that have enabled serving raw files (disabled by default), users can retrieve specific files, such as images by their cid and path where applicable. See for a more in depth example.

Retrieving IPLD blocks

For advanced IPFS and IPLD use cases, you can now retrieve individual blocks by passing an Accept header with the application/vnd.ipld.raw; format

Local Setup

SPs should try a local setup and test their HTTP retrievals before proceeding to run booster-http in production.

To build and run booster-http :

1. Clone the boost repo and checkout the latest release

1. Build the new binary

1. Collect the token information for boost, lotus-miner and lotus daemon API

1. Start the booster-http server with the above details

Running Public Boost HTTP Retrieval

• SSL

• Authentication

• Load balancing

Making HTTP Retrieval Discoverable

To enable public discovery of the Boost HTTP server, SPs should set the domain root in boostd's config.toml. Under the [DealMaking] section, set HTTPRetrievalMultiaddr to the public domain root in multi-address format.

Example config.toml section:

Clients can determine if an SP offers HTTP retrieval by running:

Clients can check the HTTP URL scheme version and supported queries

Clients can download a piece using the domain root configured by the SP:

Go JSON-RPC client

To use the Boost Go client, the Go RPC-API library can be used to interact with the Boost API node.

1. Import the necessary Go module:

go get github.com/filecoin-project/go-jsonrpc

1. Create the following script:

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"

    jsonrpc "github.com/filecoin-project/go-jsonrpc"
    boostapi "github.com/filecoin-project/boost/api"
)

func main() {
    authToken := "<value found in ~/.boost/token>"
    headers := http.Header{"Authorization": []string{"Bearer " + authToken}}
    addr := "127.0.0.1:1288"

    var api boostapi.BoostStruct
    closer, err := jsonrpc.NewMergeClient(context.Background(), "ws://"+addr+"/rpc/v0", "Filecoin", []interface{}{&api.Internal, &api.CommonStruct.Internal}, headers)
    if err != nil {
        log.Fatalf("connecting with boost failed: %s", err)
    }
    defer closer()

    // Now you can call any API you're interested in.
    netAddrs, err := api.NetAddrsListen(context.Background())
    if err != nil {
      log.Fatalf("calling netAddrsListen: %s", err)
    }
    fmt.Printf("Boost is listening on: %s", netAddrs.Addrs[0])
}

1. Run go mod init to set up your go.mod file

2. You should now be able to interact with the Boost API.

Python JSON-RPC client

The JSON-RPC API can also be communicated with programmatically from other languages. Here is an example written in Python. Note that the method must be prefixed with Filecoin.

import requests
import json

def main():
    url = "http://localhost:3051/rpc/v0"
    headers = {'content-type': 'application/json', "Authorization": "Bearer <token>"}
    payload = {
        "method": "Filecoin.BoostOfflineDealWithData",
        "params": [
            "<deal-uuid>",
            "<file-path>",
            True
        ],
        "jsonrpc": "2.0",
        "id": 1,
    }
    response = requests.post(url, data=json.dumps(payload), headers=headers)
    print(response.text)

if __name__ == "__main__":
    main()

Groups

• Actor

  • ActorSectorSize

• Auth

  • AuthNew

  • AuthVerify

• Blockstore

  • BlockstoreGet

  • BlockstoreGetSize

  • BlockstoreHas

• Boost

  • BoostDagstoreDestroyShard

  • BoostDagstoreGC

  • BoostDagstoreInitializeAll

  • BoostDagstoreInitializeShard

  • BoostDagstoreListShards

  • BoostDagstorePiecesContainingMultihash

  • BoostDagstoreRecoverShard

  • BoostDagstoreRegisterShard

  • BoostDeal

  • BoostDealBySignedProposalCid

  • BoostDummyDeal

  • BoostIndexerAnnounceAllDeals

  • BoostMakeDeal

  • BoostOfflineDealWithData

• Deals

  • DealsConsiderOfflineRetrievalDeals

  • DealsConsiderOfflineStorageDeals

  • DealsConsiderOnlineRetrievalDeals

  • DealsConsiderOnlineStorageDeals

  • DealsConsiderUnverifiedStorageDeals

  • DealsConsiderVerifiedStorageDeals

  • DealsPieceCidBlocklist

  • DealsSetConsiderOfflineRetrievalDeals

  • DealsSetConsiderOfflineStorageDeals

  • DealsSetConsiderOnlineRetrievalDeals

  • DealsSetConsiderOnlineStorageDeals

  • DealsSetConsiderUnverifiedStorageDeals

  • DealsSetConsiderVerifiedStorageDeals

  • DealsSetPieceCidBlocklist

• I

  • ID

• Log

  • LogList

  • LogSetLevel

• Market

  • MarketCancelDataTransfer

  • MarketDataTransferUpdates

  • MarketGetAsk

  • MarketGetRetrievalAsk

  • MarketImportDealData

  • MarketListDataTransfers

  • MarketListIncompleteDeals

  • MarketListRetrievalDeals

  • MarketPendingDeals

  • MarketRestartDataTransfer

  • MarketSetAsk

  • MarketSetRetrievalAsk

• Net

  • NetAddrsListen

  • NetAgentVersion

  • NetAutoNatStatus

  • NetBandwidthStats

  • NetBandwidthStatsByPeer

  • NetBandwidthStatsByProtocol

  • NetBlockAdd

  • NetBlockList

  • NetBlockRemove

  • NetConnect

  • NetConnectedness

  • NetDisconnect

  • NetFindPeer

  • NetLimit

  • NetPeerInfo

  • NetPeers

  • NetPing

  • NetProtectAdd

  • NetProtectList

  • NetProtectRemove

  • NetPubsubScores

  • NetSetLimit

  • NetStat

• Online

  • OnlineBackup

• Pieces

  • PiecesGetCIDInfo

  • PiecesGetMaxOffset

  • PiecesGetPieceInfo

  • PiecesListCidInfos

  • PiecesListPieces

• Runtime

  • RuntimeSubsystems

• Sectors

  • SectorsRefs