Boost is developing new market features on a regular basis as part of the overall market development. This section covers the experimental features released by boost along with details on how to use them.

It is not recommended to run experimental features in production environments. The features should be tested as per your requirements, and any issues or requests should be reported to the team via Github or Slack.

Once the new features have been tested and vetted, they will be released as part of a stable Boost release and all documentation concerning those features will be moved to an appropriate section of this site.

Current experimental features are listed below.

Dependencies

Local Index Directory depends on a backend database to store various indices. Currently we support two implementations - YugabyteDB or LevelDB - depending on the size of deal data and indices a storage provider holds.

LevelDB is an open source on-disk key-value store, and can be used when indices fit on a single host.

YugabyteDB is an open source modern distributed database designed to run in any public, private, hybrid or multi-cloud environment.

Hardware requirements

YugabyteDB is designed to run on bare-metal machines, virtual machines (VMs), and containers. CPU and RAM

You should allocate adequate CPU and RAM. YugabyteDB has adequate defaults for running on a wide range of machines, and has been tested from 2 core to 64 core machines, and up to 200GB RAM.

Minimum requirement

2 cores
2GB RAM

Production requirement

16+ cores
32GB+ RAM
Add more CPU (compared to adding more RAM) to improve performance.

Verify support for SSE2 and SSE4.2

YugabyteDB requires the SSE2 instruction set support, which was introduced into Intel chips with the Pentium 4 in 2001 and AMD processors in 2003. Most systems produced in the last several years are equipped with SSE2.

In addition, YugabyteDB requires SSE4.2.

To verify that your system supports SSE2, run the following command:

cat /proc/cpuinfo | grep sse2

To verify that your system supports SSE4.2, run the following command:

cat /proc/cpuinfo | grep sse4.2

Disks

SSDs (solid state disks) are required.

We recommend a minimum of 1TiB or more allocated for YugabyteDB, depending on the amount of deal data you store and its average block size.

With the release of FVM, it is now possible for smart contracts to make deal proposals on-chain. This is made possible though the DealProposal FRC.

DataDAOs, as well as other clients who want to store data on Filecoin, can now deploy a smart contract on the FVM which adheres to the DealProposal FRC, and make deal proposals that are visible to every storage provider who monitors the chain.

How to enable FVM monitoring in order to process storage deal proposals published on-chain?

Boost already has support for the DealProposal FRC.

The code for FVM monitoring resides in the latest release of the Boost. It should be used with caution for production use. SPs must enable FEVM on lotus daemon before proceeding to the next step.

To build for mainnet:

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <Release>
make build

In order to enable DealProposal FRC, you have to edit your config.toml and enable contract deal monitoring. By default it is disabled. Here is an example configuration:

[ContractDeals]
  Enabled = true
  AllowlistContracts = []
  From = "0x0000000000000000000000000000000000000000"

AllowlistContracts field could be left empty if you want to accept deals from any client. If you only want to accept deals from some clients, you can specify their contract addresses in the field.

From field should be set to your SP's FEVM address. Some clients may implement a whitelist which allows specific SPs to accept deal proposals from their contract. This field will help those clients identify your SP and match it to their whitelist.

How contract deals work in Boost

1. A contract publishes a DealProposalCreate event on the chain.

2. Boost monitors the chain for such events from all the clients by default. When such an event is detected, we go and fetch the data for the deal.

3. Deal is then run through the basic deal validation filters like clients has enough funds, SP has enough funds etc.

4. Once deal passes the validation, we create a new deal handler in Boost and pass this deal for execution like other Boost deals.

Background

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (cid).

Currently this task is performed by the DAG store component. The DAG store keeps its indexes on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools.

To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size.

Storing data on Filecoin

When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way cilents can later retrieve subsets of the original deal data, without retrieving the full deal data.

Retrieving data

When a client makes a request for data by cid, LID: - checks which piece the cid is in, and where in the piece the data is - checks which sector the piece is in, and where in the sector the piece is - reads the data from the sector

Use cases

The retrieval use cases that the Local Index Directory supports are:

Graphsync retrieval

Request one root cid with a selector, receive many blocks

LID is able to: - look up which piece contains the root cid - look up which sector contains the piece - for each block, get the offset into the piece for the block

Bitswap retrieval

Request one block at a time

LID is able to: - look up which piece contains the block - get the size of the block (Bitswap asks for the size before getting the block data) - look up which sector contains the piece - get the offset into the piece for the block

HTTP retrieval

Request a whole piece

LID is able to look up which sector contains the piece.

Request an individual block

LID is able to: - look up which piece contains the block - look up which sector contains the piece - get the offset into the piece for the block

Request a file by root cid

LID is able to: - look up which piece contains the block - look up which sector contains the piece - for each block, get the offset into the piece for the block

Considering that the Local Index Directory is a new feature, Storage Providers should initialise it after upgrading their Boost deployments.

There are two ways a Storage Provider can do that:

1. Migrate existing indices from the DAG store into LID: this solution assumes that the Storage Provider has been keeping an unsealed copy for every sector they prove on-chain, and has already indexed all their deal data into the DAG store. Typically index sizes for a given sector range between 100KiB up to 1GiB, depending on deal data and its blocks sizes. The DAG store keeps these indices in the repository directory of Boost under the ./dagstore/index and ./dagstore/datastore directories. This data should be migrated to LID with the migrate-lid utility.

2. Recreate indices for deal data based on unsealed copies of sectors: this solution assumes that the Storage Provider has unsealed copies for every sector they prove on-chain. If this is not the case, then the SP should first trigger an unseal (UNS) job on their system for every sector that contains user data and produce an unseal copy. SPs can use the boostd recover lid utility to produce an index for all deal data within an unsealed sector and store it in LID so that they enable retrievals for the data. Depending on SPs deployment and where unsealed copies are hosted (NFS, Ceph, external disks, etc.) and the performance of the hosting system, producing an index for a 32GiB sector can take anywhere from a few seconds up to a few minutes, as the unsealed copy needs to be processed by the utility.

Migrate existing indices from the DAG store into LID

TODO

Recreate indices for deal data based on unsealed copies of sectors

TODO

When designing the Local Index Directory we considered the needs of various Storage Providers (SPs) and the operational overhead LID would have on their systems. We built a solution for: - small- SPs - holding up to 1PiB), and - mid- and large- size SPs - holding anywhere from 1PiB, up to 100PiB data

Depending on underlying block size and data format, index size can vary in size. Typically block sizes are between 16KiB and 1MiB.

At the moment there are two implementations of LID: - a simple LevelDB implementation, for small SPs who want to keep all information in a single process database. - a scalable YugabyteDB implementation, for medium and large size SPs with tens of thousands of deals.

Index types

In order to support the described retrieval use cases, LID maintains the following indexes:

multihash → []piece cid

To look up which pieces contain a block

piece cid → sector information {sector ID, offset, size}

To look up which sector a piece is in

piece cid → map<mulithash → block offset / size>

To look up where in the piece a block is and the block’s size