Boost stores metadata about deals in a sqlite database in the root directory of the Boost repo.
To open the database use a sqlite client:
sqlite3 boost.db
The database tables are
Deals
metadata about Boost storage deals (eg deal proposal) and their current state (eg checkpoint)
FundsLogs
log of each change in funds reserved for a deal
FundsTagged
how much FIL is tagged for deal collateral and publish message for a deal
StorageLogs
log of each change in storage reserved for a deal
StorageTagged
how much storage is tagged for a deal
Boost keeps a separate database just for deal logs, so as to make it easier to manage log data separately from deal metadata. The logs database is named boost.logs.db
and it has a single table DealLogs
that stores logs for each deal, indexed by uuid.
Boost uses goose
(https://pressly.github.io/goose/) tool and library for handling sqlite3 migrations.
goose
can be installed following the instructions at https://pressly.github.io/goose/installation/
Migrations in Boost are stored in the /db/migrations
directory.
Boost handles database migrations on start-up. If a user is running an older version of Boost, migrations up to the latest version are automatically applied on start-up.
Developers can use goose
to inspect and apply migrations using the CLI:
The DAG store manages a copy of unsealed deal data stored as CAR files. It maintains indexes over the CAR files to facilitate efficient querying of multihashes.
By default, the dagstore root will be:
$BOOST_PATH/dagstore
The directory structure is as follows:
index
: holds the shard indices.
transients
: holds temporary shard data (unsealed pieces) while they're being indexed.
datastore
: records shard state and metadata so it can survive restarts.
.shard-registration-complete
: marker file that signals that initial migration for legacy markets deals is complete.
.boost-shard-registration-complete
: marker file that signals that initial migration for boost deals is complete.
When you first start your boost process without a dagstore repo, a migration process will register all shards for both legacy and Boost deals in lazy initialization mode. As deals come in, shards are fetched and initialized just in time to serve the retrieval.
For legacy deals, you can monitor the progress of the migration in your log output, by grepping for the keyword migrator
. Here's example output. Notice the first line, which specifies how many deals will be evaluated (this number includes failed deals that never went on chain, and therefore will not be migrated), and the last lines (which communicate that migration completed successfully):
For Boost deals, you can do the same by grepping for the keyword boost-migrator
.
Forcing bulk initialization will become important in the near future, when miners begin publishing indices to the network to advertise content they have, and new retrieval features become available (e.g. automatic shard routing).
Initialization places IO workload on your storage system. You can stop/start this command at your wish/convenience as proving deadlines approach and elapse, to avoid IOPS starvation or competition with window PoSt.
To stop a bulk initialization(see the next paragraph), press Control-C. Shards being initialized at that time will continue in the background, but no more initializations will be performed. The next time you run the command, it will resume from where it left off.
You can force bulk initialization using the boostd dagstore initialize-all
command. This command will force initialization of every shard that is still in ShardStateNew
state for both legacy and Boost deals. To control the operation:
You must set a concurrency level through the --concurrency=N
flag.
A value of 0
will disable throttling and all shards will be initialized at once. ⚠️ Use with caution!
By default, only unsealed pieces will be indexed to avoid forcing unsealing jobs. To index also sealed pieces, use the --include-sealed
flag.
In our test environments, we found the migration to proceed at a rate of 400-500 shards/deals per second, on the following hardware specs: AMD Ryzen Threadripper 3970X, 256GB DDR4 3200 RAM, Samsung 970 EVO 2TB SSD, RTX3080 10GB GPU.
The DAG store can be configured through the config.toml
file of the node that runs the boost subsystem. Refer to the [DAGStore]
section. Boost ships with sane defaults:
Shards can error for various reasons, e.g. if the storage system cannot serve the unsealed CAR for a deal/shard, if the shard index is accidentally deleted, etc.
Boost will automatically try to recover failed shards by triggering a recovery once.
You can view failed shards by using the boostd dagstore list-shards
command, and optionally grepping for ShardStateErrored
.
The boostd
executable contains a dagstore
command with several useful subcommands:
boostd dagstore list-shards
boostd dagstore initialize-shard <key>
boostd dagstore initialize-all --concurrency=10
boostd dagstore gc
Refer to the --help
texts for more information.
By default the Boost daemon repository is located at ~/.boost
It contains the following files:
api
The local multi-address of Boost's libp2p API
boost.db
The sqlite database with all deal metadata
boost.logs.db
The sqlite database with the logs for deals
config.toml
The config file with all Boost's settings
repo.lock
A lock file created when Boost is running
storage.json
Deprecated (needed by legacy markets)
token
The token used when calling Boost's JSON RPC endpoints
It has the following directories:
dagstore
Contains indexes of CAR files stored with Boost
datastore
Contains metadata about deals for legacy markets
deal-staging
The directory used by legacy markets for incoming data transfers
incoming
The directory used by Boost for incoming data transfers
journal
Contains journal events (used by legacy markets)
keystore
Contains the secret keys used by libp2p (eg the peer ID)
kvlog
Used by legacy markets datastore
The boostd
executable runs as a daemon alongside a lotus node and lotus miner. This daemon replaces the current markets subsystem in the lotus miner. The boost daemon exposes a libp2p interface for storage and retrieval deals. It performs on-chain operations by making API calls to the lotus node. The daemon hands off downloaded data to the lotus miner for sealing via API calls to the lotus miner.
boostd
has a web interface for fund management and deal monitoring. The web interface is a react app that consumes a graphql interface exposed by the daemon.
The typical flow for a Storage Deal is:
The Client puts funds in escrow with the Storage Market Actor on chain.
The Client uploads a CAR file to a web server.
The Client sends a storage deal proposal to Boost with the URL of the CAR file.
Boost checks that the client has enough funds in escrow to pay for storing the file.
Boost accepts the storage deal proposal.
Boost downloads the CAR file from the web server.
Boost publishes the deal on chain.
The client checks that the deal was successfully published on chain.
Boost exposes a libp2p interface to listen for storage deal proposals from clients. This is similar to the libp2p interface exposed by the lotus market subsystem.
Boost communicates with the lotus node over its JSON-RPC API for on-chain operations like checking client funds and publishing the deal.
Once the deal has been published, Boost hands off the downloaded file to lotus-miner
for sealing.
Boost supports the same libp2p protocols as legacy markets, and adds new versions of the protocols used to propose a storage deal and to check the deal's status.
The client makes a deal proposal over v1.2.0
or v1.2.1
of the Propose Storage Deal Protocol: /fil/storage/mk/1.2.0
or /fil/storage/mk/1.2.1
It is a request / response protocol, where the request and response are CBOR-marshalled.
There are two new fields in the Request of v1.2.1
of the protocol, described in the table below.
The client requests the status of a deal over v1.2.0
of the Storage Deal Status Protocol: /fil/storage/status/1.2.0
It is a request / response protocol, where the request and response are CBOR-marshalled.
Field | Type | Description |
---|
Field | Type | Description |
---|
Field | Type | Description |
---|
Field | Type | Description |
---|
DealUUID | uuid | A uuid for the deal specified by the client |
IsOffline | boolean | Indicates whether the deal is online or offline |
ClientDealProposal | ClientDealProposal | Same as |
DealDataRoot | cid | The root cid of the CAR file. Same as |
Transfer.Type | string | eg "http" |
Transfer.ClientID | string | Any id the client wants (useful for matching logs between client and server) |
Transfer.Params | byte array | Interpreted according to |
Transfer.Size | integer | The size of the data that is sent across the network |
SkipIPNIAnnounce (v1.2.1) | boolean | Whether the provider should announce the deal to IPNI or not (default: false) |
RemoveUnsealedCopy (v1.2.1) | boolean | Whether the provider should keep an unsealed copy of the deal (default: false) |
Accepted | boolean | Indicates whether the deal proposal was accepted |
Message | string | A message about why the deal proposal was rejected |
DealUUID | uuid | The uuid of the deal |
Signature | A signature over the uuid with the client's wallet |
DealUUID | uuid | The uuid of the deal |
Error | string | Non-empty if there's an error getting the deal status |
IsOffline | boolean | Indicates whether the deal is online or offline |
TransferSize | integer | The total size of the transfer in bytes |
NBytesReceived | integer | The number of bytes that have been downloaded |
DealStatus.Error | string | Non-empty if the deal has failed |
DealStatus.Status | string |
DealStatus.Proposal | DealProposal |
SignedProposalCid | cid | cid of the client deal proposal + signature |
PublishCid | cid | The cid of the publish message, if the deal has been published |
ChainDealID | integer | The ID of the deal on chain, if it's been published |
The that the deal has reached