New Boost Setup
How setup a new Boost instance for a new lotus-miner
Boost v2 introduces the Local Index Directory as a replacement for the DAG store. It scales horizontally and provides a more intuitive experience for users, by surfacing problems in the UI and providing repair functionality.
The Local Index Directory (LID) stores retrieval indices in YugabyteDB. Retrieval indices store the size and location of each block in the deal data.
It is recommend to run YugabyteDB on a dedicated host with SSD drives. Depending on how many blocks there are in the user data, the retrieval indices may require up to 2% of the size of the unsealed data. e.g. 1 TiB of unsealed user data may require a 20 GiB index.
YugabyteDB requires about the same amount of space as your DAG store requires today.
You can find more information about YugabyteDB in the
Follow these instructions in order to setup a new Boost instance with
- 1.Make sure you have a Lotus node and miner running
- 2.Create and send funds to two new wallets on the lotus node to be used for Boost. Boost currently uses two wallets for storage deals:
- The publish storage deals wallet - This wallet pays the gas cost when Boost sends the
- The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the
export PUBLISH_STORAGE_DEALS_WALLET=`lotus wallet new bls`
export COLLAT_WALLET=`lotus wallet new bls`
lotus send --from mywallet $PUBLISH_STORAGE_DEALS_WALLET 10
lotus send --from mywallet $COLLAT_WALLET 10
- 3.Set the publish storage deals wallet as a control wallet
lotus-miner actor control set --really-do-it $PUBLISH_STORAGE_DEALS_WALLET
boostd-datais a data proxy service which abstracts the access to LID through an established API. It makes it easier to secure the underlying database and not expose it.
boostd-datalistens to a websocket interface, which is the entrypoint which should be exposed to
boostd-dataservice with parameters to connect to YugabyteDB on its Cassandra and PostgreSQL interfaces:
./boostd-data run yugabyte \
--hosts <yugabytedb-hosts> \
boostd-dataservice can run a separate machine from Boost as long as the service is reachable via the network from the boost node
- 6.Create and initialize the Boost repository
Boost keeps all data in a directory called the repository. By default the repository is at
~/.boost. To use a different location pass the
--boost-repoparameter (must precede any particular command verb, e.g.
boostd --boost-repo=<PATH> init).
Export the environment variables needed for
boostd initto connect to the lotus daemon and lotus miner.
export $(lotus auth api-info --perm=admin)
export $(lotus-miner auth api-info --perm=admin)
Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the
boostnode to make JSON-RPC calls to the
export APISEALER=`echo $MINER_API_INFO`
export APISECTORINDEX=`echo $MINER_API_INFO`
boostd initto create and initialize the repository:
boostd --vv init \
--api-sealeris the API info for the lotus-miner instance that does sealing
--api-sector-indexis the API info for the lotus-miner instance that provides storage
--max-staging-deals-bytesis the maximum amount of storage to be used for downloaded files (once the limit is reached Boost will reject subsequent incoming deals)
ulimitfile descriptor limit if necessary. Boost deals will fail if the file descriptor limit for the process is not set high enough. This limit can be raised temporarily before starting the Boost process by running the command
ulimit -n 1048576. We recommend setting it permanently by following the Permanently Setting Your ULIMIT System Value guide.
- 8.Before you start your boostd, it is important that it is reachable from any peer in the Filecoin network. For this, you will need a stable public IP and edit your
<boostd repo>/config.tomlas follows:[Libp2p]ListenAddresses = ["/ip4/0.0.0.0/tcp/24001"] # choose a fixed portAnnounceAddresses = ["/ip4/<YOUR_PUBLIC_IP_ADDRESS>/tcp/24001"] # important!
boostd-datadetails to the
boostdrepository config (located at
<boostd repo>/config.toml) to point to the exposed
boostd-dataservice endpoint. Note that the connection must be configured to go over a websocket. For example:[LocalIndexDirectory]ServiceApiInfo = "ws://<boostd-data>:8044"
- 10.Start the
boostdprocessboostd --vv run
- 11.Make sure that the correct peer id and multiaddr for your SP is set on chain, given that
boost initgenerates a new identity. Use the following commands to update the values on chain:
lotus-miner actor set-addrs <MULTIADDR>
lotus-miner actor set-peer-id <PEER_ID>
<MULTIADDR> should be the same as the
ListenAddressesyou set in the
Libp2psection of the config.toml of Boost <PEER_ID> can be found in the output of
boostd net idcommand
At this stage you should have the latest version of Boost running with the Local Index Directory. Go to the Local Index Directory page and review the number sections:
Pieces section shows counters for total pieces of user data that your SP is storing as well as whether you are keeping unsealed and indexed copies of them.
Flagged pieces are pieces that either lack an unsealed copy or are missing an index. For the sealed-only user data, you should make sure that you unseal individual sectors if you want this data to be retrievable.
Sealed only copies of data are not retrievable and are only being proven on-chain within the corresponding deadline / window. Typically sealed only data is considered as archival as it is not immediately retrievable. If the client requests it, the SP sealing pipeline must first unseal it, which typically takes 1-2 hours, and only then the data becomes available.
Flagged (unsealed) pieces is user data that your SP is hosting, which is not indexed.
Deal Sector Copies section displays counters of your sectors state - whether you keep unsealed copies for all sectors or not. Ideally the SP should keep unsealed copies for all data that should be immediately retrievable.
Sector Proving State section displays counters of your active and inactive sectors - active sectors are those that are actively proven on-chain, inactive sectors are those that you might have failed to publish a WindowPoSt for, or are expired or removed.