HTTP Retrieval
How to configure and use HTTP retrievals in Boost
Boost is introducing a new binary,
booster-http, with release v1.2.0. This binary can be run alongside the boostd market process in order to serve retrievals over http.Currently, there is no payment method or built-in security integrated in the new binary. It can be run with any stable release of
boostd and can also be run on a separate machine from the boostd process.To build and run
booster-http :
1. Clone the main branch from the boost repogit clone https://github.com/filecoin-project/boost.git
2. Build the new binary
make booster-http
3. Collect the token information for boost, lotus-miner and lotus daemon API
boostd auth api-info --perm=admin
lotus auth api-info --perm=admin
lotus-miner auth api-info --perm=admin
Output of the above commands is in the following format.
<ENV VARIABLE NAME> = <TOKEN>:<API ADDRESS>
Please use only <TOKEN>:<API ADDRESS> part for the next step.
4. Start the
booster-http server with the above detailsbooster-http run --api-boost=$BOOST_API_INFO --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO
You can also run multiple processes on the same machine but you would need to use different port for each
booster-http instance by specifying the value using the --port flag. You can run multiple instances of the booster-http spread over multiple machines. 5. To download files from the new http-server locally, you use the following endpoints:
# Download a piece by its PieceCID
# Randomly selects the first deal/sector matching the CID
# This will transfer the entire piece including all its (compressible) padding
curl http://localhost:7777/piece/bagaSomePieceCID -o $HOME/download.piece
The booster-http server listens on localhost. To expose the server publically, SPs should run a reverse proxy such as NGINX to handle operational concerns like:
- SSL
- Authentication
- Load balancing
While booster-http may get more operational features over time, the intent is that providers who want to scale their HTTP operations will handle most of operational concerns via software in front of booster-http.
To enable public discovery of the Boost HTTP server, SPs should set the domain root in boostd's
config.toml: Under the [DealMaking] section, set HTTPRetrievalMultiaddr to the public domain root in multi-address format.Example
config.toml section:[DealMaking]
HTTPRetrievalMultiaddr = "/dns/foo.com/tcp/443/https"
Clients can determine if an SP offers HTTP retrieval by running:
boost provider retrieval-transports <miner id>
Clients can check the HTTP URL scheme version and supported queries
// Supported queries
curl https://foo.com/index
// Version
curl https://foo.com/info
Clients can download a piece using the domain root configured by the SP:
# Download a piece by its CID
curl https://foo.com/piece/bagaSomePieceCID -o $HOME/download.piece
Last modified 4mo ago