V2
Search
⌃K

HTTP retrieval

How to configure and use HTTP retrievals in Boost
Boost introduced a new binary, booster-http, with release v1.2.0. This binary can be run alongside the boostd market process in order to serve retrievals over http.
Currently, there is no payment method or built-in security integrated in the new binary. It can be run with any stable release of boostd and can also be run on a separate machine from the boostd process.
Release v1.7.0-rc1 introduced support in booster-http for running an IPFS gateway, which enables Storage Providers to serve content to their users in multiple formats as described below and demonstrated using curl.

Retrieving a full Filecoin Piece

When performing certain actions, such as replicating deals, it can be convenient to retrieve the entire Piece (with padding) to ensure commp integrity.
curl http://{SP's http retrieval URL}/piece/bagaSomePieceCID -o bagaSomePieceCID.piece

Retrieving a CAR file

To return the CAR file for a given CID, you can pass an Accept header with the application/vnd.ipld.car; format. This can be useful for retrieving the raw, unpadded data of a deal.
curl -H "Accept:application/vnd.ipld.car;" http://{SP's http retrieval URL}/ipfs/bafySomePayloadCID -o bafySomePayloadCID.car

Retrieving specific files

For Storage Providers that have enabled serving raw files (disabled by default), users can retrieve specific files, such as images by their cid and path where applicable. See Enable serving files for a more in depth example.
curl http://{SP's http retrieval URL}/ipfs/{content ID}/{optional path to resource} -o myimage.png

Retrieving IPLD blocks

For advanced IPFS and IPLD use cases, you can now retrieve individual blocks by passing an Accept header with the application/vnd.ipld.raw; format
curl -H "Accept:application/vnd.ipld.raw;" http://{SP's http retrieval URL}/ipfs/bafySomeBlockCID -o bafySomeBlockCID

Local setup

SPs should try a local setup and test their HTTP retrievals before proceeding to run booster-http in production.
The booster-http binary is built and installed with boostd binary. If you are planning to run booster-http on a different node, you can build and install the new binary. Otherwise, skip to step 3.
  1. 1.
    Clone the Boost repo and checkout the same release as your boostd version
git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>
  1. 2.
    Build and install the new binary
make build
sudo make build
  1. 3.
    Collect the token information for lotus-miner and lotus daemon API
export ENV_FULLNODE_API_INFO=`lotus auth api-info --perm=admin`
export FULLNODE_API_INFO=`echo $ENV_FULLNODE_API_INFO | awk '{split($0,a,"="); print a[2]}'`
export ENV_MINER_API_INFO=`lotus-miner auth api-info --perm=admin`
export MINER_API_INFO=`echo $ENV_MINER_API_INFO | awk '{split($0,a,"="); print a[2]}'`
  1. 4.
    Start the booster-http server with the above details
booster-http run --api-lid="ws://<boostd-data IP>:8044" --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO
You can run multiple booster-http processes on the same machine by using a different port for each instance with the --port flag. You can also run multiple instances of the booster-http on different machines.

Running public Boost HTTP retrieval

The booster-http server listens on localhost. To expose the server publically, SPs should run a reverse proxy such as NGINX to handle operational concerns like:
  • SSL
  • Authentication
  • Load balancing
While booster-http may get more operational features over time, the intent is that providers who want to scale their HTTP operations will handle most of operational concerns via software in front of booster-http. You can setup a simple NGINX proxy using the example provided in Serving files with booster-http​

Making HTTP Retrieval Discoverable

To enable public discovery of the Boost HTTP server, SPs should set the domain root in boostd's config.toml. Under the [DealMaking] section, set HTTPRetrievalMultiaddr to the public domain root in multi-address format.
Example config.toml section:
[DealMaking]
HTTPRetrievalMultiaddr = "/dns/foo.com/tcp/443/https"
Clients can determine if an SP offers HTTP retrieval by running:
boost provider retrieval-transports <miner id>
Clients can check the HTTP URL scheme version and supported queries
// Supported queries
curl https://foo.com/index
​
// Version
curl https://foo.com/info
Clients can download a piece using the domain root configured by the SP:
# Download a piece by its CID
curl https://foo.com/piece/bagaSomePieceCID -o download.piece
Previous
Retrieving data from Filecoin
Next
Serving files with booster-http
Last modified 17d ago