Boost introduced a new binary, booster-http, with release v1.2.0. This binary can be run alongside the boostd market process in order to serve retrievals over http.

Currently, there is no payment method or built-in security integrated in the new binary. It can be run with any stable release of boostd and can also be run on a separate machine from the boostd process.

Release v1.7.0-rc1 introduced support in booster-http for running an IPFS gateway, which enables Storage Providers to serve content to their users in multiple formats as described below and demonstrated using curl.

Retrieving a full Piece

When performing certain actions, such as replicating deals, it can be convenient to retrieve the entire Piece (with padding) to ensure commp integrity.

curl http://{SP's http retrieval URL}/piece/bagaSomePieceCID -o bagaSomePieceCID.piece

Retrieving a CAR file

To return the CAR file for a given CID, you can pass an Accept header with the application/vnd.ipld.car; format. This can be useful for retrieving the raw, unpadded data of a deal.

curl -H "Accept:application/vnd.ipld.car;" http://{SP's http retrieval URL}/ipfs/bafySomePayloadCID -o bafySomePayloadCID.car

Retrieving specific files

For Storage Providers that have enabled serving raw files (disabled by default), users can retrieve specific files, such as images by their cid and path where applicable. See Enable serving files for a more in depth example.

curl http://{SP's http retrieval URL}/ipfs/{content ID}/{optional path to resource} -o myimage.png

Retrieving IPLD blocks

For advanced IPFS and IPLD use cases, you can now retrieve individual blocks by passing an Accept header with the application/vnd.ipld.raw; format

curl -H "Accept:application/vnd.ipld.raw;" http://{SP's http retrieval URL}/ipfs/bafySomeBlockCID -o bafySomeBlockCID

Local Setup

SPs should try a local setup and test their HTTP retrievals before proceeding to run booster-http in production.

To build and run booster-http :

1. Clone the boost repo and checkout the latest release

git clone https://github.com/filecoin-project/boost.git
cd boost
git checkout <release>

1. Build the new binary

make booster-http

1. Collect the token information for boost, lotus-miner and lotus daemon API

export ENV_BOOST_API_INFO=`boostd auth api-info --perm=admin`
export BOOST_API_INFO=`echo $ENV_BOOST_API_INFO | awk '{split($0,a,"="); print a[2]}'`

export ENV_FULLNODE_API_INFO=`lotus auth api-info --perm=admin`
export FULLNODE_API_INFO=`echo $ENV_FULLNODE_API_INFO | awk '{split($0,a,"="); print a[2]}'`

export ENV_MINER_API_INFO=`lotus-miner auth api-info --perm=admin`
export MINER_API_INFO=`echo $ENV_MINER_API_INFO | awk '{split($0,a,"="); print a[2]}'`

1. Start the booster-http server with the above details

booster-http run --api-boost=$BOOST_API_INFO --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO

Running Public Boost HTTP Retrieval

The booster-http server listens on localhost. To expose the server publically, SPs should run a reverse proxy such as NGINX to handle operational concerns like:

• SSL

• Authentication

• Load balancing

While booster-http may get more operational features over time, the intent is that providers who want to scale their HTTP operations will handle most of operational concerns via software in front of booster-http. You can setup a simple NGINX proxy using the example provided in Serving files with booster-http

Making HTTP Retrieval Discoverable

To enable public discovery of the Boost HTTP server, SPs should set the domain root in boostd's config.toml. Under the [DealMaking] section, set HTTPRetrievalMultiaddr to the public domain root in multi-address format.

Example config.toml section:

[DealMaking]
  HTTPRetrievalMultiaddr = "/dns/foo.com/tcp/443/https"

Clients can determine if an SP offers HTTP retrieval by running:

boost provider retrieval-transports <miner id>

Clients can check the HTTP URL scheme version and supported queries

// Supported queries
curl https://foo.com/index

// Version
curl https://foo.com/info

Clients can download a piece using the domain root configured by the SP:

# Download a piece by its CID
curl https://foo.com/piece/bagaSomePieceCID -o download.piece

With the release v1.7.0-rc1 of booster-http, Storage Providers can now serve blocks and files directly over the HTTP protocol. booster-http now implements a IPFS HTTP gateway with a path resolution style. This will allow the clients to download individual IPFS blocks, car files and request uploaded files directly from their browser.

SPs can take advantage of the ecosystem of tools to manage HTTP traffic, like load balancers and reverse proxies.

Configuring what to serve

The booster-http service can be started with specific type of content on IPFS gateway API

   --serve-pieces                                           enables serving raw pieces (default: true)
   --serve-blocks                                           serve blocks with the ipfs gateway API (default: true)
   --serve-cars                                             serve CAR files with the ipfs gateway API (default: true)
   --serve-files                                            serve original files (eg jpg, mov) with the ipfs gateway API (default: false)
   --api-filter-endpoint value                              the endpoint to use for fetching a remote retrieval configuration for bitswap requests
   --api-filter-auth value                                  value to pass in the authorization header when sending a request to the API filter endpoint (e.g. 'Basic ~base64 encoded user/pass~'
   --badbits-denylists value [ --badbits-denylists value ]  the endpoints for fetching one or more custom BadBits list instead of the default one at https://badbits.dwebops.pub/denylist.json (default: "https://badbits.dwebops.pub/denylist.json")
   --help, -h                                               show help

This allows SPs to run multiple booster-http instances, each serving specific type of content like car files only or raw blocks only.

Enable serving files

In the curl request below we appended the query parameter format=raw to the URL to get the raw block data for the file.

$ curl --output /tmp/museum.jpg "http://localhost:7777/ipfs/bafybeidqindpi4ucx7kmrtnw3woc6jtl7bqvyiokrkpbbuy6gs6trn57tm/vincent/Vincent%20van%20Gogh_files/Caf%C3%A9tafel_met_absint_-_s0186V1962_-_Van_Gogh_Museum.jpg?format=raw"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11830  100 11830    0     0   140k      0 --:--:-- --:--:-- --:--:--  175k

$ open /tmp/museum.jpg

But, if we try to open the file directly in a web browser, with no extra query parameters, we get an error message:

By default booster-http does not serve files in a format that can be read by a web browser. This is to protect Storage Providers from serving content that may be flagged as illicit content.

To enable serving files to web browsers, we must pass --serve-files=true to booster-http on startup. Once, booster-http is restarted with --serve-files=true, we can open the file directly from a web browser:

We can also browse all files in the CAR archive.

Protecting booster-http with NGINX

SPs must secure their booster-http before exposing it to the public. SPs can feel free to use any tool available to limit who can download files, the number of requests per second, and the download bandwidth each client can use per second.

Users can follow this example to use NGNIX reverse proxy to secure their booster-http instance. In this section we’ve just scratched the surface of the ways in which nginx can set access limits, rate limits and bandwidth limits. In particular it’s possible to add limits by request token, or using JWT tokens. The examples in this section are adapted from Deploying NGINX as an API Gateway which goes into more detail.

By default nginx puts configuration files into /etc/nginx

The default configuration file is /etc/nginx/sites-available/default

Setup server block

Setup nginx server listen on port 7575 and forward requests to booster-http on port 7777

The IPFS gateway serves files from /ipfs. So, we will add a server block for location /ipfs/

# ipfs gateway config
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location /ipfs/ {
                proxy_pass http://127.0.0.1:7777;
        }
}

Limiting Access

Let’s limit access to the IPFS gateway using the standard .htaccess file. We need to set up an .htaccess file with a username and password. Create a user named alice

$ mkdir /etc/nginx/ipfs-gateway.conf.d

$ htpasswd -c /etc/nginx/ipfs-gateway.conf.d/.htpasswd alice
New password:
Re-type new password:
Adding password for user alice

Include the .htaccess file in the /etc/nginx/sites-available/default

 # ipfs gateway config
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location /ipfs/ {
        # htaccess authentication
                auth_basic "Restricted Server";
                auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd;
                proxy_pass http://127.0.0.1:7777;
        }
}

Now when we open any URL under the path /ipfs we will be presented with a Sign in dialog.

Rate Limiting

To prevent users from making too many requests per second, we should add rate limits.

1. Create a file with the rate limiting configuration at /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf

2. Add a request zone limit to the file of 1 request per second, per client IP

limit_req_zone $binary_remote_addr zone=client_ip_10rs:1m rate=1r/s;

1. Include ipfs-gateway.conf in /etc/nginx/sites-available/default and set the response for too many requests to HTTP response code 429

include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf;
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location /ipfs/ {
                # htaccess authentication
                auth_basic "Restricted Server";
                auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd;

                limit_req zone=client_ip_10rs;
                limit_req_status 429;
                proxy_pass http://127.0.0.1:7878;
        }
}

1. Click the refresh button in your browser on any path under /ipfs more than once per second you will see a 429 error page

Bandwidth Limiting

It is also recommended to limit the amount of bandwidth that clients can take up when downloading data from booster-http. This ensures a fair bandwidth distribution to each client and prevents situations where one client ends up choking the booster-http instance.

1. Create a new .htaccess user called bob

$ htpasswd /etc/nginx/ipfs-gateway.conf.d/.htpasswd bob

1. Add a mapping from .htaccess username to bandwidth limit in /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf

map $remote_user $bandwidth_limit {
    default  1k;
    "alice"  10k;
    "bob"    512k;
}

1. Add the bandwidth limit to /etc/nginx/sites-available/default

include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf;
server {
        listen 7575 default_server;
        listen [::]:7575 default_server;

        location /ipfs/ {
                # htaccess authentication
                auth_basic "Restricted Server";
                auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd;

                limit_rate $bandwidth_limit;

                limit_req zone=client_ip_10rs;
                limit_req_status 429;
                proxy_pass http://127.0.0.1:7878;
        }
}

1. To verify bandwidth limiting, use curl to download a file with user alice and then bob Note the difference in the Average Dload column (the average download speed).