This tutorial goes through the steps required to run our Docker monitoring setup to collect and visualize metrics for various Boost processes
The monitoring stack we will use includes:
Prometheus - collects metrics and powers dashboards in Grafana
Tempo - collects traces and powers traces search in Grafana with Jaeger
Grafana - provides visualization tools and dashboards for all metrics and traces
Lotus and Boost are already instrumented to produce traces and stats for Prometheus to collect.
The Boost team also packages a set of Grafana dashboards that are automatically provisioned as part of this setup.
This setup has been tested on macOS and on Linux. We haven’t tested it on Windows, so YMMV.
All the monitoring stack containers run in Docker.
We have tested this setup with Docker 20.10.23 on macOS and Ubuntu.
https://docs.docker.com/engine/install/
Update extra_hosts in docker-compose.yaml for prometheus, so that the Prometheus container can reach all its targets - boostd, lotus-miner, booster-bitswap, booster-http, etc.
https://github.com/filecoin-project/boost/blob/main/docker/monitoring/docker-compose.yaml#L47-L55
Depending on where your Filecoin processes (boostd, lotus, lotus-miner, booster-bitswap, etc.) are running, you need to confirm that they are reachable from Prometheus so that it can scrape their metrics.
By default the setup expects to find them within the same Docker network, so if you are running them elsewhere (i.e. on the `host` network), add the following arguments:
Confirm that Prometheus targets are scraped at http://localhost:9090 / Targets
If you are running software firewall like `ufw`, you might need to modify your iptables and allow access from the Prometheus container / network to the Filecoin stack network, for example:
sudo docker network inspect monitoring
# note the Subnet for the network
sudo ufw allow from 172.18.0.0/16
Go to Grafana at http://localhost:3333 and inspect the dashboards:
Storage providers might demand very precise and dynamic control over a combination of deal parameters.
Boost, similarly to Lotus, provides two IPC hooks allowing you to name a command to execute for every deal before the storage provider accepts it:
Filter for storage deals.
RetrievalFilter for retrieval deals.
The executed command receives a JSON representation of the deal parameters, as well as the current state of the sealing pipeline, on standard input, and upon completion, its exit code is interpreted as:
0: success, proceed with the deal.
non-0: failure, reject the deal.
The most trivial filter rejecting any retrieval deal would be something like:
RetrievalFilter = "/bin/false".
/bin/false is binary that immediately exits with a code of 1.
lets the miner deny specific clients and only accept deals that are set to start relatively soon.
You can also use a third party content policy framework like bitscreen by Murmuration Labs, or :
Here is a sample JSON representation of the input sent to the deal filter:
This tutorial goes through all the steps required to make a storage deal with Boost on Filecoin.
First, you need to initialise a new Boost client and also set the endpoint for a public Filecoin node. In this example we are using https://glif.io
The init command will output your new wallet address, and warn you that the market actor is not initialised.
Then you need to send funds to the wallet, and add funds to the market actor (in the example below we are adding 1 FIL).
You can use the boostx utilities to add funds to the market actor:
You can confirm that the market actor has funds by running boost init again.
After that you need to generate a car file for data you want to store on Filecoin, and note down its payload-cid. We recommend using go-car CLI to generate the car file.
Then you need to calculate the commp and piece size for the generated car file:
Place the generated car file on a public HTTP server, so that a storage provider can later fetch it.
Finally, trigger an online storage deal with a given storage provider:
Step by step guide to various Boost tasks
