# Setting up a monitoring stack for Boost
## Background
The monitoring stack we will use includes:
* Prometheus - collects metrics and powers dashboards in Grafana
* Tempo - collects traces and powers traces search in Grafana with Jaeger
* Grafana - provides visualization tools and dashboards for all metrics and traces
Lotus and Boost are already instrumented to produce traces and stats for Prometheus to collect.
The Boost team also packages a set of Grafana dashboards that are automatically provisioned as part of this setup.
{% hint style="info" %}
The monitoring setup should be done on the same node where `boostd` and other Boost services are running. If you are running multiple Boost services (like `boostd-data`/`booster-http`) across multiple machines then any one of those machine can be used.
{% endhint %}
## Prerequisites
This setup has been tested on macOS and on Linux. We haven’t tested it on Windows, so YMMV.
All the monitoring stack containers run in Docker.
## Steps
1. **Install Docker**
We have tested this setup with Docker 20.10.23 on macOS and Ubuntu.
2. **DNS resolution for Prometheus**
Update `extra_hosts` in `docker-compose.yaml` for `prometheus`, so that the Prometheus container can reach all its targets - `boostd`, `lotus-miner`, `booster-bitswap`, `booster-http`, etc.\
\
Depending on where your Filecoin processes (`boostd`, `lotus`, `lotus-miner`, `booster-bitswap`, etc.) are running, you need to confirm that they are reachable from Prometheus so that it can scrape their metrics.
By default the setup expects to find them within the same Docker network, so if you are running them elsewhere (i.e. on the \`host\` network), make sure to update the docker-compose file for the same.
3. **Loki plugin for Docker**
The `loki` plugin for docker is required to allow collecting logs from the services running on docker itself
4. **Storage location**
The prometheus and tempo services requires access to local storage to ensure all historical data is safe across the docker restarts. This path can be defined in the [env file](https://github.com/filecoin-project/boost/blob/main/docker/monitoring/.env).
The default value is `$HOME/.boost-monitoring`
Please ensure that the storage directory exists and has permission 0775 or 0777 depending on your user.
```
mkdir ~/.boost-monitoring
chmod 0777 ~/.boost-monitoring
```
5. **Update metrics endpoints for Prometheus**
In case, you are running all Boost services and monitoring stack on the same node, you can skip this step.
If any of your Boost services like `boostd-data` or `booster-http` are running on a different host then we must modify the prometheus configuration to update the endpoint to be scraped. Please edit file and update the IP addresses and port to scrap the metrics. Example:
```
- job_name: 'booster-bitswap'
static_configs:
- targets: [ ':' ]
labels:
host: 'local'
```
6. **Start the monitoring stack using docker-compose**
```
cd docker/monitoring
docker-compose up -d
```
## Verify Prometheus Targets
Confirm that Prometheus targets are scraped at Targets
{% hint style="info" %}
You can also access the Grafana on the localhost interface on a remote server, you can open an SSH tunnel from your local machine:
```
ssh -L 9190:localhost:9190 myserver
```
{% endhint %}
If you are running software firewall like \`ufw\`, you might need to modify your iptables and allow access from the Prometheus container / network to the Filecoin stack network, for example:
`sudo docker network inspect monitoring`\
`# note the Subnet for the network`\
`sudo ufw allow from 172.18.0.0/16`
## Grafana dashboards
Go to Grafana at and inspect the dashboards.
{% hint style="info" %}
You can also access the Grafana on the localhost interface on a remote server, you can open an SSH tunnel from your local machine:
```
ssh -L 3333:localhost:3333 myserver
```
{% endhint %}
{% hint style="info" %}
The default username and password for Grafana dashboard is "admin" and "admin" respectively.
{% endhint %}
