Troubleshooting
Inspect
The new inspect page in the Boost UI helps with debugging retrieval problems. It allows the user to check the following using a payload CID or piece CID:
Verify if the piece has been correctly added to the Piece Store
Validate if the piece is indexed in the DAG store
Check for an unsealed copy of the piece
Verify that the payload CID -> piece CID index has been created correctly
Failed to connect to peer
If the client cannot connect to Boost running on a Storage provider, with an error similar to the following:
The problem is that:
The SP registered their peer id and address on chain.
eg "Register the peer id 123abcd
at address ip4/123.456.12.345/tcp/1234
"
The SP changed their peer id locally but didn't update the peer id on chain.
The client wants to make a storage deal with peer
123abcd
. The client looks on chain for the address of peer123abcd
and sees peer123abcd
has registered an addressip4/123.456.12.345/tcp/1234
.The client sends a deal proposal for peer
123abcd
to the SP at addressip4/123.456.12.345/tcp/1234
.The SP has changed their peer ID, so the SP responds to the deal proposal request with an error:
peer id mismatch
To fix the problem, the SP should register the new peer id on chain:
Update storage provider's on chain address
Clients would not be able to connect to Boost running on a Storage provider after an IP change. This happens as clients lookup the registered peer id and address on chain for a SP. When a SP changes their IP or address locally, they must update the same on chain.
The SP should register the new peer id on chain using the following lotus-miner command
Please make sure to use the public IP and port of the Boost node and not lotus-miner
node if your miner and boostd
runs on a separate machine.
The on chain address change requires access to the worker key and thus the command lives in lotus-miner
instead of Boost.
Error in lotus-miner info output
After migrating to Boost, following error is seen when running lotus-miner info
:
Problem:
lotus-miner
is making a call on lotus-market
process which has been replaced by Boost, but lotus-miner
is not aware of the new market process.
Solution:
Export the MARKETS_API_INFO variable on your lotus-miner node.
Fix retrievals with error "failed to lookup index for mh"
The following error shows up when trying to retrieve the data from a storage provider.
The error indicates that dagstore does not have a corresponding index shard for the piece containing the requested data. When a retrieval is requested, the dagstore on storage provider side is queried and a reverse look up is used to determine the key(piece CID). This key is then used to query the piece store to find the sector containing the data and byte offset.
If for any reason the shard is not registered with the dagstore then reverse look up to find the piece CID fails and the above error is seen. The most widely know reason for not having the shard registered with dagstore is the below error.
To fix the deals where retrievals are impacted by above error, user will need to register the shards manually with dagstore:
If you have multiple deals in such state then you will need to generate a list of registered pieces with piece store and then compare with the shards available in the dagstore to create a list of missing shards.
Please stop accepting any deals and ensure all current deals are handed off to the lotus-miner (sealer) subsystem before proceeding from here.
1. Create a list of all sectors on lotus-miner
and redirect the output to a file. Copy the output file to boost node to be used by the below command.
2. Generate a list of shards to be registered
3. Register the shards with dagstore in an automated fashion.
Please note that each shard may take upto 3-5 minutes to get registered. So, the above command might take hours or days to complete depending upon the number of missing shards.
Last updated