How to re-index unsealed pieces that are flagged by LID in Boost v2

This pages explains how to re-index unsealed pieces flagged by the Piece Doctor in the Local Index Directory so that they are retrievable

As explained in the tutorial how to upgrade from Boost v1 to Boost v2, in Boost v2 the Local Index Directory periodically checks all pieces that the SP stores and confirms if there exists an unsealed copy of the data and whether it is indexed. If the index is missing, the piece is flagged, meaning that the operator of the SP should fix it if they wish to make it retrievable.

Fixing individual flagged unsealed pieces from the Boost Web UI is possible directly from the Web UI.

If the SP has a lot of flagged pieces, you can automate the re-indexing of pieces with the following commands:

Fetch the piececids of all the flagged unsealed pieces from LID

// GraphQL query to get the first 100 flagged unsealed pieces from LID.
// If you have more than 100 flagged unsealed pieces, you need to modify the query
// and adjust the pagination parameters

curl -X POST -H "Content-Type: application/json" -d '{"query":"query { piecesFlagged(hasUnsealedCopy: true, limit: 100) { totalCount pieces { CreatedAt PieceCid IndexStatus { Status Error } } } }" }' http://localhost:8080/graphql/query | jq -r ".data.piecesFlagged.pieces[] | .PieceCid"

Trigger re-indexing of each piececid

// Re-index a specific piececid.
// Assumes that the piececid has an unsealed copy.

boostd lid gen-index <piececid>

Automatically re-index marked pieces only

This little script will fetch up to a thousand flagged pieces that are not processed yet, and will process them 4 at a time. Change the -P4 parameter to change the concurrency.

curl -X POST -H "Content-Type: application/json" -d '{"query":"query { piecesFlagged(hasUnsealedCopy: true, limit: 1000) { totalCount pieces { CreatedAt PieceCid IndexStatus { Status Error } } } }" }' http://localhost:8080/graphql/query | jq -rc ".data.piecesFlagged.pieces[]" | grep Registered | jq -r ".PieceCid" | xargs -P4 -L1 boostd lid gen-index