Local Index Directory
This page describes the Local Index Directory component in Boost, what it is used for, how it works and how to start using it
Last updated
This page describes the Local Index Directory component in Boost, what it is used for, how it works and how to start using it
Last updated
Local Index Directory is not yet released. This is a placeholder page for its documentation.
The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (cid).
Currently this task is performed by the DAG store component. The DAG store keeps its indexes on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.
LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools.
To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size.
When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way cilents can later retrieve subsets of the original deal data, without retrieving the full deal data.
When a client makes a request for data by cid, LID: - checks which piece the cid is in, and where in the piece the data is - checks which sector the piece is in, and where in the sector the piece is - reads the data from the sector
The retrieval use cases that the Local Index Directory supports are:
Request one root cid with a selector, receive many blocks
LID is able to: - look up which piece contains the root cid - look up which sector contains the piece - for each block, get the offset into the piece for the block
Request one block at a time
LID is able to: - look up which piece contains the block - get the size of the block (Bitswap asks for the size before getting the block data) - look up which sector contains the piece - get the offset into the piece for the block
Request a whole piece
LID is able to look up which sector contains the piece.
Request an individual block
LID is able to: - look up which piece contains the block - look up which sector contains the piece - get the offset into the piece for the block
Request a file by root cid
LID is able to: - look up which piece contains the block - look up which sector contains the piece - for each block, get the offset into the piece for the block
