LogoLogo
v2.x (Stable)
v2.x (Stable)
  • What is Boost?
  • Features
  • Components
    • boostd
      • Repository
      • GraphQL API
      • JSON-RPC API
      • SQLite metadata database
    • Local Index Directory
      • Index types
      • Dependencies
      • Initialisation
    • boostd-data
    • YugabyteDB
    • booster-http
    • booster-bitswap
    • libp2p Protocols
  • Hardware requirements
    • YugabyteDB
  • Installation
  • New Boost Setup
  • Configuration
    • UI Settings
    • HTTP Transfer limit
    • Deal Filters
    • Remote CommP
    • Legacy Deal configuration
    • HTTP indexer announcement
    • Manual Publish Storage Deal Message
  • Monitoring
    • Setting up a monitoring stack for Boost
  • Storing data on Filecoin
  • Retrieving data from Filecoin
    • HTTP retrieval
    • Advanced Configuration of booster-http
    • Bitswap retrieval
  • Backup and Restore
  • Tutorials
    • How to upgrade from Boost v1 to Boost v2
    • How to re-index unsealed pieces that are flagged by LID in Boost v2
    • How to upgrade from v2.0.0 to v2.1.0
    • How to upgrade from v2.1.x to v2.2.0
    • Start and stop Boost processes
    • How to store files with Boost on Filecoin
    • Using filters for storage and retrieval deals
    • Migrate from Lotus to Boost
    • How to onboard data using DDO deals
  • Troubleshooting
  • Experimental Features
    • FVM Contract Deals
    • Direct Deals
    • Data Segment Indexing
  • FAQ
  • Need help?
Powered by GitBook
On this page
  • Background
  • Storing data on Filecoin
  • Retrieving data
  • Use cases
Edit on GitHub
Export as PDF
  1. Components

Local Index Directory

This page describes the Local Index Directory component in Boost, what it is used for, how it works and how to start using it

PreviousSQLite metadata databaseNextIndex types

Last updated 11 months ago

Background

The Local Index Directory (LID) manages and stores indices of deal data so that it can be retrieved by a content identifier (cid).

Currently this task is performed by the DAG store component. The DAG store keeps its indices on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB.

LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools.

To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it, the client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size.

At the moment there are two implementations of LID: - a simple LevelDB implementation, for small SPs who want to keep all information in a single process database. - a scalable YugabyteDB implementation, for medium and large size SPs with tens of thousands of deals.

When designing LID we considered the needs of various Storage Providers (SPs) and the operational overhead LID would have on their systems. We built a solution for: - small- SPs - holding up to 1PiB, and - mid- and large- size SPs - holding anywhere from 1PiB, up to 100PiB data

Depending on underlying block size and data format, index size can vary in size. Typically block sizes are between 16KiB and 1MiB.

Storing data on Filecoin

When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way clients can later retrieve subsets of the original deal data, without retrieving the full deal data.

Retrieving data

When a client makes a request for data by cid, LID: - checks which piece the cid is in, and where in the piece the data is - checks which sector the piece is in, and where in the sector the piece is - reads the data from the sector

Use cases

The retrieval use cases that the Local Index Directory supports are:

Graphsync retrieval

Request one root cid with a selector, receive many blocks

LID is able to: - look up which piece contains the root cid - look up which sector contains the piece - for each block, get the offset into the piece for the block

Bitswap retrieval

Request one block at a time

LID is able to: - look up which piece contains the block - get the size of the block (Bitswap asks for the size before getting the block data) - look up which sector contains the piece - get the offset into the piece for the block

HTTP retrieval

Request a whole piece

LID is able to look up which sector contains the piece.

Request an individual block

LID is able to: - look up which piece contains the block - look up which sector contains the piece - get the offset into the piece for the block

Request a file by root cid

LID is able to: - look up which piece contains the block - look up which sector contains the piece - for each block, get the offset into the piece for the block

How Boost stores deal data from clients
How clients retrieve their data from Boost