Skip to main content

NEAR Lake Indexer

NEAR Lake pushes the data to AWS S3

Description

NEAR Lake is an indexer built on top of NEAR Indexer Framework to watch the network and store all the events as JSON files on AWS S3.

History

We used to have NEAR Indexer for Explorer that was watching for the network and stored all the events to PostgreSQL database. PostgreSQL became the main bottleneck for us. After some brainstorming sessions and researches we decided to go with SingleStore database.

Knowing the fact that NEAR Explorer is not the only project that uses the Indexer for Explorer's database, we wanted to come up with the concept that will allow us to cover even more projects that can benefit from the data from NEAR Protocol.

That's why we decided to store the data from the blockchain as JSON files on AWS S3 bucket that can be used as a data source for different projects.

As "Indexer for Explorer Remake" project we are going to have near-lake as a data writer. There's going to be another project that will read from AWS S3 bucket and will store all the data in SingleStore database. This will replace NEAR Indexer for Explorer PostgreSQL database at some moment and will become the main source for NEAR Explorer.

How it works

note

Pagoda Inc. runs NEAR Lake nodes to store the data in JSON format on AWS S3

There is no need to run NEAR Lake by your own unless you have specific reasons to do that.

There are AWS S3 buckets created:

  • near-lake-data-testnet (eu-central-1 region)
  • near-lake-data-mainnet (eu-central-1 region)

All the buckets are set up the way the requester pays for the access. Anyone can read from these buckets by connecting to them with their own AWS credentials to be charged by Amazon.

Data structure

The data structure we use is the following:

    <block_height>/
block.json
shard_0.json
shard_1.json
...
shard_N.json

<block_height> is a 12-character-long u64 string with leading zeros (e.g "000042839521"). See this issue for a reasoning.

block_json contains JSON-serialized BlockView struct. NB! this struct might change in the future, we will announce it

shard_N.json where N is u64 starting from 0. Represents the index number of the shard. In order to find out the expected number of shards in the block you can look in block.json at .header.chunks_included

How to use it

We have created NEAR Lake Framework to have a simple straightforward way to create an indexer on top of the data stored by NEAR Lake itself.

NEAR Lake Framework announcement

We have announced the release of NEAR Lake Framework on NEAR Gov forum

Please, read the post there