NEAR Lake Indexer
NEAR Lake is an indexer built on top of NEAR Indexer Framework to watch the network and store all the events as JSON files on AWS S3.
We used to have NEAR Indexer for Explorer that was watching for the network and stored all the events to PostgreSQL database. PostgreSQL became the main bottleneck for us. After some brainstorming sessions and researches we decided to go with SingleStore database.
Knowing the fact that NEAR Explorer is not the only project that uses the Indexer for Explorer's database, we wanted to come up with the concept that will allow us to cover even more projects that can benefit from the data from NEAR Protocol.
That's why we decided to store the data from the blockchain as JSON files on AWS S3 bucket that can be used as a data source for different projects.
As "Indexer for Explorer Remake" project we are going to have
near-lake as a data writer. There's going to be another project that will read from AWS S3 bucket and will store all the data in SingleStore database. This will replace NEAR Indexer for Explorer PostgreSQL database at some moment and will become the main source for NEAR Explorer.
How it works
Pagoda Inc. runs NEAR Lake nodes to store the data in JSON format on AWS S3
There is no need to run NEAR Lake by your own unless you have specific reasons to do that.
There are AWS S3 buckets created:
All the buckets are set up the way the requester pays for the access. Anyone can read from these buckets by connecting to them with their own AWS credentials to be charged by Amazon.
The data structure we use is the following:
block_json contains JSON-serialized
BlockView struct. NB! this struct might change in the future, we will announce it
shard_N.json where N is
u64 starting from
0. Represents the index number of the shard. In order to find out the expected number of shards in the block you can look in
How to use it
We have created NEAR Lake Framework to have a simple straightforward way to create an indexer on top of the data stored by NEAR Lake itself.
NEAR Lake Framework announcement
We have announced the release of NEAR Lake Framework on NEAR Gov forum
Please, read the post there