NEAR Indexer ideas

[frol]

This is a discussion board tracking requests and ideas for future work around NEAR Indexer.

As mentioned in the README, we have plenty of ideas as to NEAR Indexer applications. Feel free to share your ideas and build your own indexers!

[stevepro-lab]

Is there Javascript or Type Script version of this indexer available?

[frol]

Well, we had an implementation in JS for NEAR Explorer, but it was built as a proof-of-concept, so the code is hacky and ultimately we just rewrote it in Rust. As of today, I am aware only of Rust indexers all based on the NEAR Indexer Framework (referenced in the topic description):

• https://github.com/near/near-indexer-for-explorer
• https://github.com/near/near-indexer-for-wallet
• https://github.com/fluxprotocol/flux-capacitor
• https://github.com/Mintbase/mintbase-near-indexer

[bowenwang1996]

@frol why is this not a discussion :)

[frol]

Moved 😉

[bowenwang1996]

time machine :)

[stevepro-lab]

@bowenwang1996 @frol We are estimating efforts to support Near Protocol on Dappquery analytics platform. In this process we are checking existing available indexers. We are already aware of Mintbase Near Indexer.

Please share any resources which we can use for further analysis.

[kcelestinomaria]

Hello guys, thinking of integrating NEAR Indexer with Hadoop or other Big Data solutions. Anyone working on this or thinking of working on this? @stevepro-lab @frol @bowenwang1996

[frol]

We don't have any plans for Hadoop or Big Data support yet. Currently, we can fit the whole history (without the contract binaries) into 200GB PostgreSQL database, so it is currently not in the realm of BigData in my opinion, but we should expect growth, and with 1k transactions per second on a single shard, we can get to the Big Data state quickly, so I would love to see your contribution in the future @kcelestinomaria 😉

[Pangstar]

Hi there, has there been any movement on the Kafka front, have you considered Apache Pulsar as an alternative?

[frol]

@Pangstar Actually, we decided not to go with Kafka since we need to optimize for long-term storage, and we don't have that huge of a volume of blocks. Thus, we went with a different strategy of having the blocks indexed into S3, and build a lightweight framework that would give you a stream of blocks by reading from S3.

We are about to finish MVP:

https://github.com/near/near-lake (we run this ourselves, but you may also run it if you wish)
https://github.com/near/near-lake-framework (this is lightweight indexer framework that is based on the data we dumped to S3; it is in Rust, but you may actually implement it in any language now)
https://github.com/near/near-lake-raw-printer (demo)

[mpc-enterprise]

Just +1 to request Kafka does not fall too far back on the NEAR roadmap. It ticks a big box for app integrations (in addition to the raw streaming capability) as many enterprises already understand that Kafka integration pattern i.e. "evolutionary architectures"....

[frol]

NEAR Lake Ecosystem was actually launched for early adopters and it was proven to be quite a success: https://gov.near.org/t/announcement-near-lake-framework-brand-new-word-in-indexer-building-approach/17668

We currently have Rust "client", but since the data is stored on S3, you can use any language you like, and we plan to release JS/TS version of it soon. Stay tuned, and read the new docs: https://near-indexers.io

[zhang-wenchao]

@Pangstar @frol @mpc-enterprise Hi, We have a project in progress, https://github.com/octopus-network/octopus-near-indexer-kafka. This project imports NEAR data into Kafka. And we have released an experimental version.

[zhang-wenchao]

Kafka is an important component in the data analysis platform.

[mpc-enterprise]

Hi Vincent, Excellent!
Thanks for the link, looks very neat. I built and started the network sync before I realised I did not have the disk space, argh...