Extending LiteDB with spatial indexing and spatial functions

[andrerav]

Hi,

First of all -- thank you for creating this super handy library! Now, down to business -- I'd like to gauge the interest for extending LiteDB with spatial capabilities, first and foremost by adding spatial indexing and some basic spatial functions. I must admit that this is not very well thought through, so I am just going to throw the idea out there and I hope you guys will chime in with feedback, corrections, suggestions or alternative ideas (also see the Questions section at the bottom).

Why

Spatial support is limited in existing NoSQL solutions, and location data (especially on mobile platforms) is getting more relevant by the day. I have come across two immediate use cases in my own projects; efficient storage and processing of location data on the edge (i.e a mobile device or a IoT device), and data engineering pipelines for transforming and processing medium-large spatial data sets (terabyte scale) in C# without resorting to PostGIS and other completely external dependencies. Also, all the building blocks are already available on the .NET platform.

The basic necessities for spatial support

• Geometry types (Point, Polygon, etc.) from NetTopologySuite are already used by big ORM frameworks like EF Core and NHibernate.Spatial, and could be a good fit for LiteDB as well
• Spatial indexing can be done by using for example quad trees from NetTopologySuite or, in a less sophisticated manner, by using geohashing. There are probably other relevant implementations available as well
• Lots of battle-tested geometry functions are readily available in NetTopologySuite, and by the looks of it I think it should be a straight-forward job to bridge them into the LiteDB query API

Required changes in LiteDB that I can think of

• Bridge spatial functions from NetTopologySuite into the query API
• Serialize mapped geometry types to/from BSON binaries using the WKB format (serializers available in NetTopologySuite)
• Add the necessary logic to QueryOptimization.ChooseIndex() to select the correct spatial index for columns where this exists
• Overload or extend LiteEngine.EnsureIndex() and add support for spatial indexes for geometry types
• The indexes must be stored on disk somehow -- I'm pretty blank on this one though, haven't dug into that part of the codebase yet :)

(...and probably lots of other small and big things I haven't thought of)

Misc

• A number of unit tests that covers all supported spatial functions and spatial index implementations must be in place (this is an obvious one, but I include it for completeness sake)

Questions

LiteDB has no third party dependencies today, and I'm going to venture a guess that you guys might not be all that keen to pull inn a bunch of dependencies for spatial support. If that is the case, do you have any thoughts on whether it would be possible to extend LiteDB as it sits today to add spatial support as outlined above?

I'd like to note that I would happily volunteer to implement this if it turns out to be a nice way to do it that aligns with your architecture principles and future plans :)

[fubar-coder]

What about a concept like "custom indexable type" to keep the amount external dependencies low?

[AleRoe]

Hi @andrerav

one approach to solving this at least partially could be to write am extension library that automatically configures the BsonMapper to properly serialize and deserialize your custom spatial type. I do this with my own custom types that are not supoorted out-of-the-box. Using an extension method would allow separating the additional libraries required from the core LiteDB code (which fortunately does not have any dependencies). Of course this would not cater for any custom indexing, which would require a plugable interface for indexing in LiteDB.
This is just of the top of my head but maybe something worth investigating.

Cheers
Alexander