Develop Vector Store Management Pattern Similar to ORM

[shoutsid]

Issue 3: Develop Vector Store Management Pattern Similar to ORM

Title: Develop Vector Store Management Pattern Similar to ORM
Description:
Develop a management pattern similar to Object-Relational Mapping (ORM) that is specifically designed to handle opensource vector space models. This pattern will aim to streamline the way we interact with vector data stores, improving data access and manipulation.

Context

The project aims to leverage opensource vector space models for various functionalities. Traditional ORMs are not suitable for this kind of data. The goal is to create a pattern similar to ORM but designed to work with vector stores.

Expected Outcomes

• Design and implement a vector store management pattern that provides CRUD (Create, Read, Update, Delete) operations for vector data.
• The pattern should be compatible with multiple types of vector space models and data stores.

Challenges

• Creating a pattern flexible enough to accommodate different types of vector space models and data stores.
• Ensuring efficient handling of large-scale vector data.

Recommended Vector Data Stores and Libraries

• Vector Data Stores
  • Weaviate for GraphQL-based vector search.
  • Pinecone for distributed vector indexing and search.

Resources

• Weaviate and Pinecone documentation for understanding their capabilities and limitations.
• Research papers or articles on efficient data access patterns could be a good starting point.

[shoutsid]

llama_index has some existing infra for this, I've added it to the requriements.txt file for now.