[FEA]: Add a .vdb_bulk_upload method to the Ingestor class.

[drobison00]

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Significant improvement

Please provide a clear description of problem this feature solves

Requirements

• Add a .vdb_bulk_upload method to the Ingestor class. This method will eliminate the need for using utility functions to create and bulk-upload to milvus.
• Remove existing usage of 'vdb_upload' from the Ingestor class, Ingest pipeline, and example docs

This would be a multi-phase method, responsible for first ingesting and propagating embeddings to minio, verifying all tasks completed successfully, and then performing the bulk upload tasks for the embedding data.

Describe the feature, and optionally a solution or implementation and any alternatives

Method API

class Ingestor():
....

    def vdb_bulk_upload(collection_ids: str | Dict[str, str], milvus_uri: str, sparse: bool, dense_dim: bool):
       ....

Usage:

collection_ids = {
 "text": ....,
 "image": ....,
 "table": ....,
 "chart": ....,
}

# Single vdb collection:
ingestor_single_collection = ( Ingestor(client=client)
 .files(....)
 .extract(...)
 .embed(...)
 .store_embedding(...)
 .vdb_bulk_upload("my_collection_name", localhost:123, True, False)
)

results_single = ingestor_single_collection.ingest()

# Multiple vdb collections:
ingestor_multi_collection = (Ingestor(client=client)
 .files(....)
 .extract(...)
 .embed(...)
 .store_embedding(...)
 .vdb_bulk_upload(collection_ids, localhost:123, True, False)
)

results_multi = ingestor_multi_collection.ingest()

Additional context

Implementation notes:

• Collection ID's may need to support simple regex, for example '(table|chart)' : 123981273129'
• Ingestor.ingest() should probably still return the first round results after finishing the bulk upload, so that someone could enable bulk uploading, but still get usable data back. If its not desirable by default, then it should be a flag.