OAI File Storage Metadata management

[dahifi]

The OAI Platform files UX is pretty lacking. https://platform.openai.com/storage/files

Right now it shows the following information in the UX:

We would like a better interface to manage files between our local file systems, git repos, Chainlit front end, &c, and make sure we're not loading the same file in multiple places. A lot of times we get random file names. And most of the other endpoints such as vector stores and annotation file citations refer to the OAI file ID. We need a way to manage all this better.

The vector storage has a slightly better interface:

It at least shows us the file names (most of the time) and now shows what assistants and threads a datastore is attached to.

I'm not sure I want to rebuild the OAI UX for all this, but we do need to do checks for file uploads to do file hashes, as well as some sort of summary or descriptive details about a file and why it was added. These can be used for rollups and the like.

We might also use some of this metadata for storing the original download URL or source, this is crucial when we start building ingestion pipelines for youtube videos and other datasources that aren't natively supported by retrieval.

[dahifi]

Requirements for OAI File Metadata Management

Description

Enhance the metadata management of files uploaded to the OAI Platform to improve UX and prevent duplication. This includes adding checks for file uploads using file hashes and storing comprehensive metadata such as the original download URL.

Acceptance Criteria

1. Metadata Enhancements

   • Store additional metadata for each file, including:
     • Original download URL
     • File hash (to check for duplicates)
     • Upload timestamp
     • File size
     • User who uploaded the file
   • Ensure metadata is retrievable via the Chainlit front end and API.

2. Duplicate File Handling

   • Implement file hash checks during the upload process.
   • Prevent upload of duplicate files based on hash comparison.
   • Provide a clear message to the user if a duplicate file is detected, suggesting actions (e.g., use the existing file or rename).

3. User Interface Improvements

   • Enhance the Chainlit front end to display the additional metadata.
   • Allow users to filter and search files based on metadata attributes (e.g., file name, upload date, uploader).

4. Backend Adjustments

   • Update the backend to handle and store the new metadata fields.
   • Ensure existing files are retroactively updated with the new metadata where possible.

5. Documentation and Testing

   • Update the documentation to reflect the changes in metadata management.
   • Write and execute unit tests to ensure the correct functionality of the new features.
   • Perform user acceptance testing (UAT) to verify the improved UX and duplicate handling.

[dahifi]

Inital plan for enhancing metadata management, handling duplicates, and improving the backend:

High-Level Plan

1. Metadata Enhancements

   • Create a FileMetadata class to store additional metadata attributes.
   • Implement methods to calculate file hashes and retrieve file metadata.

2. Duplicate File Handling

   • Calculate file hashes during the upload process.
   • Check for existing files with the same hash and prevent duplicates.

3. User Interface Improvements

   • Ensure API endpoints return the necessary metadata for the front-end to consume.
   • Update the front-end to display and handle new metadata fields.

4. Backend Adjustments

   • Update the database schema to include new metadata fields.
   • Implement scripts to update existing files with the new metadata.

5. Documentation and Testing

   • Update API documentation to reflect changes.
   • Write and execute unit tests to ensure proper functionality.