Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(document-search): automatic configuration selection based on evaluation #136

Closed
mhordynski opened this issue Oct 22, 2024 · 0 comments · Fixed by #177
Closed

feat(document-search): automatic configuration selection based on evaluation #136

mhordynski opened this issue Oct 22, 2024 · 0 comments · Fixed by #177
Assignees
@kdziedzic68
Labels
document search Changes to the document search package evals Adding new evaluation pipelines or improving existing ones feature New feature or request
Milestone
Ragbits 0.4

Comments

@mhordynski
Copy link
Member

mhordynski commented Oct 22, 2024 •
edited
Loading

Feature description

There should be a way to automatically choose most parameters like chunking, reranking, rephrasing in the DocumentSearch instance based on the evaluation.

Desired steps required to get customized DocumentSearch instance:

  1. Provide validation dataset for the DocumentSearch.
  2. Configure static fields such as VectorStore, and dynamic lists for example embedding models to test.
  3. Run evaluation
  4. Output should be a ranked list of config files sorted by the evaluation score.

Maybe it should be CLI command?

Motivation

This will make us closer to rapidly develop e2e applications with prod-like performance.

Additional context

https://github.com/KruxAI/ragbuilder has some sort of optimization method
@mhordynski mhordynski added the feature New feature or request label Oct 22, 2024
@mhordynski mhordynski self-assigned this Oct 22, 2024
@mhordynski mhordynski moved this to Backlog in ragbits Oct 22, 2024
@mhordynski mhordynski moved this from Backlog to Ready in ragbits Oct 23, 2024
@mhordynski mhordynski added this to the Ragbits 0.4 milestone Oct 23, 2024
@micpst micpst assigned kdziedzic68 and unassigned mhordynski Oct 25, 2024
@micpst micpst added document search Changes to the document search package evals Adding new evaluation pipelines or improving existing ones labels Oct 25, 2024
@micpst micpst linked a pull request Oct 31, 2024 that will close this issue
WIP: feat(document-search): automatic configuration selection based on evaluation #167
Closed
@kdziedzic68 kdziedzic68 removed a link to a pull request Nov 7, 2024
WIP: feat(document-search): automatic configuration selection based on evaluation #167
Closed
@kdziedzic68 kdziedzic68 linked a pull request Nov 7, 2024 that will close this issue
feat(document-search): automatic configuration selection based on evaluation #177
Merged
@github-project-automation github-project-automation bot moved this from In review to Done in ragbits Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kdziedzic68 kdziedzic68

Labels
document search Changes to the document search package evals Adding new evaluation pipelines or improving existing ones feature New feature or request
Projects
ragbits
Status: Done
Milestone
Ragbits 0.4
Development

Successfully merging a pull request may close this issue.

feat(document-search): automatic configuration selection based on evaluation deepsense-ai/ragbits
3 participants
@mhordynski @micpst @kdziedzic68