Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(evaluate): broken data formatting #249

Open
micpst opened this issue Dec 17, 2024 · 0 comments
Open

bug(evaluate): broken data formatting #249

micpst opened this issue Dec 17, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@micpst
Copy link
Collaborator

micpst commented Dec 17, 2024

What happened?

I noticed that there are two formats for the evaluation data - one flat and the other with an options section. When I try to run the evaluation with the neptune callback, I get an exception. We should choose one layout and fix the code for neptune.

To reproduce run example from examples/evaluation/document-search/evaluate.py and flip neptune.run flag in examples/evaluation/document-search/config/retrieval.yaml

How can we reproduce it?

No response

Relevant log output

Traceback (most recent call last):
  File "/Users/michal/repos/ragbits/examples/evaluation/document-search/evaluate.py", line 63, in main
    asyncio.run(bench(config))
  File "/Users/michal/.local/share/uv/python/cpython-3.10.15-macos-aarch64-none/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/michal/.local/share/uv/python/cpython-3.10.15-macos-aarch64-none/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/michal/repos/ragbits/examples/evaluation/document-search/evaluate.py", line 33, in bench
    run = setup_neptune(config)
  File "/Users/michal/repos/ragbits/packages/ragbits-evaluate/src/ragbits/evaluate/utils.py", line 110, in setup_neptune
    config.data.name,
omegaconf.errors.ConfigAttributeError: Key 'name' is not in struct
    full_key: data.name
    object_type=dict
@micpst micpst added the bug Something isn't working label Dec 17, 2024
@micpst micpst changed the title bug(evaluate): broken data input formatting bug(evaluate): broken data formatting Dec 17, 2024
@micpst micpst added this to ragbits Dec 17, 2024
@micpst micpst moved this to Backlog in ragbits Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Backlog
Development

No branches or pull requests

1 participant