Skip to content

Commit

Permalink
[WIP] Add evaluation (#7)
Browse files Browse the repository at this point in the history
* Ignore custom venvs

* Move order of import

* Add weblinx.processing.outputs module for processing model outputs

* Update setup to include more necessary libraries

* specify correct path

* Add evaluation and output processing scripts
  • Loading branch information
xhluca authored Feb 23, 2024
1 parent ef28cc0 commit 98a0220
Show file tree
Hide file tree
Showing 11 changed files with 1,307 additions and 7 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ celerybeat.pid
.venv
env/
venv/
venv-*/
ENV/
env.bak/
venv.bak/
Expand Down
22 changes: 21 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,27 @@ For the rest of the instructions, please take a look at the [modeling README](./

### Evaluation

Coming soon!
To install packages necessary for evaluation, run:

```bash
pip install weblinx[eval]
```

You can now access the evaluation module by importing in Python:

```python
import weblinx.eval
```

Use `weblinx.eval.metrics` for evaluation metrics, `weblinx.eval.__init__` for useful evaluation-related functions. You may also find it useful to take a look at `weblinx.processing.outputs` to get an idea of how to use the outputs of the model for evaluation.

To run the automatic evaluation, you can use the following command:

```bash
python -m weblinx.eval --help
```

Note: We are still working on the code for `weblinx.eval` and `weblinx.processing.outputs`. If you have any questions or would like to contribute docs, please feel free to open an issue or a pull request.

### Citations

Expand Down
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dev": ["black", "wheel"],
"video": ["opencv-python-headless", "numpy", "Pillow"],
"processing": ["lxml"],
"eval": ["sacrebleu", "numpy", "pandas", "tqdm"],
}
# Dynamically create the 'all' extra by combining all other extras
extras_require["all"] = sum(extras_require.values(), [])
Expand Down
2 changes: 1 addition & 1 deletion weblinx/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1333,7 +1333,7 @@ def list_demonstrations(
"""
path = Path(base_dir)
demos = [
Demonstration(demo.name, base_dir)
Demonstration(demo.name, base_dir=base_dir)
for demo in sorted(path.iterdir())
if demo.is_dir()
]
Expand Down
Loading

0 comments on commit 98a0220

Please sign in to comment.