Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: update for zeno-hub #193

Merged
merged 6 commits into from
Jan 5, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .flake8

This file was deleted.

90 changes: 0 additions & 90 deletions .github/workflows/ci.yml

This file was deleted.

26 changes: 0 additions & 26 deletions .github/workflows/release.yml

This file was deleted.

7 changes: 0 additions & 7 deletions .markdownlint-cli2.yaml

This file was deleted.

50 changes: 0 additions & 50 deletions .pre-commit-config.yaml

This file was deleted.

20 changes: 0 additions & 20 deletions CITATION.cff

This file was deleted.

2 changes: 1 addition & 1 deletion CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1 +1 @@
* @cabreraalex @neubig
* @cabreraalex @neubig @Sparkier
44 changes: 3 additions & 41 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,9 @@
If you're reading this, you're probably interested in contributing to
`zeno-build`. Thank you for your interest!

## Types of Contributions

We welcome:

- **New examples**, which you can create by navigating to the [examples/]
directory and adding a new sub-directory implementing a new example
(see the examples directory for details).
- **New reports**, which you can create by modifying the configuration and/or
modeling files under [examples/] and sending us the a link of the resulting
report.
- **New core functionality**, making `zeno-build` more feature-rich or easier to
use.
We always welcome **new examples**, which you can create by navigating to the
[examples/] directory and adding a new sub-directory implementing a new example
(see the examples directory for details).

## Contribution Guide

Expand All @@ -27,32 +18,3 @@ If you want to make a contribution you can:
We'd recommend reaching out to us first so we can help out any way we can,
but if you're confident in your contribution, you can go ahead and send a PR
directly.

## Developer Installation

If you're a developer, you should install pre-commit hooks before developing.

```bash
pip install .[dev]
pre-commit install
```

These will do things like ensuring code formatting, linting, and type checking.

You'll also want to run tests to make sure your code is working by running the
following:

```bash
pytest
```

## Making a Release

If you are an admin of the repository, you can make a new release of the
library.

We are using the [hatchling](https://github.com/pypa/hatch) build system, which
makes it easy to make new library releases. In order to do so, just create a new
version tag on github (it has to be a valid [semantic
version](https://semver.org/)) and the CI will automatically build and publish
the new version to PyPI.
105 changes: 17 additions & 88 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,104 +1,33 @@
# Zeno Build

[![PyPI version](https://badge.fury.io/py/zeno-build.svg)](https://badge.fury.io/py/zeno-build)
![Github Actions CI tests](https://github.com/zeno-ml/zeno-build/actions/workflows/ci.yml/badge.svg)
[![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://lbesson.mit-license.org/)
[![Discord](https://img.shields.io/discord/1086004954872950834)](https://discord.gg/km62pDKAkE)
[![Open Zeno](https://img.shields.io/badge/%20-Open_Zeno-612593.svg?labelColor=white&logo=data:image/svg%2bxml;base64,PHN2ZyB3aWR0aD0iMzMiIGhlaWdodD0iMzMiIHZpZXdCb3g9IjAgMCAzMyAzMyIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZD0iTTMyIDE1Ljc4NDJMMTYuNDg2MiAxNS43ODQyTDE2LjQ4NjIgMC4yNzA0MDFMMjQuMzAyIDguMDg2MTdMMzIgMTUuNzg0MloiIGZpbGw9IiM2MTI1OTMiLz4KPHBhdGggZD0iTTE1Ljc5MTcgMTUuODMxMUw4LjAzNDc5IDguMDc0MjJMMTUuNzkxNyAwLjMxNzMyOEwxNS43OTE3IDE1LjgzMTFaIiBmaWxsPSIjNjEyNTkzIiBmaWxsLW9wYWNpdHk9IjAuOCIvPgo8cGF0aCBkPSJNMTQuODY1NSAxNS44MzExTDcuNTk0ODUgMTUuODMxMUw3LjU5NDg1IDguNTYwNDJMMTQuODY1NSAxNS44MzExWiIgZmlsbD0iIzYxMjU5MyIgZmlsbC1vcGFjaXR5PSIwLjYiLz4KPHBhdGggZD0iTTYuMTEyOSAxNS44MzExTDMuMjQxNyAxNS44MzExTDMuMjQxNyAxMi44NjcyTDYuMTEyOSAxNS44MzExWiIgZmlsbD0iIzZBMUI5QSIgZmlsbC1vcGFjaXR5PSIwLjQiLz4KPHBhdGggZD0iTTIuNzMyMjggMTUuODMxTDEuNTE1NSAxNC42MTQzTDIuNzQyNzEgMTMuMzg3TDIuNzMyMjggMTUuODMxWiIgZmlsbD0iIzZBMUI5QSIgZmlsbC1vcGFjaXR5PSIwLjMiLz4KPHBhdGggZD0iTTIuMDM3NiAxNS43ODQyTDEuMTU3NzEgMTUuNzg0MkwxLjE1NzcxIDE0Ljk1MDZMMi4wMzc2IDE1Ljc4NDJaIiBmaWxsPSIjNkExQjlBIiBmaWxsLW9wYWNpdHk9IjAuMiIvPgo8cGF0aCBkPSJNMC44MzM1NjggMTUuNzg0MUwwLjUwOTM5OSAxNS40NkwwLjgzMzU2NyAxNS4xMzU4TDAuODMzNTY4IDE1Ljc4NDFaIiBmaWxsPSIjNjEyNTkzIiBmaWxsLW9wYWNpdHk9IjAuMSIvPgo8cGF0aCBkPSJNMC4xMDYxODcgMTUuNzk0NEwwLjMwMTAyNSAxNS41OTk2TDAuNDk1ODYzIDE1Ljc5NDRIMC4xMDYxODdaIiBmaWxsPSIjNjEyNTkzIiBmaWxsLW9wYWNpdHk9IjAuMSIvPgo8cGF0aCBkPSJNNi45NTIxMyAxNS44MjQ4TDMuNjQwOTkgMTIuNTEzN0w2Ljk2OTYzIDkuMTg1MDNMNi45NTIxMyAxNS44MjQ4WiIgZmlsbD0iIzYxMjU5MyIgZmlsbC1vcGFjaXR5PSIwLjUiLz4KPHBhdGggZD0iTTAuMjk0MjM1IDE2LjQ3OTVMMTUuODA4IDE2LjQ3OTVMMTUuODA4IDMxLjk5MzNMNy45OTIyMyAyNC4xNzc1TDAuMjk0MjM1IDE2LjQ3OTVaIiBmaWxsPSIjNjEyNTkzIi8+CjxwYXRoIGQ9Ik0xNi40OTU2IDE3LjI0MzZMMjMuODUwNyAyNC41ODVMMTYuNDk1NiAzMS45NEwxNi40OTU2IDE3LjI0MzZaIiBmaWxsPSIjNjEyNTkzIiBmaWxsLW9wYWNpdHk9IjAuOCIvPgo8cGF0aCBkPSJNMTYuNTMyNiAxNi40Nzk1TDI0LjQ1MTUgMTYuNDc5NUwyNC40NTE1IDI0LjAyOEwxNi41MzI2IDE2LjQ3OTVaIiBmaWxsPSIjNjEyNTkzIiBmaWxsLW9wYWNpdHk9IjAuNiIvPgo8cGF0aCBkPSJNMjYuMTgxMyAxNi40MzI2TDI5LjA1MjUgMTYuNDMyNkwyOS4wNTI1IDE5LjM5NjRMMjYuMTgxMyAxNi40MzI2WiIgZmlsbD0iIzZBMUI5QSIgZmlsbC1vcGFjaXR5PSIwLjQiLz4KPHBhdGggZD0iTTI5LjU2MTkgMTYuNDMyNkwzMC43Nzg3IDE3LjY0OTRMMjkuNTUxNSAxOC44NzY2TDI5LjU2MTkgMTYuNDMyNloiIGZpbGw9IiM2QTFCOUEiIGZpbGwtb3BhY2l0eT0iMC4zIi8+CjxwYXRoIGQ9Ik0zMC4yNTY2IDE2LjQ3OTVMMzEuMTM2NSAxNi40Nzk1TDMxLjEzNjUgMTcuMzEzMUwzMC4yNTY2IDE2LjQ3OTVaIiBmaWxsPSIjNkExQjlBIiBmaWxsLW9wYWNpdHk9IjAuMiIvPgo8cGF0aCBkPSJNMzEuNDYwNiAxNi40Nzk1TDMxLjc4NDggMTYuODAzN0wzMS40NjA2IDE3LjEyNzlMMzEuNDYwNiAxNi40Nzk1WiIgZmlsbD0iIzYxMjU5MyIgZmlsbC1vcGFjaXR5PSIwLjEiLz4KPHBhdGggZD0iTTMyLjE4OCAxNi40NjkyTDMxLjk5MzIgMTYuNjY0MUwzMS43OTgzIDE2LjQ2OTJIMzIuMTg4WiIgZmlsbD0iIzYxMjU5MyIgZmlsbC1vcGFjaXR5PSIwLjEiLz4KPHBhdGggZD0iTTI1LjM0MjEgMTYuNDM4OUwyOC42NTMyIDE5Ljc1TDI1LjMyNDYgMjMuMDc4NkwyNS4zNDIxIDE2LjQzODlaIiBmaWxsPSIjNjEyNTkzIiBmaWxsLW9wYWNpdHk9IjAuNSIvPgo8L3N2Zz4K)](https://hub.zenoml.com)

**Zeno Build** is a tool for developers who want to quickly build, compare, and
iterate on applications using large language models.
**Zeno Build** is a collection of examples using **Zeno** to evaluate generative
AI models. Use it to get started with common evaluation setups.

![Zeno Build Overview](/docs/images/zeno-build-overview.png)

It provides:

- **Simple examples** of code to build LLM-based apps. The examples are
architecture agnostic, we don't care if you are using
[OpenAI](https://openai.com/),
[LangChain](https://github.com/hwchase17/langchain), or [Hugging
Face](https://huggingface.co).
- **Experiment management** and **hyperparameter optimization** code, so you can
quickly kick off experiments using a bunch of different settings and compare
the results.
- **Evaluation** of LLM outputs, so you can check if your outputs are correct,
fluent, factual, interesting, or "good" by whatever definition of good you
prefer! Use these insights to compare models and iteratively improve your
application with model, data, or prompt engineering.
The examples in this repository are architecture agnostic, we don't care if you
are using
[OpenAI](https://openai.com/),
[LangChain](https://github.com/hwchase17/langchain), or [Hugging
Face](https://huggingface.co).

Sound interesting? Read on!

## Getting Started

To get started with `zeno-build`, install the package from PyPI:

```bash
pip install zeno-build
```

Next, _start building_! Browse to the [docs](docs/) directory to get a
primer or to the [examples/](examples/) directory, where we
have a bunch of examples of how you can use `zeno-build` for different tasks,
such as [chatbots](examples/chatbot/),
[text summarization](examples/summarization/), or [text
classification](examples/text_classification/).

Each of the examples include code for running experiments and evaluating the
results. `zeno-build` will produce a comprehensive report with the
[Zeno](https://zenoml.com/) AI evaluation platform.

## Interactive Demos/Reports

Using Zeno Build, we have generated reports and online browsing demos of
state-of-the-art systems for different popular generative AI tasks.
Check out our pre-made reports below:
## Examples

- **Chatbots** ([Report](examples/chatbot/report/),
[Browser](https://zeno-ml-chatbot-report.hf.space/)):
A report comparing different methods
for creating chatbots, including API-based models such as ChatGPT and Cohere,
with open-source models such as Vicuna, Alpaca, and MPT.
- **Translation** ([Report](examples/analysis_gpt_mt/report/),
[Browser](https://zeno-ml-translation-report.hf.space/)):
A report comparing GPT-based methods, Microsoft Translator, and the best system
from the Conference on Machine Translation.
Each of the examples in this repository is specifically designed to be
self-contained and easy to modify. To get started with
zeno, we suggest that you find the closest example to what you're trying
Sparkier marked this conversation as resolved.
Show resolved Hide resolved
to do, copy the example to the new directory, and start hacking!

## Building Your Own Apps (and Contributing Back)

Each of the examples in the [examples/](examples/) directory is specifically designed
to be self-contained and easy to modify. To get started building your own apps,
we suggest that you first click into the directory and read the general README,
find the closest example to what you're trying to do, copy the example to the
new directory, and start hacking!
## Contributing Back

If you build something cool, **we'd love for you to contribute it back**. We
welcome pull requests of both new examples, new reports for existing examples,
and new functionality for the core `zeno_build` library. If this is of interest
to you, please click through to our [contributing doc](contributing.md) doc to
learn more.

## CITATION

To cite [GPT-MT report](examples/analysis_gpt_mt/README.md), please use the
following BibTeX/APA entry.

### BibTeX

```bibtex
@misc{Neubig_Zeno_GPT_Machine_2023,
author = {
Neubig, Graham and
He, Zhiwei
},
title = {{Zeno GPT Machine Translation Report}},
year = {2023}
}
```

### APA

```bibtex
Neubig, G., & He, Z. (2023). Zeno GPT Machine Translation Report
```
welcome pull requests of both new examples and new functionality for the core
`zeno_build` library. If this is of interest to you, please click through to
our [contributing doc](contributing.md) doc to learn more.

## Get in Touch

Expand Down
Loading