Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ProjectDev to the Agent section #8

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

minhngh
Copy link

@minhngh minhngh commented Aug 27, 2024

Dear Mr. mlabonne,

I would like to request the inclusion of the ProjectDev dataset in the Agent section. This dataset is used to evaluate the performance of multi-agent systems, such as ChatDev, MetaGPT, and AgileCoder, in solving software development challenges.

Thank you very much.

@mlabonne
Copy link
Owner

Hey, is it this dataset https://huggingface.co/datasets/Fsoft-AIC/ProjectDev? If that's the case, sorry but this is too small to include in this repo. I'm mostly interested in large datasets (>10k, preferably >100k).

@minhngh
Copy link
Author

minhngh commented Aug 30, 2024

Yes, that’s correct. The dataset is pretty small because it requires manual evaluation to assess the capabilities of multi-agent systems like ChatDev and AgileCoder. Tasks within the dataset involve generating software based on user input, with user interfaces that feature complex interactions, such as chains of mouse and keyboard events, and generated programs can vary significantly in terms of code element names and implementations. Thus, it's hard to scale the dataset beyond 10,000 samples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants