Finalizing MMTEB

[KennethEnevoldsen]

This issue is to get an overview of what needs to be done before MMTEB can be finalized.

1. Adding the last remaining datasets, notably:
   • MIRACL reranking #641 fix: Add MIRACL reranking #830
   • GPT4-o generated queries for 14 languages #718
   • Adding MIRACL Retrieval #642 fix: Add MIRACL retrieval #833
2. Speeding up the benchmark
   • I believe we are only missing: Convert tasks to ClusteringFast. #660
   • see also Paper segment: Speeding up retrieval #836
   • see also Paper segment: Speeding up MTEB (English) + co2 impact #838
   • see also Paper segment: Comparing scores of Clustering to ClusteringFast #835
3. Running models #705 (partly depends on 1, 2 as well as Model which we need to implement #879)
4. Figuring out Score Aggregation for Multilingual benchmark (and in general) #752 (partly depends on 3)
5. Deciding on meaningful benchmark subsets (depends on 3)

• see Paper segment: Task selection #837

7. Paper Writing: An overview issue #896 (depends on 3, 4 and 5) (see also Paper writing #595)

• see Paper segment: Score aggregation #839

8. Updating leaderboard to new format Leaderboard changes for massive multilinguality #674 (depends on 3-6)

Is there anything else that is needed?

[vaibhavad]

Construction of MMTEB-Lite? It will be a faster version of MMTEB. Two approaches that come to mind for implementing this are -

1. Reducing the size document set of some retrieval benchmarks.
2. Reducing the number of tasks

[dokato]

Hey @KennethEnevoldsen I'd like to merge also this dataset in #773. 3 reasons: a) we don't seem to have brazilian dialect represeted, b) multilabel task doesn't have large language coverage c) I had it prepared for long time, but multilabel task got only merged last week when I was away. We only need to address a problem with stratification of the splits there.

[KennethEnevoldsen]

@vaibhavad yes, def. we need to construct the benchmarks and ideally think about downsampling some of the larger retrieval datasets. A solution might be to implement a downsample function for retrieval tasks.

Thanks @dokato - let us get it merged in as well. Looks to be in a reasonable state

[Ruqyai]

Hey @KennethEnevoldsen I read the list and I think I can help in Running models #705

[jordiclive]

@KennethEnevoldsen Is there anything meaningful new contributors can help with?

[KennethEnevoldsen]

Hi @jordiclive! I believe there are multiple avenues to take, but any of the outlines paper segment I believe is meaning (see the updated post above), implementing model (see e.g. #845, will finish it up either Monday or in the weekend), or starting work on 8)

[bwanglzu]

quick question: is there a script to select & run all MMTEB tasks? I'm a bit unclear about the difference between current development progress and how MMTEB is different from the current MTEB (in different languages).

Best

Bo

[Samoed]

@bwanglzu You can select benchmark like this:

import mteb
mteb.get_benchmark("MTEB(eng, classic)") # or get_benchmarks

Full list of bencharks here

[KennethEnevoldsen]

Will close this issue as MMTEB has been submitted, moving the public preprint release over to #1405