[Feature] Long context benchmark enhancement

[joshuayao]

HELMET supports OPEA LLM endpoint.

[minmin-intel]

I was trying to reproduce HELMET results. I implemented a model class to interact with vllm-gaudi: https://github.com/minmin-intel/GenAIEval/blob/test-helmet/evals/evaluation/HELMET/model_utils.py#L204

So far, I have tested kilt_nq dataset in RAG category. I saw accuracy numbers are close to the results published in paper when input length = 8k, but 64k results have big differences when using vllm endpoint. However, using transformer pipeline gets similar results to the paper at both 8k and 64k length. Debugging WIP.

[joshuayao]

I was trying to reproduce HELMET results. I implemented a model class to interact with vllm-gaudi: https://github.com/minmin-intel/GenAIEval/blob/test-helmet/evals/evaluation/HELMET/model_utils.py#L204

So far, I have tested kilt_nq dataset in RAG category. I saw accuracy numbers are close to the results published in paper when input length = 8k, but 64k results have big differences when using vllm endpoint. However, using transformer pipeline gets similar results to the paper at both 8k and 64k length. Debugging WIP.

@minmin-intel , have you made progress on that?