Instructions for BYOM - Bring Your Own Model - LLM

[NikosDi]

Hello.
I downloaded AI Playground v1.22.1 for desktop GPUs which has a built-in "LLM picker" but unfortunately the dGPU version does not provide Mistral model- as mentioned in the release notes (it has a download link for gemma 7B - a gated LLM with 47.7 GB download requirements but not for Mistral)

I tried to find some instructions of where to download and install Mistral or any other HuggingFace tranformers 4.39 PyTorch LLM compatible model.

So, how do I use Mistral with AI Playground v1.22.1 Arc dGPU version ?

Thank you

[Nuullll]

Sorry, the built-in "LLM picker" should have provided Mistral. We'll fix that and release a patched version.
c2a8603

[NikosDi]

Thanks for that.

But it doesn't reply generally, how could someone install a different model than those proposed by 'LLM picker" ?

Where can I find instructions?

[Nuullll]

You can try to download model files from huggingface into a dedicated folder under <InstallDir>\resources\service\models\llm\checkpoints. Then you could probably be able to pick your own model in the ANSWER tab -- however, it may not work because we didn't validate them. Feel free to file issues of course.

[NikosDi]

@Nuullll
It seems that in order to download a model e.g Mistral you have to sign-up to Huggingface, create a token and find a suitable app or run a script to download in reasonable time (using multithreaded download) the model.

And it could probably not working (not referring to Mistral obviously), eventually using AI Playground.

I think I'll wait for your next version with Mistral as a choice of "LLM picker".

Thank you