More OpenAI configurability

[KTibow]

Check for existing issues

• Completed

Describe the feature

tl;dr I'd like a way to remove models and possibly use separate APIs and reorder models

For context, if you reconfigure your OpenAI base URL, the models get really pushed down. You also can't have two different OpenAI base URLs.

Continue to the mockup section for some of my ideas.

Environment

Zed: v0.165.4 (Zed)
OS: Linux Wayland fedora 40
Memory: 15.2 GiB
Architecture: x86_64
GPU: Intel(R) Iris(R) Plus Graphics (ICL GT1.5) || Intel open-source Mesa driver || Mesa 24.1.7

If applicable, add mockups / screenshots to help present your vision of the feature

So the main thing I'd like is a way to disable models that are unneeded.
There's a few ways this could work:

• A configuration setting like

  "use_default_models": false,

  (this applies in the case that you configured OpenAI so the "gpt-4o", "gpt-3.5-turbo", etc models are broken)
• A configuration setting like

  "model_order": ["Claude 3.5 Sonnet", "Llama", "GPT-4o"]

  This has the bonus of allowing reordering of models.
• A section in the "Configuration" GUI
  This would look like the existing model chooser but with x or - buttons to remove models.
  This could have the bonus of allowing reordering of models if dragging was enabled.

The second thing I'd like is to allow multiple URLs. Both ways of doing this require Zed to store separate auth tokens for each API URL:

  "language_models": {
    "openai": {
      "available_models": [
        {
          "name": "llama-3.3-70b-specdec",
          "display_name": "Llama",
          "max_tokens": 6000,
          "api_url": "https://api.groq.com/openai/v1/"
        },
        {
          "name": "gpt-4o",
          "display_name": "GPT-4o",
          "max_tokens": 16000,
          "api_url": "https://api.openai.com/v1/"
        }
      ]
    }
  }
/* would have you configure every model you want to use */

or it could look like

  "language_models": {
    "openai": {
      "version": 2,
      "api_urls": ["https://api.openai.com/v1/", "https://api.groq.com/openai/v1/"]
    }
  }
/* would have Zed fetch all available models using the `models` endpoint, and possibly allow removing more from there with the earlier methods */

[notpeter]

Being able to configure multiple endpoints/apikeys for the same underlying "provider" is a duplicate of:

• Allow add custom AI assistant provider #15732

As the number of models/providers grows the single monolithic list of models is definitely becoming harder to use. There have been some mocks of alternatives done internally (multiple dropdowns; folding by provider; pinning specific models, hiding models, etc) but we haven't landed on anything specific yet. I'm going go ahead and migrate this to a discussion in case folks have further ideas.

Thanks for reporting.

[itsaphel]

I'd also like to see this, and I'd be happy to contribute if manpower is needed. Sort of just spitballing, in case these thoughts are helpful:

It does seem it's common in projects to have "OpenAI" configuration and then override the base URL (cline does something similar too), but this feels a bit wrong to me. For one, it means you can only use a single OpenAI-like provider at a time. The design proposed in the original comment doesn't have this limitation, but it feels wrong marking groq under a key of supported_models.openai.

To me, it feels like there are certain API 'formats'¹; this includes the authentication mechanism, and perhaps an endpoint to list the models supported. Then there are actual providers which follow some API format, and provide some models. Each model needs to define certain properties needed for Zed (things like max_tokens, and whether it supports streaming).

Under such an abstraction, I think one could easily define (an unlimited number of) additional models and providers through the config file? That frees the need of PRs like #23425 to update models, or hacks like #15732 (comment) to support additional providers.

Then, the config might be something like:

"lm_providers": {
  "crosshatch": {
    "api_format": "openai",
    "api_url": "https://api.crosshatch.app/v1"
  },
  "groq": {
    "api_format": "openai",
    "api_url": "https://api.groq.com/openai/v1"
  }
},
"language_models": {
  "lmsys-coding": {
    "display_name": "LMSys Coding",
    "provider": "crosshatch",
    "max_tokens": 10000
  },
  "llama-3.3-70b-specdec": {
    "display_name": "Llama",
    "provider": "groq",
    "max_tokens": 20000
  }
},

It'd need some more thinking on how to nicely expose this to a GUI, and how to merge user-defined configuration with defaults.

Footnotes

1. With some providers following the OpenAI API. I'm not sure how 'completely' they follow it, since it isn't standardised, but maybe the compliance is good enough for common endpoints. ↩

[notpeter]

Preserving backwards compatibility would be ideal.
I think we could do something like this:

  "language_models": {
    "open_ai": {
      "available_models": [
        {
          "name": "o1-preview-2024-09-12",
          "max_tokens": 128000,
          "display_name": "O1 Preview Old"
        }
      ],
      "version": "1"
    },
    "grok": {
      "api": "open_ai",
      "api_url": "https://api.x.ai/v1",
      "available_models": [
        {
          "name": "grok-beta",
          "display_name": "X.ai Grok (Beta)",
          "max_tokens": 131072
        }
      ],
      "version": "1"
    },
    "ollama": {
      "api_url": "http://localhost:11434",
      "available_models": [
        {
          "name": "deepseek-r1-64K",
          "display_name": "Deepseek 64K Local",
          "max_tokens": 65536
        }
      ]
    },
    "ollama_remote": {
      "api": "ollama",
      "api_url": "http://localhost:11434"
    }
  }

Where api is optional and matches the keyname if absent. There would likely need to be some fiddling with only showing the hardcoded models OpenAI in certain circumstance or perhaps a setting like show_default_models or hide_default_models.

[itsaphel]

I like that format - seems sensible to me.