Managing models

Your organization’s models decide what your fleet can serve and what your applications can request. Admins manage them on the console’s Models page. Members can see the models and their usage, but can’t change them.

Add models

New organizations start with no models. To add one:

Open Models and click Add models. If the organization has no models yet, click Browse model catalog.
Search the catalog, for example llama, qwen coder or gemma. You can filter by publisher and sort by Most downloaded, Most liked or Recently updated.
Click Add on each model you want, then click Done.

The catalog lists open text models from Hugging Face in GGUF format, quantized to Q4_K_M, from the ggml-org, bartowski and unsloth publishers.

Added models start enabled. Devices with enough memory start serving them shortly afterwards.

Model names

Each model has a name in the form publisher/repository:Q4_K_M, for example bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M. This is the value you pass as model in API requests. You can find it on the Models page, or list it through the API.

Choosing models

Memory decides where a model can run. Each model card shows Memory, the memory a device needs to serve it. A model can only run on devices with enough memory. See How devices get models.
Fewer models can mean more throughput. Each device serves one model at a time. If you enable many models, your devices are split between them. With a few models, each one has more devices serving it.

Enable or disable a model

Use the toggle on a model’s card, or on its detail page.

When you disable a model:

API requests for it are rejected immediately with model_disabled.
Devices stop serving it. Within about a minute, devices that were serving it switch to another enabled model that fits them. Devices that don’t fit any enabled model unload it and sit idle.
It stays in your organization’s list, with its usage history, so you can turn it back on at any time.

When you enable it again, devices start loading it again as needed.

Remove a model

Click the trash icon on a model’s card, or Delete on its detail page, and confirm.

Removing a model has the same effect on devices and requests as disabling it, and it also takes the model out of your organization’s list. Usage history and statistics are kept, and come back if you add the model again.

Model details and usage

Click View details on a model card to see:

Devices serving: how many devices are serving it right now.
Tokens generated: total output across your fleet.
Memory required: what a device needs to serve it.
Devices over time and Tokens over time charts, for the last day, 7 days or month.