How devices get models

You don’t choose which device serves which model: Nodea assigns them for you. This page explains how, so you can predict what your fleet will do.

Memory budget

Each device can use up to about 80% of its memory for a model. On devices where the GPU shares the system memory, a device with 16 GB can serve a model that needs up to roughly 12–13 GB.

A device can only serve models whose Memory requirement (shown on the Models page) fits in that budget.

Assignment

• When a device connects, it gets an enabled model that fits its memory budget, preferring the largest one that fits.
• While devices are connected, Nodea keeps rebalancing. It looks at which models are being requested and moves devices toward those models, so busy models get more devices serving them.
• When you disable or remove a model, devices serving it move to another enabled model that fits. Devices that don’t fit any enabled model unload it.
• A device downloads each model once and keeps it cached. Switching back to a model it already has only takes a reload.

Changes are spread out over time, so your whole fleet doesn’t reload at once.

A device that shows “Unassigned”

In the console’s Fleet page, a device marked Unassigned is connected but isn’t serving a model. The usual reasons are:

• No enabled model fits its memory. Add or enable a smaller model.
• The organization has no enabled models. See Add models.

What this means for API requests

A request for a model is sent to a device that’s serving that model.

• Requests are spread across the devices serving the model, least busy first. Each device generates one response at a time, so extra requests wait their turn. A request that gets no output within 180 seconds fails with timeout.
• If no device is serving the model at all, the request waits up to 120 seconds for one to become available, for example while a device loads it. After that it fails with model_unavailable, which you can retry.

To handle more requests at once for a model, connect more devices with enough memory for it, or enable fewer models so more devices serve each one.