> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nerdit.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Serve models and connect AI services

> Bind applications to an Ollama or vLLM model, or to an external OpenAI-compatible API.

An application declares AI bindings in `nerdit.toml`. Nerdit resolves them at launch and injects endpoint, key, and model variables. External bindings send application requests to the configured provider; they do not keep those requests on your machine.

## Serve a local model

Ollama supports CPU or GPU serving. Choose a model that fits available RAM or VRAM and allow time and disk space for its initial download:

```bash theme={null}
nerdit serve llama3.2:1b --gpus 0
nerdit models list
```

On a machine with a working NVIDIA container stack, a GPU example is:

```bash theme={null}
nerdit serve llama3.1:8b --gpus 1
```

Read the created service name and readiness from the output; inspect problems with `nerdit diagnose SERVICE_NAME` and `nerdit logs SERVICE_NAME`. The model must be running and ready before the application can use it.

```toml nerdit.toml theme={null}
[ai.default]
provider = "ollama"
model = "llama3.2:1b"
```

Use the model reference you served. The `ollama` binding provider selects a locally managed model; the serving backend itself can be Ollama or vLLM.

## Use vLLM

vLLM requires a GPU. Models use Hugging Face references:

```bash theme={null}
nerdit serve Qwen/Qwen2.5-0.5B-Instruct --backend vllm --gpus 1 \
  --max-model-len 4096 --gpu-memory-utilization 0.8
```

These bounds control context length and the fraction of VRAM the engine may claim. Reduce them when diagnostics show insufficient memory. A compatible GPU, driver, and model architecture are still required; the command is not a hardware compatibility guarantee.

Gated repositories require you to accept the model's license and provide an authorized Hugging Face token through the shared `HF_TOKEN` secret. Shared secrets require admin access. Keep it out of the model reference and command arguments.

## Connect an external API

```toml nerdit.toml theme={null}
[ai.default]
provider = "api"
base_url = "https://api.example.com/v1"
model = "YOUR_MODEL_ID"
api_key = "${secrets.AI_API_KEY}"
```

Replace the endpoint and model with values supported by your provider, then set `AI_API_KEY` using the [secret store](/engine/security#application-secrets). This path requires neither a local model container nor a GPU. Nerdit injects the endpoint; it does not add a separate local gateway in front of the external provider.

## Read bindings in the application

| Binding                 | Injected variables                                                       |
| ----------------------- | ------------------------------------------------------------------------ |
| Every named binding     | `NERDIT_AI_<NAME>_URL`, `NERDIT_AI_<NAME>_KEY`, `NERDIT_AI_<NAME>_MODEL` |
| Binding named `default` | Also `OPENAI_BASE_URL`, `OPENAI_API_KEY`, `OPENAI_MODEL`                 |

Names become uppercase in environment variables. For a Python application with the OpenAI SDK installed:

```python theme={null}
import os
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model=os.environ["OPENAI_MODEL"],
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(response.choices[0].message.content)
```

Model capabilities and API extensions differ across providers. Use a request supported by your chosen model. Changing a binding requires a restart or redeploy to update the application's environment.

## Troubleshoot and remove models

If a binding fails, check the exact model reference, readiness, container logs, and bridge connectivity. Container clients cannot reach a host service just because it listens on host loopback; `[models].bridge_host` controls the shared container-to-resource reachability configuration.

Manage the returned model service with `nerdit services stop`, `restart`, or `rm`. Removing a model used by running applications is refused unless forced. Update dependent bindings first. Model weights are outside control-plane backups and need separate preservation if re-downloading is unacceptable.