Serve models and connect AI services

An application declares AI bindings in nerdit.toml. Nerdit resolves them at launch and injects endpoint, key, and model variables. External bindings send application requests to the configured provider; they do not keep those requests on your machine.

Serve a local model

Ollama supports CPU or GPU serving. Choose a model that fits available RAM or VRAM and allow time and disk space for its initial download:

nerdit serve llama3.2:1b --gpus 0
nerdit models list

On a machine with a working NVIDIA container stack, a GPU example is:

nerdit serve llama3.1:8b --gpus 1

Read the created service name and readiness from the output; inspect problems with nerdit diagnose SERVICE_NAME and nerdit logs SERVICE_NAME. The model must be running and ready before the application can use it.

nerdit.toml

[ai.default]
provider = "ollama"
model = "llama3.2:1b"

Use the model reference you served. The ollama binding provider selects a locally managed model; the serving backend itself can be Ollama or vLLM.

Use vLLM

vLLM requires a GPU. Models use Hugging Face references:

nerdit serve Qwen/Qwen2.5-0.5B-Instruct --backend vllm --gpus 1 \
  --max-model-len 4096 --gpu-memory-utilization 0.8

These bounds control context length and the fraction of VRAM the engine may claim. Reduce them when diagnostics show insufficient memory. A compatible GPU, driver, and model architecture are still required; the command is not a hardware compatibility guarantee. Gated repositories require you to accept the model’s license and provide an authorized Hugging Face token through the shared HF_TOKEN secret. Shared secrets require admin access. Keep it out of the model reference and command arguments.

Connect an external API

nerdit.toml

[ai.default]
provider = "api"
base_url = "https://api.example.com/v1"
model = "YOUR_MODEL_ID"
api_key = "${secrets.AI_API_KEY}"

Replace the endpoint and model with values supported by your provider, then set AI_API_KEY using the secret store. This path requires neither a local model container nor a GPU. Nerdit injects the endpoint; it does not add a separate local gateway in front of the external provider.

Read bindings in the application

Binding	Injected variables
Every named binding	`NERDIT_AI_<NAME>_URL`, `NERDIT_AI_<NAME>_KEY`, `NERDIT_AI_<NAME>_MODEL`
Binding named `default`	Also `OPENAI_BASE_URL`, `OPENAI_API_KEY`, `OPENAI_MODEL`

Names become uppercase in environment variables. For a Python application with the OpenAI SDK installed:

import os
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model=os.environ["OPENAI_MODEL"],
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
)
print(response.choices[0].message.content)

Model capabilities and API extensions differ across providers. Use a request supported by your chosen model. Changing a binding requires a restart or redeploy to update the application’s environment.

Troubleshoot and remove models

If a binding fails, check the exact model reference, readiness, container logs, and bridge connectivity. Container clients cannot reach a host service just because it listens on host loopback; [models].bridge_host controls the shared container-to-resource reachability configuration. Manage the returned model service with nerdit services stop, restart, or rm. Removing a model used by running applications is refused unless forced. Update dependent bindings first. Model weights are outside control-plane backups and need separate preservation if re-downloading is unacceptable.

Start here

Build and operate

Access and interfaces

HTTP endpoints

Get help

Serve models and connect AI services

Serve a local model

Use vLLM

Connect an external API

Read bindings in the application

Troubleshoot and remove models

​Serve a local model

​Use vLLM

​Connect an external API

​Read bindings in the application

​Troubleshoot and remove models

Serve a local model

Use vLLM

Connect an external API

Read bindings in the application

Troubleshoot and remove models