AiIntermediate

Docker Model Runner Cheatsheet

Pull and run open-source LLMs locally with an OpenAI-compatible endpoint.

What it is

Docker Model Runner lets you run large language models (LLMs) locally inside Docker containers with a standard OpenAI-compatible REST API, no GPU required.

Installation

Available in Docker Desktop 4.40+. Enable Model Runner under Features in Development. Verify with docker model version.

Quick start

docker model pull ai/llama3.2

Download a model from the Docker Model catalog.

docker model run ai/llama3.2

Start an interactive chat session.

docker model list

Show downloaded models.

docker model serve ai/llama3.2

Expose the model as an OpenAI-compatible API on port 8080.

Common commands

Task	Command	Description
Pull model	`docker model pull <model>`	Download a model from the Docker Hub catalog.
Run interactively	`docker model run <model>`	Start an interactive chat in the terminal.
Serve as API	`docker model serve <model>`	Launch an OpenAI-compatible HTTP server.
List models	`docker model list`	Show locally downloaded models.
Remove model	`docker model rm <model>`	Delete a model from local storage.
Model info	`docker model inspect <model>`	Show model metadata and capabilities.

Real-world examples

Pull and serve Llama 3.2 as an OpenAI API

docker model pull ai/llama3.2
docker model serve ai/llama3.2
# API available at http://localhost:8080/v1

Use with OpenAI Python SDK

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")
response = client.chat.completions.create(
  model="ai/llama3.2",
  messages=[{"role":"user","content":"Hello!"}]
)
print(response.choices[0].message.content)

Best practices

Use docker model serve so any OpenAI-compatible tool can connect to your local model.
Check model size before pulling — larger models need more RAM.
Use quantized (Q4/Q8) variants for faster inference on CPU.

Related cheatsheets

Last reviewed: 2026-06-15