Getting Started with Ollama: Run LLMs Locally with Python, Docker & REST API

Table of Contents

Ollama is the fastest-growing tool for running large language models (LLMs) locally. Whether you’re using Python, Docker, or the REST API, this hands-on guide covers the most popular Ollama use cases with real, runnable code.

1. Install Ollama

macOS / Linux

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

Windows

# Download the installer from:
# https://ollama.com/download/windows

# Or via winget
winget install Ollama.Ollama

2. Pull and Run Your First Model

# Pull Llama 3.2 (3B – runs on 8GB RAM)
ollama pull llama3.2

# Pull Mistral 7B
ollama pull mistral

# Pull Google's Gemma 3
ollama pull gemma3

# Pull Qwen3 (code-focused)
ollama pull qwen3

# List downloaded models
ollama list

# Run interactive chat
ollama run llama3.2

3. Ollama REST API

Ollama exposes a local REST API on port 11434. Use it directly with curl or any HTTP client.

Generate a Response (non-streaming)

curl http://localhost:11434/api/generate   -H "Content-Type: application/json"   -d '{
    "model": "llama3.2",
    "prompt": "Explain Docker in 3 bullet points",
    "stream": false
  }'

Streaming Response

curl http://localhost:11434/api/generate   -H "Content-Type: application/json"   -d '{
    "model": "mistral",
    "prompt": "Write a Python function to reverse a linked list",
    "stream": true
  }'

Chat Completions (OpenAI-compatible)

curl http://localhost:11434/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "llama3.2",
    "messages": [
      {"role": "system", "content": "You are a helpful DevOps assistant."},
      {"role": "user", "content": "What is a Kubernetes pod?"}
    ]
  }'

List Available Models via API

curl http://localhost:11434/api/tags

Generate Embeddings

curl http://localhost:11434/api/embeddings   -H "Content-Type: application/json"   -d '{
    "model": "nomic-embed-text",
    "prompt": "Ollama makes running LLMs easy"
  }'

4. Ollama Python Library

# Install the library
pip install ollama

Basic Chat

import ollama

response = ollama.chat(
    model='llama3.2',
    messages=[
        {'role': 'user', 'content': 'What is Kubernetes?'}
    ]
)
print(response['message']['content'])

Streaming Chat Response

import ollama

stream = ollama.chat(
    model='mistral',
    messages=[{'role': 'user', 'content': 'Write a Dockerfile for a Node.js app'}],
    stream=True
)

for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)

Generate Text

import ollama

response = ollama.generate(
    model='llama3.2',
    prompt='Explain CI/CD pipeline in simple terms'
)
print(response['response'])

Generate Embeddings with Python

import ollama

result = ollama.embeddings(
    model='nomic-embed-text',
    prompt='Docker container orchestration'
)
print(result['embedding'][:5])  # Print first 5 dimensions

Using Tool Calling (Function Calling)

import ollama
import json

def get_pod_status(namespace: str, pod_name: str) -> str:
    # Simulate Kubernetes pod status lookup
    return json.dumps({"status": "Running", "restarts": 0, "age": "2d"})

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_pod_status",
            "description": "Get the status of a Kubernetes pod",
            "parameters": {
                "type": "object",
                "properties": {
                    "namespace": {"type": "string"},
                    "pod_name": {"type": "string"}
                },
                "required": ["namespace", "pod_name"]
            }
        }
    }
]

response = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'Check status of pod nginx-abc in default namespace'}],
    tools=tools
)

# Process tool calls
if response['message'].get('tool_calls'):
    for tool_call in response['message']['tool_calls']:
        fn_name = tool_call['function']['name']
        fn_args = tool_call['function']['arguments']
        if fn_name == 'get_pod_status':
            result = get_pod_status(**fn_args)
            print(f"Tool result: {result}")

Async Python Client

import asyncio
import ollama

async def chat_async():
    client = ollama.AsyncClient()
    response = await client.chat(
        model='llama3.2',
        messages=[{'role': 'user', 'content': 'What is Helm in Kubernetes?'}]
    )
    print(response['message']['content'])

asyncio.run(chat_async())

5. Run Ollama in Docker

CPU Only

docker run -d   -v ollama:/root/.ollama   -p 11434:11434   --name ollama   ollama/ollama

With NVIDIA GPU

docker run -d   --gpus=all   -v ollama:/root/.ollama   -p 11434:11434   --name ollama   ollama/ollama

Docker Compose: Ollama + Open WebUI


services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    depends_on:
      - ollama
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open_webui_data:/app/backend/data
    restart: unless-stopped

volumes:
  ollama_data:
  open_webui_data:

docker compose up -d

# Pull a model after containers are up
docker exec ollama ollama pull llama3.2

6. Create a Custom Model with Modelfile

# Create a Modelfile
cat <<EOF > Modelfile
FROM llama3.2

# Set system prompt
SYSTEM """
You are a DevOps expert specializing in Docker, Kubernetes, and CI/CD pipelines.
Answer concisely with code examples whenever possible.
"""

# Set parameters
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
EOF

# Build the custom model
ollama create devops-expert -f Modelfile

# Run it
ollama run devops-expert

Modelfile with a GGUF base model

cat <<EOF > Modelfile
FROM ./my-custom-model.gguf

SYSTEM "You are a helpful coding assistant."
PARAMETER temperature 0.5
PARAMETER stop "<|end|>"
EOF

ollama create my-model -f Modelfile
ollama run my-model

7. Ollama OpenAI-Compatible API with Python

pip install openai

from openai import OpenAI

# Point to local Ollama server
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Required but not used
)

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "You are a Kubernetes expert."},
        {"role": "user", "content": "How do I scale a deployment to 5 replicas?"}
    ]
)
print(response.choices[0].message.content)

Streaming with OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

stream = client.chat.completions.create(
    model="mistral",
    messages=[{"role": "user", "content": "Write a GitHub Actions workflow for Python testing"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end='', flush=True)

8. Build a RAG Pipeline with Ollama + LangChain

pip install langchain langchain-ollama langchain-community chromadb

from langchain_ollama import OllamaLLM, OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain.schema import Document

# Sample documents (replace with your own)
docs = [
    Document(page_content="Docker is a platform for building, shipping, and running containers."),
    Document(page_content="Kubernetes orchestrates containerized workloads at scale."),
    Document(page_content="Ollama lets you run LLMs locally on your own hardware."),
]

# Split documents
splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)
splits = splitter.split_documents(docs)

# Create vector store with Ollama embeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(documents=splits, embedding=embeddings)

# Create retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})

# Set up RAG chain
llm = OllamaLLM(model="llama3.2")
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True
)

# Query
result = qa_chain.invoke({"query": "What is Ollama used for?"})
print(result['result'])

9. Multi-Modal: Vision with LLaVA

# Pull the LLaVA vision model
ollama pull llava

# Run interactively with an image
ollama run llava "Describe this image" --image /path/to/image.jpg

Vision via Python API

import ollama
import base64

# Load and encode image
with open("architecture-diagram.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode()

response = ollama.chat(
    model='llava',
    messages=[
        {
            'role': 'user',
            'content': 'What does this architecture diagram show?',
            'images': [image_data]
        }
    ]
)
print(response['message']['content'])

10. Ollama on Kubernetes

# ollama-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
            - containerPort: 11434
          volumeMounts:
            - name: ollama-storage
              mountPath: /root/.ollama
          resources:
            requests:
              memory: "4Gi"
              cpu: "2"
            limits:
              memory: "8Gi"
              cpu: "4"
      volumes:
        - name: ollama-storage
          persistentVolumeClaim:
            claimName: ollama-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: ollama-service
  namespace: ai
spec:
  selector:
    app: ollama
  ports:
    - protocol: TCP
      port: 11434
      targetPort: 11434
  type: ClusterIP
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-pvc
  namespace: ai
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

kubectl create namespace ai
kubectl apply -f ollama-deployment.yaml

# Pull a model into the running pod
kubectl exec -n ai deployment/ollama -- ollama pull llama3.2

# Port-forward to access locally
kubectl port-forward -n ai svc/ollama-service 11434:11434

Ollama with GPU on Kubernetes (NVIDIA)

# Add GPU resource request to the container spec
resources:
  limits:
    nvidia.com/gpu: 1
  requests:
    nvidia.com/gpu: 1

# Make sure NVIDIA device plugin is installed
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.0/nvidia-device-plugin.yml

11. Useful Ollama CLI Commands

# List running models
ollama ps

# Show model details
ollama show llama3.2

# Remove a model
ollama rm mistral

# Copy/rename a model
ollama cp llama3.2 my-llama

# Pull a specific version
ollama pull llama3.2:1b

# Push to Ollama registry (requires account)
ollama push username/my-model

# Start Ollama server manually
ollama serve

# Run with a system prompt
ollama run llama3.2 "You are a Linux sysadmin. Help me debug this error: connection refused"

12. Environment Variables

# Change default model storage location
export OLLAMA_MODELS=/mnt/large-disk/ollama/models

# Bind to all interfaces (for remote access)
export OLLAMA_HOST=0.0.0.0:11434

# Enable verbose logging
export OLLAMA_DEBUG=1

# Set number of parallel requests
export OLLAMA_NUM_PARALLEL=4

# Keep models loaded in memory (seconds, 0 = always, -1 = never unload)
export OLLAMA_KEEP_ALIVE=300

# Start server with custom settings
OLLAMA_HOST=0.0.0.0:11434 OLLAMA_MODELS=/data/models ollama serve

13. Build a Simple ChatBot with Ollama + FastAPI

pip install fastapi uvicorn ollama

# app.py
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
import ollama

app = FastAPI(title="Ollama Chatbot API")

class ChatRequest(BaseModel):
    message: str
    model: str = "llama3.2"
    system: str = "You are a helpful assistant."

@app.post("/chat")
async def chat(req: ChatRequest):
    response = ollama.chat(
        model=req.model,
        messages=[
            {"role": "system", "content": req.system},
            {"role": "user", "content": req.message}
        ]
    )
    return {"response": response['message']['content']}

@app.post("/chat/stream")
async def chat_stream(req: ChatRequest):
    def generate():
        stream = ollama.chat(
            model=req.model,
            messages=[
                {"role": "system", "content": req.system},
                {"role": "user", "content": req.message}
            ],
            stream=True
        )
        for chunk in stream:
            yield chunk['message']['content']

    return StreamingResponse(generate(), media_type="text/plain")

@app.get("/models")
async def list_models():
    models = ollama.list()
    return {"models": [m['name'] for m in models['models']]}

# Run: uvicorn app:app --reload

# Test the API
curl -X POST http://localhost:8000/chat   -H "Content-Type: application/json"   -d '{"message": "What is a Docker volume?", "model": "llama3.2"}'

# List available models
curl http://localhost:8000/models

Quick Reference: Popular Models

# General Purpose
ollama pull llama3.2        # Meta Llama 3.2 3B (2GB) — fast, runs on 4GB RAM
ollama pull llama3.3        # Meta Llama 3.3 70B (43GB) — best quality
ollama pull mistral         # Mistral 7B (4GB) — great all-rounder
ollama pull gemma3          # Google Gemma 3 (2GB) — efficient
ollama pull phi4            # Microsoft Phi-4 (9GB) — strong reasoning

# Code Models
ollama pull qwen3           # Alibaba Qwen3 — excellent for coding
ollama pull deepseek-coder  # DeepSeek Coder 6.7B
ollama pull codellama       # Meta Code Llama

# Embeddings
ollama pull nomic-embed-text     # Best for RAG, 274MB
ollama pull mxbai-embed-large    # High quality embeddings

# Vision
ollama pull llava           # LLaVA multimodal (image + text)
ollama pull moondream       # Lightweight vision model

Ollama makes local LLM deployment accessible to every developer. From quick prototyping with the CLI to production-grade Kubernetes deployments, the tools shown in this tutorial cover the most commonly searched and used Ollama patterns. Start with ollama pull llama3.2 and build from there.