5 Reasons to Switch from Ollama to Docker Model Runner

In recent months, the LLM deployment landscape has been evolving rapidly, with users experiencing frustration with some existing solutions. A Reddit thread titled “How to move on from Ollama?” highlights growing discontent with Ollama’s performance and reliability issues. As Docker enters this space with Model Runner, it’s worth examining why many users are making the switch.

The Growing Frustration with Ollama

One community member mentioned, “I moved over to this after trouble with ollama and qwen3 and my problems immediately went away. I like the priority vs even distribution of work option for the GPU offload. Works well and gained some speed with my mixed GPU server.”

Recent GitHub issues for Ollama have documented multiple problems, including performance degradation, compatibility issues, and various bugs that affect user experience. With these challenges in mind, Docker Model Runner emerges as a compelling alternative. Let’s explore why.

1. Seamless Integration with Docker Ecosystem

Docker Model Runner introduces a first-class citizen approach with the new docker model CLI, treating AI models similar to how Docker treats containers, images, and volumes. This integration means you can leverage your existing Docker knowledge and workflows without learning a completely new tool.

# Check available models
docker model ls

# Pull a model from Docker Hub
docker model pull ai/llama3.2:1B-Q8_0

# Run a model for a quick response
docker model run ai/llama3.2:1B-Q8_0 "What is containerization?"

# Interactive chat mode
docker model run ai/llama3.2:1B-Q8_0

Unlike Ollama, which requires its own separate CLI and processes, Docker Model Runner works within your familiar Docker environment, making it a more cohesive solution for teams already using Docker for their development workflow.

2. OCI Artifact Support for Models

Docker Model Runner stores models as OCI artifacts in Docker Hub or any other Docker Registry. This standardized format provides several advantages:

• No unnecessary compression of model weights (which are largely uncompressible)
• Faster deployments
• Lower disk requirements
• Support for private registries within your organization

This approach is particularly beneficial compared to Ollama’s proprietary model storage format, as it leverages existing container ecosystem standards and allows for better integration with CI/CD pipelines.

3. Multiple Connection Methods

Docker Model Runner provides flexible options for connecting to your models:

# Option 1: From within containers via internal DNS
# http://model-runner.docker.internal/

# Option 2: From the host via Docker Socket
curl --unix-socket /var/run/docker.sock \
    localhost/exp/vDD4.40/engines/llama.cpp/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"ai/llama3.2:1B-Q8_0","messages":[{"role":"user","content":"Hello"}]}'

# Option 3: From the host via TCP (when enabled)
# Direct connection on port 12434 or through a helper container:
docker run -d --name model-runner-proxy -p 8080:80 \
  alpine/socat tcp-listen:80,fork,reuseaddr tcp:model-runner.docker.internal:80

These multiple connection methods provide greater flexibility than Ollama’s more limited REST API approach, allowing for more diverse integration patterns in your applications.

4. OpenAI API Compatibility

Docker Model Runner implements OpenAI-compatible endpoints, making it easier to migrate existing applications that use the OpenAI API:

# Example API endpoint structure
GET /engines/{backend}/v1/models
POST /engines/{backend}/v1/chat/completions
POST /engines/{backend}/v1/completions
POST /engines/{backend}/v1/embeddings

This compatibility means you can more easily switch between cloud-based OpenAI services and local inference when needed, providing greater flexibility for development, testing, and deployment scenarios.

5. Optimized for Apple Silicon with More Platforms Coming Soon

While currently optimized for Apple Silicon Macs (Windows support already arrived in April 2025), Docker Model Runner makes excellent use of Metal API for GPU acceleration on M-series chips:

# Pull a model
docker model pull ai/llama3.2:1B-Q8_0

# Start a chat session and observe GPU performance in Activity Monitor
docker model run ai/llama3.2:1B-Q8_0

The direct access to GPU acceleration without containerization overhead results in efficient inference performance, often matching or exceeding Ollama’s performance on the same hardware.

A Quick Example: Building a GenAI App with Docker Model Runner

Let’s look at how easy it is to build an application with Docker Model Runner:

# Step 1: Pull the model
docker model pull ai/llama3.2:1B-Q8_0

# Step 2: Clone the demo repository
git clone https://github.com/dockersamples/genai-app-demo
cd genai-app-demo

# Step 3: Configure environment variables
# Set in backend.env:
# BASE_URL=http://model-runner.docker.internal/engines/llama.cpp/v1/
# MODEL=ai/llama3.2:1B-Q8_0
# API_KEY=${API_KEY:-ollama}

# Step 4: Start the application
docker compose up -d

# Step 5: Access the application at http://localhost:3000

The application runs a Go backend that connects to Model Runner, providing a React frontend for user interaction. You can monitor real-time GPU usage in Activity Monitor to see the efficient performance.

Addressing Common Ollama Pain Points

It’s worth noting that Docker Model Runner appears to directly address some of the specific issues that Ollama users have reported:

1. Stability: The Docker infrastructure provides a more reliable foundation compared to the issues reported in Ollama’s GitHub repository.
2. Performance: By running directly on the host, Docker Model Runner’s inference server can access Apple’s Metal API for direct GPU acceleration without containerization overhead. Users can observe GPU usage in Activity Monitor when queries are being processed.
3. Compatibility: Docker Model Runner works with a variety of models hosted on Docker Hub, including ai/gemma3, ai/llama3.2, ai/qwq, ai/mistral-nemo, ai/mistral, ai/phi4, ai/qwen2.5, and ai/deepseek-r1-distill-llama.
4. User-friendly: Docker Desktop makes enabling Model Runner simple, either through CLI or the Docker Dashboard, with straightforward options for configuration.

Conclusion

While Ollama has been a popular choice for running LLMs locally, Docker Model Runner offers several compelling advantages, especially for teams already working within the Docker ecosystem. Its seamless Docker integration, OCI artifact support, flexible connection methods, OpenAI API compatibility, and optimized performance make it a strong contender for your local LLM needs.

The growing frustration in the Ollama community, as evidenced by recent GitHub issues and Reddit discussions, suggests this might be the perfect time to explore alternatives. Docker Model Runner provides a promising solution that leverages Docker’s robust infrastructure while addressing many of the pain points Ollama users have experienced.

As Docker continues to expand Model Runner’s capabilities with additional inference engines and platform support, it’s positioned to become a powerful tool in the generative AI development workflow.

Ready to make the switch? Docker Model Runner is available in Docker Desktop 4.40+ with more features on the way.