Deploying an LLM API-Backed Microservice with Docker and CI/CD

Most teams integrating language model capabilities into their stack eventually hit the same wall: the initial prototype works, but moving it into a shared, reliable, observable service is a different problem entirely. Dropping an OpenAI call into a Flask app is straightforward. Getting that call into a container, behind proper retries and rate limiting, with a CI pipeline that actually runs security scans before pushing to a registry, that’s where the real work lives.

This post walks through a complete production-grade setup: a Python FastAPI microservice that wraps an external LLM API, containerized with Docker, and deployed through a GitHub Actions CI/CD pipeline. Everything here is based on patterns that hold up under real production conditions, not just demo traffic.

Why a Dedicated Microservice for LLM Capabilities?

Before jumping to code, it’s worth being clear about why teams build a separate service for this rather than calling an LLM provider directly from their application.

The short answer: control. When LLM calls live inside a dedicated service, you get a single place to enforce rate limiting, centralize API key rotation, add request logging, implement retry logic, and swap providers without touching downstream clients. If you’re calling an LLM from five different application services directly, rotating a compromised API key means touching five codebases.

There’s also the testability argument. A microservice with a well-defined REST interface is far easier to mock in integration tests than inline LLM calls scattered across business logic.

The tradeoff is latency, you’re adding a network hop, and operational overhead. For most teams, the tradeoff is worth it once you’re past the prototype stage.

Architecture Overview

Here’s how the pieces fit together:

The LLM microservice is the only component that holds the API key. Clients authenticate to the microservice using internal tokens, they never see the upstream provider credentials. The CI/CD pipeline builds and pushes a new image on every merge to main, with the deployment stage gated behind the test and scan steps.

Building the LLM API-Backed Service

Let’s build the service. We’ll use FastAPI, it gives us async support, automatic OpenAPI docs, and request validation through Pydantic with very little boilerplate.

Project structure:

llm-service/
├── app/
│   ├── __init__.py
│   ├── main.py
│   ├── models.py
│   ├── llm_client.py
│   └── config.py
├── tests/
│   └── test_main.py
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .github/
    └── workflows/
        └── ci.yml

app/config.py — Environment variable handling in one place:

from pydantic_settings import BaseSettings
from functools import lru_cache

class Settings(BaseSettings):
    llm_api_key: str
    llm_api_base_url: str = "https://api.aimlapi.com/v1"
    llm_model: str = "gpt-4o-mini"
    max_tokens: int = 1024
    request_timeout: int = 30
    service_api_key: str
    log_level: str = "INFO"

    class Config:
        env_file = ".env"

@lru_cache()
def get_settings():
    return Settings()

Using pydantic-settings here means startup fails immediately if a required variable is missing, rather than failing silently at request time. The lru_cache avoids re-reading the environment on every request.

app/models.py — Request and response schemas:

from pydantic import BaseModel, Field
from typing import Optional

class CompletionRequest(BaseModel):
    prompt: str = Field(..., min_length=1, max_length=8000)
    system_prompt: Optional[str] = Field(None, max_length=2000)
    temperature: float = Field(0.7, ge=0.0, le=2.0)
    max_tokens: Optional[int] = Field(None, ge=1, le=4096)

class CompletionResponse(BaseModel):
    content: str
    model: str
    usage: dict
    request_id: str

Integrating an External LLM API

app/llm_client.py — The actual API abstraction:

import httpx
import logging
import uuid
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

from .config import get_settings
from .models import CompletionRequest, CompletionResponse

logger = logging.getLogger(__name__)

class LLMClient:
    def __init__(self):
        self.settings = get_settings()
        self.client = httpx.AsyncClient(
            base_url=self.settings.llm_api_base_url,
            headers={"Authorization": "Bearer " + self.settings.llm_api_key},
            timeout=self.settings.request_timeout,
        )

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=2, max=10),
        retry=retry_if_exception_type((httpx.TimeoutException, httpx.ConnectError)),
    )
    async def complete(self, request, request_id):
        messages = []
        if request.system_prompt:
            messages.append({"role": "system", "content": request.system_prompt})
        messages.append({"role": "user", "content": request.prompt})

        payload = {
            "model": self.settings.llm_model,
            "messages": messages,
            "temperature": request.temperature,
            "max_tokens": request.max_tokens or self.settings.max_tokens,
        }

        response = await self.client.post("/chat/completions", json=payload)
        response.raise_for_status()

        data = response.json()
        logger.info("llm_response", extra={
            "request_id": request_id,
            "usage": data.get("usage", {}),
        })

        return CompletionResponse(
            content=data["choices"][0]["message"]["content"],
            model=data["model"],
            usage=data.get("usage", {}),
            request_id=request_id,
        )

    async def close(self):
        await self.client.aclose()

The retry decorator from tenacity handles transient network failures — timeouts and connection errors — with exponential backoff. It does not retry on 4xx responses, which is intentional. A 400 from the LLM provider means our request was malformed; retrying won’t help.

app/main.py — The FastAPI application:

from fastapi import FastAPI, HTTPException, Depends, Request
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
import logging
import uuid

from .config import get_settings
from .models import CompletionRequest, CompletionResponse
from .llm_client import LLMClient

logging.basicConfig(level=get_settings().log_level)
logger = logging.getLogger(__name__)

app = FastAPI(title="LLM API Service", version="1.0.0")
security = HTTPBearer()
llm_client = LLMClient()

def verify_api_key(credentials: HTTPAuthorizationCredentials = Depends(security)):
    if credentials.credentials != get_settings().service_api_key:
        raise HTTPException(status_code=401, detail="Invalid API key")
    return credentials.credentials

@app.get("/health")
async def health():
    return {"status": "ok"}

@app.post("/v1/complete", response_model=CompletionResponse)
async def complete(
    request: Request,
    body: CompletionRequest,
    _: str = Depends(verify_api_key),
):
    request_id = str(uuid.uuid4())
    try:
        response = await llm_client.complete(body, request_id)
        return response
    except Exception as e:
        logger.error("completion_failed", extra={"error": str(e)})
        raise HTTPException(status_code=502, detail="LLM API request failed")

A few deliberate choices here: the lifespan context manager handles client setup and teardown cleanly, the /health endpoint is unauthenticated (load balancers need it), and all upstream errors are caught and returned as 502 rather than leaking internal details to callers.

Integrating an External LLM API

The LLMClient in the code above is deliberately thin. It implements the OpenAI-compatible chat completions format, which is supported by a wide range of providers. This means swapping providers is mostly an environment variable change, update LLM_API_BASE_URL and LLM_MODEL, redeploy, done.

Teams that want to avoid managing GPU infrastructure entirely can consume models through providers such as AI/ML API, which offers a unified endpoint across many open and proprietary models. This lets the microservice layer focus entirely on business logic, observability, and rate limiting, without any concern for the underlying serving infrastructure.

The pattern matters more than the specific provider. What you want is a client abstraction that:

1. Holds credentials in one place.
2. Enforces timeouts on every call.
3. Retries only on transient errors, not on validation failures.
4. Logs enough structured data to reconstruct cost and latency per request.

If you later need to add a fallback provider, that logic belongs in LLMClient, not scattered across application code.

Containerizing the Service with Docker

Production Dockerfiles for Python services should be multi-stage. The build stage installs everything; the runtime stage copies only what’s needed to run the application. The difference in image size is meaningful — often 400–600 MB versus under 200 MB.

# Build stage
FROM python:3.12-slim as builder

WORKDIR /build
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

# Runtime stage
FROM python:3.12-slim

RUN addgroup --gid 1001 appgroup && \
    adduser --uid 1001 --gid 1001 --no-create-home --disabled-password appuser

WORKDIR /app
COPY --from=builder /root/.local /home/appuser/.local
COPY app/ ./app/

RUN chown -R appuser:appgroup /app

USER appuser

ENV PATH=/home/appuser/.local/bin:$PATH
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Non-root execution is the most consistently skipped security practice in Docker images. If a vulnerability in your application or its dependencies is exploited, running as appuser instead of root significantly limits blast radius. There’s no good reason not to do this for a web service.

The –no-create-home flag keeps the filesystem clean, and copying only the installed packages (not the entire pip toolchain) from the builder stage keeps the runtime image lean.

Local Development Workflow

Build and run manually:

docker-compose.yml for a more complete local environment: 

version: "3.9"
services:
  llm-service:
    build: .
    ports:
      - "8000:8000"
    environment:
      - LLM_API_KEY=${LLM_API_KEY}
      - SERVICE_API_KEY=${SERVICE_API_KEY}
      - LLM_MODEL=gpt-4o-mini
      - LOG_LEVEL=DEBUG
    volumes:
      - ./app:/app/app

Keep a .env.example in the repo with dummy values. Never commit a real .env file. Add it to .gitignore on day one, not after the incident.

CI/CD Automation with GitHub Actions

Here’s a complete pipeline that runs tests, lints, scans for secrets and vulnerabilities, builds the image, pushes to GHCR, and deploys:

yaml 

# .github/workflows/ci.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}/llm-service

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install pytest pytest-asyncio httpx ruff

      - name: Lint with ruff
        run: ruff check app/ tests/

      - name: Run tests
        env:
          LLM_API_KEY: test-key
          SERVICE_API_KEY: test-service-key
          LLM_API_BASE_URL: https://api.example.com
          LLM_MODEL: test-model
        run: pytest tests/ -v --tb=short

  security-scan:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4

      - name: Scan for secrets with Gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Run Trivy vulnerability scan on filesystem
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: fs
          scan-ref: .
          severity: CRITICAL,HIGH
          exit-code: 1

  build-and-push:
    runs-on: ubuntu-latest
    needs: [test, security-scan]
    if: github.ref == 'refs/heads/main'
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract Docker metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=sha-
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Run Trivy vulnerability scan on image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
          severity: CRITICAL
          exit-code: 1

  deploy:
    runs-on: ubuntu-latest
    needs: build-and-push
    if: github.ref == 'refs/heads/main'
    environment: production

    steps:
      - name: Deploy to production
        run: |
          echo "Deploy step: pull new image and restart service"
          # kubectl set image deployment/llm-service llm-service=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:sha-${{ github.sha }}
          # or: ssh deploy@host 'docker pull ... && docker compose up -d'

A few things worth pointing out. The security-scan job runs Gitleaks to catch secrets committed to the repository, this catches the most common mistake teams make when first building these pipelines. Trivy runs twice: once on the filesystem to catch dependency vulnerabilities before building, and once on the final image to catch anything introduced by the build itself. The deploy job uses a GitHub Environment named production, which lets you add required reviewers or protection rules before anything reaches your live environment.

The build cache with type=gha is also worth keeping. Without it, every CI run rebuilds from scratch. The first cached run shaves two to three minutes off typical Python service builds.

Production Deployment Considerations

Secrets management. Never pass API keys through environment variables set directly in a container orchestrator’s deployment YAML. Use a secrets manager — AWS Secrets Manager, GCP Secret Manager, Vault, or Kubernetes Secrets mounted as files. The distinction matters: environment variables can leak into logs, crash dumps, and debug output. Files mounted to a non-world-readable path are significantly harder to accidentally expose.

Rate limiting. Your microservice should implement rate limiting before calls reach the upstream LLM API. The upstream provider will rate-limit you too, but by that point you’ve already consumed threads and potentially generated partial costs. Use a middleware like slowapi for FastAPI:

Observability. Structured JSON logs are a prerequisite for anything running at scale. Every request to /v1/complete should log: request ID, caller identity, model used, token counts, latency, and whether the request succeeded. Token counts are particularly important — they’re your cost signal. Without them, you’ll notice an unexpectedly large bill before you understand why.

For metrics, expose a /metrics endpoint using prometheus-fastapi-instrumentator. Track llm_request_duration_seconds and llm_tokens_total as histograms and counters respectively. Alert on p99 latency and error rate, not just averages.

Horizontal scaling. LLM microservices are stateless if you’ve built them correctly — no local state, no session pinning. This means they scale horizontally without coordination. Set a Kubernetes HPA targeting 70% CPU or, better, a custom metric based on queue depth if you’re using async processing. Start with two replicas minimum so you’re never fully down during a rolling update.

Cost optimization. Cache responses for identical prompts using Redis with a TTL appropriate to your freshness requirements. Even a 10–15% cache hit rate on common prompts makes a measurable difference at scale. Use a hash of (model, system_prompt, prompt, temperature, max_tokens) as the cache key; temperature is included because different temperature values produce different results even for the same prompt.

Common Pitfalls

API key leakage. The most damaging mistake, and it happens consistently. Keys end up in environment variable dumps, application logs, Docker inspect output, or — worst — committed to git. Use Gitleaks in pre-commit hooks and in CI. Rotate keys on any doubt. Store them in a secrets manager, not in your deployment manifests.

Missing timeouts. The httpx.AsyncClient default is no timeout. An LLM provider experiencing degraded performance can hold a connection open for minutes, exhausting your async worker pool and bringing down the service. Set explicit connect and read timeouts, and make them shorter than your load balancer’s idle timeout.

No retry differentiation. Retrying a 429 (rate limited) immediately makes the problem worse. Retrying a 400 (bad request) is pointless. Retrying a 503 with exponential backoff is exactly right. The tenacity library makes this easy, use retry_if_exception_type to be selective.

Container bloat. Including build tools, test dependencies, and documentation in runtime images is common and expensive. Use multi-stage builds. If your runtime image is over 300 MB for a Python web service, it’s carrying weight it shouldn’t. Bloated images mean slower pulls during scaling events, when speed matters most.

CI/CD anti-patterns. The most common one: deploying on every push to main, including commits that only change documentation. Use path filters in your workflow triggers. The second: no deployment gates. If your test job completes in under a minute, it’s probably not running enough tests. Add integration tests that exercise the actual API contract, not just unit tests for individual functions.

Conclusion

FastAPI service wrapping an external LLM API, containerized with a hardened Dockerfile, tested and scanned in CI, deployed through an automated pipeline. It’s boring infrastructure work, which is exactly what you want when the LLM call in the middle of it fails at 2am.

The key takeaways: isolate your LLM API key behind a service, not directly in client code. Build retries and timeouts into the client from day one, not as an afterthought. Use multi-stage Docker builds and non-root users as defaults, not improvements. And wire up structured logging that captures token usage, your future self will want that data when the billing report looks wrong.

Everything else — provider selection, model choice, caching strategy — can change. The container boundary and the CI pipeline are what make those changes safe to make.

Related Posts

Dive deeper into Docker, Kubernetes, and AI microservices with these Collabnix articles:

• Ollama Docker Sandbox: Run Local Models Safely — Run AI coding agents isolated in a Docker sandbox with a local Ollama model.
• Getting Started with NVIDIA DGX Spark: Your Personal AI Supercomputer — Run frontier-scale AI models locally with the NVIDIA DGX Spark hardware platform.
• Kubernetes for Generative AI: Complete Guide to Deploying LLMs at Scale — Take your Docker-based LLM service to the next level with Kubernetes orchestration.
• Mastering OpenClaw Multi-Agent Workflows: Coordinating Multiple AI Agents — Learn how to orchestrate multiple AI agents in coordinated workflows using OpenClaw.