Docker GenAI Stack on Windows using Docker Desktop

Table of Contents

Approaching 2000 GitHub stars, Docker GenAI Stack repository is gaining widespread traction, especially within the data science community. This stack is a collection of pre-configured Docker services designed to streamline the incorporation of large language models (LLMs) and AI workflows into your development setup. It offers a user-friendly solution for handling and executing AI applications, eliminating the necessity for in-depth knowledge of LLMs or containerization.

In October 2023, at the Dockercon event, Docker unveiled the groundbreaking GenAI Stack for the very first time. This collaborative effort brought together the expertise of Docker, Neo4j, LangChain, and Ollama. The primary objective behind this partnership was to forge a pre-configured GenAI stack comprising top-tier technologies. This stack not only boasts seamless integration but also includes sample applications, simplifying the onboarding process for developers and enabling them to dive into their projects effortlessly.

What is GenAI Stack composed of?

This stack comprises a set of Docker containers designed to simplify the process of experimenting with the development and execution of Generative AI (GenAI) applications. Within these containers, a pre-built support agent application is available, catering to data import and response generation use-cases. The key components of the stack include:

1. Ollama

A management tool facilitating the handling of local Large Language Models (LLMs).

2. Neo4j

A database crucial for grounding.

3. GenAI apps based on LangChain

Applications leveraging LangChain as the foundation for Generative AI functionalities.

4. Pre-configured LLMs

Jumpstart your AI projects with preconfigured Large Language Models such as Llama2, GPT-3.5, and GPT-4, embedded within the stack to enhance your development experience.

Why Ollama, Neo4j and LangChain?

The collaboration with LangChain and Ollama stemmed from their specialized knowledge in Large Language Models (LLMs). LangChain serves as a programming and orchestration framework tailored for LLMs, while Ollama excels as a tool specifically designed for the local management and operation of LLMs.

Neo4j’s involvement in the collaboration was driven by its proficiency in graph databases and knowledge graphs. Recognizing the potent synergy between graphs and LLMs, Neo4j identified the capability to construct highly accurate and reliable Generative AI (GenAI) applications through this amalgamation. The combined expertise of Ollama, Neo4j, and LangChain thus forms a robust foundation for the GenAI Stack, ensuring a comprehensive and effective solution for developers.

Refer this link if you’re interested to setup Docker GenAI Stack on Mac or Linux system

Getting Started

To kick off your journey with the GenAI Stack, follow these simple steps:

Install Docker Desktop for Windows

Ensure you have Docker Desktop installed on your Windows machine. You can download it from the official Docker website.

$ docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:44 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:02 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Install Git

Git is a fundamental version control tool. If you don’t have it installed, grab it from the official Git website to empower collaborative development.

Install WinGet

WinGet is a package manager for Windows that streamlines the installation of software packages. If you haven’t installed WinGet yet, do so to enhance your system management capabilities.

With these essential tools in place, you’re ready to dive into the world of Generative AI using the Docker GenAI Stack. Let the exploration begin!

Configuration changes in Docker Desktop Settings

To optimize your Docker Desktop experience, follow these configuration changes:

Navigate to Docker Desktop and select Settings.
Under the General tab, locate the option: *”Add the .docker.internal names to the host’s /etc/hosts file.”
Enable this option:
Enabling this setting allows the resolution of *.docker.internal DNS names from both the host machine and your containers.

By incorporating this configuration change, you enhance the communication capabilities between your host and containers, fostering a smoother and more integrated Docker environment. Save the changes and elevate your Docker experience!

Check if WSL2 is enabled or not

$ wsl --list --verbose
NAME STATE VERSION

docker-desktop-data Running 2
docker-desktop Running 2

Clone the repository

git clone https://github.com/docker/genai-stack
cd genai-stack

Bringing up the Stack

docker compose up -d --build

Results:

 Container genai-stack-database-1  Creating
 Container genai-stack-pull-model-1  Creating
 Container genai-stack-pull-model-1  Created
 Container genai-stack-database-1  Created
 Container genai-stack-pdf_bot-1  Creating
 Container genai-stack-loader-1  Creating
 Container genai-stack-bot-1  Creating
 Container genai-stack-api-1  Creating
 Container genai-stack-pdf_bot-1  Created
 Container genai-stack-loader-1  Created
 Container genai-stack-api-1  Created
 Container genai-stack-front-end-1  Creating
 Container genai-stack-bot-1  Created
 Container genai-stack-front-end-1  Created
 Container genai-stack-pull-model-1  Starting
 Container genai-stack-database-1  Starting
 Container genai-stack-pull-model-1  Started
 Container genai-stack-database-1  Started
 Container genai-stack-database-1  Waiting
 Container genai-stack-pull-model-1  Waiting
 Container genai-stack-database-1  Waiting
 Container genai-stack-pull-model-1  Waiting
 Container genai-stack-database-1  Waiting
 Container genai-stack-pull-model-1  Waiting
 Container genai-stack-database-1  Waiting
 Container genai-stack-pull-model-1  Waiting
 Container genai-stack-pull-model-1  service "pull-model" didn't complete successfully: exit 1
 Container genai-stack-pull-model-1  service "pull-model" didn't complete successfully: exit 1
 Container genai-stack-pull-model-1  service "pull-model" didn't complete successfully: exit 1
 Container genai-stack-pull-model-1  service "pull-model" didn't complete successfully: exit 1
 Container genai-stack-database-1  Error
 Container genai-stack-database-1  Error
 Container genai-stack-database-1  Error
 Container genai-stack-database-1  Error
service "pull-model" didn't complete successfully: exit 1

The error message was expected.
Reason – Ollama support for Windows is not yet available.
So, we might need to make changes and replace ollama with GPT-3.5 or GPT-4.0

Here’s the new Docker Compose file:

services:

  pull-model:
    image: genai-stack/pull-model:latest
    build:
      context: .
      dockerfile: pull_model.Dockerfile
    environment:
      - LLM=${LLM-gpt-3.5}
    networks:
      - net
    tty: true

  database:
    image: neo4j:5.11
    ports:
      - 7687:7687
      - 7474:7474
    volumes:
      - $PWD/data:/data
    environment:
      - NEO4J_AUTH=${NEO4J_USERNAME-neo4j}/${NEO4J_PASSWORD-password}
      - NEO4J_PLUGINS=["apoc"]
      - NEO4J_db_tx__log_rotation_retention__policy=false
    healthcheck:
        test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
        interval: 5s
        timeout: 3s
        retries: 5
    networks:
      - net

  loader:
    build:
      context: .
      dockerfile: loader.Dockerfile
    volumes:
      - $PWD/embedding_model:/embedding_model
    environment:
      - NEO4J_URI=${NEO4J_URI-neo4j://database:7687}
      - NEO4J_PASSWORD=${NEO4J_PASSWORD-password}
      - NEO4J_USERNAME=${NEO4J_USERNAME-neo4j}
      - OPENAI_API_KEY=${OPENAI_API_KEY-}
      - EMBEDDING_MODEL=${EMBEDDING_MODEL-sentence_transformer}
      - LANGCHAIN_ENDPOINT=${LANGCHAIN_ENDPOINT-"https://api.smith.langchain.com"}
      - LANGCHAIN_TRACING_V2=${LANGCHAIN_TRACING_V2-false}
      - LANGCHAIN_PROJECT=${LANGCHAIN_PROJECT}
      - LANGCHAIN_API_KEY=${LANGCHAIN_API_KEY}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
    networks:
      - net
    depends_on:
      database:
        condition: service_healthy
      pull-model:
        condition: service_completed_successfully
    x-develop:
      watch:
        - action: rebuild
          path: .
          ignore:
            - bot.py
            - pdf_bot.py
            - api.py
            - front-end/
    ports:
      - 8081:8080
      - 8502:8502


  bot:
    build:
      context: .
      dockerfile: bot.Dockerfile
    volumes:
      - $PWD/embedding_model:/embedding_model
    environment:
      - NEO4J_URI=${NEO4J_URI-neo4j://database:7687}
      - NEO4J_PASSWORD=${NEO4J_PASSWORD-password}
      - NEO4J_USERNAME=${NEO4J_USERNAME-neo4j}
      - OPENAI_API_KEY=${OPENAI_API_KEY-}
      - LLM=${LLM-gpt-3.5}
      - EMBEDDING_MODEL=${EMBEDDING_MODEL-sentence_transformer}
      - LANGCHAIN_ENDPOINT=${LANGCHAIN_ENDPOINT-"https://api.smith.langchain.com"}
      - LANGCHAIN_TRACING_V2=${LANGCHAIN_TRACING_V2-false}
      - LANGCHAIN_PROJECT=${LANGCHAIN_PROJECT}
      - LANGCHAIN_API_KEY=${LANGCHAIN_API_KEY}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
    networks:
      - net
    depends_on:
      database:
        condition: service_healthy
      pull-model:
        condition: service_completed_successfully
    x-develop:
      watch:
        - action: rebuild
          path: .
          ignore:
            - loader.py
            - pdf_bot.py
            - api.py
            - front-end/
    ports:
      - 8501:8501

  pdf_bot:
    build:
      context: .
      dockerfile: pdf_bot.Dockerfile
    environment:
      - NEO4J_URI=${NEO4J_URI-neo4j://database:7687}
      - NEO4J_PASSWORD=${NEO4J_PASSWORD-password}
      - NEO4J_USERNAME=${NEO4J_USERNAME-neo4j}
      - OPENAI_API_KEY=${OPENAI_API_KEY-}
      - LLM=${LLM-gpt-3.5}
      - EMBEDDING_MODEL=${EMBEDDING_MODEL-sentence_transformer}
      - LANGCHAIN_ENDPOINT=${LANGCHAIN_ENDPOINT-"https://api.smith.langchain.com"}
      - LANGCHAIN_TRACING_V2=${LANGCHAIN_TRACING_V2-false}
      - LANGCHAIN_PROJECT=${LANGCHAIN_PROJECT}
      - LANGCHAIN_API_KEY=${LANGCHAIN_API_KEY}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
    networks:
      - net
    depends_on:
      database:
        condition: service_healthy
      pull-model:
        condition: service_completed_successfully
    x-develop:
      watch:
        - action: rebuild
          path: .
          ignore:
            - loader.py
            - bot.py
            - api.py
            - front-end/
    ports:
      - 8503:8503

  api:
    build:
      context: .
      dockerfile: api.Dockerfile
    volumes:
      - $PWD/embedding_model:/embedding_model
    environment:
      - NEO4J_URI=${NEO4J_URI-neo4j://database:7687}
      - NEO4J_PASSWORD=${NEO4J_PASSWORD-password}
      - NEO4J_USERNAME=${NEO4J_USERNAME-neo4j}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - LLM=${LLM-gpt-3.5}
      - EMBEDDING_MODEL=${EMBEDDING_MODEL-sentence_transformer}
      - LANGCHAIN_ENDPOINT=${LANGCHAIN_ENDPOINT-"https://api.smith.langchain.com"}
      - LANGCHAIN_TRACING_V2=${LANGCHAIN_TRACING_V2-false}
      - LANGCHAIN_PROJECT=${LANGCHAIN_PROJECT}
      - LANGCHAIN_API_KEY=${LANGCHAIN_API_KEY}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
      - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
    networks:
      - net
    depends_on:
      database:
        condition: service_healthy
      pull-model:
        condition: service_completed_successfully
    x-develop:
      watch:
        - action: rebuild
          path: .
          ignore:
            - loader.py
            - bot.py
            - pdf_bot.py
            - front-end/
    ports:
      - 8504:8504
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 http://localhost:8504/ || exit 1"]
      interval: 5s
      timeout: 3s
      retries: 5

  front-end:
    build:
      context: .
      dockerfile: front-end.Dockerfile
    x-develop:
      watch:
        - action: sync
          path: ./front-end
          target: /app
          ignore:
            - ./front-end/node_modules/
        - action: rebuild
          path: ./front-end/package.json
    depends_on:
      api:
        condition: service_healthy
    networks:
      - net
    ports:
      - 8505:8505

networks:
  net:

Re-building the GenAI Stack

docker compose up -d --build

Verify the container services

docker compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
genai-stack-api-1 genai-stack-api "uvicorn api:app --h…" api About an hour ago Up About an hour (healthy) 0.0.0.0:8504->8504/tcp
genai-stack-bot-1 genai-stack-bot "streamlit run bot.p…" bot About an hour ago Up About an hour (healthy) 0.0.0.0:8501->8501/tcp
genai-stack-database-1 neo4j:5.11 "tini -g -- /startup…" database About an hour ago Up About an hour (healthy) 0.0.0.0:7474->7474/tcp, 7473/tcp, 0.0.0.0:7687->7687/tcp
genai-stack-front-end-1 genai-stack-front-end "npm run dev" front-end About an hour ago Up About an hour 0.0.0.0:8505->8505/tcp
genai-stack-loader-1 genai-stack-loader "streamlit run loade…" loader About an hour ago Up About an hour (unhealthy) 0.0.0.0:8502->8502/tcp, 0.0.0.0:8081->8080/tcp
genai-stack-pdf_bot-1 genai-stack-pdf_bot "streamlit run pdf_b…" pdf_bot About an hour ago Up About an hour (healthy) 0.0.0.0:8503->8503/tcp