Ollama With Docker

Run Ollama in Docker with a persistent volume for models and GPU passthrough for fast inference.

TL;DR

  1. Run Ollama in Docker with the ollama/ollama image.
  2. Mount a volume at /root/.ollama so weights persist.
  3. Add --gpus=all for GPU access via the NVIDIA toolkit.

Run The Container

    Pull Image

    Get the official Ollama Docker image.

    docker pull ollama/ollama
    Run Detached

    Start the server mapped to port 11434.

    docker run -d -p 11434:11434 ollama/ollama
    Exec A Model

    Run a model inside the container.

    docker exec -it ollama ollama run llama3.2
    Stop The Container

    Stop the running container by name.

    docker stop ollama

Persist Weights

    Named Volume

    Mount a volume so models persist.

    -v ollama:/root/.ollama
    List Volumes

    See all Docker volumes on the host.

    docker volume ls

GPU Access

    NVIDIA Toolkit

    Install the toolkit on the host first.

    # Install nvidia-container-toolkit
    --gpus=all

    Expose all GPUs to the container.

    docker run --gpus=all ollama/ollama
    Verify GPU

    Check the container sees the GPU.

    docker exec ollama nvidia-smi

Full Command

    Everything Together

    Name, volume, port, and GPU in one command.

    docker run -d --name ollama --gpus=all \
      -v ollama:/root/.ollama \
      -p 11434:11434 ollama/ollama
    Verify It Runs

    Confirm the named container started cleanly.

    docker ps --filter name=ollama
    Kubernetes

    Mount a PVC for weights and request a GPU in the pod spec.

    volumeMounts: [{ name: ollama, mountPath: /root/.ollama }]
    resources: { limits: { nvidia.com/gpu: 1 } }
    Environment Variables

    Configure Ollama via env vars inside the container.

    docker run -e OLLAMA_HOST=0.0.0.0:11434 ollama/ollama

Tips

  1. Mount a named volume at /root/.ollama, so pulled models survive container restarts instead of downloading again every time.
  2. Install the NVIDIA Container Toolkit first, then pass --gpus=all, so the container can actually see and use your GPU.

Warnings

  1. Without a volume at /root/.ollama, every container restart loses downloaded models and has to re-pull them from scratch.
  2. Omitting --gpus=all makes the container run on CPU only; the NVIDIA Container Toolkit must also be installed on the host.

In Practice

FAQ