How do I run Ollama in Docker?

Run the official image: docker run -d -p 11434:11434 ollama/ollama . Then run a model inside it with docker exec -it ollama ollama run llama3.2 . Add a volume and GPU flags for real use.

How do I keep models from re-downloading in Docker?

Mount a persistent volume at /root/.ollama , where Ollama stores weights. With -v ollama:/root/.ollama , models survive restarts instead of being pulled again each time the container starts.

How do I give a Docker container GPU access?

Install the NVIDIA Container Toolkit on the host, then add --gpus=all to the run command. Verify it worked by running nvidia-smi inside the container.

How do I run Ollama on Kubernetes?

Use the same image in a Deployment, mount a PersistentVolumeClaim at /root/.ollama , and request GPU resources in the pod spec. The container behaves the same as under Docker.

Build / AI / Ollama

Ollama With Docker

Run Ollama in Docker with a persistent volume for models and GPU passthrough for fast inference.

TL;DR

Run Ollama in Docker with the ollama/ollama image.
Mount a volume at /root/.ollama so weights persist.
Add --gpus=all for GPU access via the NVIDIA toolkit.

Jump to section

Run The Container
Persist Weights
GPU Access
Full Command
Tips
Warnings
In Practice
FAQ

Run The Container

Pull Image

Get the official Ollama Docker image.

docker pull ollama/ollama

Run Detached

Start the server mapped to port 11434.

docker run -d -p 11434:11434 ollama/ollama

Exec A Model

Run a model inside the container.

docker exec -it ollama ollama run llama3.2

Persist Weights

Named Volume

Mount a volume so models persist.

-v ollama:/root/.ollama

Host Path

Or bind a host directory for weights.

-v /data/ollama:/root/.ollama

Why It Matters

Without it, restarts lose all models.

# No volume = re-pull every restart

GPU Access

NVIDIA Toolkit

Install the toolkit on the host first.

# Install nvidia-container-toolkit

--gpus=all

Expose all GPUs to the container.

docker run --gpus=all ollama/ollama

Verify GPU

Check the container sees the GPU.

docker exec ollama nvidia-smi

Full Command

Everything Together

Volume, port, and GPU in one command.

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 ollama/ollama

Name It

Give the container a stable name.

--name ollama

Kubernetes

Mount a PVC and request GPU in the pod.

# PVC at /root/.ollama + GPU limits

Tips

Mount a named volume at /root/.ollama, so pulled models survive container restarts instead of downloading again every time.
Install the NVIDIA Container Toolkit first, then pass --gpus=all, so the container can actually see and use your GPU.

Warnings

Without a volume at /root/.ollama, every container restart loses downloaded models and has to re-pull them from scratch.
Omitting --gpus=all makes the container run on CPU only; the NVIDIA Container Toolkit must also be installed on the host.

In Practice

Start the official image with a persistent volume and full GPU access, then run a model inside it.

The NVIDIA Container Toolkit is a one-time host prerequisite.
The volume at /root/.ollama keeps models across restarts.
--gpus=all gives the container access to the host GPUs.
nvidia-smi inside the container confirms the GPU is visible.

# 1. Install the NVIDIA Container Toolkit
#    (one-time host setup)

# 2. Start Ollama with volume + GPU
docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama ollama/ollama

# 3. Pull and run a model in the container
docker exec -it ollama ollama run llama3.2

# 4. Confirm the GPU is visible
docker exec ollama nvidia-smi