Can one Ollama instance handle concurrent requests?

Yes. Set OLLAMA_NUM_PARALLEL above 1 and a single loaded model serves that many requests at once. The default can be 1, which queues extra requests, so raise it when memory allows.

How do I scale Ollama for many users?

First raise OLLAMA_NUM_PARALLEL for in-process concurrency. For more load, run several Ollama instances and put an Nginx reverse proxy in front to balance requests across them.

What does OLLAMA_NUM_PARALLEL do?

It sets how many requests one loaded model handles simultaneously. Each slot uses its own KV cache memory, so higher concurrency costs more VRAM. Tune it to your hardware.

How do I load balance multiple Ollama servers?

Define an Nginx upstream block listing each instance, then proxy_pass to it. Nginx round-robins requests across the backends by default, spreading load evenly.

Build / AI / Ollama

Scaling With A Proxy

Serve many users by raising per-instance concurrency and load balancing Ollama replicas with Nginx.

TL;DR

Let one server handle several requests with OLLAMA_NUM_PARALLEL.
Run multiple Ollama replicas behind an Nginx proxy.
Spread load across replicas with an Nginx upstream block.

Jump to section

In-Process Concurrency
Scale Out
Nginx Upstream
Production Tips
Tips
Warnings
In Practice
FAQ

In-Process Concurrency

OLLAMA_NUM_PARALLEL

Set how many requests one model serves at once.

export OLLAMA_NUM_PARALLEL=4

Default

Defaults to 1 or 4 based on free memory.

# 1 serializes; raise for concurrency

Memory Cost

Each slot needs its own KV cache.

# More parallel = more VRAM used

Scale Out

Multiple Replicas

Run several Ollama containers in parallel.

# docker compose up --scale ollama=3

Same Models

Each replica pulls the same models.

# Share a models volume across replicas

Front With Nginx

Put a reverse proxy in front of them.

# Nginx balances across the replicas

Nginx Upstream

upstream Block

List the backend Ollama instances.

upstream ollama {
  server a:11434;
  server b:11434;
}

proxy_pass

Forward requests to the upstream group.

location / {
  proxy_pass http://ollama;
}

Round Robin

Nginx rotates requests by default.

# Default balances requests evenly

Production Tips

Health Checks

Probe each instance before routing to it.

curl http://a:11434/api/tags

Keep Models Warm

Pin models so replicas respond instantly.

export OLLAMA_KEEP_ALIVE=-1

One Per GPU

Run one instance per available GPU.

# 1 replica per GPU, not oversubscribed

Tips

Set OLLAMA_NUM_PARALLEL above 1 so a single instance serves several requests at once, as long as memory allows the extra KV cache.
Scale horizontally by running several Ollama containers and balancing them with an Nginx upstream block for round-robin routing.

Warnings

By default OLLAMA_NUM_PARALLEL may be 1, so a single instance queues extra requests; raise it or add replicas for concurrency.
Each parallel slot needs its own KV cache memory; setting OLLAMA_NUM_PARALLEL too high can exhaust VRAM and crash model loads.

In Practice

Raise per-instance concurrency, then put two Ollama replicas behind an Nginx upstream for round-robin routing.

Raising OLLAMA_NUM_PARALLEL lets each replica serve several requests.
The upstream block lists both Ollama backends.
Nginx round-robins requests across the two servers.
proxy_pass forwards client traffic to the upstream group.

# Each Ollama instance: raise concurrency
#   export OLLAMA_NUM_PARALLEL=4

# nginx.conf: balance two replicas
upstream ollama {
    server 10.0.0.1:11434;
    server 10.0.0.2:11434;
}

server {
    listen 80;
    location / {
        proxy_pass http://ollama;
    }
}