Guardrails For Small Models
Constrain small local models with rigid prompts, stop tokens, and examples to stop rambling and filler.
TL;DR
- Give small models rigid, explicit rules in the
SYSTEMprompt. - Cut rambling early with a
PARAMETER stopsequence. - Lower randomness with
PARAMETER temperature 0for steady output.
Write Rigid System Prompts
Be ExplicitState the exact task and format precisely.
SYSTEM """Answer in one sentence only."""Ban FillerForbid preambles and sign-offs directly.
SYSTEM """No preamble. Output only code."""One JobGive the model a single, narrow task.
SYSTEM """Return valid JSON. Nothing else."""Constrain Output
temperature 0Remove randomness for consistent formatting.
PARAMETER temperature 0stopEnd generation at a known boundary token.
PARAMETER stop "```"num_predictCap tokens to prevent runaway output.
PARAMETER num_predict 200Prime With Examples
MESSAGE pairShow one ideal question and answer.
MESSAGE user "2+2"
MESSAGE assistant "4"Match The FormatMake the example output the exact target shape.
MESSAGE assistant """{"ok": true}"""Keep It ShortOne or two pairs is enough for small models.
# 1-2 pairs, not tenStructure The Prompt
Linear StepsAsk for one action at a time.
ollama run llama3.1:8b "List 3 fruits"Explicit FormatTell the model exactly how to format.
"Reply as a numbered list, no intro"Test And TightenIterate on wording until output is stable.
# Rerun, then tighten the SYSTEM ruleTips
- Spell out the exact output format for small models, since an
8Bmodel follows rigid, concrete rules far better than vague ones. - Show one or two
MESSAGEexamples of the ideal answer, which anchors the format better than description alone for small models.
Warnings
- Small models drift with long, multi-part prompts; split complex tasks into single, linear steps to keep them on track.
- Without a
stopsequence, a small model may add filler like "Sure, here is" or keep talking well past the answer.
In Practice
Combine a rigid system prompt, zero temperature, and one example to make a small model emit only JSON.
- The system rule bans preamble and demands JSON only.
temperature 0keeps the format from drifting between runs.- One example pair shows the exact JSON shape to copy.
- A new input returns JSON in the same shape, without filler.
cat > Modelfile <<'EOF'
FROM llama3.1:8b
SYSTEM """Output only valid JSON. No preamble."""
PARAMETER temperature 0
MESSAGE user "apple, red"
MESSAGE assistant """{"item":"apple"}"""
EOF
ollama create json8b -f Modelfile
ollama run json8b "carrot, orange"FAQ
Smaller models follow loose instructions poorly, so they pad answers with preambles and keep going. Rigid rules, a low temperature, and a stop sequence keep them focused on just the answer.
Add an explicit SYSTEM rule like "No preamble, output only the answer." Reinforce it with a MESSAGE example that jumps straight to the result, and set a stop sequence for the boundary.
Be concrete and linear. Give one task at a time, state the exact output format, set temperature 0, and show an example. Small models reward rigid, unambiguous prompts far more than clever ones.
Yes. One or two MESSAGE example pairs anchor the format better than description alone. Keep them short, since too many examples eat the context window and slow each request.