Ollama

Ollama bundles model weights, configurations, and runtime dependencies into a streamlined CLI and local server for running open-source LLMs locally with native tool-calling.

When to Use

• Local LLM Execution: Running Llama 3, DeepSeek-R1, Mistral, and Qwen locally with a single CLI command.
• Zero-Cloud Air-Gapped Deployments: Privacy-critical enterprise environments requiring 100% on-premise execution.
• Custom Modelfile Configuration: Tailoring system prompts, temperature, context length, and stop sequences.
• REST & OpenAI-Compatible API: Seamless drop-in replacement for OpenAI SDKs in local test and dev pipelines.

Quick Start

# Run open-weight models locally with zero setup
ollama run llama3.2:3b "Write a Python script to compute Fibonacci numbers."

# Query local Ollama instance via Python SDK
import ollama

response = ollama.chat(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Explain zero-trust security."}]
)
print(response['message']['content'])

Core Concepts

Local Inference with the Python Ollama SDK

Querying local models with streaming and structured JSON output:

import ollama

# Basic chat completion
response = ollama.chat(
    model='llama3.1:8b',
    messages=[
        {'role': 'system', 'content': 'You are an expert DevOps engineer.'},
        {'role': 'user', 'content': 'Write a minimal production Dockerfile for a Go API.'}
    ],
    options={
        'temperature': 0.2,
        'num_ctx': 8192
    }
)

print(response['message']['content'])

Structured Output with Pydantic & JSON Schema

Enforcing typed JSON responses from local models:

from pydantic import BaseModel
import ollama

class ThreatAnalysis(BaseModel):
    threat_level: str
    vulnerabilities: list[str]
    mitigation: str

response = ollama.chat(
    model='llama3.1:8b',
    messages=[
        {'role': 'user', 'content': 'Analyze the security of an unauthenticated Redis instance exposed to 0.0.0.0.'}
    ],
    format=ThreatAnalysis.model_json_schema(),
    options={'temperature': 0.1}
)

analysis = ThreatAnalysis.model_validate_json(response['message']['content'])
print("Threat Level:", analysis.threat_level)
print("Mitigation:", analysis.mitigation)

Custom Modelfile Authoring

Creating tailored local model artifacts:

# Modelfile
FROM llama3.1:8b

PARAMETER temperature 0.3
PARAMETER num_ctx 16384
PARAMETER stop "<|eot_id|>"

SYSTEM '''
You are Antigravity Code Guardian. You analyze git diffs for security vulnerabilities,
performance regressions, and design pattern anti-patterns. Respond in concise markdown tables.
'''

# Build and register the custom model
ollama create code-guardian -f ./Modelfile
ollama run code-guardian "Review this diff..."

Common Patterns

Custom System Instructions and Temperature via Modelfile

Problem: Re-sending identical system prompts and temperature settings with every API request.

Solution: Create a custom Modelfile:

FROM llama3.2:3b

# Set model temperature
PARAMETER temperature 0.2

# Set system instruction
SYSTEM """
You are a senior DevOps engineer. Always provide answers as copy-pasteable bash or Terraform snippets.
"""

Build model: ollama create devops-assistant -f ./Modelfile

Best Practices

Do:

• Set OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS to handle concurrent local requests efficiently.
• Use format=Schema.model_json_schema() for reliable structured outputs without regex parsing.
• Offload models to Apple Silicon Metal or NVIDIA CUDA automatically with proper driver setups.
• Utilize ollama pull and model tags to test quantized variants (:q4_K_M, :q8_0).

Don't:

• Expose Ollama port (11434) to the public internet without an authenticating reverse proxy (Nginx, Caddy).
• Assume default context is unlimited; explicitly set num_ctx: 16384 or higher in options if needed.
• Run heavy 70B models on systems with less than 64GB unified memory/RAM.

Troubleshooting

References

• Ollama Website