agentleFS
THEROCKSSSTHEROCKSSS/hermes-skills-portfolio/skills/ollama-local/SKILL.md

ollama-local

Skill1 starsChanged 3 months agoMITView on GitHub

Use when the user wants to run an LLM locally without cloud API costs or data leaving their machine — installing Ollama, pulling/managing models, calling the REST API (generate, chat, embeddings, streaming), picking a model for their RAM budget, or wiring Ollama into Hermes as a provider.

  • Pipes a download into a shell
  • Installs packages

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Port conflict on 11434: If another service already binds Ollama's default port, the server fails to start silently in some setups.

    line 204

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

ollama-local

Overview

Set up and use Ollama for running large language models locally. Ollama runs models on your machine — no API keys, no cloud, no per-token costs. The agent installs Ollama, pulls models, and shows you how to use them via the REST API or command line.

When to Use

  • The user wants to run an LLM locally without paying for API access.
  • The user wants privacy — no data leaves their machine.
  • The user wants to use a local model with their agent or application.
  • The user says "set up Ollama", "run a local LLM", or "I want offline AI".

Installation

Linux

curl -fsSL https://ollama.com/install.sh | sh

macOS

# Via Homebrew
brew install ollama

# Or download from https://ollama.com/download

Windows

Download from https://ollama.com/download and run the installer. Ollama runs as a background service on Windows.

Verify installation

ollama --version
# ollama version is 0.x.x

Model Management

Pull a model

# Small, fast model (good for testing)
ollama pull llama3.2:3b

# Medium model (good balance of speed and quality)
ollama pull llama3.1:8b

# Large model (best quality, needs 16GB+ RAM)
ollama pull llama3.1:70b

# Coding-focused model
ollama pull qwen2.5-coder:7b

# Embedding model
ollama pull nomic-embed-text

List installed models

ollama list

Run a model (interactive chat)

ollama run llama3.1:8b
>>> Tell me about quantum computing

Remove a model

ollama rm llama3.2:3b

API Usage

Ollama exposes a REST API at http://localhost:11434:

Generate a response

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain recursion in one sentence.",
  "stream": false
}'

Chat (multi-turn)

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [
    {"role": "user", "content": "What is 2+2?"},
    {"role": "assistant", "content": "4"},
    {"role": "user", "content": "What about 3+5?"}
  ],
  "stream": false
}'

Generate embeddings

curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "The quick brown fox jumps over the lazy dog."
}'

Python client

import requests

response = requests.post('http://localhost:11434/api/generate', json={
    'model': 'llama3.1:8b',
    'prompt': 'Write a haiku about the ocean.',
    'stream': False
})
print(response.json()['response'])

Streaming responses

import requests

response = requests.post('http://localhost:11434/api/generate', json={
    'model': 'llama3.1:8b',
    'prompt': 'Tell me a story.',
    'stream': True
}, stream=True)

for line in response.iter_lines():
    if line:
        import json
        chunk = json.loads(line)
        print(chunk.get('response', ''), end='', flush=True)

Integration with Hermes

Configure Hermes to use the local Ollama instance as a provider:

# Set Ollama as a custom provider
hermes config set model.provider custom
hermes config set model.base_url http://localhost:11434/v1
hermes config set model.api_key ollama  # Ollama doesn't require a real key
hermes config set model.default llama3.1:8b

Or use Ollama for specific tasks (like auxiliary/compression) while keeping a cloud model for main reasoning:

hermes config set auxiliary.compression.provider custom
hermes config set auxiliary.compression.base_url http://localhost:11434/v1
hermes config set auxiliary.compression.model llama3.2:3b

Model Selection Guide

ModelSizeRAM neededBest for
llama3.2:3b2 GB4 GBFast responses, simple tasks
llama3.1:8b5 GB8 GBGeneral purpose, good balance
qwen2.5-coder:7b5 GB8 GBCode generation, debugging
llama3.1:70b40 GB64 GBHigh quality, complex reasoning
nomic-embed-text0.3 GB1 GBEmbeddings for RAG/search

Performance Tips

  • Use GPU if available — Ollama auto-detects NVIDIA/AMD GPUs and Apple Silicon. GPU inference is substantially faster than CPU; measure on your own hardware, since the gap depends on the model, quantisation, and VRAM.
  • Match model size to your RAM — A model that doesn't fit in RAM will spill to disk and become extremely slow. Check ollama ps to see if the model is fully in memory.
  • Use smaller models for simple tasks — Don't use a 70B model for a one-sentence answer. Use 3B or 8B for quick tasks.
  • Keep models loaded — Ollama keeps models in memory for 5 minutes after last use by default. Increase this with OLLAMA_KEEP_ALIVE env var if you're making frequent requests.
  • Quantization — Ollama uses 4-bit quantization by default, which reduces memory usage by ~70% with minimal quality loss. No configuration needed.

Common Pitfalls

  1. Model larger than available RAM. Ollama will fall back to disk swap and performance becomes unusable rather than failing outright — check ollama ps to confirm the model is fully resident in memory, and use a smaller model or add RAM if not.
  2. First pull looks "stuck". The first ollama pull downloads the full model file (multiple GB); this can take minutes on a slow connection. Subsequent runs use the cached model and start instantly — don't kill the process assuming it's hung.
  3. Port conflict on 11434. If another service already binds Ollama's default port, the server fails to start silently in some setups. Set OLLAMA_HOST=0.0.0.0:11435 before starting and update client URLs to match.
  4. GPU not detected. On Linux, missing NVIDIA drivers/CUDA toolkit means Ollama silently falls back to CPU (much slower) instead of erroring. Verify with nvidia-smi before assuming GPU is in use.
  5. Prompts exceed local context comfortably. Local models advertise large context windows (e.g., 128k for Llama 3.1) but running at full context requires far more RAM than the base model size suggests. Keep prompts under ~8k tokens for 8B-class models in practice.
  6. Requests queue instead of running in parallel. Ollama processes requests sequentially by default, so concurrent callers block each other. Set OLLAMA_NUM_PARALLEL if concurrency is needed.

Verification Checklist

  • ollama --version succeeds and ollama list shows the pulled model
  • ollama ps confirms the model is loaded and shows a reasonable memory footprint (not swapping)
  • A test curl http://localhost:11434/api/generate call returns a non-empty response field
  • If GPU acceleration was expected, ollama ps or system GPU monitor (nvidia-smi) confirms it's actually being used
  • If wired into Hermes as a provider, hermes config get model.base_url reflects the correct local URL and a real Hermes call round-trips successfully

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.