Benchmark Memory System

Systematic benchmarking framework to measure retrieval quality, compare configurations, and identify optimal parameters for the Local Brain Search memory system.

Purpose

1. Measure retrieval quality objectively using LLM-as-judge scoring
2. Compare different configuration settings (spreading vs static, parameter sweeps)
3. Identify optimal parameters for different query types (factual, conceptual, synthesis)
4. Generate reproducible results against frozen test datasets

Design Principles

• Contained: Skill + sub-agent + bundled scripts
• Reproducible: Test against frozen Brain snapshot
• Automated: LLM-as-judge for relevance scoring
• Analyzable: CSV output for analysis

State Dependencies

Prerequisites

• Local Brain Search system indexed (resources/local-brain-search/data/brain.faiss)
• Python venv at resources/local-brain-search/venv/ with search dependencies
• A container-valid LLM-judge channel (see LLM-as-Judge Scoring below - the channel differs between workstation and Trinity)

LLM-as-Judge Scoring

The judge channel is environment-dependent. Pick it before you run anything.

Why claude -p fails on Trinity (measured 2026-09-16, retrieval-signal-hierarchy AM-10)

A Trinity agent's turn is subscription-authenticated by the platform at process spawn, but that credential is turn-scoped by design and never written to disk in refreshable form. The container does ship ~/.claude/.credentials.json, which is why this looks like it should work - but its accessToken and refreshToken are empty strings. A child claude -p has nothing to read and dies with Not logged in in about a second. claude --version still succeeds, so it is not a valid readiness check - a binary on $PATH proves nothing about auth.

Do not work around this by shelling out anyway. Per Trinity ent#643, agents must not spawn the claude CLI: a subprocess escapes the execution ledger, cost attribution, autonomy gates and permission enforcement. Task and chat_with_agent are the sanctioned paths.

Container-valid option A - in-process Task subagent

For a judging pass driven from inside this turn, spawn a subagent with the Task tool. It inherits the turn's authentication, stays inside the execution ledger, and needs no key. Best when the scoring set is small enough to fan out over a handful of agents and return scores as text.

Container-valid option B - chat_with_agent for delegated/batch work

For a long batch (the 50-query x 15-config sweep), drive chat_with_agent over Trinity MCP. TRINITY_MCP_URL and TRINITY_MCP_API_KEY are in every agent's environment, so a stdlib-only Python script can call tools/call -> chat_with_agent over plain HTTP and get a real, subscription-backed execution. No SDK, no installs, no key. Reference client: a Trinity MCP client of your own (protocol: Trinity MCP tools/call).

Benchmark-critical hazard: identical messages are DEDUPLICATED into one execution. Same message -> same execution_id -> the same cached response text; the second call never runs. A benchmark that scores the same (query, result) pair twice - score-rescore, rubric perturbation, self-consistency, or simply two configs that retrieve the same note - will record perfect agreement as a pure artifact, and a failed judge call will "retry" forever against its own cached failure. Always put a unique nonce in the message, derived from (query id, config id, model, attempt) so it stays deterministic and resumable.

Two further measured behaviours that matter here: allowed_tools is not enforced (never treat a judge as sandboxed or blind), but tool use is auditable - get_execution_result(include_log=True) returns the tools the turn actually held, so verify the judge used none rather than assuming it.

• Default model: sonnet (good quality) - haiku is faster/cheaper, opus stronger
• JSON output via prompt engineering for reliable scoring

Installing Dependencies

Dependencies are installed in the local-brain-search venv:

cd resources/local-brain-search
source venv/bin/activate
pip install pandas tqdm  # no anthropic SDK and no API key needed on either channel

Sub-Commands

/benchmark-memory setup

Create a frozen Brain snapshot and build its index.

cd .claude/skills/benchmark-memory/scripts
./run_benchmark.sh --list-snapshots  # Check existing
python3 create_snapshot.py           # Create new snapshot

What it does:

1. Creates snapshot directory with date stamp
2. Copies Brain folder (excluding .obsidian, .trash)
3. Builds FAISS index for the snapshot
4. Creates SNAPSHOT-INFO.md with metadata

/benchmark-memory create-queries [--count N]

Generate or manage test query sets.

cd .claude/skills/benchmark-memory/scripts
python build_query_set.py --count 50 --output ../query-sets/core-50.json

Query Categories (50 total):

/benchmark-memory run [--config CONFIG] [--snapshot SNAPSHOT]

Execute benchmark with specified configuration.

cd .claude/skills/benchmark-memory/scripts
./run_benchmark.sh --config focused --snapshot brain-snapshot-2026-02-18
./run_benchmark.sh --dry-run --config focused  # Preview without execution
./run_benchmark.sh --list-configs              # List available configs

Configurations:

• focused: 15 key configurations (recommended for initial benchmarking)
• single:CONFIG_NAME: Run single configuration
• all: Full parameter sweep (expensive)

Estimated cost per run:

• 50 queries x 10 results x 15 configs = 7,500 LLM scores
• Using Sonnet (default): ~$75
• Using Haiku: ~$7.50

/benchmark-memory analyze [--results FILE]

Generate analysis summary from benchmark results.

cd .claude/skills/benchmark-memory/scripts
python analyze_results.py --results ../results/benchmark-*.csv

Outputs:

• Summary by configuration
• Summary by query category
• Best config per intent
• Recommendations

Workflow

Step 1: Setup (one-time)
/benchmark-memory setup
    |
    v
Step 2: Create Queries (one-time)
/benchmark-memory create-queries --count 50
    |
    v
Step 3: Run Benchmark (per experiment)
/benchmark-memory run --config focused
    |
    v
Step 4: Analyze Results
/benchmark-memory analyze

Metrics Collected

Performance Metrics

Quality Metrics

LLM-as-Judge Scoring Scale

Configurations to Test

Baseline

• static_baseline: Traditional vector search
• spreading_default: Spreading activation with defaults

Parameter Sweeps

• Iteration count: 2, 5, 7
• Inhibition strength: 0.1, 0.3, 0.5
• Temporal decay: 0.8, 0.9, 0.95
• Q-weight: 0.0, 0.3, 0.5

Optimized Combinations

• synthesis_optimized: max_iterations=7, inhibition=0.1
• factual_optimized: max_iterations=2, inhibition=0.5
• balanced_optimized: max_iterations=5, inhibition=0.2, decay=0.85

Output Files

Results CSV

results/benchmark-YYYY-MM-DD-HHMMSS.csv

Schema:

timestamp,config_name,query_id,query_category,mode,max_iterations,
inhibition_strength,latency_ms,result_1_note,result_1_score,...,
precision_at_5,precision_at_10,mrr,ndcg_at_10,avg_score

Analysis Report

analysis/report-YYYY-MM-DD.md

Error Recovery

Partial benchmark run

Results are appended incrementally. Resume by running with --resume:

python run_benchmark.py --config focused --resume

API rate limits

Built-in retry with exponential backoff. Adjust --delay if needed:

python run_benchmark.py --config focused --delay 1.0

Invalid snapshot

Re-create snapshot:

python create_snapshot.py --force

Success Criteria

• Snapshot creation produces valid index
• Query set covers all 6 categories (50+ queries)
• LLM judge produces consistent scores (>80% agreement on re-run)
• All 15 focused configs can be benchmarked
• Results CSV is valid and analyzable
• Analysis identifies best config per query type

Expected Insights

After running benchmarks, answer these questions:

1. Does spreading beat static? For which query types?
2. What's the optimal iteration count for synthesis queries?
3. Does high inhibition help factual queries?
4. Does q_weight > 0 improve results over time?
5. What settings work best for each query type?

Cost Management

Recommendation: Start with Haiku judge, validate sample against Sonnet.

Directory Structure

.claude/skills/benchmark-memory/
├── SKILL.md                    # This file
├── requirements.txt            # Python dependencies
├── scripts/
│   ├── run_benchmark.sh        # Wrapper script (uses venv Python)
│   ├── create_snapshot.py      # Create frozen Brain snapshot
│   ├── build_query_set.py      # Generate/manage query test set
│   ├── run_benchmark.py        # Execute benchmark with config
│   ├── score_results.py        # LLM-as-judge scoring
│   ├── compute_metrics.py      # Calculate evaluation metrics
│   └── analyze_results.py      # Generate analysis summary
├── configs/
│   ├── focused_configs.json    # Test configurations (15 configs)
│   └── judge_prompt.txt        # LLM judge prompt template
├── snapshots/                  # Frozen Brain copies
├── query-sets/                 # Test queries (core-50.json included)
├── results/                    # Benchmark CSVs
└── analysis/                   # Analysis reports

Tested Components