agentleFS
cybertronaicybertronai/SutroYaro/CLAUDE.md

SutroYaro

CLAUDE.md16 starsChanged 6 months agoUnlicenseView on GitHub

This is a research workspace for the Sutro Group, a study group exploring energy-efficient AI training. The group meets weekly at South Park Commons in San Francisco. SutroYaro is the lab's memory and dispatcher — not the research front, not the metric, not the benchmark problems. Five things it does: What this repo is NOT (so you don't accidentally do work in the wrong place): If you're about to write experiment code or a new benchmark stub here, stop and…

  • Pipes a download into a shell
  • Reads credentials

The file

We split this CLAUDE.md into the rules it gives an agent

6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    All 4 local learning rules (Hebbian, Predictive Coding, Equilibrium Propagation, Target Propagation) fail at chance level -- parity …

    line 104
  2. Distinctive:

    Skills: run-experiment (two-phase protocol), weekly-catchup, prepare-meeting

    line 141
  3. Very specific:

    Sync Google Docs: if meeting notes may have changed: python3 src/sync_google_docs.py

    line 187
  4. Very specific:

    Sync Telegram: if group discussion may have new messages: bun run sync_telegram.ts

    line 188
  5. Very specific:

    Check docs/index.md: if findings or status changed -- homepage should reflect current state

    line 189
  6. Distinctive:

    When reviewing PRs: check that results are reproducible, findings follow the template, and DISCOVERIES.md is updated if the experiment …

    line 211

≈ “Common” is an estimate: at least 1 in 5 of the 28,029 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
CLAUDE.md

CLAUDE.md - Sutro Group Research Workspace

Project Context

This is a research workspace for the Sutro Group, a study group exploring energy-efficient AI training. The group meets weekly at South Park Commons in San Francisco.

Scoped role of this repo (read this first)

SutroYaro is the lab's memory and dispatcher — not the research front, not the metric, not the benchmark problems. Five things it does:

  1. Lab memory — DISCOVERIES.md is the curated "what's proven" record. New agents read this BEFORE running experiments to avoid spending tokens on settled work.
  2. Agent-teams dispatcher — the patterns in .claude/skills/, bin/, hooks, and the agent-teams machinery here are how new builds (e.g. hinton-problems, schmidhuber-problems) get launched.
  3. Cross-repo index — docs/related-repos.md is the curated map of the 8-repo cybertronai org.
  4. Public face — docs/ + mkdocs build the site shown at Monday meetings.
  5. Telegram + Google Docs + GitHub sync — bin/tg-sync, src/sync_google_docs.py, daily catchups.

What this repo is NOT (so you don't accidentally do work in the wrong place):

  • NOT the active research front. That's at cybertronai/ByteDMD/experiments/grid (Yaroslav) and cybertronai/simplified-dally-model (Yaroslav, 2D grid cost model).
  • NOT the home for benchmark problems. Those are in dedicated repos: sparse-parity-challenge, hinton-problems, schmidhuber-problems, sutro-problems.
  • NOT the cost metric definition. That's in ByteDMD and simplified-dally-model.

If you're about to write experiment code or a new benchmark stub here, stop and check whether one of the per-purpose repos is the right home. SutroYaro should accumulate findings, not source code that belongs elsewhere. A reshuffle to align the codebase with this scoped role is tracked as a meta issue.

Read These First

  • docs/challenges.md — The four-challenge index: what each challenge is, where it lives, current state
  • LAB.md — Protocol for running experiments (templates, lifecycle, rules)
  • AGENT.md — Machine-executable experiment loop for autonomous sessions
  • DISCOVERIES.md — What's proven so far (read before every experiment)
  • CONTRIBUTING.md — How external contributors submit experiments and findings
  • TODO.md — Open research tasks
  • docs/tasks/INDEX.md — Current task tracker with priorities
  • docs/research/survey.md — Practitioner's Field Guide ranking all 37 experiments
  • docs/research/peer-research-protocol.md — Full design doc for multi-researcher autonomous research

Core Concepts

The lab now runs four challenges (see docs/challenges.md). The concepts below are from Challenge #1, sparse parity, where most of the recorded findings live.

  • Sparse Parity: Challenge #1's task — learn XOR/parity from random {-1,+1} inputs. n=20 bits, k=3 secret, 17 noise. The "drosophila" of energy-efficient training.
  • Average Reuse Distance (ARD): Proxy metric for energy efficiency. Small ARD = data stays in cache = cheap. Large ARD = expensive external memory access.
  • Data Movement Complexity (DMC): Better proxy metric (Ding et al., arXiv:2312.14441). DMC = sum of sqrt(stack_distance) for all float accesses. Tracks alongside ARD in MemTracker. Baseline: ARD 4,104 / DMC 300,298.
  • Cache Energy Model: register 5pJ, L1 (64KB) 20pJ, L2 (256KB) 100pJ, HBM 640pJ per float access (Bill Dally numbers).
  • CacheTracker: Extended MemTracker with LRU cache simulation for realistic energy estimates.
  • TrackedArray / Auto DMD: TrackedArray wraps numpy arrays so every operation (ufuncs, indexing, slicing) auto-records reads and writes on an LRUStackTracker. Removes manual instrumentation errors. Legacy element-level metric. See docs/research/tracked-numpy.md.
  • ByteDMD (primary metric, new): Pure Python tracer at src/bytedmd/. Wraps Python values; traces Python-level ops via dunders. Byte-granularity LRU stack with simultaneous pricing and eager liveness compaction. Reads cost ceil(sqrt(depth)); writes are free. Spec/reference: https://github.com/cybertronai/ByteDMD. Use bytedmd(func, args) for cost or traced_eval(func, args) for the trace. Submissions to the challenge should use Python ops (no numpy) to be fully tracked.

Current Best Methods

⚠️ Pre-ByteDMD numbers. The DMC values below were measured under the legacy element-level TrackedArray. ByteDMD (byte-granularity) became the primary metric on 2026-04-15 (PR #80) and these methods have not been re-measured under it yet. Treat as historical rankings; absolute numbers and relative ordering may shift. See docs/research/bytedmd.md for the current metric.

MethodTime (n=20/k=3)ARDDMC (legacy)Notes
KM-min (1 sample)~0.001s203,578New DMC leader. 1 influence sample suffices for parity.
GF(2) Gaussian Elimination509 us~420~203K240x faster than SGD, k-independent. Auto-tracked via TrackedArray; old harness reported 8,607.
KM Influence Estimation0.001-0.006s9220,633ARD leader. 5 influence samples per bit.
SMT Backtracking0.002s3,360348,336Constraint satisfaction approach
SGD (baseline)0.12s8,5041,278,460LR=0.1, batch=32, hidden=200

GF(2) solves n=100/k=10 in 703 microseconds. Parity is linear over the binary field -- the neural network was solving an easy problem the hard way.

SGD Config (when using neural nets)

n_bits=20, k_sparse=3, hidden=200, lr=0.1, wd=0.01,
batch_size=32, n_train=1000, max_epochs=200

Solves in ~40 epochs / 0.12s with numpy (fast.py).

Nix Development Shell (Optional)

For NixOS users (or those with flakes), a flake.nix provides a reproducible environment with python3 + numpy. Non-NixOS users can ignore the nix files.

nix develop
python3 bin/reproduce-all

Or one-liner:

nix develop --command python3 bin/reproduce-all

For reproducibility, macOS/Linux users can install Nix via the Determinate Systems installer:

curl --proto '=https' --tlsv1.2 -sSf https://install.determinate.systems/nix | sh -s -- install

This is informational (tells agents nix is available) rather than controlling (instructing behavior). Non-NixOS users can ignore and run python3 directly.

Key Findings

Phase 1 (16 experiments, SGD optimization):

  • LR=0.1 is critical (0.5 overshoots, never triggers phase transition)
  • W1 dominates 75% of all float reads -- limits ARD optimization to ~10%
  • L2 cache (256KB) eliminates ALL cache misses for both single-sample and batch
  • Curriculum learning (n=10 then expand to n=50) gives 14.6x speedup at scale
  • SGD breaks when n^k exceeds ~100,000 gradient steps

Phase 2 (17 experiments, broad search):

  • Algebraic/exact methods (GF(2), KM, SMT) solve instantly -- they exploit that parity is linear over GF(2)
  • All 4 local learning rules (Hebbian, Predictive Coding, Equilibrium Propagation, Target Propagation) fail at chance level -- parity requires k-th order interaction detection
  • Information-theoretic methods (MI, LASSO, MDL, Random Projections) all solve it but none beats Fourier meaningfully
  • RL sequential Q-learning achieves ARD of 1 at inference (reads exactly k=3 bits per prediction)

Autonomous Research Infrastructure

FilePurpose
AGENT.mdAgent-executable experiment loop (machine protocol)
src/harness.pyLocked evaluation harness (DO NOT MODIFY in experiment PRs)
research/search_space.yamlBounded mutation space per challenge
research/questions.yamlDependency graph of open research questions
research/log.jsonlAppend-only experiment log (machine-readable)
results/scoreboard.tsvHuman-readable leaderboard (auto-generated)
checks/env_check.pyPre-flight environment check
checks/baseline_check.pyRe-establish baselines on this machine
bin/run-agentLaunch autonomous agent cycle
bin/merge-findingsImport contributor log entries via PR

See docs/research/peer-research-protocol.md for the full design.

Eval Environment

The eval environment tests whether an AI agent can do energy-efficient ML research.

  • Guide: See AGENT_EVAL.md for adding challenges, methods, running evals
  • Quick test: PYTHONPATH=src python3 -c "import gymnasium as gym; import sparse_parity.eval; env = gym.make('SutroYaro/SparseParity-v0', metric='dmc', budget=10); obs, info = env.reset(); obs, r, _, _, info = env.step(5); print(info)"
  • Environments: SutroYaro/SparseParity-v0 (single challenge), SutroYaro/MultiChallenge-v0 (all three)
  • Ground truth: 37 experiments, 72-point grading rubric (12 categories)
  • Docs: docs/research/eval-environment.md

Agent Infrastructure

Hooks, rules, and skills that make Claude Code sessions more effective. See docs/research/agent-infrastructure.md for the full docs.

  • Hooks: session-start (shows status), security-guard (blocks locked measurement code and destructive commands), session-end (session summary)
  • Rules: experiment reproducibility (seeds, config, environment), agent coordination (parallel dispatch, file ownership)
  • Skills: run-experiment (two-phase protocol), weekly-catchup, prepare-meeting
  • Config: .claude/settings.json

Other coding agents (Gemini, Codex) don't run the hooks but can read the rules and skills.

Automation

ScriptWhat it doesDocs
bin/tg-syncSyncs Telegram to local SQLite (incremental)docs/tooling/telegram-setup.md
bin/tg-postPosts to Telegram forum topics via Bot APIdocs/tooling/telegram-setup.md
src/sync_google_docs.pyPulls Google Docs to local markdowndocs/tooling/automation.md
.traces/export_sessions.pyExports Claude Code session tracesdocs/tooling/automation.md

Telegram Quick Reference

# First time: install deps and authenticate
bun install
# On NixOS: credentials come from sops-nix via flake shellHook, no .env needed
# On other systems: cp .env.example .env and fill in credentials
bin/tg-auth

# Sync messages to SQLite (incremental)
bin/tg-sync
# Database: telegram.db (project root, .gitignored)

# Query messages
sqlite3 telegram.db "SELECT date, sender, text FROM messages ORDER BY date DESC LIMIT 10"

# Post via your own bot (requires TELEGRAM_BOT_TOKEN in .env)
bin/tg-post --topic agent-updates "Experiment completed"

Working Style

  • Iteration time must stay under 2 seconds (use fast.py for numpy speed)
  • Change one thing at a time (correctness, then speed, then energy)
  • Priority: correctness > wall-clock time > energy usage
  • One hypothesis per experiment, always compare against baseline
  • Record everything -- failed hypotheses are findings too
  • Apply anti-slop writing rules to all prose (no em dashes, no AI vocabulary)

Before Pushing

  • Update docs/changelog.md with what changed (bump version, add section)
  • Sync Google Docs if meeting notes may have changed: python3 src/sync_google_docs.py
  • Sync Telegram if group discussion may have new messages: bun run sync_telegram.ts
  • Check docs/index.md if findings or status changed -- homepage should reflect current state
  • Check GitHub for PRs/issues: gh pr list --repo cybertronai/SutroYaro

Full sync workflow: docs/tooling/sync-runbook.md

People

  • Yad (repo creator, SutroYaro) — Built the Claude Code autonomous research lab, parallel agent experiments
  • Yaroslav (Sutro Group founder) — Technical sprints, algorithm work, cybertronai/sutro
  • Emmett — Aster agentic loop framework, 2x energy improvement on microgpt
  • G B — Architecture experiments (depth-1/hidden-64, ARD ~33-35)
  • Germaine, Andy, Seth, Barak, Jamie Simon — Group members

Contributing

Multiple people contribute via PRs (fork and branch). See CONTRIBUTING.md for the full guide and docs/branch-workflow.md for branch naming, locked files, and agent permissions.

  • contributions/ — Drop raw results here in any format. No template needed.
  • findings/_template.md — Standalone findings template for structured reports.
  • DISCOVERIES.md — Shared knowledge base. Anyone can PR new bullets.
  • Metric isolation (LAB.md rule #9) — Never modify tracker.py, cache_tracker.py, data.py, config.py, harness.py in experiment PRs.

When reviewing PRs: check that results are reproducible, findings follow the template, and DISCOVERIES.md is updated if the experiment answers an open question.

Related Repos

Full curated map with descriptions: docs/related-repos.md. Quick list:

Active research front (May 2026):

Companion baseline catalogs (shipped May 2026):

Together: representational + algorithmic baseline pair, both pure numpy + matplotlib. Tracking issues for v2 ByteDMD instrumentation: hinton-problems #45, schmidhuber-problems #17.

Adjacent (not in cybertronai org):

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.