run-bench
Skill5 starsChanged 53 days agoMITView on GitHub
Use when deciding or re-checking which model/effort should back explore, read_slice, web_lookup, run_filtered, fast_delegate or the FAST_CANDIDATES order in polyagent-mcp, when comparing speed or accuracy of engines/models on real bridge calls, or when the user asks to measure, benchmark or "testar de verdade" a model before adopting it.
In this repository4 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- add-engineSkill
- model-refreshSkill
- shipSkill
The file
Files
run-bench — medir no caminho real antes de decidir
Leaderboard mede a API; o chamador do polyagent espera o CLI inteiro (subida + loop do agente +
modelo). Em 2026-09-23 codex exec levou 24 s para responder "OK" — a latência de API do mesmo
modelo é ~6 s. Por isso decisão de tool auxiliar sai de medição própria.
Rodar
npm run bench # 2 repetições, todos os candidatos
node bench/aux-bench.mjs 1 luna-low,haiku-low # recorte: reps + ids de candidatoCandidatos, tarefas e gabarito: bench/aux-bench.mjs. Formato da saída:
research/bench/README.md. Rode em background (dezenas de chamadas em série; minutos).
Enquanto roda
- Não edite
src/nemtest/: o gabarito foi tirado do repo no início. Trabalhe em docs/skills ou espere. - Cota esgotada aparece como
err: "... quota exhausted"— descarte a engine da comparação, não conte como erro do modelo. Grok estava sem cota em 2026-09-23. - opencode/OpenRouter cobra por chamada e tem cauda longa de latência: mercury-2 variou de ~15 s a
160 s na mesma tarefa. Reporte mediana e pior caso, não só média.
Concluir
- Resuma por tarefa × candidato: acertos/rodadas, mediana e máximo de
ms, custo (assinatura vs pay-per-token). - Cruze com as notas de uso (
bridge_stats), quando houver volume. - Estudo em
research/AAAA-MM-DD-<assunto>.md+ linha emresearch/README.md. - Mudança de default: skill
model-refresh(passo 3) eship.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.