serve-model-vibe-test
Skill309 starsChanged 4 months agoApache-2.0View on GitHub
Stand up a PUBLIC, shareable inference endpoint for an HF/gs model on an Iris TPU so people on the internet can vibe-test it in a browser or via an OpenAI-compatible API. Use when asked to "serve a model for testing", "throw up a public endpoint", "let people play with model X", or to demo a checkpoint. Combines marin-serve (marin#6556) + a Pinggy tunnel from our endpoint bank.
- Reads credentials
In this repository50 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- analyze-datagen-campaign-summarySkill
- analyze-dataset-token-lengthSkill
- analyze-id-eval-rankingSkill
- analyze-job-history-irisSkill
- analyze-rl-behaviorSkill
- analyze-training-run-irisSkill
- build-tpu-image-irisSkill
- code-create-staged-planSkill
- code-execute-staged-planSkill
- commitSkill
- crud-archive-runSkill
- crud-otagent-supabaseSkill
- crud-purge-below-gate-evalsSkill
- crud-purge-stale-eval-placeholdersSkill
- datagen-create-task-datasetSkill
- datagen-job-cleanupSkill
- datagen-launch-irisSkill
- datagen-launchSkill
- datagen-reduce-dataset-snapshotsSkill
- datagen-standard-launchSkill
- debugSkill
- docs-deslopSkill
- eval-agentic-cleanupSkill
- eval-agentic-launch-irisSkill
- eval-agentic-launchSkill
- eval-standard-cleanupSkill
- eval-standard-launchSkill
- file-issueSkill
- monitor-cron-sweep-irisSkill
- monitor-cron-sweepSkill
- monitor-job-tablesSkill
- monitor-restore-irisSkill
- monitor-restoreSkill
- monitor-restore-unifiedSkill
- rl-agentic-job-cleanupSkill
- rl-agentic-launch-irisSkill
- rl-agentic-launch-jupiterSkill
- rl-job-health-deep-diveSkill
- rl-standard-job-cleanupSkill
- rl-standard-launch-leonardoSkill
- sft-cleanup-hf-onlySkill
- sft-job-cleanupSkill
- sft-launchSkill
- supervisor-initSkill
- utils-cleanup-stale-sandboxesSkill
- utils-fix-permissionsSkill
- utils-reclaim-stale-snapshotsSkill
- write-testsSkill
- writing-styleSkill
The file
We split this skill into the rules it gives an agent
5 rules. About 0 are common, 1 is distinctive, and 4 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
4
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 41
Use a pair that shows down/502/503 (free).
- Very specific: line 56
Keep the process running because it holds both tunnels.
- Distinctive: line 78
Verify once the public URL prints:
- Very specific: line 97
Dead tunnel reported live (
psstateT): run kill -CONT <ssh_pid> <loop_pid>, then re-probe. - Very specific: line 105
Local DNS poisoning: verify through the real edge with dig @1.1.1.1 and curl --resolve, or use 1.1.1.1/8.8.8.8 DNS.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
serve-model-vibe-test
Spin up a public endpoint for a HuggingFace repo or gs:// checkpoint. marin-serve runs vLLM on a
single-host Iris TPU slice behind the controller proxy; a Pinggy tunnel makes its proxied port public at
https://<id>.a.pinggy.link/proxy/serve.<ep>/, with a dashboard and OpenAI-compatible API.
scripts/inference/serve_public.py wraps marin-serve, hpc.pinggy_utils.PinggyTunnel, and the Pinggy bank.
⚠️ Security — read first
The public URL is unauthenticated. Use it only for throwaway vibe testing; keep --timeout-hours short,
use an unused Pinggy pair, and tear it down when done.
Prerequisites
marin-serveavailable (marin#6556).serve_public.pyinvokes~/Documents/marin/.venv/bin/python -m marin.inference.quick_serve_cliby default. Override with--marin-serve-binorMARIN_SERVE_BIN.- Run from the marin repo root (
~/Documents/marin), not from this repo. marin-serve bundlesPath.cwd()as the worker workspace (quick_serve_cli.py:227) and the worker runsuv sync --all-packages --extra tpu --extra vllmagainst it. Only the marin workspace defines thetpu/vllmextras (onmarin-core); any other CWD →Extra 'tpu' is not defined in any project's 'optional-dependencies' tableand the job dies before vLLM starts.serve_public.pyresolves its own repo root viaparents[2]for thehpcimport, so invoke it by absolute path from the marin CWD. - Iris controller access (the laptop's step-ca/GCP creds — same as
irisCLI). $DC_AGENT_SECRET_ENVsourced (Pinggy uses your~/.sshidentity; SSH topro.pinggy.io:443).- Pinggy bank at
~/Documents/notes/ot-agent/pinggy_bank.md(override--pinggy-bank/PINGGY_BANK). - Single-host TPU slices only (
v6e-8,v6e-4,v5litepod-8, …); multi-host is rejected.
Pick an unused Pinggy pair
The bank has 10 pairs. Before launching, pick one that isn't already serving — a quick probe (a free pair returns a Pinggy "tunnel not found"/503 or connection error; an in-use one returns your model):
for i in 1 2 3; do u=$(sed -n "/## Pair $i$/,/pinggy.link/p" ~/Documents/notes/ot-agent/pinggy_bank.md | grep pinggy.link); \
echo "pair $i: $u -> $(curl -s -o /dev/null -w '%{http_code}' --max-time 6 https://$u/ || echo down)"; doneUse a pair that shows down/502/503 (free). Pass it as --pair N.
Launch (the one-liner)
cd ~/Documents/marin # MUST be the marin repo — see Prerequisites
set -a; source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"; set +a
python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py <MODEL> --tpu <SLICE> [--region <R>] \
[--chat-template <FILE-or-URL>] --pair <N> --timeout-hours 6It launches marin-serve, waits for READY — dashboard: http://127.0.0.1:<port>/proxy/serve.<ep>/, opens the
Pinggy tunnel, and prints:
dashboard : https://<id>.a.pinggy.link/proxy/serve.<ep>/
OpenAI : https://<id>.a.pinggy.link/proxy/serve.<ep>/v1Keep the process running because it holds both tunnels. For unattended operation, use tmux, not
nohup setsid … on macOS:
tmux new-session -d -s serve-public \
"source \"${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}\" && cd ~/Documents/marin && \
python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py <MODEL> ... 2>&1 | tee /tmp/serve_public.log"
# then poll /tmp/serve_public.log (or `tmux attach -t serve-public`) for the "OpenAI :" lineWorked example — the Delphi 9.7B SFT canary (marin#6545)
This is the reference model: laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft.
It needs its own chat template (the repo ships a plain Llama-3 one); use delphi_v0.jinja2.
marin-serve auto-derives the 4k context (malformed RoPE) and TP=2 (30 heads on a 4-chip slice).
# run from ~/Documents/marin (the bundled workspace); script is invoked by absolute path
cd ~/Documents/marin && source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}"
python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py \
laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft \
--tpu v6e-4 --region europe-west4 \
--chat-template https://raw.githubusercontent.com/open-thoughts/OpenThoughts-Agent/ed4d6f483151f14d6d78cf732f04cd3c8ff5c606/chat_templates/delphi_v0.jinja2 \
--pair 1 --timeout-hours 6Verify once the public URL prints:
BASE=https://<id>.a.pinggy.link/proxy/serve.<ep>
# from the laptop you must force the real Pinggy edge IP (ISP DNS poisons *.a.pinggy.link); see Gotchas
curl "$BASE/v1/models"
curl "$BASE/v1/chat/completions" -H 'content-type: application/json' \
-d '{"model":"laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft",
"messages":[{"role":"user","content":"Give me a fun fact about otters."}]}'(Base/midtrained checkpoints with no chat template auto-default the dashboard to completion mode — omit --chat-template.)
Teardown
- Foreground:
Ctrl-C(tears down the Pinggy tunnel and the marin-serve job it launched). - Detached/other host:
iris job stop <job> --cluster marin(the job name is in the log /iris query), then kill theserve_public.pyPID. The slice also self-stops at--timeout-hours.
Gotchas
- Run from the marin repo or the build fails before vLLM starts.
Path.cwd()becomes the worker workspace; only marin defines the needed extras. Invokeserve_public.pyby absolute path from~/Documents/marin. - Dead tunnel reported live (
psstateT): runkill -CONT <ssh_pid> <loop_pid>, then re-probe.PinggyTunnel.start()prevents this with detached stdin,ssh -n, andsetsid;serve_public.pyprintsLIVE ✓only after a DNS-aware/v1/modelsHTTP 200 probe. - Cold compile: first boot of a model can take tens of minutes;
--ready-timeout(default 2700s) bounds the wait. - Don't use marin-serve
--no-waitfor the public path — the local proxied port only exists while the launching process runs;--no-waitreturns immediately and there's nothing to tunnel. - Pinggy pair collisions: two jobs on the same pair clobber each other — always pick a free pair.
- Single-host concurrency: one slice = limited throughput; fine for a few testers, not a viral launch (that's inference-broker territory).
- The public URL carries the
/proxy/serve.<ep>/base path (it's the controller proxy path), so the OpenAI base ishttps://<id>.a.pinggy.link/proxy/serve.<ep>/v1, not/v1. - Local DNS poisoning: verify through the real edge with
dig @1.1.1.1andcurl --resolve, or use1.1.1.1/8.8.8.8DNS. See.agents/projects/pinggy/pinggy.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

