agentleFS
open-thoughtsopen-thoughts/OpenThoughts-Agent/.agents/skills/serve-model-vibe-test/SKILL.md

serve-model-vibe-test

Skill309 starsChanged 4 months agoApache-2.0View on GitHub

Stand up a PUBLIC, shareable inference endpoint for an HF/gs model on an Iris TPU so people on the internet can vibe-test it in a browser or via an OpenAI-compatible API. Use when asked to "serve a model for testing", "throw up a public endpoint", "let people play with model X", or to demo a checkpoint. Combines marin-serve (marin#6556) + a Pinggy tunnel from our endpoint bank.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 1 is distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use a pair that shows down/502/503 (free).

    line 41
  2. Very specific:

    Keep the process running because it holds both tunnels.

    line 56
  3. Distinctive:

    Verify once the public URL prints:

    line 78
  4. Very specific:

    Dead tunnel reported live (ps state T): run kill -CONT <ssh_pid> <loop_pid>, then re-probe.

    line 97
  5. Very specific:

    Local DNS poisoning: verify through the real edge with dig @1.1.1.1 and curl --resolve, or use 1.1.1.1/8.8.8.8 DNS.

    line 105

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

serve-model-vibe-test

Spin up a public endpoint for a HuggingFace repo or gs:// checkpoint. marin-serve runs vLLM on a single-host Iris TPU slice behind the controller proxy; a Pinggy tunnel makes its proxied port public at https://<id>.a.pinggy.link/proxy/serve.<ep>/, with a dashboard and OpenAI-compatible API.

scripts/inference/serve_public.py wraps marin-serve, hpc.pinggy_utils.PinggyTunnel, and the Pinggy bank.

⚠️ Security — read first

The public URL is unauthenticated. Use it only for throwaway vibe testing; keep --timeout-hours short, use an unused Pinggy pair, and tear it down when done.

Prerequisites

  • marin-serve available (marin#6556). serve_public.py invokes ~/Documents/marin/.venv/bin/python -m marin.inference.quick_serve_cli by default. Override with --marin-serve-bin or MARIN_SERVE_BIN.
  • Run from the marin repo root (~/Documents/marin), not from this repo. marin-serve bundles Path.cwd() as the worker workspace (quick_serve_cli.py:227) and the worker runs uv sync --all-packages --extra tpu --extra vllm against it. Only the marin workspace defines the tpu/vllm extras (on marin-core); any other CWD → Extra 'tpu' is not defined in any project's 'optional-dependencies' table and the job dies before vLLM starts. serve_public.py resolves its own repo root via parents[2] for the hpc import, so invoke it by absolute path from the marin CWD.
  • Iris controller access (the laptop's step-ca/GCP creds — same as iris CLI).
  • $DC_AGENT_SECRET_ENV sourced (Pinggy uses your ~/.ssh identity; SSH to pro.pinggy.io:443).
  • Pinggy bank at ~/Documents/notes/ot-agent/pinggy_bank.md (override --pinggy-bank / PINGGY_BANK).
  • Single-host TPU slices only (v6e-8, v6e-4, v5litepod-8, …); multi-host is rejected.

Pick an unused Pinggy pair

The bank has 10 pairs. Before launching, pick one that isn't already serving — a quick probe (a free pair returns a Pinggy "tunnel not found"/503 or connection error; an in-use one returns your model):

for i in 1 2 3; do u=$(sed -n "/## Pair $i$/,/pinggy.link/p" ~/Documents/notes/ot-agent/pinggy_bank.md | grep pinggy.link); \
  echo "pair $i: $u -> $(curl -s -o /dev/null -w '%{http_code}' --max-time 6 https://$u/ || echo down)"; done

Use a pair that shows down/502/503 (free). Pass it as --pair N.

Launch (the one-liner)

cd ~/Documents/marin                               # MUST be the marin repo — see Prerequisites
set -a; source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"; set +a
python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py <MODEL> --tpu <SLICE> [--region <R>] \
    [--chat-template <FILE-or-URL>] --pair <N> --timeout-hours 6

It launches marin-serve, waits for READY — dashboard: http://127.0.0.1:<port>/proxy/serve.<ep>/, opens the Pinggy tunnel, and prints:

  dashboard : https://<id>.a.pinggy.link/proxy/serve.<ep>/
  OpenAI    : https://<id>.a.pinggy.link/proxy/serve.<ep>/v1

Keep the process running because it holds both tunnels. For unattended operation, use tmux, not nohup setsid … on macOS:

tmux new-session -d -s serve-public \
  "source \"${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}\" && cd ~/Documents/marin && \
   python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py <MODEL> ... 2>&1 | tee /tmp/serve_public.log"
# then poll /tmp/serve_public.log (or `tmux attach -t serve-public`) for the "OpenAI :" line

Worked example — the Delphi 9.7B SFT canary (marin#6545)

This is the reference model: laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft. It needs its own chat template (the repo ships a plain Llama-3 one); use delphi_v0.jinja2. marin-serve auto-derives the 4k context (malformed RoPE) and TP=2 (30 heads on a 4-chip slice).

# run from ~/Documents/marin (the bundled workspace); script is invoked by absolute path
cd ~/Documents/marin && source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV first}"
python -u ~/Documents/OpenThoughts-Agent/scripts/inference/serve_public.py \
    laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft \
    --tpu v6e-4 --region europe-west4 \
    --chat-template https://raw.githubusercontent.com/open-thoughts/OpenThoughts-Agent/ed4d6f483151f14d6d78cf732f04cd3c8ff5c606/chat_templates/delphi_v0.jinja2 \
    --pair 1 --timeout-hours 6

Verify once the public URL prints:

BASE=https://<id>.a.pinggy.link/proxy/serve.<ep>
# from the laptop you must force the real Pinggy edge IP (ISP DNS poisons *.a.pinggy.link); see Gotchas
curl "$BASE/v1/models"
curl "$BASE/v1/chat/completions" -H 'content-type: application/json' \
  -d '{"model":"laion/delphi-1e22-p33m67-32p07b-lr0_67-54770ae7-wc386k_lr1e5-sft",
       "messages":[{"role":"user","content":"Give me a fun fact about otters."}]}'

(Base/midtrained checkpoints with no chat template auto-default the dashboard to completion mode — omit --chat-template.)

Teardown

  • Foreground: Ctrl-C (tears down the Pinggy tunnel and the marin-serve job it launched).
  • Detached/other host: iris job stop <job> --cluster marin (the job name is in the log / iris query), then kill the serve_public.py PID. The slice also self-stops at --timeout-hours.

Gotchas

  • Run from the marin repo or the build fails before vLLM starts. Path.cwd() becomes the worker workspace; only marin defines the needed extras. Invoke serve_public.py by absolute path from ~/Documents/marin.
  • Dead tunnel reported live (ps state T): run kill -CONT <ssh_pid> <loop_pid>, then re-probe. PinggyTunnel.start() prevents this with detached stdin, ssh -n, and setsid; serve_public.py prints LIVE ✓ only after a DNS-aware /v1/models HTTP 200 probe.
  • Cold compile: first boot of a model can take tens of minutes; --ready-timeout (default 2700s) bounds the wait.
  • Don't use marin-serve --no-wait for the public path — the local proxied port only exists while the launching process runs; --no-wait returns immediately and there's nothing to tunnel.
  • Pinggy pair collisions: two jobs on the same pair clobber each other — always pick a free pair.
  • Single-host concurrency: one slice = limited throughput; fine for a few testers, not a viral launch (that's inference-broker territory).
  • The public URL carries the /proxy/serve.<ep>/ base path (it's the controller proxy path), so the OpenAI base is https://<id>.a.pinggy.link/proxy/serve.<ep>/v1, not /v1.
  • Local DNS poisoning: verify through the real edge with dig @1.1.1.1 and curl --resolve, or use 1.1.1.1/8.8.8.8 DNS. See .agents/projects/pinggy/pinggy.md.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.