agentleFS
hail-ishail-is/hail/.claude/skills/hail-batch/SKILL.md

hail-batch

Skill1.1k starsChanged 6 days agoMITView on GitHub

Submit jobs, debug failures, and write Hail Batch submission scripts as an end user

Trending

Less talk lately.about the whole repository

Dec 2025: 0 mentionsJan 2026: 0 mentionsFeb 2026: 0 mentionsMar 2026: 0 mentionsApr 2026: 0 mentionsMay 2026: 0 mentionsJun 2026: 2 mentionsJul 2026: 0 mentionsAug 2026: 2 mentionsSep 2026: 0 mentionsOct 2026: 0 mentionsNov 2026: 1 mention
Dec 2025Jun 2026Nov 2026
Mentions a month of the hail-is/hail repository that our weekly crawl found. There is no adoption data for this file yet.

Who's talking about hail-is/hail

The biggest 5 of 54 mentions linking to the repository that our crawl found. Each line is their own headline.

See all 54

Talked about on GitHub, Stack Overflow, Hacker News and Bluesky (54 times).

On GitHub · 50

45 more on GitHub

On Hacker News · 1

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 1 is distinctive, and 6 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 6

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If hailctl isn't found, it may be in a virtual environment — look for one and activate it.

    line 9
  2. Very specific:

    Never suppress stderr: when running hailctl or other Hail tools — important connection info and warnings appear there.

    line 11
  3. Very specific:

    Always specify --container — the bare command dumps all containers as escaped YAML and is hard to read.

    line 51
  4. Very specific:

    Use the hailgenetics/hail:<version> image.

    line 110
  5. Distinctive:

    When given a batch ID:

    line 178
  6. Very specific:

    Run hailctl batch get BATCH_ID — report state, n_jobs, n_succeeded, n_failed, cost

    line 179
  7. Very specific:

    Run hailctl batch jobs BATCH_ID --state bad to find failed/errored jobs

    line 180

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

You are helping a user work with Hail Batch — a job orchestration service for running analysis pipelines on cloud compute.

Running hailctl commands

If hailctl isn't found, it may be in a virtual environment — look for one and activate it. Once activated, it stays active for the session.

Never suppress stderr when running hailctl or other Hail tools — important connection info and warnings appear there. Do not use 2>&1 or 2>/dev/null.

Authentication

Users need two things on GCP:

# 1. GCP Application Default Credentials (for cloud storage access)
gcloud auth application-default login

# 2. Hail service auth (reads ~/.hail/tokens.json)
hailctl auth login

Both are needed. If a user gets auth errors, check which one is missing.

Investigating batches: use the CLI

For inspection tasks (checking status, reading logs, diagnosing failures), always prefer hailctl batch CLI commands — they're simpler and don't require writing Python.

List recent batches

hailctl batch list                          # last 50 batches
hailctl batch list --limit 20 --query 'state=failure'
hailctl batch list -o json                  # machine-readable

Get batch status

hailctl batch get BATCH_ID                  # full details (yaml)
hailctl batch get BATCH_ID -o json

Inspect a specific job

hailctl batch job BATCH_ID JOB_ID           # status + spec (yaml)

Get logs

Always specify --container — the bare command dumps all containers as escaped YAML and is hard to read. If a job has been retried, use --attempt to get the right one (defaults to most recent).

hailctl batch attempts BATCH_ID JOB_ID                              # list attempts + IDs (only if needed)
hailctl batch log BATCH_ID JOB_ID --container main                  # user code (start here)
hailctl batch log BATCH_ID JOB_ID --container main --attempt ID     # specific attempt
hailctl batch log BATCH_ID JOB_ID --container input                 # file copies in
hailctl batch log BATCH_ID JOB_ID --container output                # file copies out

Wait for a batch to complete

hailctl batch wait BATCH_ID
hailctl batch wait BATCH_ID -o json

Cancel or delete

hailctl batch cancel BATCH_ID
hailctl batch delete BATCH_ID

Billing

hailctl batch billing list
hailctl batch billing get BILLING_PROJECT

Writing batch submission scripts: use the Python API

For writing scripts that submit batches programmatically, use hailtop.batch:

import hailtop.batch as hb

b = hb.Batch(
    name='my-analysis',
    backend=hb.ServiceBackend(
        billing_project='my-project',
        remote_tmpdir='gs://my-bucket/tmp',
    ),
)

j = b.new_job(name='step1')
j.image('hailgenetics/hail:0.2.x')
j.memory('4Gi')
j.cpu(2)
j.command('python3 my_script.py')

b.run()

For quick one-off jobs, hailctl batch submit works too:

hailctl batch submit --image hailgenetics/hail:0.2.x --memory 4Gi python3 my_script.py

Hail Query within a Batch job

Use the hailgenetics/hail:<version> image. Write outputs to cloud storage and read them back:

# In the batch job script (runs on the worker):
import hail as hl
hl.init()
mt = hl.read_matrix_table('gs://bucket/input.mt')
results = mt.some_analysis()
results.write('gs://bucket/output.ht')

# Back in the parent script after b.run():
results = hl.read_table('gs://bucket/output.ht')

Direct API access with hailctl curl

For endpoints not covered by hailctl batch subcommands, use hailctl curl — it handles auth automatically:

hailctl curl default batch /PATH

To discover all available endpoints, fetch the OpenAPI spec:

hailctl curl default batch /openapi.yaml

Listing jobs within a batch

hailctl batch jobs BATCH_ID                          # first 50 jobs
hailctl batch jobs BATCH_ID --state bad              # failed or errored jobs
hailctl batch jobs BATCH_ID --state bad -o json      # machine-readable
hailctl batch jobs BATCH_ID --exit-code 137          # OOM kills
hailctl batch jobs BATCH_ID --name my-step           # by job name
hailctl batch jobs BATCH_ID --limit 0                # all jobs (no limit)
hailctl batch jobs BATCH_ID --limit 50 --last-job-id 50  # pagination: next page after job 50

Valid --state values: pending, ready, creating, running, live (ready+creating+running), cancelled, error, failed, bad (error+failed), success, done (all terminal states).

Flags can be combined: --state bad --exit-code 1 returns jobs that are bad AND have exit code 1.

Job states

StateMeaning
PendingWaiting on dependencies
ReadyDependencies met, queued for a worker
CreatingWorker being provisioned
RunningExecuting on a worker
SuccessExited 0
FailedUser code exited non-zero
ErrorInfrastructure/platform failure (not user code)
CancelledCancelled before completion

Common failure patterns

SymptomLikely causeAction
Failed, exit code 1Script errorhailctl batch log BATCH JOB --container main
Failed, exit code 137OOM killIncrease j.memory() in the submission script
Error stateInfrastructure failureCheck --container input or --container output logs; may be transient
Jobs stuck in PendingDependency failure or quotaCheck upstream jobs
High cost, slow progressInsufficient parallelismIncrease job fan-out in the script

How to investigate a specific batch

When given a batch ID:

  1. Run hailctl batch get BATCH_ID — report state, n_jobs, n_succeeded, n_failed, cost
  2. Run hailctl batch jobs BATCH_ID --state bad to find failed/errored jobs
  3. For each bad job (up to ~5), run hailctl batch log BATCH_ID JOB_ID --container main
  4. Identify the root cause and suggest next steps

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.