hail-batch-dev
Skill1.1k starsChanged 6 days agoMITView on GitHub
Work on the Hail system itself — CI pipeline, flaky tests, Batch service internals, dev namespace deployments, deployed infrastructure management
Trending
Less talk lately.about the whole repository
Who's talking about hail-is/hail
- ipython/ipython“In ipython 8.18 (but not ipython 8.16), the current event loop is overwritten in each cell”GitHub · Jan 17, 2024 · 7 reactions
- @stephenturner.us“The Scalable Variant Call Representation: Enabling Genetic Analysis Beyond One Million Genomes …”Bluesky · Jan 12, 2024 · 7 likes
- has2k1/plotnine“Request: re-license under MIT?”GitHub · Dec 6, 2021 · 3 reactions
- Stack Overflow“Get relative path from inside a scala library (while developing that library)”Stack Overflow · Nov 27, 2018 · 3 votes
- d12frosted/homebrew-emacs-plus“Fix linking how-to in formula caveats”GitHub · Dec 21, 2023 · 2 reactions
The biggest 5 of 54 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 54
Talked about on GitHub, Stack Overflow, Hacker News and Bluesky (54 times).
On GitHub · 50
- Jan 17, 2024In ipython 8.18 (but not ipython 8.16), the current event loop is overwritten in each cell — ipython/ipython7 reactions · 2 comments
- Dec 6, 2021Request: re-license under MIT? — has2k1/plotnine3 reactions · 1 comment
- Dec 21, 2023Fix linking how-to in formula caveats — d12frosted/homebrew-emacs-plus2 reactions
- Dec 14, 2023Incorrect lint in 0.1.8? RUF006 Store a reference to the return value of `asyncio.create_task` — astral-sh/ruff2 reactions · 9 comments
- Nov 4, 2026New hail-is/hail version available: 0.2.140 — TheBoutrosLab/docker-Hail1 reaction
45 more on GitHub
- Aug 20, 2026New hail-is/hail version available: 0.2.139 — TheBoutrosLab/docker-Hail1 reaction
- Feb 7, 2021Hail developer set-up — populationgenomics/team-docs1 reaction
- Aug 4, 2026Bump the pip group across 1 directory with 5 updates — broadinstitute/variant-curation-portal0 reactions
- Jun 24, 2026Update hadoop to hailtop after hail's hadoop deprecation — broadinstitute/gnomad_methods0 reactions · 2 comments
- Jun 16, 2026[skill-suggestion] hail: scalable genomic analysis and GWAS engine (Broad Institute) — stanfish06/skillquarium0 reactions
- Sep 11, 2025[explore] Variant store: Hail — tidywf/nemo0 reactions
- Jul 25, 2025Add LEN to compute_stats_per_ref_site after hail updates — broadinstitute/gnomad_methods0 reactions
- Mar 14, 2025Temporarily revert username validation, as new rule disallows dashes — populationgenomics/hail0 reactions
- Jan 10, 2025Refine combiner usage — populationgenomics/production-pipelines0 reactions · 1 comment
- Dec 16, 2024fix: remove obsolete Hortonworks repositories — h2oai/h2o-30 reactions · 6 comments
- Dec 15, 2024Cherry-pick upstream fix: Set StreamReadConstraints maxStringLength as early as possible — populationgenomics/hail0 reactions
- Sep 20, 2024Bump the pip group with 4 updates — broadinstitute/variant-curation-portal0 reactions · 2 comments
- Jul 15, 2024chore(deps): bump the pip group across 2 directories with 15 updates — broadinstitute/gnomad-browser0 reactions · 4 comments
- Jul 5, 2024chore(deps): bump the pip group across 2 directories with 14 updates — broadinstitute/gnomad-browser0 reactions · 1 comment
- Jun 17, 2024chore(deps): bump the pip group across 3 directories with 14 updates — broadinstitute/gnomad-browser0 reactions · 1 comment
- Jun 11, 2024chore(deps): bump the pip group across 3 directories with 15 updates — broadinstitute/gnomad-browser0 reactions · 2 comments
- Jun 11, 2024chore(deps): bump the pip group across 3 directories with 14 updates — broadinstitute/gnomad-browser0 reactions · 1 comment
- Jun 10, 2024chore(deps): bump the pip group across 3 directories with 13 updates — broadinstitute/gnomad-browser0 reactions · 1 comment
- Jun 6, 2024chore(deps): bump the pip group in /data-pipeline with 12 updates — broadinstitute/gnomad-browser0 reactions · 1 comment
- May 14, 2024Fix syntax — broadinstitute/seqr-loading-pipelines0 reactions
- May 1, 2024Local_to_global: local alleles not strictly increasing: [0,2,1] — iris-garden/test-process0 reactions
- May 1, 2024Where to find data for practicing using Hail — iris-garden/test-process0 reactions
- Apr 30, 2024De novo confidence queries — iris-garden/test-process0 reactions
- Apr 30, 2024DNANexus: hail installation on custom spark cluster — iris-garden/test-process0 reactions
- Apr 30, 2024Hail architecture or design philosophy — iris-garden/test-process0 reactions
- Apr 30, 2024Assertion Error during entries().show() — iris-garden/test-process0 reactions
- Apr 30, 2024Dataproc connect should support --project command line option — iris-garden/test-process0 reactions
- Apr 30, 2024CollectionExpression join — iris-garden/test-process0 reactions
- Apr 30, 2024Use agg.linreg for GWAS — iris-garden/test-process0 reactions
- Apr 30, 2024Limit memory usage — iris-garden/test-process0 reactions
- Apr 30, 2024ArrayIndexOutOfBoundsException error in hail 0.2.40 — iris-garden/test-process0 reactions
- Apr 30, 2024Import_bgen 16 bits error — iris-garden/test-process0 reactions
- Apr 30, 2024VCF4.1, is.hail.utils.HailException: array index out of bounds — iris-garden/test-process0 reactions
- Apr 30, 2024How to get the ration transition-transversion on all dataset? — iris-garden/test-process0 reactions
- Apr 30, 2024GQ set to NA when PL is NA after split_multi_hts — iris-garden/test-process0 reactions
- Apr 30, 2024Download for tutorial notebooks — iris-garden/test-process0 reactions
- Apr 30, 2024Typo in impute_sex() description — iris-garden/test-process0 reactions
- Apr 30, 2024Hail V01 to V02 missing genotypes? — iris-garden/test-process0 reactions
- Apr 30, 2024Import_table on gs with wildcard — iris-garden/test-process0 reactions
- Apr 30, 2024Hail 0.2.10 patch notes — iris-garden/test-process0 reactions
- Apr 30, 2024Interval.parse in v0.2 — iris-garden/test-process0 reactions
- Apr 30, 2024Hail 0.2.6 patch notes — iris-garden/test-process0 reactions
- Apr 30, 2024Hail 0.2.5 patch notes — iris-garden/test-process0 reactions
- Apr 30, 2024Chained logistic regression — iris-garden/test-process0 reactions
- Apr 30, 2024[Breaking change] partition keys removed from python interface — iris-garden/test-process0 reactions
On Stack Overflow · 2
On Hacker News · 1
- Aug 28, 2020Hail.is: open-source library for exploring genomic data1 point
On Bluesky · 1
- Jan 12, 2024The Scalable Variant Call Representation: Enabling Genetic Analysis Beyond One Million Genomes www.biorxiv.org/content/10.1... 🧬🖥️… — @stephenturner.us7 likes and reposts
In this repository5 other files it gives its agents.
- hail/AGENTS.mdAGENTS.md
- hail/CLAUDE.mdCLAUDE.md
- flamegraphSkill
- hail-batchSkill
- release-changelogSkill
The file
We split this skill into the rules it gives an agent
9 rules. About 0 are common, 2 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 11
Read the matching runbook before starting one of these tasks, and follow it.
- Very specific: line 23
Never suppress stderr: when running hailctl or other Hail tools — important connection info and warnings appear there.
- Distinctive: line 86
If you get "no upstream", "connection refused", or "batch not found" errors, the most likely causes are:
- Very specific: line 91
Check with hailctl dev config list first before digging deeper.
- Distinctive: line 93
If the namespace has expired or doesn't exist, it needs to be started.
- Very specific: line 116
Prefer a native hailctl command when one exists.
- Very specific: line 137
If you find yourself making the same hailctl curl call repeatedly, it may be worth adding a native command (see *Enhancing the hailctl CLI* …
- Very specific: line 214
When users do hl.init(backend='batch'), Hail Query uses ServiceBackend to:
- Very specific: line 293
If you hit a gap in the CLI during an investigation (something you can only reach through the Python client or hailctl curl), it may be …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
You are helping a Hail service engineer investigate Hail Batch infrastructure or develop improvements to it.
For end-user batch investigation (logs, job status, writing scripts), also invoke /hail-batch.
Vetted runbooks
Read the matching runbook before starting one of these tasks, and follow it. These have been checked step by step against production; other docs in dev-docs/ may be out of date.
When you finish a runbook, report your results, then suggest a few ways to improve the next run: insights or hints that would have changed how this run went, rather than nits. When an improvement is a small, concrete change with a big payoff (e.g. one extra pattern in a grep), give the exact change.
| Task | Runbook | Last verified |
| Finding flaky CI tests and why they fail | dev-docs/services/flaky_tests.md | 2026-10-14 |
Running hailctl commands
hailctl is likely in a virtual environment — if it isn't found, look for one and activate it. Once activated, it stays active for the session.
Never suppress stderr when running hailctl or other Hail tools — important connection info and warnings appear there. Do not use 2>&1 or 2>/dev/null.
What this adds over /hail-batch
- Namespace/deploy config targeting (non-default namespaces)
- Service-level debugging (batch-driver, workers, CloudSQL)
- ServiceBackend internals (how Hail Query drives Batch)
- CI batch investigation
- Developing and testing changes to the batch service itself
Authentication
Same as end users, plus namespace-specific tokens:
gcloud auth application-default login # GCP ADC for cloud storage
hailctl auth login # Hail service authTo check which namespaces you have tokens for and confirm your current identity:
hailctl auth list # lists namespaces (* = current)
hailctl auth user # shows current user identityTo switch namespaces persistently:
hailctl dev config set default_namespace my-namespace
hailctl dev config list # show current settingsOr per-command with an env var:
HAIL_DEFAULT_NAMESPACE=my-namespace hailctl batch list
HAIL_DEFAULT_NAMESPACE=my-namespace hailctl batch get BATCH_IDFor engineers working across multiple namespaces regularly, config profiles are useful:
hailctl config profile create my-namespace
hailctl config set batch/billing_project my-project
hailctl config profile load my-namespace
hailctl config profile list # * marks active profile
hailctl config profile load default # switch backOr in Python:
from hailtop.config import DeployConfig
from hailtop.batch_client import client as bc
deploy_config = DeployConfig(
location='external',
domain='hail.is',
base_path='',
default_namespace='my-namespace',
)
with bc.BatchClient(billing_project='my-project', deploy_config=deploy_config) as client:
batch = client.get_batch(BATCH_ID)Troubleshooting connection errors
If you get "no upstream", "connection refused", or "batch not found" errors, the most likely causes are:
- The target namespace is not running (dev namespaces are ephemeral and may have expired)
hailctl dev configis pointing at the wrong namespace — runhailctl dev config listto check- The
HAIL_DEFAULT_NAMESPACEenv var is overriding your config unexpectedly
Check with hailctl dev config list first before digging deeper.
If the namespace has expired or doesn't exist, it needs to be started. There are two options:
Start from scratch (deploys services via CI into a fresh namespace — takes a while, returns a batch ID to track progress):
HAIL_DEFAULT_NAMESPACE=default hailctl dev deploy \
-b <github_user>/hail:<branch> \
-s deploy_batch,add_developers # adjust steps as neededThis prints a batch ID. Give the user the exact command to monitor it in a separate terminal:
hailctl batch wait <id>Then ask whether they'd like you to watch it too and verify the deployment looks good when it comes up (useful if they want to iterate on failures). Waiting can take 20-30 minutes, so only do it if the user explicitly asks.
Update a running namespace live (faster, redeploys a single service):
make -C <service> deploy NAMESPACE=<my-namespace> # e.g. make -C batch deploy NAMESPACE=chrislImportant: both commands affect live infrastructure. Do not run them without explicit confirmation from the user — either ask the user to run the command themselves, or show the exact command and wait for approval before executing.
Direct API access with hailctl curl
Prefer a native hailctl command when one exists. When it doesn't, hailctl curl calls any service endpoint and handles auth for you:
# hailctl curl NAMESPACE SERVICE PATH [curl args]
hailctl curl default ci /openapi.yaml -s -o ci-openapi.yaml # extra curl flags go after the pathDiscovering endpoints
Each service publishes an OpenAPI spec, which is the authoritative list of its endpoints (prefer it over any list in documentation):
hailctl curl default batch /openapi.yaml
hailctl curl default auth /openapi.yaml
hailctl curl default ci /openapi.yaml
hailctl curl default batch-driver /openapi.yaml
hailctl curl default monitoring /openapi.yamlSpecs list mutating operations (POST, DELETE, ...) right next to read-only ones. hailctl curl sends a GET by default; make any other call only with the user's explicit go-ahead.
If you find yourself making the same hailctl curl call repeatedly, it may be worth adding a native command (see Enhancing the hailctl CLI below).
When adding a new endpoint, update the service's openapi.yaml in the same PR. The specs are Jinja templates next to each service's other templates (e.g. batch/batch/front_end/templates/openapi.yaml, ci/ci/templates/openapi.yaml).
When an endpoint doesn't match the spec (wrong path, missing params, incorrect response shape), fix the spec too — a drift between implementation and spec is a bug.
Investigating with the CLI (prefer this for inspection)
The hailctl batch CLI works against any namespace via HAIL_DEFAULT_NAMESPACE. Prefer it for inspection — see /hail-batch for the full command reference.
To list jobs within a batch, use hailctl batch jobs — it pages through results for you:
hailctl batch jobs BATCH_ID --state bad # failed or errored jobs
hailctl batch jobs BATCH_ID --state live --limit 0 # everything still pending/running
hailctl batch jobs BATCH_ID --name test_batch_db -o json | jq '.[] | {job_id, state, exit_code}'If you need the raw API, use /api/v2alpha/.../jobs (what the CLI uses). Its q filter takes newline-separated terms like state=bad or exit_code=1. Don't send that syntax to the v1alpha endpoints: v1 reads key=value as a job attribute match, so state=failed silently returns no jobs (v1 wants a bare word like failed).
For anything the CLI can't reach (job attempts, raw DB state, worker-level details), drop into the Python client:
from hailtop.batch_client import client as bc
with bc.BatchClient(billing_project='my-project') as client:
job = client.get_job(BATCH_ID, JOB_ID)
attempts = job.attempts() # retry history
usage = job.resource_usage() # per-container resource metrics
status = job.status() # full status dict including worker nameOOM diagnosis
status = job.status()
# Worker-level OOM (job killed by cgroup, not user code):
oom = status.get('status', {}).get('main', {}).get('out_of_memory')
# Or use the helper:
from hailtop.batch_client.aioclient import Job
oom = Job._get_out_of_memory(status, 'main')Worker identification
worker = status.get('status', {}).get('main', {}).get('worker')
# Use this to find worker VM logs in Cloud LoggingWorker-level logs (GCP Cloud Logging)
Worker logs aren't in the batch API — go to Cloud Logging:
gcloud logging read \
'resource.type="gce_instance" resource.labels.instance_name="<worker-name>"' \
--project=<gcp-project> --limit=100Batch service architecture
batch: API server — handles client requests, validates, persists to MySQLbatch-driver: Schedules jobs, manages the state machine, provisions/deprovisions workers- workers: GCE VMs that pull and execute jobs
State machine:
Pending → Ready → Creating → Running → {Success, Failed, Error, Cancelled}Error = infrastructure failure (batch-driver or worker), not user code.
Key source directories:
batch/batch/— API server and driver Python codebatch/jvm-entryway/— Scala JVM worker entrypoint (for Query jobs)batch/worker/— Worker Python code
ServiceBackend internals
When users do hl.init(backend='batch'), Hail Query uses ServiceBackend to:
- Serialize the query IR to cloud storage
- Submit a Batch job via
hailtop.batch_client.aioclient.BatchClient - The JVM worker (
jvm-entryway) reads the IR, executes, writes results back - Python side reads results from cloud storage
Key source: hail/python/hail/backend/service_backend.py
To find the batch submitted by a Query operation, look for batches with names/attributes matching the IR token logged by the ServiceBackend.
Database
Batch state lives in MySQL (Google CloudSQL). Schema: batch database.
Key tables: batches, jobs, job_groups, attempts, resources, billing_projects
To connect (requires port-forward or make local-mysql):
kubectl -n <namespace> port-forward svc/db 3306:3306
mysql -h 127.0.0.1 -P 3306 -u batch -pbatch batchkubectl
# Pod logs
kubectl -n <namespace> logs -l app=batch --tail=200
kubectl -n <namespace> logs -l app=batch-driver --tail=200
# Pod status
kubectl -n <namespace> get pods -l app=batch
kubectl -n <namespace> get pods -l app=batch-driver
# Port-forward batch API
kubectl -n <namespace> port-forward svc/batch 5000:5000Developing changes to batch
Local dev loop
# Sync Python changes to running pods without rebuilding images
python3 devbin/sync.py \
--namespace <ns> \
--app batch --app batch-driver \
--path batch/batch /opt/venv/lib/python3.11/site-packages/
# Deploy a full image rebuild to a dev namespace
HAIL_DEFAULT_NAMESPACE=default hailctl dev deploy \
-b <github_user>/hail:<branch> \
-s deploy_batchRunning batch tests
# Integration tests against a deployed namespace
make -C hail pytest-qob NAMESPACE=default
# Batch service tests
make -C hail jvm-test # if touching JVM/Scala sideDB migrations
New migrations go in batch/sql/. They're applied by the *_database CI steps. For dev namespaces, apply manually:
mysql -h ... batch < batch/sql/your-migration.sqlCI batch investigation
CI submits batches to the ci billing project. To investigate a CI build failure:
- Find the batch ID from the CI UI or
ci/ci/ci.pylogs - Use
HAIL_DEFAULT_NAMESPACE=default hailctl batch get BATCH_ID - CI batches are large — filter to failed jobs and check their logs
- Job names correspond to step names in
build.yaml
Enhancing the hailctl CLI
If you hit a gap in the CLI during an investigation (something you can only reach through the Python client or hailctl curl), it may be worth adding the missing command. The CLI lives in hail/python/hailtop/hailctl/ and is straightforward to extend.
That said, CLI additions should be a separate PR from whatever you're currently working on — note it as a follow-up rather than doing it now.
Common dev-level failure patterns
| Symptom | Likely cause | Action |
Many Error jobs | batch-driver bug or worker crash | Check batch-driver pod logs |
Jobs stuck in Creating | Worker pool exhausted or GCE quota | Check batch-driver scheduling logs |
| Worker VM crash (OOM) | Worker itself OOM (not the job) | Cloud Logging for the worker VM |
| Auth failures in jobs | SA key missing/expired | Check create_test_gsa_keys CI step |
| ServiceBackend hangs | IR too large or batch not completing | Find the submitted batch; check driver logs |
make local-mysql fails | Port conflict or prior container | docker ps and kill the old container |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

