AGENTS.md

Guidance for AI agents working in this repository.

Working principles

• No overengineering, and no MVP shortcuts. Hold the middle path: don't build infrastructure before there's a concrete need (note the seam for later instead), and don't ship quick-and-dirty or "for now" hacks. Build each feature correctly and idiomatically — neither gold-plated nor a placeholder.
• MVP stage — keep the architecture fluid. The current structure is not load-bearing legacy. When a new feature doesn't fit cleanly, prefer reshaping the affected part over bolting on an awkward special case.
• Surgical changes. Clean up what your change orphaned; leave pre-existing dead code alone. Prefer a library's intended API over a clever shim.
• English only. All code, comments, identifiers, docs, and commits.

What this is

freehire (freehire.me) is an open-source IT job aggregator. Many source parsers feed a pipeline that normalizes jobs into one schema, deduplicates them, and enriches them with AI; served over an HTTP API with rich filters, consumed by a SvelteKit app under web/.

Stack: Go + Fiber v2, PostgreSQL, sqlc, Meilisearch, Docker Compose, langchaingo.

Layout

internal/<block>/<pkg>/ — every package sits in exactly one of eleven blocks, and the blocks form eight layers. A block may import only blocks strictly below it; two blocks sharing a layer may not see each other in either direction. The substantial packages carry their own AGENTS.md (see the table below), and each block carries one saying what it is and what it may reach for.

8. api           handlers, middleware, realtime, OG images
7. engage        notifications, digests, onboarding, referrals
   ingest        source adapters, crawl pipeline, link import, manual intake
6. application   tracking, stages, the event ledger, mail and calendar
   search        Meili topology, the drain, saved searches, intent
5. job           the posting: identity, facets, wire shape, dedup, ghost signal
4. candidate     CV, structured extraction, experience bank, PII, matching
3. ai            enrichment, embeddings, assistant, speech, plan limits
   identity      accounts, auth, profile
2. dict          the facet dictionaries and normalisation under them
1. platform      db, worker plumbing, outbox, cache, HTTP and LLM transport

The rule is enforced twice: depguard in .golangci.yml fails on the offending import line, and internal/platform/arch/layering holds the same table and reports the whole graph at once — including imports that exist only in test files, and only because run.build-tags names integration and llmlive. Without those tags both guards go blind over 222 files. Adding a package means adding it to the table in internal/platform/arch/layering/blocks.go; a package in neither fails the guard.

Two placements surprise people, and both are in the table's comments: the LLM client (platform/llm) is transport, not AI — it knows nothing about the domain, like safehttp; and moderator-authored vacancies plus the public submission queue are ingest, because they are manual job intake rather than applications.

cmd/<name>/ — every binary is a run-once-and-exit worker except cmd/server, cmd/mail-ingest (a long-lived SES inbound daemon, Restart=always), and cmd/auto-apply-orchestrate (serves the Inngest function's own HTTP callback protocol between steps, needs no DATABASE_URL). The rest are cron-driven, not daemons; they need DATABASE_URL and exit non-zero on failure.

Non-obvious:

• migrations/ — the source for both sqlc codegen and Postgres initdb. Never edit an applied migration; add a new file. pnpm check:sql now ENFORCES that: it fails when a change modifies, deletes or renames a migration main already holds, unless a commit in the change carries a Migration-Edit: <reason> trailer — for the rare edit that is right (a comment, a file that never ran anywhere). A migration the branch itself added is new at the merge base, so editing it is never caught. pnpm check:sql (squawk, via scripts/check-migrations.mjs) lints the ones a change ADDS — the applied history holds 1322 findings that stay, because rewriting an applied file is a worse hazard than any of them. Whether a file runs inside a transaction is a property of the FILE (the migrate: no-transaction marker, which must be a comment line of its OWN — quoted inside a sentence it does not count, and used to), so the check runs two passes; .squawk.toml carries the argument. Suppress a rule with -- squawk-ignore <rule-name> on the line before the statement, with the reason beside it.
• Dead JS code and unused dependencies: pnpm check:dead (knip, the dead-code CI job) gates everything knip reports — files, dependencies, binaries AND exports across web/, design-system/ and the root scripts. An exported name nothing imports fails the build, an exported type included; knip.config.js argues why the line ended up there rather than at the usual "report exports, do not gate them", and the two deliberate exemptions carry a JSDoc @public tag beside the code rather than an entry in the config. (This paragraph used to describe a narrower gate and a pnpm check:dead:exports script that no longer exists — the config is the current answer.) CI only, since knip needs both packages' dependencies installed and a fresh worktree usually has neither.
• Dead Go code: deadcode -test -tags=integration,llmlive ./... (the dead-code-go CI job), which builds the whole call graph and reports what no binary and no test can reach. Both flags are load-bearing. Without the tags it does not parse the build-tagged tests and calls their helpers dead — the same blindness .golangci.yml's run.build-tags documents for depguard. Without -test every test-only helper is dead. The residue it cannot see is a function whose only caller is its own test; that one still needs a reviewer. To run it locally: go install golang.org/x/tools/cmd/deadcode@v0.52.0 — with a toolchain at least as new as go.mod's go directive, or the binary's type checker rejects the module and reports "package requires newer Go version" instead of findings.
• .github/workflows/ — linted by actionlint (the workflows CI job), which also runs shellcheck over every run: block, and audited by zizmor --offline in the same job. Every uses: is pinned to a commit SHA with a # vX.Y.Z comment, which Dependabot moves together; a new action goes in the same way or zizmor fails. actions/checkout takes persist-credentials: false unless a later step pushes. A finding wrong for one line carries # zizmor: ignore[<audit>] with the reason beside it. Nothing else reads a workflow until GitHub runs it, so a mistake in one that fires on a tag or a schedule surfaces late.
• The instruction files are linted by agnix --strict (the docs CI job): an unclosed code fence turns the rest of an AGENTS.md into one code block, and that is invisible to a reader skimming the rendered file. .agnix.toml argues every rule it turns off.
• The public API contract is checked against main. scripts/check-openapi-breaking.sh (the api-breaking CI job) runs oasdiff over web/static/openapi.yaml and fails on a change that would break an existing client — a removed field, a narrowed bound, a weaker response promise. A deliberate break passes with an API-Break: <reason> commit trailer.
• The links in these documents are checked. pnpm check:links (the docs CI job and a pre-commit hook) resolves every relative Markdown link in the repository. The table below is the map an agent is meant to follow, so a link that no longer resolves does not just fail — it sends the reader somewhere confidently. The hook carries no glob on purpose: what breaks a link is renaming its target, usually a .go or .svelte file.
• What we hand to someone else is linted too (the artifacts CI job): web/static/install.sh is served at freehire.me/install.sh to be piped into sh, so shellcheck covers every tracked *.sh; both Dockerfiles go through hadolint (.hadolint.yaml argues its two exclusions, and a finding wrong for one line carries # hadolint ignore= beside it); and web/static/openapi.yaml is validated against the OpenAPI specification, since it is a static asset nothing else parses.
• There is no sources/ directory. The board catalog is the boards table (internal/ingest/boardcatalog) and the Telegram channel list is telegram_channels — both moved out of YAML, and nothing in the repository parses a board file any more. internal/ingest/sources is the adapters, not the catalog.
• design-system/ — a separate pnpm package, sibling to web/ and extension/, linked via pnpm's link:../design-system (web/) or npm's file:../design-system (extension/) — both are symlinks, not copies. Install it before building either consumer — neither package manager installs a linked/file: package's own dependencies for you. Being a symlink also means each of the three installs its OWN node_modules, so a package all of them depend on sits on disk more than once — free for a value, fatal for a TYPE, since TypeScript identifies a type by where it was declared and two copies of LucideIcon are two unrelated types sharing a name. pnpm check:deps (scripts/check-shared-deps.mjs, the shared-deps CI job) holds the three manifests to the same version for exactly the packages the design system imports as types — a set it DERIVES from design-system/src rather than listing, so a new type import joins the rule by existing. It reads no lockfile and compares the declared range literally: ^1.25.0 and ^1.45.0 overlap perfectly well and are the two ranges that put @lucide/svelte on disk twice (freehire#2836), so "the ranges are compatible" is the wrong question. peerDependencies is exempt — a library's peer range is deliberately wider than what it develops against. Dependabot cannot see any of this: it reads three package.json files as three unrelated projects, so it will keep proposing a bump to one of them, and that PR is correct once the other two move with it.
• extension/ — the browser extension (WXT + Svelte side-panel agent client), npm-managed unlike the rest of the JS in this repo. See extension/AGENTS.md.
• internal/platform/db/ — generated; edit internal/platform/db/queries/*.sql and run make sqlc. The pre-commit hook and the sqlc CI job regenerate and diff, so a query edited without regenerating no longer ships the old Go with every check green. Both use make sqlc, which holds the only version pin — a second pin would be a second answer, and the drift between them would look exactly like stale code.
• services/pii-filter — a standalone service, not a Go package.
• There is no deploy/ snapshot of the host any more. The systemd units, the nginx snippets, the operator scripts (release.sh, autodeploy.sh, gen-ingest-timers.sh) and the prose explaining them all live in the private freehire-ops repository, under provision/host2/ and scripts/host2/ — that is the only place any of them is edited, and provision/host2/AGENTS.md is the document to read about this machine. This repo carried a third copy of all of it until 2026-09-18. It drifted from the host in BOTH directions and was deleted rather than kept in step: measured against the running host, this repo's copies were the more accurate ones for the scripts (14 of 17), for most units and for all three hand-edited nginx files, while a Sentry gate merged here had never once run because the host had never heard of it. Both being partly right, with nothing able to say which half, is the whole argument for one copy. deploy/rollback/ stays: those are a migration's SQL, not host configuration.
• Nothing in freehire-ops deploys itself either. release.sh builds and flips the app; it never touches a unit, a snippet or a script. Editing one there is half the job — the other half is copying it to the host and running systemctl daemon-reload. Its own scripts/host2/drift-check.sh reports what has moved apart.

Commands

make up / make down / make logs       # start / stop / tail app + postgres in Docker
make run / make psql / make sqlc      # run server on host / psql into DB / regenerate internal/platform/db
make reindex                          # rebuild the Meilisearch jobs index from Postgres
go build ./...  &&  go vet ./...
go test ./...                             # unit tests (no external deps)
go vet -tags=integration ./...            # compiles the tagged tests — run before EVERY push
go test -tags=integration ./internal/platform/db/  # queue integration tests (needs Docker; testcontainers)
golangci-lint run                         # .golangci.yml; CI only fails on issues new-from-main (ratchet)

Before committing any *.go file: gofmt -w those paths (gofmt -l . must print nothing), then go vet ./... and go test ./.... Do not commit if they fail. Skip this suite when the commit has no Go. Integration-tagged tests stay push-time (go vet -tags=integration ./... before every push; the full tagged suite when behaviour changed).

Pre-commit hooks: lefthook (go install github.com/evilmartians/lefthook@latest, then lefthook install once per clone) runs gofmt/vet/golangci-lint on staged Go files and eslint on staged files in web/, extension/, design-system/ — the same ratchet policy as CI (only new issues fail the commit), so it won't block on the pre-existing backlog.

Secrets are the one thing the hooks prevent rather than report: gitleaks (brew install gitleaks) scans the staged index on every commit and fails hard if the binary is missing, because a scanner that quietly skips makes a commit look examined. The gitleaks workflow scans the whole history on top, since --no-verify walks past the hook and a credential removed in a later commit is still leaked. A finding that survives .gitleaks.toml means revoke the credential — rewriting history does not un-leak anything. Suppressions in that file allowlist a line shape, never a path: gitleaks ORs paths with regexes and prunes the file before reading it, so a path entry switches the scanner off for that file entirely.

go test ./... compiles no //go:build integration file, and those files are not confined to internal/platform/db — there are 187 of them across 20 packages, and internal/api/handler holds 78, which call unexported constructors like newCVHandlers. A changed signature therefore passes every command above except the vet line, then fails CI, which runs go test -tags=integration ./... over the whole module. The vet line is the cheap guard: seconds, no Docker. Run the full tagged suite when you change behaviour rather than a signature.

Worker gotchas (go run ./cmd/<name>, all need DATABASE_URL; run ls cmd/ for the full list):

• migrate — run before deploying code that reads new schema. -baseline records on-disk files without executing; a pre-runner database auto-baselines on first run.
• ingest — takes one PROVIDER NAME: go run ./cmd/ingest <provider> (or INGEST_PROVIDER), and reads that provider's live boards from the boards table. Handing it a file path is not an error — the path becomes the provider it looks up, which finds nothing, logs providers=0 and exits 0. INGEST_RUN_TIMEOUT_SECONDS (the unit's own timeout) gives the crawl a deadline INGEST_SWEEP_RESERVE_SECONDS (default 600) short of it, so the stale sweep still runs when the crawl overruns — the scheduler passes it on every launch; a static unit must set it itself, net of any slot wait, or its crawl stays unbounded and a timed-out run sweeps only in the 90s after SIGTERM. A run stopped early closes through the board scope only (see docs/agents/job-lifecycle.md (1c)). HYDRATION_RETRY_DAYS widens the window (default 14 days, measured from created_at) during which a stored row with NO description is withheld from the seen-set so a hydrating adapter re-attempts its detail fetch. Set it only for a deliberate repair run — HYDRATION_RETRY_DAYS=365 ingest hh re-offers every body-less row of that provider, one extra detail request each (hh's detail pages are ~1 MB), and expect to repeat it: a run that hits its timeout still persists what it hydrated and the fixed rows leave the set, so successive runs shrink the backlog. An unparseable or non-positive value fails the run rather than falling back, because a typo here would look exactly like a repair that found nothing. INGEST_REFETCH_ALL=1 empties the seen-set for the run, so every listed posting is treated as new and the pipeline re-WRITES it instead of only refreshing its liveness. That is the repair path for an adapter fix that changes what the LISTING yields (a mis-read remote flag, a location built from the wrong place): the fix reaches new postings on the next crawl and reaches the stored ones never, and backfill-derive cannot recover them either — it re-derives FROM the stored columns, which hold the wrong reading. Costs one detail request per stored posting (nofluffjobs: ~19k postings, ~20 min at 8 workers), re-enqueues only what actually changed for search, and re-enqueues nothing for enrichment (that queue is gated on enrichment_version, not on the write). A failed detail fetch cannot wipe a body — UpsertJob keeps the stored description when the incoming one is empty — so a run that hits its timeout is simply repeated. BODY_REFRESH_DAYS re-reads a stored body that has gone stale, so an employer's later EDIT reaches the catalogue at all: a hydrating adapter fetches detail only while the catalogue lacks the posting, so once stored a body is never read again and a posting that gains a "this position is 100% on-site" paragraph keeps whatever we read the day we first saw it (freehire#2555). It ships UNSET, which is disabled — turning it on costs one detail request per re-read posting against a crawl fleet that is already the host's bottleneck, so enable it one provider at a time and watch the load — in /opt/freehire/ingest-env/<provider>.env on the crawl host, which the scheduler hands each run of that provider after the shared .env (a value there wins; no file, no change). Putting it in the shared .env turns it on for the whole fleet. BODY_REFRESH_SLICE (default 30) bounds one run: a posting belongs to one slice for its life and a run takes the slice named by the day of the year, so a full sweep takes SLICE days and no run faces a provider's whole catalogue (1.27M postings on workday). Setting the slice without the days fails the run rather than doing nothing. What records a body's age is jobs.hydrated_at (migration 0144), written by both ingest write paths and NOT by the liveness-only TouchJob; updated_at cannot serve, because the cheap write deliberately leaves it alone and a body that came back identical is the common case.
• enrich / tg-extract — need LLM_BASE_URL / LLM_API_KEY / LLM_MODEL.
• embed / search-drain / rollup-facets / reindex-companies — need MEILI_URL / MEILI_MASTER_KEY. search-drain drains search_outbox (queued by cmd/ingest, atomically with each write) into the live facet index in batches — run it frequently (e.g. every 1-2 min); see internal/search/searchdrain/AGENTS.md.
• reindex — rebuilds the Meilisearch jobs index. REINDEX_DEDUP=1 additionally refreshes the duplicate markers (role clusters, aggregator suppression, fuzzy collapse) before the rebuild; without it the rebuild uses the markers the last dedup run left. Off by default since 2026-08-16: aggregator suppression alone measured ~23h against a 12h unit timeout, so the run was cancelled mid-dedup and never reached the rebuild — 3 days with zero successful reindexes. Run the dedup invocation on its own, rarer schedule. REINDEX_DEDUP_ONLY=1 runs just the marker passes and exits — no Meilisearch client, no disk guard, no rebuild, so no need to pause search-drain for it. It exists so the markers can refresh on a tighter cadence than the full REINDEX_DEDUP=1 invocation: the plain (no-flag) rebuild already runs every few hours on its own schedule and picks up whatever markers are current, so a fresher marker-only pass alone shrinks the window a repost sits undeduped in search. Takes precedence over REINDEX_DEDUP if both are set. The fuzzy pass works on REINDEX_FUZZY_WORKERS companies at once (default 4, strict reader) and RESUMES: it walks the companies in sorted order, saves the last one it finished (the oldest unfinished, never the newest — freehire#2869) in marker_pass_cursors as it goes, so a run its unit's timeout kills is picked up by the next one after that company, and a run that reaches the end clears the point so the next starts from the top — before this, every run walked the same head of ~383k companies into its 4h timeout and the tail was never deduplicated. A rebuild also writes the match sort's skill vectors (internal/dict/skillvec), which grows the index by roughly 10 GB and makes the rebuild materially slower — the cost is HNSW graph construction, so it lands here and not on the incremental search-drain pushes. Schedule it deliberately, and note the embedder must be in the LIVE index settings before a binary that queries it rolls out — see internal/search/search/AGENTS.md.
• backfill-derive — re-derives every deterministic column (facets, role_fingerprint, company_slug) in one keyset pass. It keeps a stored country and region (and a set work_mode) as the structured signal, because many adapters state the country and the pass cannot tell theirs from a derived one — re-deriving it moved SEEK's "Perth WA" rows to the US and blanked ~16k SEEK/JobStreet rows; a row with no stored country is re-derived in full, and correcting a stored one is INGEST_REFETCH_ALL=1's job. Never public_slug: that is a published URL minted once at insert, and re-deriving it from a title the source changed later 404'd every link to the posting — one run on 2026-10-19 moved 42,226 before it was stopped (freehire#3284). The job page resolves such a dead slug through its last eight characters (the shortcode, derived from (source, external_id) alone) and 301s to the current one; BACKFILL_CONCURRENCY tunes the pool. Follow with REINDEX_DEDUP=1 make reindex — that is what collapses newly-clustered reposts and unions their geography. It is RESUMABLE, and that is not a convenience: the pass walks 12.7M rows de-TOASTing a description each, which outlasts any sane unit timeout, and until freehire#2864 it had no resume point — afterID started at 0 every time. A hand-made transient taxonomy-backfill-night unit (systemd-run, so in /run and in nobody's git) therefore ran it nightly under TimeoutStartSec=10h, got killed about two thirds through, restarted at the first id the next night and never once finished — ~29h of CPU a night on the host whose bottleneck is the crawl fleet, while five merged dictionary fixes waited on it. BACKFILL_DERIVE_FROM_ID names the first id a run must DO — inclusive, like the sibling pass's BACKFILL_REQUIREMENTS_FROM_ID, because two knobs sharing a name and a shape while disagreeing about whether the id is done or to-do lose exactly one row per hop — and BACKFILL_DERIVE_MAX bounds how many rows it scans, unset for unbounded, never 0, the same strict-reader rule backfill-clearance documents. BACKFILL_DERIVE_CLOSED_WITHIN_DAYS narrows the scan to the rows that can still reach the catalogue — open postings, plus ones closed within that many days — which is what makes a dictionary fix affordable: measured 2026-09-23 the table holds ~12.7M rows against 1.9M open ones, so an unbounded pass spends ~85% of its ~30h on postings no search, sitemap or page will read. It is expressed in DAYS CLOSED and not as an open-only flag on purpose: a closed posting is not permanently gone, since ingest's Toucher refreshes liveness "(last_seen_at, reopen if closed) ... WITHOUT rewriting its content" and a posting that merely drifts out of a feed for 48h is routinely closed and reopened as it drifts back (see the notes in sources/seek.go and sources/whatjobs.go) — so a pass that skipped on closed_at IS NULL would hand those rows back to the catalogue carrying the OLD dictionary's facets, with nothing downstream reporting the staleness. The number is how far back a posting might still return from; unset keeps the whole table, so the cautious behaviour stays the default and the knob is the deliberate act. A run that stops early says so in as many words and prints the id to continue from — from ONE report every exit path reaches, because deciding that at each return is what made the cursor vanish twice in one evening: the scan failing is only one ending, and the companies reconcile that follows it is two more (freehire#2876); a run that reached the end prints done, so "there is more" and "that was all" are never inferred from a row count. The resume point is the oldest row the run did not FINISH, never the newest row its reader read — the two differ by up to a full page and that gap is silent data loss: jobsCh is buffered at backfillBatchSize, so the reader hands over a whole page and advances its keyset cursor past it long before a worker writes any of it, and the review that caught this measured a cursor of 500 while id 5 was still unwritten (freehire#2869). A row whose write failed is deliberately left unfinished so the resume point falls before it. Re-reading what an interrupted run had in flight costs nothing, since the write is changed-guarded. There is no timer — freehire-ops' provision/host2/systemd/freehire-backfill-derive.service ships without one, because systemd has nowhere to keep the cursor between runs and a schedule that cannot advance it re-derives the same first rows forever.
• backfill-clearance — one-off: fills jobs.requires_clearance for the rows that predate the column (migration 0119). It does NOT walk the catalogue: a description predicate de-TOASTs all 8M rows to find the ~38k that mention a clearance, and backfill-derive (which would also pick the column up) runs ~15h. Meilisearch already indexes the description, so the worker names its candidates there and reads only those bodies. Over-fetching candidates is free — the dictionary decides, and a declined row simply keeps NULL — so the queries are broad single tokens rather than a second, drifting copy of the phrase list. Idempotent (IS DISTINCT FROM-guarded), so a re-run writes nothing and stopping it is free; BACKFILL_CLEARANCE_MAX caps one run — leave it unset for unbounded, never 0. Every one-off pass's numeric knob now goes through one strict reader (worker.EnvInt64, or worker.EnvInt32 for a knob that must fit a narrower API like pgtype.Interval.Days — parsing directly at that bit size is what turns an out-of-range value into a refusal instead of a silent wraparound), which takes only positive integers because for the chunk-width knobs it also serves a 0 is an infinite loop; a set-but-unreadable value fails the run with the value named rather than quietly taking the default, which is the one outcome an operator who typed a number would never notice. Follow it with a full make reindex: incremental pushes only send documents whose content_hash moved and the new column is not in that hash, so without the rebuild the facet is empty for every pre-existing posting — the same trap is_tech fell into. Needs DATABASE_URL, MEILI_URL, MEILI_MASTER_KEY.
• backfill-company-type-hint — one-off: force-requeues the existing open postings of the companies in enrich.CompanyTypeHints (a small, hand-curated company_slug → outsource/outstaff map — there is no open dataset of IT outsourcing/outstaffing companies to import instead) back into enrichment_outbox, so the next cmd/enrich runs rewrite their company_type using the new "Known company type" prompt hint. A single job posting rarely states its employer's business model in so many words, so this field came back null for these companies far more often than the rest of the enum. Run once after adding an entry to the map — the ordinary EnqueuePendingJobs only re-queues rows BELOW the target enrichment version, which skips a company's postings already enriched under it. Idempotent (ON CONFLICT-guarded on (job_id, target_version)), so re-running before the drain finishes costs nothing. Follow the drain with a full make reindex: SetJobEnrichment never pushes to search_outbox, so the corrected values only reach Meilisearch on the next full rebuild — the same gap backfill-clearance documents. Needs DATABASE_URL.
• backfill-profession-it-tech — one-off: sets is_tech = true on the Profession itdev/itops rows stored before that source began asserting the signal at ingest (issue #2601). Profession crawls only these two dedicated IT category boards, so a posting filed under either is confirmed technical by the source's own filing even when the title dictionary resolves no category for it — but without the signal such a row's is_tech stayed unknown, EnqueuePendingJobs's is_tech IS TRUE gate never queued it for enrichment, and search.CategoryUnresolved excluded it from search forever, not just until the next enrichment cycle. A single UPDATE, not chunked: the affected set is bounded to two boards, unlike the multi-million-row backfills that need an id-range walk. Idempotent (IS DISTINCT FROM-guarded), so a re-run writes nothing and stopping it is free. Follow it with a full make reindex: is_tech is not part of content_hash, so an incremental push alone would never reach these pre-existing rows — the same trap backfill-clearance documents. Needs only DATABASE_URL.
• close-apploi-misattributed — one-off: closes every open apploi posting and retires the provider's 5833 boards, because what the catalogue holds about them is wrong and cannot be repaired in place. Measured 2026-09-15: apploi's API stopped honouring its employer parameter — employer=39092, employer=999999999 and no parameter at all return byte-identical pages — so every board fetched the same GLOBAL catalogue, and since the adapter attributes with firstNonEmpty(e.Company, j.BrandName) and all 5833 board rows carry a company, the real brand_name was never once stored. 1,565,701 rows stood for 3,024 real jobs (518 copies each, under 518 mostly-wrong employers), 264,022 of them reaching live search — 12.5% of everything a visitor could find. Repair is impossible (the correct employer is in no stored field, and there is no way to choose which of 518 companies to keep) and the adapter rewrite is not worth it: 5,074 of the 1.47M open rows are is_tech, 0.34%, which at 518 copies is on the order of ten real technical jobs. Soft close only, with its own closed_reason = 'source_misattributed' (migration 0165) — that label IS the rollback, being the only thing separating these rows from ordinarily-closed ones. Chunked over an id RANGE, paced, idempotent (closed_at IS NULL-guarded); CLOSE_APPLOI_CHUNK (default 50k) sets the span and CLOSE_APPLOI_FROM_ID resumes at the logged cursor. Deliberately does NOT push to search_delete_outbox, unlike every other close: 1.47M entries against a queue whose ordinary depth is ~7k would sit in front of every other Meilisearch task, and the rebuild reads open rows from Postgres anyway — the same trade merge-companies makes. Follow it with a reindex, or wait for the scheduled one; until then the postings stay searchable. Needs only DATABASE_URL.
• backfill-remote-perk-false-positive — one-off: corrects jobs.work_mode for the rows freehire#2696 mis-set to remote before #2710 fixed WorkModeFromDescription's "work from anywhere" false positive (a bounded travel/PTO perk phrase, not a work-arrangement statement). backfill-derive cannot reach these rows — it feeds a row's CURRENTLY STORED work_mode back in as the structured-signal input, so the old bug's output is indistinguishable from a real ATS signal and is carried forward unchanged — and ordinary re-ingest doesn't either, since a stored, still-open posting's description is never re-read (see BODY_REFRESH_DAYS above). Candidates come from Meilisearch (work_mode=remote + the text query "work from anywhere"), the same over-fetch-and-let-the-dictionary-decide shape backfill-clearance uses; the write recomputes work_mode from the row's stored location/description alone (no structured-signal input — the whole point is to stop trusting what's stored) and only touches rows where that recompute no longer says remote. A row whose current remote came from a real ATS signal that also happens to match the candidate query is a false correction in principle, but self-heals on its next ordinary crawl, which passes ingest the adapter's live signal directly; a genuinely wrongly-tagged row has no such backstop, which is why erring toward correction here is deliberate. Idempotent (IS DISTINCT FROM-guarded). Follow it with a full make reindex: work_mode is not part of content_hash, the same trap backfill-clearance documents. BACKFILL_REMOTE_PERK_MAX caps one run. Needs DATABASE_URL, MEILI_URL, MEILI_MASTER_KEY.
• backfill-remote-region-restriction — one-off: corrects jobs.countries/regions for the rows internal/dict/location.Parse's now-removed bare-remote-to-global default mis-set before the fix landed (found via /moderation?tab=reports, freehire job_reports 29-34: a region-restricted "Remote" posting — title [Remote-US], or a description stating "hiring in the US, Canada, Argentina, or Brazil only" — was showing as globally open). backfill-derive cannot reach these rows — it feeds a row's CURRENTLY STORED countries/regions back in as the structured-signal input, so the old bug's global is indistinguishable from a real ATS signal and is carried forward unchanged — and ordinary re-ingest doesn't either, since a stored, still-open posting's description is never re-read (see BODY_REFRESH_DAYS above). Candidates come from Meilisearch (regions=global + the text query "remote"), the same over-fetch-and-let-the-dictionary-decide shape backfill-clearance uses; the write recomputes geography via jobderive.Derive from the row's stored title/location/description alone (no structured-signal input — the whole point is to stop trusting what's stored) and only touches rows where that recompute no longer says bare global. A row whose current global came from a real ATS signal that also happens to match the candidate query is a false correction in principle, but self-heals on its next ordinary crawl, which passes ingest the adapter's live signal directly; a genuinely wrongly-tagged row has no such backstop, which is why erring toward correction here is deliberate. Idempotent (IS DISTINCT FROM-guarded). Follow it with a full make reindex: countries/regions are not part of content_hash, the same trap backfill-clearance documents. BACKFILL_REMOTE_REGION_MAX caps one run. Needs DATABASE_URL, MEILI_URL, MEILI_MASTER_KEY.
• backfill-telegram-prefilter — run it after any change to the Telegram prefilter's markers, which is what makes it recurring rather than one-off. cmd/tg-ingest decides a crawled post with telegram.AdmitsPost and, when it declines, InsertTelegramPost stamps extracted_at at INSERT time; nothing revisits that row. So a widened marker reaches NEW posts only, and the stored ones stay refused forever with nothing reporting it. Measured 2026-09-23 over the 15,203 posts the filter had declined: 96 (all job_it_junior) already matched the then-current markers and were merely crawled under an older set, and the Spanish empresa: marker that landed the same day left 6,335 more behind it — 42% of everything the filter had ever refused, real vacancies carrying a LinkedIn or Workday link. The pass keyset-walks the declined posts, re-asks the SAME rule the crawl uses — AdmitsPost, then the Jev vacancy judge when TYPESAFE_API_KEY is set — so it can never requeue what the next crawl would refuse, and clears extracted_at on the ones it admits. Reports by default; --apply writes — the markers are hand-maintained, and the per-channel breakdown in the report is where a widening that suddenly admits an unintended channel becomes visible. BACKFILL_TG_PREFILTER_MAX bounds one run, unset for unbounded, never 0, through the same strict reader every one-off knob uses; a bounded run prints the (channel, msg_id) to continue from as the exact environment to set, and a run that reached the end prints done, so "there is more" is never inferred from a count. BACKFILL_TG_PREFILTER_AFTER_CHANNEL / _AFTER_MSG_ID are what make that bound a bound rather than a trap: a post the filter still REFUSES never leaves the predicate, so a second bounded run starting from the top rescans exactly what the first one rejected — and a dry run, where nothing leaves the predicate at all, repeats its report forever. That is the failure backfill-derive carried until freehire#2864, where afterID started at 0 every time. The two are ONE cursor and the run refuses half of it, because a channel with no message id re-reads that channel whole while an id with no channel lands on the first channel alphabetically, and neither looks like a mistake in the output. They are EXCLUSIVE — the first row done is the one after the key — and named AFTER rather than FROM for exactly that reason: the sibling passes' BACKFILL_*_FROM_ID is inclusive, and two knobs sharing a name and a shape while disagreeing about whether the key is done or to-do lose one row per hop. Idempotent and free to interrupt: the UPDATE repeats the read's predicate, so a post already requeued — or claimed by cmd/tg-extract since this run read it — changes nothing. Needs no reindex: a requeued post reaches the catalogue through the extractor's ordinary write path. Needs only DATABASE_URL. See internal/ingest/telegram/AGENTS.md.
• rekey-gupy-boards — one-off: moves every gupy board from its numeric companyId to the career-page subdomains its postings live on, and the postings with it. Gupy's companyId listing has answered 404 since 2026-09-30 and the only public listing left is per career page (see the gupy traps in internal/ingest/sources/AGENTS.md). The companyId -> pages mapping is read from the stored posting URLs. Two passes, and the split is load-bearing: pass 1 rewrites each posting's external_id to <page>:<id> and retires the companyId board, one transaction per board (the row keeps its id, so the first crawl updates it instead of duplicating it); pass 2 — only when EVERY board moved — adds a pending board per page namespace the postings now carry. A page board added early could be crawled before a second companyId sharing that page moved (datalawyer is shared), and that move would then collide on (source, external_id) forever; so a failed board is logged, the rest continue, pass 2 is skipped and the run exits non-zero naming it. Pass 2 also asks Gupy about each page: an empty page's open postings close as feed_empty (no crawl can sweep a zero-yield board, and close-chronic-boards' empty-feed pass runs report-only on prod), a 404 page's as board_unreachable with no board added. Reports by default; --apply writes. Idempotent — a moved board is retired, a page with a live board is skipped. Needs no reindex (closes queue search_delete_outbox). Needs only DATABASE_URL.
• backfill-brazil-state-codes — one-off: re-derives the places (countries, regions, cities, metros) of the gupy/vagas/solides/inhire/quickin postings stored before internal/dict/location read a Brazilian state code as the state, so "Porto Alegre, RS, Brasil" stopped carrying Serbia beside Brazil (measured 2026-10-30: 13,936 open rows carried such a country, 13,672 read differently today). backfill-derive cannot reach them — it keeps a stored country — and a fresh crawl rewrites only new postings. Dropping the stored country is safe ONLY because none of those adapters states a structured one, so a source that does must never join its list. Candidates by SQL over jobs_source_idx (~29k rows, open and closed), by primary key after that; never moves a company slug, so no companies sync. Reports by default; --apply writes. Idempotent (IS DISTINCT FROM-guarded). Follow it with a full make reindex — none of the columns is in content_hash. Needs only DATABASE_URL.
• backfill-slug-folded — one-off: fills jobs.company_slug_folded for the rows that predate the column (migration 0109). Chunked, paced, and idempotent — the chunk UPDATE is IS DISTINCT FROM-guarded, so a re-run writes nothing and stopping it mid-way is free. Until it completes the aggregator-suppression pass simply matches fewer rows, never wrong ones. Needs only DATABASE_URL.
• backfill-semantic-input-hash — one-off: fills jobs.semantic_embedded_input_hash (migration 0192) for the jobs embedded before it existed. cmd/embed re-embeds a job only when that hash of the text it embeds (title, company, description) moved, not merely its content_hash, which also moves on every derived facet — the 2026-10-21 dictionary/geo deploys queued ~92k unchanged texts for a TEI re-embed that way. A job with no stored hash cannot be told apart and re-embeds, so run this after the migration lands, or the next dictionary deploy repeats the wave. Fills only jobs whose stamp is current (a stale one's text may have moved). Chunked over an id RANGE (BACKFILL_SEMANTIC_INPUT_HASH_CHUNK, default 1M ids), paced, idempotent (IS NULL-guarded). Needs no reindex. Needs only DATABASE_URL. cmd/embed itself reads EMBED_MAX_RUN_SECONDS: set it on the hourly unit, below its TimeoutStartSec, so a backlog run stops claiming and exits 0 instead of being killed mid-wave — see internal/ai/embed/AGENTS.md.
• backfill-requirements — one-off: fills jobs.requirements_derived for the OPEN postings that predate the column (migration 0139) by re-reading each description with internal/job/reqextract. Deliberately NOT folded into backfill-derive: that pass walks all ~11M rows for ~15h and rewrites every derived column, while this walks the ~4.6M open ones and writes one. Chunked over an id RANGE, paced, and idempotent — the chunk UPDATE is IS DISTINCT FROM-guarded, so a re-run writes nothing and stopping it mid-way is free. BACKFILL_REQUIREMENTS_CHUNK (default 5k, deliberately narrower than the slug pass's 50k because this de-TOASTs a description per row) sets the id span per chunk, BACKFILL_REQUIREMENTS_MAX (default 200k) caps one run, and BACKFILL_REQUIREMENTS_FROM_ID resumes at the id the previous run logged — a row that yields nothing keeps the '[]' it already had, so it is indistinguishable from an unvisited one and only that cursor stops a re-run re-reading it. Needs no reindex: the job page reads the row from Postgres and this field is never a filter, so the is_tech/requires_clearance trap does not apply. Keep concurrency at 1 — BACKFILL_CONCURRENCY=6 has degraded prod before and this is the same shape of load. Needs only DATABASE_URL.
• backfill-experience-dates — one-off: fills experience_employments' structured period_start_year/period_start_month/period_end_year/period_end_month (migration 0135) by parsing the free-text period_start/period_end columns they replace (see internal/candidate/perioddate). A label that parses fills exactly what it says; an empty or "Present"-reading label is left unset (a real absence, not a parse failure); genuinely garbled text falls back to the row's own created_at year. Idempotent — SetExperienceEmploymentBackfilledDates only touches rows still missing both structured periods, so a re-run (or the ordinary write paths, once deployed) never overwrites a filled row. Run once after migration 0135 lands and before the follow-up migration drops the old text columns. Needs only DATABASE_URL.
• backfill-duplicate-marker-owner — one-off: seeds jobs.duplicate_of_{aggregator,role,fuzzy} from the single duplicate_of that predates them (migration 0114). Runs BETWEEN migrations 0114 and 0115 — 0115 derives duplicate_of from those three, and a derivation over three empty columns would clear every marker in the catalogue. Chunked over an id RANGE, paced, and idempotent: a row that already has an owned column set no longer matches, so a re-run writes nothing, stopping mid-way is free, and re-running it after 0115 lands IS the reconcile sweep for rows written while it was walking. BACKFILL_MARKER_CHUNK (default 50k) widens the id span per statement — the id sequence runs far ahead of the live row count, so most chunks cover empty stretches. Provenance is not recoverable from a stored marker, so the seed goes by shape and fuzzy markers land in the role column; the first marker refresh after deploy corrects that in one cycle. Needs only DATABASE_URL.
• backfill-talent-handle — one-off: mints the Talent Network catalogue handle for the members who joined before handles existed. Migration 0148 rewrote every retired public row to anonymous and 0149 added the column, but neither could mint: a handle's readable part is the category a job title resolves to through internal/dict/classify, which SQL cannot reach. Those accounts came out of the migrations as members with NO public address — ListTalentNetworkMembers requires talent_handle IS NOT NULL, so they are absent from the catalogue and their card 404s until they happen to toggle the control again. Not chunked: the population is bounded by how many people opted in while the control was unreachable from the account navigation. Idempotent (the claim is talent_handle IS NULL-guarded), so a re-run mints nothing and stopping it mid-way is free; a per-account failure is counted and stepped over. Run it AFTER the deploy, not before — it calls the same talentnetwork.Join the handler does. Needs no reindex: the catalogue is served from Postgres. Needs only DATABASE_URL.
• backfill-company-info-wikipedia — periodic (daily, freehire-backfill-company-info-wikipedia.timer), unlike the backfills above it: fills companies.tagline/company_info for companies no curated company-info source has ever matched (the YC directory, an external company-info dump — see openspec/specs/company-info/spec.md), a population that keeps growing as ATS crawling discovers new companies rather than being a fixed historical backlog. Looks each eligible company up against Wikidata (internal/job/wikicompany, no API key, no billing) and accepts a match only when the resolved entity's own type data — walked transitively via wdt:P31/wdt:P279*, both a positive business/organization walk AND a negative geographic/administrative-entity exclusion (a same-named commune or country is rejected even when Wikidata's multi-parent hierarchy also places it under organization — found in production, not the design spike: Lookup("Nissan") first resolved to a French commune), never a keyword scan of the description. Reports by default; --apply writes. WIKIPEDIA_BACKFILL_MAX_PER_RUN (default 500, the deployed timer sets 3000) bounds one run — daily rather than the design's original "e.g. monthly" guess, once rollout measured the real backlog (~341k companies, 2026-09-09) and the real throughput (~3s/candidate against Wikidata's public endpoints); monthly would take years to make a dent. A per-company failure (a slow or erroring Wikidata request) is counted and stepped over, never fatal to the run, and is retried on the next one — the company is not marked checked. Idempotent by construction: companies.company_info_wikipedia_checked_at (migration 0154) records that a company was resolved — matched or rejected — so it is never looked up twice, the same shape jobs.hydrated_at gives ingest. Never overwrites a tagline/company_info key another source already set. Needs no reindex: neither field is part of content_hash or a Meilisearch facet. Needs only DATABASE_URL.
• publish-logo-domains — daily: writes the company-name-to-domain map logo.freehire.me consults, so a logo resolves by the employer's own domain instead of by a name string. Company logos are requested by NAME (web/src/lib/logo.ts), taken verbatim from whichever adapter ingested the row, and one employer arrives spelled several ways — g2i, G2i and G2i Inc. all carry company_slug = 'g2i', and the first two resolved to an unrelated leadership-training firm at HTTP 200, so no consumer could detect it and fall back to its monogram. The proxy has accepted ?domain= since it was written and folds it into its cache key (so a name that gains a domain gets a fresh key and the stale rendition simply goes cold — nothing to bust); nothing ever supplied one. The fix is deliberately not at the call sites: of ~25 places that draw a logo, roughly fifteen never see a company slug (the experience bank, search suggestions, community subjects, the tailored-CV list), so threading a domain would fix a third of the surfaces and leave the rest guessing. Covers the 17,859 of 239,879 job-carrying companies whose company_info records a website; the rest keep today's name guess, and backfill-company-info-wikipedia grows that population daily for free. A name two DIFFERENT companies share while disagreeing about the domain is dropped, never resolved — a wrong domain returns a confident logo belonging to someone else, which is the failure this removes. The spellings query is a deliberate sequential scan (53s, 412,648 rows): joining it to the companies that have a website is four times slower (200s), because the narrow plan drives 17,859 index searches fetching ~102 heap rows each at random — 1.59M blocks of random I/O against the seq scan's sequential read of the same heap. Fewer rows, more work. A run that would publish an empty map refuses to swap, the rule cmd/build-suggestions already follows. Needs DATABASE_URL; without LOGO_DOMAIN_MAP_OUT it is a no-op that never opens the pool, which is also the rollback. The consumer is a different repository (freehire-logo, whose internal/domainmap reads the file) that release.sh does not carry and that is copied to the host by hand — so this half alone changes nothing visible, and a half-delivered change looks exactly like a working one.
• capture-apply-form — drains the apply-form capture queue: fetches each queued posting's application form from greenhouse/ashby/workable/lever and stores it in apply_forms. Needs nothing but DATABASE_URL; APPLY_FORM_CONCURRENCY (default 4) bounds how hard one run leans on a platform and APPLY_FORM_MAX_PER_RUN (default 5000) how much of the backlog it takes — the second matters because the first drain faces a ~185k backlog and an unbounded run would work for hours, which Type=oneshot turns into silently skipped timer firings. recruitee forms never reach this queue — its listing carries them, so ingest writes them directly.
• auto-apply — drains auto_apply_queue: resolves a candidate's known profile answers against a job's live application form via a headless Chrome (internal/api/atsapply, chromedp — no second language or process) and submits only when every required question is answered; otherwise it touches nothing on the page. Needs DATABASE_URL, a Chrome/Chromium binary on the host, and (to fill the résumé field at all) a TYPST_BIN the CV renderer resolves — an entry only ever reaches this worker with an approved tailored CV attached (openspec/changes/auto-apply-tailored-resume), rendered on demand to a temp PDF; nothing else it fills needs S3_*, which stays read-but-not-required. The queue is populated by the job page's auto-apply button and by the signed-in MCP server's auto_apply_start tool (behind the account's agents_may_apply switch, openspec/changes/auto-apply-over-mcp); every queue UPDATE lands in auto_apply_events through a trigger, while the deletes (submission, cancel) and a retry write their own event in the same statement — a new delete path must too — and a stopped attempt is re-armed by the candidate's retry, never by this worker. See internal/application/autoapply/AGENTS.md and internal/api/atsapply/AGENTS.md.
• adzuna-liveness — closes the adzuna and wellfound postings the source itself has taken down (it began with adzuna and kept the name, since the binary and its crawl-host unit are deployed under it). Neither existing close reaches them: Adzuna's feed is far deeper than the crawl's request budget (157k US it-jobs postings in 14 days, 750 read per run), so "unseen" there is not evidence, and cmd/liveness's age rule is a guess Adzuna's own retention defeats (57% of its live US feed is older than 21 days). Adzuna's hosted /details/<id> page IS evidence — 404/410 for a removed posting, 200 with a JobPosting block for a live one — but sits behind a challenge about nine requests in ten fail, so each posting is retried from up to 25 exit IPs through the Scrapling tier's rotating proxy. A run samples ADZUNA_LIVENESS_MAX_PER_RUN (default 2000) open postings unseen for 3+ days in RANDOM order, because an unanswered probe leaves the row untouched and a fixed order would hand every run the same unanswered rows. Closes on ONE 404/410 (closed_reason = 'probe_expired'): it is Adzuna's own statement, and a posting the crawl lists again is reopened by it. Runs on the crawl host, where Scrapling listens on loopback — not on fsn beside cmd/liveness; without SCRAPLING_FETCH_URL it is a no-op that never opens the pool. First measured run (2026-10-23, 50 postings): 41 closed, 7 live, 2 unanswered, 98 s. Wellfound runs in the same worker, side by side with adzuna: its role-slice crawl is swept by company, so a posting whose company left every slice is out of every run's scope (measured 2026-10-30: 954 open postings unseen since mid-September, 20 of 20 sampled answering 410 on Wellfound's own page). Its page is the STORED url and never one rebuilt from the id — Wellfound answers 404 for a live posting asked for without its title slug — so a row whose url does not name exactly its own posting id is skipped. One request in flight, like the crawl's own cap, at most WELLFOUND_LIVENESS_MAX_PER_RUN (default 1000) a run.
• hydrate-adzuna-description — drains the Adzuna full-description capture queue (cmd/ingest enqueues eligible postings; see internal/ingest/sources/AGENTS.md). ADZUNA_DESCRIPTION_MAX_PER_RUN (default 500) is deliberately conservative — untested against Adzuna at real crawl-host volume. seed-adzuna-description-queue is the one-off companion that queues the pre-existing backlog; run it once, then let the cron drain handle the rest.
• build-suggestions — rebuilds the search box's suggestion dictionary: the posting titles the catalogue actually carries, plus the skill/category/company vocabularies, each with its open-posting count and how often visitors have searched for it. Needs DATABASE_URL, MEILI_URL, MEILI_MASTER_KEY; SUGGEST_TITLE_FLOOR (default 25) and SUGGEST_COMPANY_FLOOR (default 5) decide how much of the long tail becomes vocabulary. It writes a SEPARATE index (suggestions) through the same rebuild-and-swap the jobs index uses, so it must not be scheduled on top of freehire-reindexw — Meilisearch runs one serial task queue, and a swap queued behind a 2.5h rebuild holds a Type=oneshot unit open and looks like a hang. Daily at 06:45 UTC, in the clear window between reindex runs. A run that would produce an EMPTY dictionary refuses to swap: a catalogue yielding nothing is a failed measurement, not an empty catalogue. See internal/search/AGENTS.md.
• llm-probe — asks every model alias this deployment routes through a trivial question three times each and publishes what came back as Prometheus gauges (freehire_llm_probe_{attempts,ok,slowest_seconds}, one series per alias, labelled model) via the node_exporter textfile collector. Every 5 minutes. Needs LLM_BASE_URL/LLM_API_KEY/LLM_MODEL, reads ASSISTANT_MODEL and SEARCH_INTENT_MODEL beside them (each falling back to LLM_MODEL, as cmd/server does, and deduped so an unset one is not probed twice), and is a no-op that spends nothing without PROM_TEXTFILE_DIR. Watching one alias was not enough: measured 2026-09-09, flagship answers from z.ai and carries the assistant, the fit analysis and enrichment, while fast answers from Gemini and carries the AI search filter — a dead fast is a 500 on every AI search, and the old single-alias probe would have gone on reporting the gateway healthy. It exists because nothing measured what a provider ANSWERS: three times in the week to 2026-09-08 a provider behind the gateway stopped serving while the gateway reported it active — Cerebras on an empty wallet (402), then two Z.ai accounts, one under a Fair Usage Policy (429) and one with a revoked key (401) — and each was found by a person noticing a broken feature. It probes the ALIAS, not individual keys: only the gateway host holds the raw keys (the admin API returns them masked), so it cannot name which key is dead, only what share of requests are served at all. A failing alias exits 0 — a red unit meaning "the thing I watch is broken" is indistinguishable from one meaning "I am broken". See internal/platform/worker/AGENTS.md.
• queue-metrics — measures outbox depth, board-fleet health, catalogue freshness, and employment-type boilerplate (freehire_employment_boilerplate), and publishes them as Prometheus gauges via the node_exporter textfile collector. Per-source data quality (freehire_source_quality_gap{source,check} — open postings with no work mode, no geography, no company, unseen 7+ days, posted 60+ days ago) is published here too but MEASURED by cmd/rollup-stats, in the per-source scan it already runs every 3 hours: that scan reads the whole open catalogue and does not fit a one-minute worker, so this reads the source_stats snapshot and publishes its age beside it (freehire_source_stats_measured_timestamp_seconds) — a stopped rollup shows as an ageing snapshot, not as frozen counts. Needs DATABASE_URL, and is a no-op that never opens the pool without PROM_TEXTFILE_DIR. Read-only and lock-free by design — run it every minute; see internal/platform/worker/AGENTS.md for the published names.
• quality-audit — weekly, MEASURE ONLY: checks a sample of the catalogue against the world, the layer above source-quality-metrics (which counts what is missing; this counts what is wrong). Liveness: for every source with source_stats.open_jobs >= 500 it probes ~30 random open postings' stored URLs and counts how many the source answers gone (404/410) or live (2xx on the posting's own URL) — reading the STATUS only, because the shared body classifier read arbeitnow's live pages as closed. A verdict counts only on a host that first 404'd a canary (the URL with its posting id rewritten to one that cannot exist), so a source answering 200 for any id (jobtech) or walled off (adzuna, wellfound) reads as unmeasured, never 0% dead; hh/djinni/torre keep CLOSED postings at 200 and are declared unmeasurable outright (statusBlind). robots.txt and Crawl-delay are honoured, one request at a time per host spaced 2s, a 403/429 ends the host, and QUALITY_AUDIT_MAX_REQUESTS (default 15000) bounds the run. Remote: ~200 postings tagged remote a week (QUALITY_AUDIT_REMOTE_MAX, _PER_SOURCE default 20; sources drawn weighted by their remote count) are read by an LLM judge on the SERVICE credential (feature: quality-audit) that must quote the posting or answer unclear; judgements land in remote_audit_judgements and the gauges cover 56 days. Publishes freehire_audit_{liveness,remote,global}_*{source} and freehire_audit_measured_timestamp_seconds{check} to freehire-quality-audit.prom — raw counts, the alert owns thresholds, and live + dead = 0 means unmeasurable. Never closes or retags anything. A no-op that never opens the pool without PROM_TEXTFILE_DIR; -dry-run runs both passes, stores nothing and prints the exposition. Needs DATABASE_URL and LLM_* (without them the remote pass is skipped). See internal/job/qualityaudit and openspec/changes/add-sampled-quality-audit/design.md.
• search-settings-drift — checks whether the live Meilisearch jobs and companies indexes carry every sortable attribute, filterable attribute, and embedder the deployed binary may ask for, and publishes the gap count (freehire_search_settings_drift_count) via the node_exporter textfile collector. Every 5 minutes. It exists because a sortable/filterable attribute or an embedder rolled out in the binary before its settings patch reaches the live index turns every request for it into a Meili 400, which the search package maps to a 500 for every caller of the affected sort/filter/match ranking — not only the one who picked the new value, since a settings patch replaces sortableAttributes/filterableAttributes wholesale (see internal/search/search/AGENTS.md, "Adding a sortable/filterable attribute" and "Skill vectors and the match sort"). It does not fix the ordering — settings-before-binary is still a human decision — it makes a gap visible on a schedule instead of by a caller hitting the 500 first. Read-only; it only reports the hazardous direction (a live index still declaring something the binary no longer asks for is never drift). Needs MEILI_URL/MEILI_MASTER_KEY, and is a no-op that opens no Meilisearch connection without PROM_TEXTFILE_DIR or MEILI_MASTER_KEY.
• serve-probe — samples ~500 random open postings (SERVE_PROBE_SAMPLE) and checks that every field a posting STORES reaches what it SERVES, publishing freehire_serve_probe_gap{layer,field} beside its denominator freehire_serve_probe_stored via the node_exporter textfile collector. Hourly. It exists because the opposite failure is silent: an empty value is omitted from the JSON and every endpoint still answers 200, so the ATS salary that only the enrichment write copied into the payload went unserved on 872,117 of 891,859 open postings until a user asked (freehire#3964). Two layers, read differently: projection runs jobview.FromRow over the row — a gap there is a code bug; index reads the live Meilisearch document — a gap there with none in projection means a reindex is owed. A posting with no document is counted apart, never as a field gap, under the reason it has none (freehire_serve_probe_unindexed{reason}): one of search.WithheldBy's rules — the same predicate every index writer asks — or missing when none holds, which is the only one that means the index LOST a posting (or the drain has not reached it yet). Measured 2026-10-31 over 2,000 rows: 798 had no document, 699 for no category, 97 unseen, 1 no body, 1 missing — so the unsplit count read as a 40% loss while the real one was zero. A new served column belongs in internal/search/serveprobe's rule table. A gap exits 0, the same rule search-settings-drift follows. Needs DATABASE_URL, MEILI_URL/MEILI_MASTER_KEY; a no-op that opens nothing without PROM_TEXTFILE_DIR or MEILI_MASTER_KEY. First run on prod (2026-10-31, 500 postings): projection 0 gaps; index salary 23/27, employment_type 1/103.
• search-ping — announces the newest job pages to the external search engines that accept being TOLD rather than waiting to be crawled: Google's Indexing API and IndexNow (Bing/Yandex/Seznam/Naver). Hourly. It exists because waiting does not work at this catalogue's size — measured 2026-09-14, Googlebot fetched 65 distinct job pages in a day against ~690k in the sitemap, and Search Console's URL Inspection answers URL is unknown to Google for postings sampled from it (pages Google has never once fetched) while the site publishes ~14k new technical postings daily. The budget is the design constraint, not the plumbing. Google grants 200 publish calls a day by default, so this is deliberately a LEDGER (job_search_pings, migration 0162) and not an outbox: an outbox fed by cmd/ingest would grow without bound forever and its oldest row would never be reached, while recording what was SENT lets each run choose the newest eligible postings afresh. The day's remaining allowance is read back from that ledger because the API offers no way to ask — and the day is measured to midnight Pacific, where Google resets it, not UTC. Eligibility is what the search index admits — open, canonical, public, with a body and a resolved category, technical or NOT — and lives in the SQL beside the query, because the API's terms admit only JobPosting and BroadcastEvent pages and the penalty for anything else is the quota. is_tech orders the budget, it does not gate it: each run asks for the technical tier first and offers the postings beyond IT (healthcare, retail, logistics and the rest) only what that left of Google's day, so at 200/day the second tier waits until technical intake no longer fills it; IndexNow, unbounded, takes a full batch of each tier every run. Enrichment, embeddings and the social digest stay is_tech-gated — they cost money or speak for the brand. Company pages are announced too, and only through IndexNow — measured 2026-09-15 via the Bing Webmaster API, Bing holds 255,038 of this site's pages and its highest-impression ones are /companies/<slug>, answering the query shape "<company name> careers"; Google's own August data agrees, and bingbot crawls company pages half again as often as job pages. Google's Indexing API may NOT be told about them — it admits only JobPosting and BroadcastEvent, and the penalty for anything else is the quota — so Engine.Accepts declines that pass for Google and the runner skips it without reporting it. Eligibility is companies.job_count > 0, the same gate the sitemap uses; their ledger is company_search_pings (migration 0164) with no kind column, because a company page has one event and its URL survives its postings closing. Each posting is announced twice — once when it appears and once after it closes, both as URL_UPDATED: a closed posting's page stays at HTTP 200 with its JobPosting markup and a validThrough moved into the past, which is one of the three ways Google documents for retiring a posting, so a closure needs a re-crawl and URL_DELETED would be a misuse of the API against a live page. The two events SHARE one engine's daily allowance and new postings take it first — a new posting brings a visitor, a closure only tidies an index we do not own — so at 200/day the closure pass does nothing until the quota grows, which is correct. SEARCH_PING_BATCH (default 200) bounds one run per engine and per event (each tier of new postings counting as one); GOOGLE_INDEXING_DAILY_BUDGET (default 200) tracks an approved increase. An engine with no credential is skipped without error, and with neither configured the run is a no-op that never opens the pool — which is also the rollback. Google needs GOOGLE_INDEXING_KEY_FILE (a service account key ON THE HOST, not in .env, which cannot hold a multi-line value; the account must be an owner in Search Console, not merely a full user — a full user gets Permission denied. Failed to verify the URL ownership.). IndexNow needs INDEXNOW_KEY, whose matching web/static/<key>.txt is public by design; the worker fetches that file once at startup and refuses to run if the two have drifted, because IndexNow's own answer to a mismatch is a 403 on every submission. -dry-run prints the exact URLs and sends nothing. See internal/job/searchping/AGENTS.md.
• billing-sync — reconciles Pro subscriptions against Stripe: applies billing_events the webhook recorded but could not apply, then re-reads subscription state for accounts whose pro_until is within a day of lapsing. Hourly, and not optional — the provider retries a webhook for up to three days and can disable an endpoint it decides is broken sooner than that, and past either point this worker is the only path by which a paid subscription becomes Pro. Needs DATABASE_URL and the STRIPE_* credentials; without them it is a no-op that never opens the pool. BILLING_SYNC_MAX_PER_RUN (default 500) bounds one run. The second pass reaches candidates through users.stripe_customer_id, so it walks the accounts that have actually transacted rather than every account. A third pass settles referral rewards — it grants the ones whose invitee has an invoice that actually collected (amount_paid > 0, never merely an active subscription) and places the granted ones on their referrers' balances, creating a provider customer for a referrer who has never bought. Stripe only: a store subscription produces no invoice we can read. INVITE_REWARD_MAX_PER_USER (default 12) bounds how many rewards one referrer may ever earn, and an unparseable value falls back and logs rather than stopping the run that also reconciles subscriptions. See internal/identity/billing/AGENTS.md and internal/identity/promo/AGENTS.md.
• discord-sync — keeps the paid role on the community Discord server in step with the subscription: for each linked account it resolves the tier the ordinary way (plan.TierOf over users.pro_until/ultra_until — never a payment provider, which would be wrong for every app-store subscriber) and grants or revokes ONE role. Hourly. Needs DATABASE_URL and all five DISCORD_CLIENT_ID / DISCORD_CLIENT_SECRET / DISCORD_BOT_TOKEN / DISCORD_GUILD_ID / DISCORD_PAID_ROLE_ID; any one missing and it is a no-op that never opens the pool, which is both how the feature ships before the Discord application exists and how it is rolled back. Not a pass inside cmd/billing-sync even though it reconciles the same subscriptions on the same schedule: billing is identity (layer 3), this is engage (layer 7), and the layering guard forbids the import. DISCORD_SYNC_MAX_PER_RUN (default 500) bounds one run and loses nothing — the page is ordered by least-recently-synced, so a run that cannot reach everybody leaves the rest at the front of the queue. A per-account failure is counted and stepped over; only failing to read the work at all exits non-zero. Which channels the role opens, and whether the bot may move it, are configured by hand in Discord — see freehire-ops' provision/host2/AGENTS.md and internal/engage/discordlink/AGENTS.md.
• pro-welcome-mail — welcomes an account the first time it becomes entitled to a paying tier (Pro or Ultra) with a one-time, personal email from the founder (Reply-To a human inbox, not the sending address — the same tone internal/engage/onboarding's signup sequence uses, not the machine-toned digest mailer). Every 10 minutes, deliberately tighter than billing-sync/discord-sync's hourly cadence: a purchase confirmation that arrives an hour late reads as broken in a way a Discord role does not. Resolves the tier the same way discord-sync does — plan.TierOf over users.pro_until/ultra_until, never a payment provider — and for the same layering reason lives in internal/engage/prowelcome rather than as a pass inside cmd/billing-sync: billing is identity (layer 3), this is engage (layer 7), and the layering guard forbids the import. Idempotent on users.pro_welcome_sent_at (nullable, stamped only after a successful send, never cleared) — a renewal, a tier change, or a lapse-and-resubscribe never re-sends it, which is deliberate: a returning subscriber is not a new one. A failed send is counted and stepped over, leaving the stamp unset so the next run retries — the opposite of onboarding's own delivery, which burns the slot on a failed send because that sequence is a courtesy and this one is a payment confirmation. Needs DATABASE_URL, AWS_REGION/NOTIFY_EMAIL_FROM (the SES transport — a no-op that never opens the pool without them) and ONBOARDING_REPLY_TO (the same human inbox cmd/onboarding already answers from — a missing value here is a hard failure, not a soft-skip, because this letter asks for a reply). PRO_WELCOME_MAX_PER_RUN (default 200) bounds one run. See internal/engage/AGENTS.md.
• mentorship-remind — sends the 24-hour and 1-hour reminders a booked mentorship session is due. Every 10 minutes. For each offset it takes the confirmed sessions starting inside a ONE-HOUR window ending at that offset, CLAIMS each reminder, and sends only the ones it won — the claim comes first because a worker that sends and then records sends twice whenever the second step fails, and on this schedule "whenever" is often. The window is bounded at BOTH ends: without a lower bound the 24-hour reminder fires for a session three hours away and calls it "in 24 hours", so a session booked closer than an offset simply never gets that reminder. A session already STARTED is never reminded about — a late run does nothing rather than catching up, and the timer is therefore Persistent=false, unlike the reconciling workers beside it. A failed DELIVERY releases its claim so the next run retries: a missing reminder costs somebody the session, a duplicate costs them a duplicate. Needs DATABASE_URL and the mail configuration, and the unit must read /opt/freehire/.env.notify as well as .env — without the mail credentials it is a no-op that never opens the pool, which would otherwise mean every run exiting 0 having reminded nobody, indefinitely. MENTORSHIP_REMIND_MAX_PER_RUN (default 500) bounds one run per offset. The same run reminds buddy mock interviews (buddy.MockService.SendDueMockReminders: same offsets, windows and claim order); a failure in one pass does not skip the other, and the run exits non-zero if either could not read its work. See internal/engage/mentorship/AGENTS.md and internal/engage/buddy/AGENTS.md.
• message-notify — tells people about conversation messages they have left unread, for every messaging kind (today only interview buddies). Every 5 minutes. A participant is due once the oldest message another participant wrote since its read cursor is five minutes old, and is not due again until it has read and a newer message ages the same way (conversation_participants.notified_at is compared with that message, not cleared on read). Open conversations notify every participant, a pending one only those who did not start it, so a starter is never told about their own request; declined and closed never. Claim first: the participant's stamp is moved before anything is sent and only the run that moved it sends; when every channel fails the claim is released (on a context the run's own cancellation cannot kill) so the next run retries, and one channel succeeding counts as sent. Delivers by email (honouring the activity mail group), Telegram when linked, and a notification-center row, worded by the conversation's kind. The notice carries no message text and no account detail — only that a message or request is waiting and a link (for buddies, /buddies/chats?c=<id>). A failed delivery exits non-zero. Needs DATABASE_URL, AWS_REGION/NOTIFY_EMAIL_FROM, FRONTEND_ORIGIN (https, or it refuses), JWT_SECRET (32+ bytes, or it refuses); TELEGRAM_BOT_TOKEN optional; the unit must read /opt/freehire/.env.notify as well as .env — without the mail transport it is a no-op that never opens the pool and exits 0. MESSAGE_NOTIFY_MAX_PER_RUN (default 500) bounds one run. See internal/engage/messaging/AGENTS.md.
• mentor-busy-sync — reads every mentor who has opted in to busy-time sync's Google Calendar free/busy state and reconciles it into mentor_busy_intervals, the table the slot engine has read since migration 0145 and nothing wrote to until now. Hourly, offset from cal-sync/gmail-sync. Each run replaces the WHOLE 60-day forward window per mentor in one transaction (delete then re-insert), never a row-by-row diff — the table holds only bounds, no title or attendee, since Google's freeBusy endpoint (not events.list) has nothing else to offer. Best-effort per mentor: a revoked grant marks needs_reconsent and the run continues; a non-revocation failure (Google having a bad day) never touches that flag, since it is SHARED with the account's other Google grants. Needs DATABASE_URL, the Google OAuth client, and GMAIL_TOKEN_KEY — without them it is a no-op that exits 0. See internal/engage/mentorship/AGENTS.md.
• social-digest — publishes the day's ten most-viewed postings to the configured social channels (today: a Discord webhook). Ranks on job_daily_views.page_uniques, never on uniques — uniques fuses bot-filtered page opens with unfiltered API reads, and crawlers are most of this host's traffic, so a list built on it publishes what robots fetched as though it were what people liked. The day is discovered, not computed: it reads the freshest day job_daily_views holds, because whether cmd/rollup-views (02:30 UTC, rotated logs) has yesterday or the day before depends on the host's logrotate timing; a freshest day more than 3 days old fails the run rather than publishing something stale. -dry-run renders each channel's real payload and sends nothing. The view floor was 10 and is now 3, lowered 2026-09-08 against a measurement: on 2026-09-07 exactly ONE posting in the catalogue cleared 10 and the best scored 12, so the floor was not selecting the popular from the ordinary, it was emptying the list. Revisit it upward as page traffic grows. The quarantine window (7 days) is still a guess. -day YYYY-MM-DD replays one day. Idempotent on (day, channel), so a re-run publishes nothing twice and a run that posted to one channel and failed on another retries only the second. Needs DATABASE_URL and FRONTEND_ORIGIN; a channel with no credentials is skipped, and with none configured the run is a clean no-op. The second channel is our own LinkedIn company page (LINKEDIN_CLIENT_ID / LINKEDIN_CLIENT_SECRET / LINKEDIN_REDIRECT_URI / LINKEDIN_ORGANIZATION_ID — all four or the channel is off). Its access token is NOT configuration: it expires every 60 days and lives in social_tokens, put there by cmd/linkedin-auth and renewed by cmd/linkedin-token-refresh. See internal/engage/socialdigest/AGENTS.md.
• linkedin-auth — run by hand, roughly once every sixty days: grants, records and inspects the credential the digest posts to LinkedIn with. -url prints the address to open in a browser, -code <value> exchanges what that browser handed back, -token <value> stores one minted in the Developer Portal's own token generator instead, -status reports what is stored and when it dies (never the token itself). Not a route on purpose — a sign-in flow for an audience of one, whose authorization would then be whatever that route's turned out to be; a command needs SSH, which is authorization we already have. The exchange is also the only way to learn whether LinkedIn issued a refresh token, which decides whether the credential renews itself or needs a person; it logs the answer explicitly. Needs DATABASE_URL and the four LINKEDIN_* values.
• linkedin-token-refresh — daily: renews that credential a fortnight before expiry when LinkedIn permits it, and warns when it does not. Both paths ship because programmatic refresh tokens go to approved Marketing Developer Platform partners only and the Community Management API is not that program — so a null refresh token is the EXPECTED case, not an error. A warning is a published gauge, not a red unit: this fleet already runs ~37 permanently failing ingest units, so a non-zero exit repeated daily for two weeks trains an operator to ignore it. freehire_social_token_expires_in_seconds, freehire_social_token_grant_expires_in_seconds and freehire_social_token_renewable go to the node_exporter textfile collector when PROM_TEXTFILE_DIR is set (negative seconds mean "no credential stored", which must not look like a broken exporter). A non-zero exit is reserved for a credential that is ALREADY dead. Needs DATABASE_URL and the four LINKEDIN_* values; without them it is a no-op that never opens the pool. See internal/engage/linkedinauth/AGENTS.md.
• merge-companies — collapses the company slugs that are one employer written more than one way (ringcentral + ringcentral-inc, dollar-tree + dollartree). Most groups it finds by folding the NAME; the pairs no fold can reach — one employer whose own boards disagree about the name, Exadel beside Exadel Inc (Website) — come from the hand-written list in cmd/merge-companies/curated.go, which argues why a list and not a rule (Intel and intel471 are different employers). Reports by default; --apply writes. --min-jobs N bounds a wave to the folded groups whose combined open jobs reach N, which is how the backlog is taken in reviewed passes (1000, then 100, then 10, then 1) rather than one 333k-row leap. Chunked and idempotent — the chunk statement selects rows still carrying the retired slug, so a re-run moves nothing and an interrupted wave resumes. Do NOT reindex afterwards: a facet-index push costs 90-180s regardless of batch size, so a wave pushed through search_outbox is tens of hours; the scheduled freehire-reindexw picks it up, and until it does a merged company under-counts its jobs for a few hours without anything 404ing. REINDEX_DEDUP stays unset. Needs only DATABASE_URL. See docs/agents/company-identity.md.
• close-chronic-boards — the safety net for a board (or boardless provider) that has proven unreachable for a long time, not merely cooling down from a recent run of failures (issue #2017): the ordinary per-run unseen sweep never closes a board's jobs unless that run proved it covered the board (shouldSweep/boardQualifies/sweepableCompanies), which is correct against a transient failure but leaves a permanently broken board's jobs open forever, since it can never re-qualify. board_health.ListChronicBoards answers two separate windows off the same evidence (last_success_at, or first_seen_at for a board that has never once succeeded): CHRONIC_BOARD_WINDOW_DAYS (default 30) is what cmd/ingest's per-run health summary reports for curation — fix the adapter, retire the board, or wait; CHRONIC_BOARD_CLOSE_WINDOW_DAYS (default 60, deliberately longer, giving a curator the gap between the two to act first) is what this worker acts on, closing the board's own open jobs with closed_reason = 'board_unreachable' — distinct from the sweep's 'unseen', so the two are never conflated. Scoped to exactly the (provider, board) health record that qualified: a multi-board provider's other boards are untouched; a boardless provider's single record already stands for its whole crawl, so going chronic there closes the whole provider. Refuses a region-ambiguous board name outright — a board id repeated per region for one provider (e.g. Adzuna's it-jobs once per country) has no region in jobs.external_id, so a board-scoped close cannot tell one region's postings from another's; such a board is skipped and counted separately in the run's report rather than risking a healthy region's jobs, the same refusal the ordinary sweep's ambiguousRegionBoards already makes. Dry-run by default, reporting what it would close; --apply writes — ships dry-run-only on its timer until a human has read at least one full closure window's report (freehire-ops' provision/host2/systemd/freehire-close-chronic-boards.service's own comment is the reminder to flip it). Idempotent (closed_at IS NULL guarded) and cheap regardless of when it runs — daily. Needs only DATABASE_URL. A SECOND pass in the same worker closes the opposite failure: a board whose crawls all SUCCEED and whose feed carries nothing. last_success_at cannot see it — an empty-but-reachable feed refreshes that column on every single run, leaving consecutive_failures at 0 and the unit green — so board_health.last_yield_at (migration 0158) records when a board last actually YIELDED a posting, written only by a crawl whose LISTING named one (pipeline.boardListedAnyPosting — ingested, rejected, already-covered, unreadable OR failed-to-save, deliberately the most generous reading available and deliberately NOT the boardReachedPostings permission gate beside it: there, false costs a crawl cycle, but stamped here false accumulates into "this feed is empty" and eventually CLOSES, so a board whose listing works while its detail requests all fail would lose every live posting on it). EMPTY_FEED_CLOSE_WINDOW_DAYS (default 30, deliberately SHORTER than the unreachable window — that one waits out a problem on OUR side, this one has already been answered successfully on every run) closes with closed_reason = 'feed_empty', a third distinct label because "every crawl succeeded and there was nothing there" points at the source's inventory while 'board_unreachable' points at our access. The pass refuses a board with NULL last_yield_at permanently: that means no yield has been OBSERVED, not that the feed is empty, and it was true of all 155k rows the day the column landed — which is why migration 0158's backfill (last_yield_at = last_success_at) is load-bearing, not cosmetic, and why nothing can be closed by this pass until it has been observably empty for a full window AFTER that migration. Found by the WhatJobs incident of 2026-09-10: a market's publisher account was emptied, 107 postings sat open for a month, and neither existing net could reach them. See docs/agents/job-lifecycle.md.
• report-skill-gaps / report-classify-drift — hand-run, read-only reports (internal/job/dictgap) that mine LLM enrichment output nothing else reads: enrichment.skills is a free-text list the model writes per posting, and enrichment.seniority/category is the model's own guess at the same facets internal/dict/classify derives independently from the title. report-skill-gaps ranks the raw skill phrases internal/dict/skilltag resolves to no canonical, by frequency; report-classify-drift recomputes classify.Parse for every distinct enriched title and ranks the titles where that answer disagrees with what enrichment recorded, tracked separately for seniority and category. Neither writes anything — no database row, no dictionary source file, no --apply flag — and neither is on a timer: a curator runs one, reads the ranked list, and decides by hand what belongs in dictionaries.go/labels.go/descriptions.tsv or the classify title tables. No reindex needed either, since nothing they touch changes. Needs only DATABASE_URL. report-geo-gaps is the third of them, for internal/dict/location: it re-runs location.Parse on every distinct location string among publishable postings and compares it with enrichment.countries, printing GAPS (the dictionary places the location nowhere — no country and no region — while the model names a country) and CONFLICTS (the dictionary names countries most of the location's postings contradict, the georgia shape). Its share column is the confidence signal: the model is also asked about Remote-only postings and answers those with a guess, so a 40% share is noise and a 97% one is a missing name.
• Nothing holds a flock — no file locks in Go or in the systemd units; the word survives only in a few stale comments. What keeps a cron worker from stacking on itself is systemd: a Type=oneshot unit will not start a second instance while the first is active. That protects the TIMER path only, so a run started by hand has no lock at all. Never stack reindex-companies with make reindex — Meilisearch runs one serial task queue, so the second rebuild queues behind the first and looks like a hang, and a swap transiently holds ~2x the index's disk. Two workers additionally take a Postgres advisory lock (cmd/liveness, cmd/ghost-crosscheck); their keys are listed in internal/platform/migrate.
• sweep-tracer-clicks — daily: deletes CV tracer-link clicks (cv_link_clicks) older than TRACER_CLICK_RETENTION_DAYS (default 180, read through the strict worker reader). The privacy policy promises those records are gone after 180 days, so it deletes by default — there is no dry run, because a retention promise that only reports keeps nothing. The tracer tokens stay: an old PDF must keep redirecting. Idempotent, so a missed or repeated run costs nothing. Needs only DATABASE_URL.

Module files

Each is self-contained and can be read independently.

Conventions

• Response shapes: Lists: {"data": ..., "meta": {...}}; single items: {"data": ...}; errors: {"error": msg}
• A dropped filter says so: any endpoint whose answer WIDENS when it does not understand a query param reports the params it did not read, in meta.ignored_params (omitted entirely when there are none). Refusing them instead would break saved searches and shared links that still carry a retired facet; staying silent is what let a mistyped country=it pass for a search of Italy. search.UnknownParams owns the SEARCH filter's vocabulary and only that, so an endpoint reading a different one says so itself — /companies through UnknownCompanyParams, /jobs (which reads no filter at all) through its own two-param list. Its value-level twin is meta.ignored_values: a closed-vocabulary facet value outside internal/dict/vocab (work_mode=foo) still narrows — to nothing — and search.IgnoredValues names it rather than letting total: 0 pass for an empty market. The same rule decides what may be SUBSCRIBED to: search.Narrows asks whether a saved search selects less than the catalogue, and a subscription to one that does not is refused rather than reported, because nobody reads a digest's meta
• Dedup key: jobs.UNIQUE (source, external_id) — UpsertJob is ON CONFLICT on it
• Company key: normalize.CompanySlug, never normalize.Slug — the corporate form is not part of who the employer is. One legal-form vocabulary exists (internal/dict/normalize) and a test walks the module to keep it that way; there were four, they disagreed, and every resulting miss was silent. Spellings a rule cannot collapse (dollartree vs dollar-tree) resolve through company_slug_aliases, the one company-adjacent table that is NOT derived from jobs
• Auth: JWT in httpOnly cookie, same-origin, carrying the account's token_version so sessions are revocable. RequireAuth (cookie only) / RequireAuthOrKey (cookie or full-scope Bearer) / RequireAuthOrScopedKey (also admits a narrow key)
• Email ownership: users.email_verified; a password registration starts unverified and is confirmed by a mailed six-digit code. An unverified, password-backed account is seized (password cleared, sessions revoked, API keys deleted) when a provider-verified OAuth identity arrives for its address — the account-pre-hijacking defence
• API keys: Hashed at rest (SHA-256), scoped full or cv, and mintable only by an account with a verified address. Key management (create/list/revoke) and password change are cookie-only. A key does not carry the session generation, so a token_version bump does not revoke it — that is intentional for sign-out-everywhere and wrong for a takeover, so the seizure and the mailed-code password reset delete the rows in the same statement
• Enrichment: Queue-driven (enrichment_outbox), provider-agnostic LLM, Sanitize + Validate gate
• Embeddings: Queue-driven (semantic_outbox), incremental (cmd/embed) — pgvector-backed job_semantic_chunks plus a legacy single-vector column, no search index. Reconciled by bumping the embedder-model version string, which re-enqueues the whole catalogue through the existing staleness check
• Catalogue scale: Every public figure describing how big the catalogue is — GET /api/v1/stats/catalog, the jobs list's meta.total, the /about and /open strips — reads ONE snapshot published by cmd/rollup-stats (internal/ingest/catalogstats). Never count on a request path: catalogstats.Load takes no exact counter, so it cannot. A read never fails; a cold cache, an unreachable Redis or no cache at all degrades to the planner estimate with exact: false, which zeroes the figures that exist only in the database — pass exact through to whatever renders them, because a zero must not reach a page as if it were a measurement. The snapshot's open_jobs is the raw count (every open posting, reposts across boards included, never duplicate-suppressed) — unique_open_jobs is the de-duplicated figure Meilisearch's own index holds, what /jobs itself would show unfiltered. It is measured the same way as telegram_channels: cmd/rollup-stats makes one search.Client.Search call (limit 1, no filters) before publishing, best-effort — a missing MEILI_MASTER_KEY or an unreachable Meilisearch costs that one figure, not the snapshot
• Per-source figures: GET /api/v1/sources and the public /sources page read ONE snapshot, source_stats, rebuilt by a pass inside cmd/rollup-stats (internal/ingest/sourcestats) as an atomic delete-and-reinsert. It carries the same two-count pair catalogstats does and for the same reason: open_jobs is the raw open count — duplicates included — and is the denominator the ATS-overlap figures are arithmetic on, while browsable_jobs is Meilisearch's de-duplicated count, which is what /jobs?source=<key> will show and therefore what the PAGE displays. A card whose number the search it links to contradicts is the trap the pair exists to avoid. browsable_jobs is NULL when Meilisearch could not be measured, never 0 — an unreachable index costs that one figure, not the run, and a 0 would state on a public page that every source carries nothing. The snapshot's spine is sourcestats.Union over THREE sets, each covering a hole the others leave: the adapter registry (without it a source with postings and no adapter — telegram, an extraction pipeline absent from sources.Taxonomy() — is dropped), this run's scan (without it every adapter whose postings have all closed is dropped, which is the one case the page most needs to state), and the KEYS of the snapshot being replaced (without it a source in neither of the first two vanishes the moment it goes quiet — the same silent drop, one closure later). Only names carry over; every figure is re-measured. GET /api/v1/sources calls the same Union rather than keeping its own copy of the rule. The ATS-overlap figure counts rows the dedup pass matched to a first-party ATS posting (jobs.duplicate_of_aggregator); the remainder is ats_unmatched and is never called exclusive, on the wire or on the page, because the pass finding no pair is the absence of evidence — two tests enforce the wording — one over the rendered card, one over the source of every file on the surface, including the page's own prose. It is carried for aggregators ONLY: the suppression pass never marks any other row, so a 0 there is arithmetic and would read as "fully exclusive". The aggregate scan reads no description column (a guard test on the generated SQL) and excludes is_private. Source logos are resolved in the browser from the source's DISPLAY NAME (sourceLogoUrl), never from a posting host: the first version sampled min(url) per source and served the WRONG brand, because an ATS posting's URL usually lives on the employer's own domain — measured on prod, greenhouse sampled bankrate.com and successfactors a staffing agency — while a per-tenant subdomain (jobs.smartrecruiters.com) simply 404s. By name, 14 of 15 resolve, and 7 of 8 sampled sources return the byte-identical image whether asked by name or by an explicit ?domain= — a platform brand is far less ambiguous than an employer name. That margin is not free, though: the proxy answers a name it resolves WRONGLY with a 200, so a mismatch is invisible to every consumer (see the g2i case behind cmd/publish-logo-domains). SOURCE_LOGO_DOMAINS (web/src/lib/logo.ts) is that escape hatch, curated and measured: of the 145 sources carrying 500+ jobs, 14 had no logo by name and all 14 resolve with the platform's own domain passed as ?domain=, which wins over the proxy's name map. successfactors settles why a nicer NAME was never the whole fix — its label was already correct and the proxy still had nothing. It is an OVERRIDE, not a registry: a source missing from it resolves by name as before, so the map going stale costs a logo rather than breaking one — the deliberate difference from the posting-host sampling it replaced, which was wrong for every source it covered. A curated domain implies a curated display name, and a test holds the two maps together. A miss still 404s, so every call site renders through the EntityLogo primitive rather than a hand-rolled <img>: besides the onerror fallback it catches a 404 that landed BEFORE hydration, which a server-rendered page needs precisely because the browser fetches the image while no handler exists yet. A user's excluded_sources are seeded into the search filter by filtersFromProfile, the same function the saved-search alert is built from — so avoiding a source from the job page's provenance row (JobSourceRow) reaches both the feed and the digest. Before that seeding the list was stored, editable in the profile, and read by nobody
• Dictionaries: All facet dictionaries are dict-only in production — never guess, emit nothing for unknowns
• View counts: anything describing how much a posting was LOOKED AT reads job_daily_views.page_uniques, never uniques. uniques fuses bot-filtered page opens with UNFILTERED API reads, and crawlers are most of this host's traffic, so a figure built on it reports what robots fetched. The rule used to live only in social-digest, which is why /api/v1/stats/engagement and the /open grid published 11,027,722 against the real 5,401,347 for months (freehire#2945) — a rule held by one reader is not a rule. Seed the two columns to DIFFERENT totals in any test that asserts on a view figure: page_uniques carries a DEFAULT 0, so a test seeding only uniques passes whichever column the query reads. page_uniques is also every VISITOR, signed in or not, so it never belongs beside signed-in counters without its label saying so. job_daily_views.agent_uniques counts postings served to agents (MCP, OJCP, /agent/jobs/search, via internal/application/agentview); jobs.view_count = uniques + agent_uniques. jobs.view_count therefore mixes people, API reads and agents: it is a total-reach figure (the most-viewed sort and the job page show it as such), and the only human-only figure remains page_uniques
• Job deletion: The lifecycle only soft-closes, and nothing hard-deletes jobs. cmd/prune used to, stripping postings that did not belong on an IT board; it was removed once the catalogue became broader than IT. The rows it took are archived in pruned_jobs (kept), and the applications it orphaned still carry a NULL job_id, which the tracking code handles
• In-app assistant: a bounded tool-calling loop in-process (internal/ai/assistant), streamed over SSE, open to every signed-in user. Tools act as the authenticated caller — no credential is minted for an agent
• No free AI — the paid gate: every model call made for a user's action needs a live Pro or Ultra entitlement — with ONE named exception, parsing a stored CV, which every account gets (below). That is a product invariant, not a quota: there is no free allowance and no override (PLAN_FREE_DAILY_<F> is gone and no longer read), and plan.decide refuses an unpaid tier before it reads any allowance. It is enforced twice in internal/api/handler: the mw.paidAI(feature) middleware (paid_ai.go) answers an unpaid caller 402 — in the same body every plan refusal carries, before any stream opens — and an unreadable plan 503, never a pass; and the user-call binding (llmBinding.bind/caller, user_llm.go) refuses to hand out a model client for a context the gate did not mark (plan.WithPaidAI), so a new model-calling route mounted without the middleware fails its first test. A test also forbids llmkey.Bind/Caller anywhere in the handler package outside the binding. Callers the middleware cannot reach ask plan.Store.RequirePaid themselves: the auto-apply orchestrator's tailor run (for the queue entry's owner) and cmd/auto-apply's drafting (a lapsed candidate's attempt parks). The CV parse is the exception, and it is a mark, not a missing check: the upload's background extraction and its retry run for every account under plan.WithFreeAI, which the binding accepts beside plan.WithPaidAI, on their own cheap model (RESUME_EXTRACT_MODEL, falling back to LLM_MODEL). It was paid until 2026-10-19 and that cost paying customers: a CV uploaded before upgrading was never parsed, upgrading did not re-run it, and the experience bank every paid feature reads stayed empty — tailoring and the fit analysis failed for somebody who had just paid. Cached AI results, deterministic scoring, search and the public CV roast stay open to free. Ownerless background work (enrichment, embeddings, Telegram) is unaffected.
• Plan limits: every metered AI action draws on a per-feature, per-UTC-day allowance (internal/ai/plan), on top of the paid gate above. There is no currency and no balance: a paid plan differs in how MUCH of a feature it allows in a day. Tailoring carries two bounds (sessions per day, turns per session) because either alone leaves the hole this replaced. Enforcement is a per-feature switch — every metered feature ships ENFORCING today, so a refusal is real from day one; PLAN_ENFORCE remains the lever that turns a FUTURE feature on one at a time after its own shadow run has been read, which is how auto-apply and then the rest arrived at this state without ever sharing a deploy. Refusal is HTTP 402, and on a stream it must precede the stream. THREE tiers since migration 0141: free, pro and ultra, resolved by TierOf as the better one still live (users.pro_until and users.ultra_until, each derived from three source columns). PLAN_PRO_DAILY_<F> and PLAN_ULTRA_DAILY_<F> move a paid tier's figure without a deploy; STRIPE_ULTRA_PRICE_IDS is what makes Ultra sellable at all, and unset it simply is not offered. auto-apply was the first feature that shipped ENFORCING, because pro's ceiling on it is what Ultra is sold on and a ceiling that only counts sells nothing — its route already refused the free tier with 402 before the allowance took that gate's place
• LLM spend attribution: every model call made for a signed-in user goes out on that user's OWN gateway credential (internal/ai/llmkey — minted lazily on first use, never shown to them) and carries a feature: tag. Work that belongs to nobody — enrichment, Telegram, embeddings — keeps the service credential, and a test enforces that background entrypoints never resolve a user's. Attribution fails open: it can never refuse or fail a call. It measures and does not bound — what BOUNDS a turn is the plan (internal/ai/plan)
• Experience provenance: every banked achievement records whether the CANDIDATE asserted it (cv_import/stated_in_chat/manual) or the MODEL did (agent_inferred). Only the former may be written into a CV, and the check lives in the service path, not in a system prompt
• Sentry: Opt-in, env-gated, errors-only — sentry.Init with SendDefaultPII:false
• Naming — "CV", not "résumé": Default new surfaces to CV. Don't mass-rename the existing resume/resumeextract packages and columns — churn without value