jobs-ingest
Skill0 starsChanged 3 months agoMITView on GitHub
Promote Network Jobs triage batches into the local searchable corpus. Use ONLY after the user explicitly confirms a triage folder should be ingested — never auto-chain from careers-discover.
In this repository7 other files it gives its agents.
- AGENTS.mdAGENTS.md
- worker/AGENTS.mdAGENTS.md
- careers-discoverSkill
- intro-email-generatorSkill
- network-jobs-importSkill
- network-jobs-setupSkill
- network-jobsSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
15 rules. About 0 are common, 9 are distinctive, and 6 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈9
Distinctive · estimate
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 22
One question per call: Ask it, wait for the answer, then ask the next.
- Distinctive: line 23
Never bundle two questions into one prompt or one tool call, even when the tool accepts several.
- Distinctive: line 27
Put the deciding data in the option label (listing counts, department, date), never in surrounding prose the user has to correlate.
- Distinctive: line 29
Always keep a free-text / "none of these" path.
- Distinctive: line 30
No such tool in this host?
- Distinctive: line 38
When the user has not already named the set, get that confirmation as one question (matches / all / department, counts in labels, matches …
- Very specific: line 39
Default ingest set is matches (index/matches.json) when present.
- Distinctive: line 40
Not confirmation: the user earlier saying “run careers-discover”, finishing background agents, or you planning a full pipeline.
- Distinctive: line 41
If unsure, list staged dirs + listing counts and ask: one question per decision — first which triage dir (if more than one), then which set …
- Very specific: line 47
Default / “ingest these” / “ingest matches” → index/matches.json (.jobs)
- Very specific: line 50
If crawl-state said unchanged: true, skip ingest (delta empty) unless the user insists.
- Very specific: line 61
Keep any description / department the triage rows already carry — they ride through classify, merge, and shards untouched.
- Very specific: line 65
Write the chosen batch to a working file (e.g.
- Very specific: line 71
Run helpers/rebuild-corpus.sh on that file:
- Distinctive: line 105
Leave triage dirs in place (do not delete unless user asks)
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Jobs Ingest
Normalize staged listings → ~/.network-jobs/corpus/ shards + manifest.json.
Read SCHEMA.md first.
Asking the user
Whenever the next step is picking from a known set, use the host's choice tool —
AskUserQuestion (Claude Code), question (opencode), ask_user_choice, or the
harness equivalent — instead of a prose list.
- One question per call. Ask it, wait for the answer, then ask the next. Never bundle two questions into one prompt or one tool call, even when the tool accepts several.
- Recommend, and say why. Lead with the option you would pick and a one-line reason.
- Put the deciding data in the option label (listing counts, department, date), never in surrounding prose the user has to correlate.
- Always keep a free-text / "none of these" path.
- No such tool in this host? Use a short numbered list, recommendation first, never more than 4 options.
Gate
Do not ingest without explicit user confirmation for the triage path(s).
- Confirmation looks like: “ingest these”, “ingest matches”, “ingest all”, “promote the Slack batch”, “yes, put them in the corpus”.
- When the user has not already named the set, get that confirmation as one question (matches / all / department, counts in labels, matches recommended) — one decision per call, never bundled with the dir choice.
- Default ingest set is matches (
index/matches.json) when present. “ingest all” useslistings.json. A department name slices whichever set you chose. - Not confirmation: the user earlier saying “run careers-discover”, finishing background agents, or you planning a full pipeline.
- If unsure, list staged dirs + listing counts and ask: one question per decision — first which triage dir (if more than one), then which set (see below). Put the counts in the option labels.
Workflow
- Select triage dir(s) under
$DATA/triage/careers-* - Choose the ingest set — ask one question, never a prose “matches/all/department” list, with counts in the labels and matches recommended:
- Default / “ingest these” / “ingest matches” →
index/matches.json(.jobs) - “ingest all” →
index/listings.json - “ingest Product” / a department name → filter that set by
department - If crawl-state said
unchanged: true, skip ingest (delta empty) unless the user insists.
- Default / “ingest these” / “ingest matches” →
- Read the chosen listings +
INVENTORY.md(quote N of M match prefs, not the raw board) - Classify with the helper (required) — do not category-map the whole board in the model:
python3 "$SUITE/skills/jobs-ingest/helpers/classify-listings.py" \
--input "$DATA/corpus/.work/raw.json" --out "$DATA/corpus/.work/batch.json" \
--company "$NAME"Persist category, track, seniority, senioritySignals (intern / staff+), parsed locations[], locationBuckets (hybrid NYC-or-Remote hits both), and fingerprint.
Keep any description / department the triage rows already carry — they ride through classify, merge, and shards untouched.
LLM only for needsLlm titles (ambiguous lead/head/player-coach). Leave high-confidence rows alone.
Fill companyDomain from the graph when known. Keep salary / postedAt only if present in triage.
5. Merge into corpus via helper (required):
- Write the chosen batch to a working file (e.g.
$DATA/corpus/.work/batch.json). - A batch of new/updated jobs only is a delta. Pass
--delta; expiry is then skipped by design (a delta is not evidence that unseen roles closed, and--force-expiredoes not override it). To expire unseen roles, write the full classified board batch (every role on the board, including unchanged rows) and do not pass--delta. - Read
index/pagination.json(or INVENTORY). Pass--pagination-completeonly whencompleteis true andstatusis neitherblockednorcaptured, and never for a user-capture run (status: captured) — a blocked board captured zero rows because it could not be read, and a capture is one saved page, not the board; neither is evidence that roles closed.status: truncatedbehaves likecomplete: false; the helper also refuses expiry on it unless--force-expireis passed.status: emptyis a clean, parsed board listing zero roles (completeis true, so--expect-count 0works). If the batch has zero incoming rows for that company and it already had any open corpus rows, rebuild skips expiry and warns — whatever the status (omitted, jqnull, orempty). A filtered or region-limited URL can look empty. Pass--force-expireonly when the board really closed every role. A company that had zero open rows can still record empty. - Pass
--pagination-status "$(jq -r '.status' "$TRIAGE/index/pagination.json")"with every rebuild: the helper refuses expiry onblocked/capturedeven if--pagination-completeslipped through. - Always pass
--expect-countwith the board's own size:max(index/listings.json length, pagination.reportedTotal)whenreportedTotalis present, else the listings length. The listing count alone is self-referential — a crawl wrongly marked complete and its subset guard come from the same short set — while the board's reported total makes the guard bite. Do not subtractdeduped: it also includes rows that normalization dropped, so subtracting it can make a short crawl look complete. The guard counts only incoming fingerprints for--expire-company, so rows for another company in the same batch cannot satisfy this company's total. A matches-only batch over a bigger board then skips expiry loudly instead of closing live roles; only--force-expireoverrides (use when you know the board genuinely shrank). - Read
index/pagination.jsonforfiltered(older files may omit it; default false) and pass--filteredwhen true. A filtered crawl's query/location filter is not known to cover the company, so expiry is skipped.--force-expireis the explicit operator override: use it only when you know the filter covers every role for that company. - Run
helpers/rebuild-corpus.shon that file:
N_LISTINGS=$(jq 'length' "$TRIAGE/index/listings.json")
REPORTED=$(jq -r '.reportedTotal // empty' "$TRIAGE/index/pagination.json")
EXPECT="$N_LISTINGS"
if [[ "$REPORTED" =~ ^[0-9]+$ ]] && (( REPORTED > EXPECT )); then
EXPECT="$REPORTED"
fi
STATUS=$(jq -r '.status' "$TRIAGE/index/pagination.json")
FILTERED=$(jq -r '.filtered // false' "$TRIAGE/index/pagination.json")
FILTER_ARG=()
if [[ "$FILTERED" == "true" ]]; then
FILTER_ARG=(--filtered)
fi
# Choose ONE:
# Full classified board (not a delta): expiry may run.
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
--expire-company "$SLUG" --pagination-complete --expect-count "$EXPECT" \
--pagination-status "$STATUS" "${FILTER_ARG[@]}"
# OR delta (new/updated jobs only): expiry is always skipped.
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
--expire-company "$SLUG" --pagination-complete --expect-count "$EXPECT" \
--pagination-status "$STATUS" --delta "${FILTER_ARG[@]}"
# or, when truncated / incomplete / blocked / captured:
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
--expire-company "$SLUG" --pagination-incomplete --pagination-status "$STATUS" \
"${FILTER_ARG[@]}"- The helper merges by fingerprint with existing
corpus/jobs-all.json(incoming wins,firstSeenkept), rewrites shards + manifest, and deletes stale shard files - Unseen jobs for that company are marked
closedonly when--pagination-completeis set, the batch is not--delta, the company's own rows cover the expected board size,--pagination-statusis notblocked/captured, and the crawl is not--filtered(unless--force-expire). Never expire on incomplete crawls, user-capture runs, delta batches, or filtered batches. A batch with zero incoming rows for that company and prior open roles also skips expiry unless--force-expire, even if--pagination-statusis omitted ornull.
- Report incoming count, total after merge, removed stale shards; show
manifest.totalJobsandlastUpdated. If rebuild setexpirySkipped, tell the user expiry was skipped and quoteexpiryReason— do not present a zero-row ingest as a clean empty board. - Leave triage dirs in place (do not delete unless user asks)
Categories
engineering | product | design | data | ai-ml | sales | marketing | customer-success | operations | finance | people | legal | it-security | retail | other
Location heuristics
Buckets remain the fast path. The classifier also parses city/region onto locations[].
- NYC / New York / Brooklyn / Manhattan →
nyc - SF / San Francisco / Bay Area / Palo Alto / Mountain View / Oakland →
sf - Remote / Distributed / Work from home →
remote New York, NY or Remote→ bothnycandremoteshards- else →
other
Honor preferences.onsiteLocations when matching hybrid/onsite (explicit cities, not every office).
Helper
DATA="${NETWORK_JOBS_HOME:-$HOME/.network-jobs}" # always resolve from the env; never export a literal DATA
SUITE="$(cat "$DATA/suite-root" 2>/dev/null || true)"
SUITE="${NETWORK_JOBS_SUITE:-${SUITE:-}}"
python3 "$SUITE/skills/jobs-ingest/helpers/classify-listings.py" \
--input "$DATA/corpus/.work/raw.json" --out "$DATA/corpus/.work/batch.json" --company "$NAME"
# After classifying (and optionally LLM-fixing needsLlm rows):
N_LISTINGS=$(jq 'length' "$TRIAGE/index/listings.json")
REPORTED=$(jq -r '.reportedTotal // empty' "$TRIAGE/index/pagination.json")
EXPECT="$N_LISTINGS"
if [[ "$REPORTED" =~ ^[0-9]+$ ]] && (( REPORTED > EXPECT )); then
EXPECT="$REPORTED"
fi
STATUS=$(jq -r '.status' "$TRIAGE/index/pagination.json")
FILTERED=$(jq -r '.filtered // false' "$TRIAGE/index/pagination.json")
FILTER_ARG=()
if [[ "$FILTERED" == "true" ]]; then
FILTER_ARG=(--filtered)
fi
# Full classified board (not a delta); add --delta instead for new/updated-only rows.
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
--expire-company "$SLUG" --pagination-complete --expect-count "$EXPECT" \
--pagination-status "$STATUS" "${FILTER_ARG[@]}" # only if pagination.complete and status is not blocked/capturedThe classify helper is deterministic. Only send needsLlm titles to the model. Rebuild merges by fingerprint, shards by category/location/seniority (hybrid jobs land in every locationBuckets shard), and closes unseen jobs for --expire-company only when --pagination-complete is set, the batch is the full classified board (no --delta), the company's own rows cover the expected count, the status is not blocked/captured, and the crawl is not --filtered (unless --force-expire). A zero-row batch with prior open roles also skips expiry unless --force-expire, even if status is omitted or null.
Helpers
helpers/classify-listings.py— title/department → category, track, senioritySignals, locationBucketshelpers/rebuild-corpus.sh— fingerprint merge + shards; expire only if pagination.complete, the batch is not--delta, the company's rows cover the expected count, status is notblocked/captured, and the crawl is not--filtered(or--force-expire); zero incoming rows with prior open roles needs--force-expire
Related
- Upstream: careers-discover
- Downstream search: network-jobs
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

