agentleFS
hirefrankhirefrank/network-jobs/skills/jobs-ingest/SKILL.md

jobs-ingest

Skill0 starsChanged 3 months agoMITView on GitHub

Promote Network Jobs triage batches into the local searchable corpus. Use ONLY after the user explicitly confirms a triage folder should be ingested — never auto-chain from careers-discover.

The file

Show its rules and text

We split this skill into the rules it gives an agent

15 rules. About 0 are common, 9 are distinctive, and 6 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈9

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 6

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    One question per call: Ask it, wait for the answer, then ask the next.

    line 22
  2. Distinctive:

    Never bundle two questions into one prompt or one tool call, even when the tool accepts several.

    line 23
  3. Distinctive:

    Put the deciding data in the option label (listing counts, department, date), never in surrounding prose the user has to correlate.

    line 27
  4. Distinctive:

    Always keep a free-text / "none of these" path.

    line 29
  5. Distinctive:

    No such tool in this host?

    line 30
  6. Distinctive:

    When the user has not already named the set, get that confirmation as one question (matches / all / department, counts in labels, matches …

    line 38
  7. Very specific:

    Default ingest set is matches (index/matches.json) when present.

    line 39
  8. Distinctive:

    Not confirmation: the user earlier saying “run careers-discover”, finishing background agents, or you planning a full pipeline.

    line 40
  9. Distinctive:

    If unsure, list staged dirs + listing counts and ask: one question per decision — first which triage dir (if more than one), then which set …

    line 41
  10. Very specific:

    Default / “ingest these” / “ingest matches” → index/matches.json (.jobs)

    line 47
  11. Very specific:

    If crawl-state said unchanged: true, skip ingest (delta empty) unless the user insists.

    line 50
  12. Very specific:

    Keep any description / department the triage rows already carry — they ride through classify, merge, and shards untouched.

    line 61
  13. Very specific:

    Write the chosen batch to a working file (e.g.

    line 65
  14. Very specific:

    Run helpers/rebuild-corpus.sh on that file:

    line 71
  15. Distinctive:

    Leave triage dirs in place (do not delete unless user asks)

    line 105

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Jobs Ingest

Normalize staged listings → ~/.network-jobs/corpus/ shards + manifest.json.

Read SCHEMA.md first.

Asking the user

Whenever the next step is picking from a known set, use the host's choice tool — AskUserQuestion (Claude Code), question (opencode), ask_user_choice, or the harness equivalent — instead of a prose list.

  • One question per call. Ask it, wait for the answer, then ask the next. Never bundle two questions into one prompt or one tool call, even when the tool accepts several.
  • Recommend, and say why. Lead with the option you would pick and a one-line reason.
  • Put the deciding data in the option label (listing counts, department, date), never in surrounding prose the user has to correlate.
  • Always keep a free-text / "none of these" path.
  • No such tool in this host? Use a short numbered list, recommendation first, never more than 4 options.

Gate

Do not ingest without explicit user confirmation for the triage path(s).

  • Confirmation looks like: “ingest these”, “ingest matches”, “ingest all”, “promote the Slack batch”, “yes, put them in the corpus”.
  • When the user has not already named the set, get that confirmation as one question (matches / all / department, counts in labels, matches recommended) — one decision per call, never bundled with the dir choice.
  • Default ingest set is matches (index/matches.json) when present. “ingest all” uses listings.json. A department name slices whichever set you chose.
  • Not confirmation: the user earlier saying “run careers-discover”, finishing background agents, or you planning a full pipeline.
  • If unsure, list staged dirs + listing counts and ask: one question per decision — first which triage dir (if more than one), then which set (see below). Put the counts in the option labels.

Workflow

  1. Select triage dir(s) under $DATA/triage/careers-*
  2. Choose the ingest set — ask one question, never a prose “matches/all/department” list, with counts in the labels and matches recommended:
    • Default / “ingest these” / “ingest matches” → index/matches.json (.jobs)
    • “ingest all” → index/listings.json
    • “ingest Product” / a department name → filter that set by department
    • If crawl-state said unchanged: true, skip ingest (delta empty) unless the user insists.
  3. Read the chosen listings + INVENTORY.md (quote N of M match prefs, not the raw board)
  4. Classify with the helper (required) — do not category-map the whole board in the model:
python3 "$SUITE/skills/jobs-ingest/helpers/classify-listings.py" \
  --input "$DATA/corpus/.work/raw.json" --out "$DATA/corpus/.work/batch.json" \
  --company "$NAME"

Persist category, track, seniority, senioritySignals (intern / staff+), parsed locations[], locationBuckets (hybrid NYC-or-Remote hits both), and fingerprint. Keep any description / department the triage rows already carry — they ride through classify, merge, and shards untouched. LLM only for needsLlm titles (ambiguous lead/head/player-coach). Leave high-confidence rows alone. Fill companyDomain from the graph when known. Keep salary / postedAt only if present in triage. 5. Merge into corpus via helper (required):

  • Write the chosen batch to a working file (e.g. $DATA/corpus/.work/batch.json).
  • A batch of new/updated jobs only is a delta. Pass --delta; expiry is then skipped by design (a delta is not evidence that unseen roles closed, and --force-expire does not override it). To expire unseen roles, write the full classified board batch (every role on the board, including unchanged rows) and do not pass --delta.
  • Read index/pagination.json (or INVENTORY). Pass --pagination-complete only when complete is true and status is neither blocked nor captured, and never for a user-capture run (status: captured) — a blocked board captured zero rows because it could not be read, and a capture is one saved page, not the board; neither is evidence that roles closed. status: truncated behaves like complete: false; the helper also refuses expiry on it unless --force-expire is passed. status: empty is a clean, parsed board listing zero roles (complete is true, so --expect-count 0 works). If the batch has zero incoming rows for that company and it already had any open corpus rows, rebuild skips expiry and warns — whatever the status (omitted, jq null, or empty). A filtered or region-limited URL can look empty. Pass --force-expire only when the board really closed every role. A company that had zero open rows can still record empty.
  • Pass --pagination-status "$(jq -r '.status' "$TRIAGE/index/pagination.json")" with every rebuild: the helper refuses expiry on blocked / captured even if --pagination-complete slipped through.
  • Always pass --expect-count with the board's own size: max(index/listings.json length, pagination.reportedTotal) when reportedTotal is present, else the listings length. The listing count alone is self-referential — a crawl wrongly marked complete and its subset guard come from the same short set — while the board's reported total makes the guard bite. Do not subtract deduped: it also includes rows that normalization dropped, so subtracting it can make a short crawl look complete. The guard counts only incoming fingerprints for --expire-company, so rows for another company in the same batch cannot satisfy this company's total. A matches-only batch over a bigger board then skips expiry loudly instead of closing live roles; only --force-expire overrides (use when you know the board genuinely shrank).
  • Read index/pagination.json for filtered (older files may omit it; default false) and pass --filtered when true. A filtered crawl's query/location filter is not known to cover the company, so expiry is skipped. --force-expire is the explicit operator override: use it only when you know the filter covers every role for that company.
  • Run helpers/rebuild-corpus.sh on that file:
N_LISTINGS=$(jq 'length' "$TRIAGE/index/listings.json")
REPORTED=$(jq -r '.reportedTotal // empty' "$TRIAGE/index/pagination.json")
EXPECT="$N_LISTINGS"
if [[ "$REPORTED" =~ ^[0-9]+$ ]] && (( REPORTED > EXPECT )); then
  EXPECT="$REPORTED"
fi
STATUS=$(jq -r '.status' "$TRIAGE/index/pagination.json")
FILTERED=$(jq -r '.filtered // false' "$TRIAGE/index/pagination.json")
FILTER_ARG=()
if [[ "$FILTERED" == "true" ]]; then
  FILTER_ARG=(--filtered)
fi

# Choose ONE:
# Full classified board (not a delta): expiry may run.
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
  --expire-company "$SLUG" --pagination-complete --expect-count "$EXPECT" \
  --pagination-status "$STATUS" "${FILTER_ARG[@]}"
# OR delta (new/updated jobs only): expiry is always skipped.
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
  --expire-company "$SLUG" --pagination-complete --expect-count "$EXPECT" \
  --pagination-status "$STATUS" --delta "${FILTER_ARG[@]}"
# or, when truncated / incomplete / blocked / captured:
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
  --expire-company "$SLUG" --pagination-incomplete --pagination-status "$STATUS" \
  "${FILTER_ARG[@]}"
  • The helper merges by fingerprint with existing corpus/jobs-all.json (incoming wins, firstSeen kept), rewrites shards + manifest, and deletes stale shard files
  • Unseen jobs for that company are marked closed only when --pagination-complete is set, the batch is not --delta, the company's own rows cover the expected board size, --pagination-status is not blocked/captured, and the crawl is not --filtered (unless --force-expire). Never expire on incomplete crawls, user-capture runs, delta batches, or filtered batches. A batch with zero incoming rows for that company and prior open roles also skips expiry unless --force-expire, even if --pagination-status is omitted or null.
  1. Report incoming count, total after merge, removed stale shards; show manifest.totalJobs and lastUpdated. If rebuild set expirySkipped, tell the user expiry was skipped and quote expiryReason — do not present a zero-row ingest as a clean empty board.
  2. Leave triage dirs in place (do not delete unless user asks)

Categories

engineering | product | design | data | ai-ml | sales | marketing | customer-success | operations | finance | people | legal | it-security | retail | other

Location heuristics

Buckets remain the fast path. The classifier also parses city/region onto locations[].

  • NYC / New York / Brooklyn / Manhattan → nyc
  • SF / San Francisco / Bay Area / Palo Alto / Mountain View / Oakland → sf
  • Remote / Distributed / Work from home → remote
  • New York, NY or Remote → both nyc and remote shards
  • else → other

Honor preferences.onsiteLocations when matching hybrid/onsite (explicit cities, not every office).

Helper

DATA="${NETWORK_JOBS_HOME:-$HOME/.network-jobs}"  # always resolve from the env; never export a literal DATA
SUITE="$(cat "$DATA/suite-root" 2>/dev/null || true)"
SUITE="${NETWORK_JOBS_SUITE:-${SUITE:-}}"
python3 "$SUITE/skills/jobs-ingest/helpers/classify-listings.py" \
  --input "$DATA/corpus/.work/raw.json" --out "$DATA/corpus/.work/batch.json" --company "$NAME"
# After classifying (and optionally LLM-fixing needsLlm rows):
N_LISTINGS=$(jq 'length' "$TRIAGE/index/listings.json")
REPORTED=$(jq -r '.reportedTotal // empty' "$TRIAGE/index/pagination.json")
EXPECT="$N_LISTINGS"
if [[ "$REPORTED" =~ ^[0-9]+$ ]] && (( REPORTED > EXPECT )); then
  EXPECT="$REPORTED"
fi
STATUS=$(jq -r '.status' "$TRIAGE/index/pagination.json")
FILTERED=$(jq -r '.filtered // false' "$TRIAGE/index/pagination.json")
FILTER_ARG=()
if [[ "$FILTERED" == "true" ]]; then
  FILTER_ARG=(--filtered)
fi
# Full classified board (not a delta); add --delta instead for new/updated-only rows.
"$SUITE/skills/jobs-ingest/helpers/rebuild-corpus.sh" "$DATA/corpus/.work/batch.json" \
  --expire-company "$SLUG" --pagination-complete --expect-count "$EXPECT" \
  --pagination-status "$STATUS" "${FILTER_ARG[@]}"   # only if pagination.complete and status is not blocked/captured

The classify helper is deterministic. Only send needsLlm titles to the model. Rebuild merges by fingerprint, shards by category/location/seniority (hybrid jobs land in every locationBuckets shard), and closes unseen jobs for --expire-company only when --pagination-complete is set, the batch is the full classified board (no --delta), the company's own rows cover the expected count, the status is not blocked/captured, and the crawl is not --filtered (unless --force-expire). A zero-row batch with prior open roles also skips expiry unless --force-expire, even if status is omitted or null.

Helpers

  • helpers/classify-listings.py — title/department → category, track, senioritySignals, locationBuckets
  • helpers/rebuild-corpus.sh — fingerprint merge + shards; expire only if pagination.complete, the batch is not --delta, the company's rows cover the expected count, status is not blocked/captured, and the crawl is not --filtered (or --force-expire); zero incoming rows with prior open roles needs --force-expire

Related

  • Upstream: careers-discover
  • Downstream search: network-jobs

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.