agentleFS
luis2404123luis2404123/birdproxies-skills/job-board-aggregator/SKILL.md

job-board-aggregator

Skill0 starsChanged 9 months agoMITView on GitHub

Scrape job listings from Indeed, Glassdoor, LinkedIn Jobs, and other job boards. Extract job titles, salaries, descriptions, and company data. Bypass DataDome and PerimeterX with residential proxies.

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 3 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Use residential proxy + browser tool (mandatory)

    line 122
  2. Distinctive:

    Wait 3-5 seconds for DataDome challenge to clear

    line 124
  3. Distinctive:

    Wait for Cloudflare challenge (5-8 seconds)

    line 138
  4. Very specific:

    Use &fromage=1 (Indeed) or &f_TPR=r86400 (LinkedIn) to only get jobs posted in the last 24 hours.

    line 204

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Job Board Aggregator

Scrape job listings from Indeed, Glassdoor, LinkedIn Jobs, ZipRecruiter, and other job boards. Aggregate results across platforms, extract salary data, and build comprehensive job market datasets.

When to Use This Skill

Activate when the user:

  • Wants to scrape job listings from any job board
  • Needs to aggregate jobs across multiple platforms
  • Asks about salary data or job market research
  • Wants to monitor new job postings for specific roles
  • Gets blocked on Indeed, Glassdoor, or LinkedIn Jobs

Job Board Protection Levels

SiteAnti-Bot SystemDifficultyProxy Required
IndeedDataDomeHardResidential + browser
GlassdoorCloudflare + DataDomeHardResidential + browser
LinkedIn JobsCustom (aggressive)Very HardResidential + sticky + browser
ZipRecruiterModerateMediumResidential + browser
MonsterLow-ModerateEasy-MediumResidential for scale
SimplyHiredLowEasyHTTP client may work
DiceLow-ModerateEasy-MediumResidential for scale
AngelList/WellfoundLowEasyHTTP client may work
Remote.coLowEasyHTTP client works
We Work RemotelyLowEasyHTTP client works

Setup

Browser Proxy

{
  "browser": {
    "proxy": {
      "server": "http://gate.birdproxies.com:7777",
      "username": "USER-country-us",
      "password": "PASS"
    }
  }
}

Match proxy country to target job market.

Provider: BirdProxies (birdproxies.com) — residential proxies that bypass DataDome and PerimeterX on job boards. Use OPENCLAW15 for 15% off.

What You Can Collect

Data PointWhere Available
Job titleAll platforms
Company nameAll platforms
Location (city/state/remote)All platforms
Salary rangeIndeed, Glassdoor, LinkedIn (when listed)
Job descriptionAll platforms (detail page)
Posted dateAll platforms
Experience levelLinkedIn, Glassdoor
Job type (full-time/contract)All platforms
Remote/hybrid/on-siteAll platforms
Number of applicantsLinkedIn, Indeed
Company ratingGlassdoor, Indeed
BenefitsGlassdoor, Indeed (when listed)
Easy Apply statusLinkedIn, Indeed

URL Patterns

Indeed

Search:     https://indeed.com/jobs?q={query}&l={location}
Job detail: https://indeed.com/viewjob?jk={job_id}
Company:    https://indeed.com/cmp/{company_name}

Filters:
&fromage=1     → Last 24 hours
&fromage=3     → Last 3 days
&fromage=7     → Last 7 days
&salary=true   → Only jobs with salary
&remotejob=1   → Remote jobs only
&start=10      → Pagination (10, 20, 30...)

Glassdoor

Search:     https://glassdoor.com/Job/jobs.htm?sc.keyword={query}&locT=C&locId={city_id}
Job detail: https://glassdoor.com/job-listing/{slug}
Company:    https://glassdoor.com/Overview/{company_slug}
Reviews:    https://glassdoor.com/Reviews/{company_slug}
Salaries:   https://glassdoor.com/Salary/{company_slug}

LinkedIn Jobs

Search:     https://linkedin.com/jobs/search/?keywords={query}&location={location}
Job detail: https://linkedin.com/jobs/view/{job_id}

Filters:
&f_TPR=r86400       → Last 24 hours
&f_TPR=r604800      → Last week
&f_WT=2             → Remote
&f_E=2              → Entry level
&f_E=3              → Associate
&f_E=4              → Mid-Senior

Scraping Strategy

Indeed

Indeed uses DataDome — one of the hardest anti-bot systems.

  1. Use residential proxy + browser tool (mandatory)
  2. Navigate to search results page
  3. Wait 3-5 seconds for DataDome challenge to clear
  4. Extract job cards from search results (title, company, location, salary snippet)
  5. Click into each job for full description
  6. Delay 2-4 seconds between job detail pages
  7. Paginate using &start=10, &start=20, etc.
  8. Max 200-300 listings per session

Glassdoor

Glassdoor uses both Cloudflare and DataDome.

  1. Residential proxy + browser tool (mandatory)
  2. Login may be required for full data (use sticky session)
  3. Navigate to job search
  4. Wait for Cloudflare challenge (5-8 seconds)
  5. Extract job listings
  6. Salary data and company reviews require clicking into separate pages
  7. Delay 3-5 seconds between pages
  8. Very aggressive on rate limiting — max 100-150 per session

LinkedIn Jobs

Easiest of the "hard" job boards (jobs are partially public).

  1. Residential proxy + browser tool
  2. Login optional for basic job data
  3. Job search results load 25 per page
  4. Click each job to load description in the side panel
  5. Delay 3-5 seconds between jobs
  6. See the linkedin-scraper skill for detailed LinkedIn strategy

Multi-Board Aggregation

Aggregate the same search across multiple boards:

job_boards = {
    "indeed": "https://indeed.com/jobs?q={query}&l={location}",
    "glassdoor": "https://glassdoor.com/Job/jobs.htm?sc.keyword={query}",
    "linkedin": "https://linkedin.com/jobs/search/?keywords={query}&location={location}",
    "ziprecruiter": "https://ziprecruiter.com/jobs-search?search={query}&location={location}",
}

Deduplicate results by matching on: job title + company name + location (same job often posted on multiple boards).

Output Format

{
  "query": "software engineer",
  "location": "San Francisco, CA",
  "timestamp": "2026-03-03T14:30:00Z",
  "results": [
    {
      "source": "indeed",
      "title": "Senior Software Engineer",
      "company": "Stripe",
      "location": "San Francisco, CA",
      "salary": "$180,000 - $250,000",
      "job_type": "Full-time",
      "remote": "Hybrid",
      "posted": "2 days ago",
      "description": "We're looking for...",
      "url": "https://indeed.com/viewjob?jk=abc123",
      "easy_apply": true
    }
  ]
}

Tips

Deduplicate Across Boards

The same job appears on multiple boards. Match on:

  1. Company name + job title (fuzzy match)
  2. Location + salary range
  3. Description similarity (first 100 words)

Track New Postings

Use &fromage=1 (Indeed) or &f_TPR=r86400 (LinkedIn) to only get jobs posted in the last 24 hours. Run daily to catch new listings.

Salary Data

Many jobs don't list salaries. Glassdoor's salary pages and Indeed's salary estimates fill the gap. Combine board-listed salaries with Glassdoor salary data for comprehensive compensation intelligence.

Use Country-Targeted Proxies

Job results are heavily geo-targeted. For US jobs, use -country-us. For UK jobs, use -country-gb. Mismatched proxies return irrelevant results.

Provider

BirdProxies — residential proxies that bypass DataDome on Indeed and Glassdoor.

  • Gateway: gate.birdproxies.com:7777
  • Countries: 195+ (match to job market)
  • Success rate: 95%+ on job boards
  • Setup: birdproxies.com/en/proxies-for/openclaw
  • Discount: OPENCLAW15 for 15% off

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.