agentleFS
Sign inStart free
XuHo-ITXuHo-IT/Content-Agent-Kit/skills/crawl-and-queue/SKILL.md

crawl-and-queue

Skill3 starsChanged 3 months agoMITView on GitHub

Run the crawl4ai discovery pipeline to fill the idea queue for an agent built with content-agent-kit. Use when the user says "crawl for ideas / news", "refresh the idea queue", or when setting up scheduled discovery. Crawls sources.yaml, dedups against the queue API, pushes fresh excerpts.

  • Installs packages

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Copyright: never store or republish full source text.

    line 18

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Crawl → idea queue

  1. Check sources.yaml exists and each source has a list_url + a tuned link_pattern. Loose patterns catch navigation links — verify with a dry run: python scripts/crawl/crawl.py --dry-run (prints found links only, pushes nothing).
  2. Ensure deps (once): pip install -r scripts/crawl/requirements.txt && crawl4ai-setup (installs Chromium — run in CI or on a real machine, not serverless).
  3. Ensure env: SITE_URL, INGEST_API_TOKEN (+ QUEUE_PATH if not /api/queue).
  4. Run: python scripts/crawl/crawl.py [--limit N]. It:
    • GETs ?known=1 to skip URLs the server already has (server = dedup memory),
    • extracts new article links, crawls them, keeps excerpt ≤1500 chars + link only,
    • POSTs { items } to the queue (upsert, unique on source_url).
  5. Copyright: never store or republish full source text. The publishing agent writes ORIGINAL content from the idea. Prefer public-domain / permitted sources.
  6. For automation, install the cron workflow from templates/workflows/crawl.yml.template and add SITE_URL + INGEST_API_TOKEN to repo secrets.

Antigravity: same steps, run the commands directly.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free