agentleFS
Sign inStart free
zedarvateszedarvates/botte-secrete/skills/ingest/SKILL.md

ingest

Skill2 starsChanged 3 months agoMITView on GitHub

Local-first web scraping and source ingestion — fetch a URL, extract clean text locally (0 cloud tokens), optionally structure it with a local model, and store it in a Qdrant collection (the second-brain "foundation") for later recall. Use for /web-scraping, /ingest-source, building a knowledge foundation, or recalling past ingested sources. Web fetch + Qdrant over stdlib HTTP.

88 other projects use a version of this, 1 of them edited.

Adopted by

Most recently taken up by

Used with

Projects with it also use daily-arxiv, exp-eval and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • LambenthanSkill

    daily-arxiv

    每日从 arXiv 拉取新论文，过滤相关性，auto-ingest 高优先级论文，检测 SOTA 更新

    Also in 5 projects that use this

  • LambenthanSkill

    exp-eval

    实验判决门：Review LLM 独立评判实验结果 → 4 种判决路径 → 自动更新 claims confidence、ideas status、graph edges

    Also in 5 projects that use this

  • phanghonghaoSkill

    poster

    HTML/CSS A4 海报生成。从沙龙/活动 Markdown 文件生成科技风 A4 海报 PDF/PNG。Use when user mentions: poster, 海报, 海报生成, make poster, generate poster, 活动海报, 沙龙海报.

    Also in 5 projects that use this

  • skyllwtSkill

    exp-pilot-eval

    Pilot result evaluation — read pilot results, apply success criteria, update idea page (pilot_result, failure_reason if failed), generate PILOT_VERDICT_REPORT. Called by /ideate Phase 5 after /exp-pilot-run.

    Also in 4 projects that use this

  • skyllwtSkill

    exp-pilot-run

    Pilot experiment execution — read Pilot Spec YAML, write pilot code, run experiment(Confirm with the user before operation and require the applicant to conduct manual inspection), return results. Called by /ideate Phase 5. Does NOT modify wiki pages or judge pass/fail.

    Also in 4 projects that use this

  • +6 more from Lambenthan
  • +1 more from skyllwt

The file

Create a free account to mount
Files
SKILL.md

ingest — scrape the web & build a knowledge foundation, locally

Scraping and ingesting are extraction/transformation — keep them off the cloud.

python -m skills.ingest.cli scrape  https://example.com --structure   # local summary+entities
python -m skills.ingest.cli ingest  https://example.com --collection foundation
python -m skills.ingest.cli ingest  ./notes.md --file
python -m skills.ingest.cli search  "topic"  --collection foundation
  • scrape — fetch (browser UA) + stdlib HTML→text (drops script/style, keeps title). --structure asks a LOCAL model for a 3-bullet summary + key entities. 0 cloud tokens.
  • ingest — scrape (or a file/raw text) → reflect locally → upsert into Qdrant (192.168.1.47:6333 by default). Builds the "foundation"/historical store.
  • search — recall from a collection.

Embeddings: auto-resolve a local /v1/embeddings endpoint from the backend registry (any reachable backend exposing an embedding model → real semantic vectors), falling back to a deterministic hash n-gram (256-dim) when none is available — so it always works, 0 cloud tokens either way. Override with --embed-url/--embed-model. The result reports embed: endpoint|hash. Degrades gracefully when Qdrant is down (scrape still works).

python -m skills.ingest.cli ingest https://example.com --embed-url http://127.0.0.1:1234/v1/embeddings

Exposed via [[llm_mcp]] as scrape and ingest_source. Related: [[hermes-second-brain]] (the foundation concept), media_loader, [[auto_router]].

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free