project-scraper
Skill0 starsChanged 3 months agoMITView on GitHub
Orchestrates project search across multiple sources
In this repository6 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- awesome-searchSkill
- github-searchSkill
- project-recommenderSkill
- user-profileSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 3 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈3
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 25
Every /discover run is logged to project_scraper/search_history.json.
- Distinctive: line 63
Learn preferences: if user consistently likes lightweight tools, boost similar projects
- Distinctive: line 64
Refine queries: if "React" searches return too many boilerplates, add "production-ready" filter
- Distinctive: line 65
Track skill evolution: if user starts searching for "Rust" after initially saying they don't know it, update inferred skills
- Very specific: line 81
After user feedback: update the last entry's user_feedback field
- Very specific: line 86
Keep the last 50 search entries in search_history.json.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Project Scraper Skill
Orchestrates search across GitHub, Awesome lists, and curated sources. Logs every search for future reference.
When to use
/discovercommand- Background project discovery
Sources
- GitHub Search API — repositories by topic, language, stars, activity
- Awesome lists — curated lists on GitHub (awesome-react, awesome-nodejs, etc.)
- Topic exploration — discover related topics from high-scoring projects
Deduplication
Projects may appear in multiple sources. Dedup by full-name (owner/repo).
Track seen projects in project_scraper/seen_projects.json.
Search History
Every /discover run is logged to project_scraper/search_history.json. This file grows over time and becomes the agent's memory of past searches.
What gets logged
Each search entry records:
{
"id": "2026-08-26T14:30:00",
"query": {
"topics": ["react", "state management"],
"languages": ["TypeScript"],
"min_stars": 100,
"filters": {}
},
"context": {
"skills_used": ["React", "TypeScript", "Node.js"],
"interests_matched": ["web development", "developer tools"],
"reason": "User asked for React state management libraries"
},
"results": {
"total_found": 47,
"top_picks": [
{"repo": "zustandjs/zustand", "score": 92, "reason": "Perfect stack match, active community"},
{"repo": "pmndrs/jotai", "score": 88, "reason": "Atomic model, good first issues"}
]
},
"user_feedback": {
"liked": ["zustandjs/zustand"],
"disliked": [],
"notes": "Prefers minimal API, lightweight solutions"
}
}How history influences future searches
- Avoid repetition: skip repos already recommended (unless user asks to re-scan)
- Learn preferences: if user consistently likes lightweight tools, boost similar projects
- Refine queries: if "React" searches return too many boilerplates, add "production-ready" filter
- Track skill evolution: if user starts searching for "Rust" after initially saying they don't know it, update inferred skills
- Identify patterns: "user always rejects projects with >50k stars" → they prefer smaller, approachable projects
History file structure
project_scraper/
├── search_history.json # All past searches (gitignored)
├── seen_projects.json # Dedup index (gitignored)
└── README.md # This file (tracked)When to update history
- Before searching: load
search_history.jsonto inform query strategy - After searching: append the new search entry
- After user feedback: update the last entry's
user_feedbackfield - On
/discover --fresh: still log the search, but ignoreseen_projects.jsonfor dedup
History limits
Keep the last 50 search entries in search_history.json. When the file exceeds 50 entries, archive older ones to project_scraper/archive/ (also gitignored). This prevents the file from growing unbounded while preserving recent context.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

