add-source
Skill2.6k starsChanged 3 months agoGPL-3.0View on GitHub
Create or fix a source crawler — base-class choice (SoupTemplate/shared templates), selectors, cleaner tuning, reading a chapter list correctly, testing via CLI. Use when adding a new source site, fixing a broken crawler, or working in sources/ or lncrawl/templates/.
11 other projects use a version of this.
Trending
Quiet since March.about the whole repository
Most recently taken up by
- dosu-ai/decanttheir own version, first seen Nov 6, 2026
- seonwooj0810/careerlooptheir own version, first seen Oct 20, 2026
- IvanWng97/pixtuoidtheir own version, first seen Oct 14, 2026
- pixtuoid/pixtuoidtheir own version, first seen Oct 13, 2026
- bruin-data/ingestrtheir own version, first seen Oct 8, 2026
- lncrawl/lightnovel-crawlerword for word, first seen Oct 8, 2026
Who's talking about dipu-bd/lightnovel-crawler
- @mara.x0f.nl“this seems useful to back up things from scribblehub/etc github.com/dipu-bd/ligh...”Bluesky · Mar 2, 2025 · 12 likes
- lncrawl/lightnovel-crawler“Version 3”GitHub · Feb 10, 2020 · 4 reactions
- lncrawl/lightnovel-crawler“www.webnovel.com _csrfToken”GitHub · Jun 19, 2024 · 3 reactions
- lncrawl/lightnovel-crawler“Fix https://puretl.com/”GitHub · Sep 26, 2022 · 2 reactions
- lncrawl/lightnovel-crawler“Update source link: allnovelfull.com ==>> allnovelfull.net”GitHub · Oct 17, 2023 · 1 reactions
The biggest 5 of 53 mentions linking to the repository that our crawl found. Each line is their own headline.
See all 53
Talked about on GitHub, Stack Overflow and Bluesky (53 times).
On GitHub · 50
- Feb 10, 2020Version 3 — lncrawl/lightnovel-crawler4 reactions · 7 comments
- Jun 19, 2024www.webnovel.com _csrfToken — lncrawl/lightnovel-crawler3 reactions · 5 comments
- Sep 26, 2022Fix https://puretl.com/ — lncrawl/lightnovel-crawler2 reactions
- Oct 17, 2023Update source link: allnovelfull.com ==>> allnovelfull.net — lncrawl/lightnovel-crawler1 reaction
- Sep 26, 2022[template] WordPress MangaStream (WP MangaStream) — lncrawl/lightnovel-crawler1 reaction · 16 comments
45 more on GitHub
- Sep 10, 2022New source https://novelsala.com/ — lncrawl/lightnovel-crawler1 reaction · 8 comments
- Jan 25, 2022Fix this source https://www.royalroad.com/ — lncrawl/lightnovel-crawler1 reaction · 2 comments
- Nov 4, 2019Bug: Fanfiction.net hardcoded to specific story — lncrawl/lightnovel-crawler1 reaction · 1 comment
- Oct 22, 2019Cannot download novel with an apostrophe — lncrawl/lightnovel-crawler1 reaction · 8 comments
- Mar 15, 2026Error/Bug when trying to crawl on WuxiaWorld — lncrawl/lightnovel-crawler0 reactions · 1 comment
- Jan 22, 2026Fix this source: royalroad.com — lncrawl/lightnovel-crawler0 reactions
- Aug 25, 2025Fix this source: https://novelfull.com/ — lncrawl/lightnovel-crawler0 reactions · 1 comment
- Jul 14, 2025Wuxiaworld.com KeyError — lncrawl/lightnovel-crawler0 reactions · 1 comment
- Jul 7, 2025Fix this source: ncode.syosetu.com — lncrawl/lightnovel-crawler0 reactions · 1 comment
- Jun 16, 2025Update lightnovel-crawler requirement from ~=3.8.0 to ~=3.10.1 — Tech-TTGames/KumoFeaturedBot0 reactions · 1 comment
- Jun 2, 2025Update lightnovel-crawler requirement from ~=3.8.0 to ~=3.9.4 — Tech-TTGames/KumoFeaturedBot0 reactions · 1 comment
- Apr 23, 2025Fix this source reaperscans.com — lncrawl/lightnovel-crawler0 reactions · 2 comments
- Mar 18, 2025WanderingInn: retain colored text — lncrawl/lightnovel-crawler0 reactions · 1 comment
- Jan 13, 2025Update lightnovel-crawler requirement from ~=3.7.2 to ~=3.7.5 — Tech-TTGames/KumoFeaturedBot0 reactions · 3 comments
- Dec 30, 2024Update lightnovel-crawler requirement from ~=3.7.2 to ~=3.7.4 — Tech-TTGames/KumoFeaturedBot0 reactions · 1 comment
- Dec 10, 2024Add wtrlab source — lncrawl/lightnovel-crawler0 reactions
- Jul 8, 2024Update lightnovel-crawler requirement from ~=3.5.1 to ~=3.7.2 — Tech-TTGames/KumoFeaturedBot0 reactions · 2 comments
- Jun 18, 2024https://www.69shuba.pro/ — lncrawl/lightnovel-crawler0 reactions · 2 comments
- May 13, 2024Update lightnovel-crawler requirement from ~=3.5.1 to ~=3.7.1 — Tech-TTGames/KumoFeaturedBot0 reactions · 1 comment
- May 6, 2024Update lightnovel-crawler requirement from ~=3.5.1 to ~=3.6.0 — Tech-TTGames/KumoFeaturedBot0 reactions · 1 comment
- Apr 8, 2024Update lightnovel-crawler requirement from ~=3.5.0 to ~=3.5.1 — Tech-TTGames/KumoFeaturedBot0 reactions · 1 comment
- Mar 12, 2024Novelupdates Fix this source — lncrawl/lightnovel-crawler0 reactions · 2 comments
- Mar 8, 2024Fix this source wuxiaworld.com — lncrawl/lightnovel-crawler0 reactions · 2 comments
- Feb 26, 2024Update lightnovel-crawler requirement from ~=3.4.0 to ~=3.5.0 — Tech-TTGames/KumoFeaturedBot0 reactions · 3 comments
- Feb 19, 2024Update lightnovel-crawler requirement from ~=3.4.0 to ~=3.4.2 — Tech-TTGames/KumoFeaturedBot0 reactions · 1 comment
- Feb 12, 2024Fix this source — lncrawl/lightnovel-crawler0 reactions · 5 comments
- Dec 25, 2023Feature Request: Add filename formatting — lncrawl/lightnovel-crawler0 reactions
- Oct 26, 2023INSPIRATION FOR THE CREATOR — Taraflex/ranobe-ebook-loader0 reactions
- Oct 21, 2023Fix sources and Added new sources — lncrawl/lightnovel-crawler0 reactions · 3 comments
- Oct 14, 2023Novelsemperor "No chapters found" — lncrawl/lightnovel-crawler0 reactions · 2 comments
- Aug 31, 2023[novelbin.net] Fix this source — lncrawl/lightnovel-crawler0 reactions · 7 comments
- Aug 9, 2023Novelupdates Segmentation fault — lncrawl/lightnovel-crawler0 reactions · 2 comments
- Jul 20, 2023Fix this source — lncrawl/lightnovel-crawler0 reactions · 3 comments
- Jun 4, 2023Fix this source mtlnation.com — lncrawl/lightnovel-crawler0 reactions · 1 comment
- May 21, 2023Fix this source — lncrawl/lightnovel-crawler0 reactions · 1 comment
- May 20, 2023Error processing HTML sources with non-BMP characters — lncrawl/lightnovel-crawler0 reactions · 1 comment
- Nov 24, 2022[www.scribblehub.com] Detected a Cloudflare version 2 challenge — lncrawl/lightnovel-crawler0 reactions
- Nov 7, 2022Fix novelcool — lncrawl/lightnovel-crawler0 reactions
- Oct 10, 2022New template MangaStream — lncrawl/lightnovel-crawler0 reactions · 37 comments
- Oct 4, 2022New source https://novelhulk.com/ — lncrawl/lightnovel-crawler0 reactions · 1 comment
- Oct 3, 2022New source https://freewebnovel.com/ — lncrawl/lightnovel-crawler0 reactions
- Oct 2, 2022New source https://snowycodex.com/ — lncrawl/lightnovel-crawler0 reactions · 4 comments
- Oct 1, 2022New source https://novelfull.me/ — lncrawl/lightnovel-crawler0 reactions
- Sep 30, 2022[403 error] New source https://novel35.com/ — lncrawl/lightnovel-crawler0 reactions · 4 comments
- Sep 30, 2022New source https://novelnext.com/ — lncrawl/lightnovel-crawler0 reactions
On Stack Overflow · 2
- Feb 22, 2020Smartphone as a server1 vote
- Mar 27, 2020Heroku Git object enumeration is wrong, counting files that were already pushed0 votes
On Bluesky · 1
- Mar 2, 2025this seems useful to back up things from scribblehub/etc github.com/dipu-bd/ligh... — @mara.x0f.nl12 likes and reposts · 1 reply
Other versions11 projects use a version of this.
Word for word · 1
- lncrawl/lightnovel-crawler★ 2.6k100% same
Edited · 0
None found.
Wrote their own version · 10
- bruin-data/ingestr★ 4ksame name
- IvanWng97/pixtuoid★ 551same name
- pixtuoid/pixtuoid★ 551same name
- dosu-ai/decant★ 74same name
- NVIDIA/physicsnemo-curator★ 67same name
- assaio/assaio★ 63same name
- qbeka/jev-job-search★ 5same name
- jacksoswag/ObsidAgent★ 1same name
- Miracle-Aligner/job-search-hq★ 1same name
- seonwooj0810/careerloop★ 0same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository8 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- add-api-endpointSkill
- add-job-typeSkill
- db-migrationSkill
- output-and-translationSkill
- releasingSkill
- triage-source-issuesSkill
The file
We split this skill into the rules it gives an agent
7 rules. 0 are common, 1 is distinctive, and 6 are very specific to this project.
0
Common
Most of the 6 edited versions we read keep these.
1
Distinctive
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
Licensed with conditions — read it at the source. We count its rules but don't quote them. Read them on GitHub.
“Common” is measured: kept by at least 6 in 10 of the 6 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.