debug-site-quirks
Skill0 starsChanged 5 months agoApache-2.0View on GitHub
GEODoctor 个别真实站点的审计怪象排查手册——403 拒绝、SPA 空壳误判、乱码、超时、sitemap 找不到时使用(多数站点都异常则去 debug-audit-pipeline)。含 openai.com 案例与不许伪装 UA 的红线。
In this repository14 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- acceptance-gatesSkill
- architecture-contractsSkill
- bilingual-docs-contractSkill
- debug-audit-pipelineSkill
- debug-test-failuresSkill
- diagnostic-commandsSkill
- evidence-registry-contractSkill
- failure-archaeologySkill
- release-versioning-controlSkill
- rule-change-controlSkill
- security-boundariesSkill
- showcase-refresh-protocolSkill
- skill-mapSkill
The file
Files
真实站点怪象排查
适用场景
某个具体网站的审计结果反常,且已确认不是本工具代码 bug。
怪象速查
403 / 直接拒绝(openai.com 现象)
我们的 UA 是真实身份:Mozilla/5.0 (compatible; GEODoctor/0.1; +repo链接)。反爬严格的站(openai.com 实测)会直接拒绝。处置:
- 这是有效的审计发现,不是 bug——用与 AI 爬虫同类的方式访问被拒,说明该站对 AI 爬虫大概率同样不友好。showcase 榜单对此的呈现惯例是"拒绝访问"行(见 showcase-refresh-protocol)
- 红线:不许改 UA 伪装成浏览器绕过。这是白帽定位的工程体现,PR 里出现伪装 UA 直接拒(security-boundaries 有完整边界)
SPA 空壳的误判两侧
access.js-dependency 的判定是启发式:bodyText < 150 units 且(存在近空的 #root/#app/#__next/#___gatsby 容器 或 scriptBytes > 100KB)→ 0 分。两个已知误差方向:
- 漏报:SSR 首屏 + 客户端补水的站判 pass——正确,AI 爬虫确实读得到首屏
- 误报:极简落地页(正文本来就 < 150 units)+ 大 bundle 会被当空壳。注意这个 150 是 js-dependency 的判 0 分线,与内容规则的 150 适用门槛(na 线)语义不同,只是数值巧合(辨析见 rule-change-control)。核对方法:
curl -s <url> | grep -c "关键正文词",HTML 里真有正文就是误报,考虑给该规则提 issue 而不是现场改阈值(改阈值走 rule-change-control)
中文站乱码 / charset 怪象
解码固定 TextDecoder('utf-8', { fatal: false })——GBK 站点会解出乱码但不会崩,随后 structure.lang-charset 会警告。这是接受的行为:主流 AI 爬虫同样默认 UTF-8。若乱码导致正文规则全失真,在报告里体现为多条规则 finding 异常,不要单独"修"某一条。
超时与慢站
-t(默认 15000ms)是单请求超时,不是总预算——3 页 + robots + llms + sitemap 最多 6+ 次请求,慢站总耗时可到分钟级。批量跑榜单用 -t 8000 截断长尾(showcase 的既定参数)。timeout 错误在 FetchResult.error 里是字符串 'timeout'(describeFetchError 归一)。
sitemap 明明有却报没有
resolveSitemap 是二选一,不是两处都查:robots.txt 有 Sitemap 声明 → 只试声明的前 2 条,失败不回退 /sitemap.xml;无声明 → 才试 /sitemap.xml。推论:站点 robots 声明了一条坏 Sitemap(404 或被 SSRF 过滤),哪怕 /sitemap.xml 完好也会判"无 sitemap"。另不递归 sitemapindex、不猜 /sitemap_index.xml——以上均为 v0.1 已知局限。另外 robots 声明的 sitemap URL 要过 safePublicUrl SSRF 过滤,指向内网/元数据地址的声明会被静默丢弃(安全设计,见 security-boundaries)。
什么时候不该用这份文档
- 所有站都不对(不是个别站)→ debug-audit-pipeline,那是代码问题
- 本地 fixture/测试问题 → debug-test-failures
- 想为怪象改规则阈值 → rule-change-control(有 fixture 余量和榜单失效链约束)
姊妹文档
debug-audit-pipeline(分层探针)· security-boundaries(UA 伦理与 SSRF 过滤)· showcase-refresh-protocol(榜单如何呈现拒绝访问)
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

