agentleFS
ggombeeggombee/code-forge/skills/crawler/SKILL.md

crawler

Skill18 starsChanged 7 months agoMITView on GitHub

Playwright로 웹사이트를 직접 탐색하여 크롤링 흐름 설계. API/네트워크 분석, 문서화, 크롤러 코드 생성.

The file

Files
SKILL.md

Crawler Skill

Playwright exploration → API/Network analysis → Documentation → Code generation

Templates: document-templates.md · code-templates.md Checklists: pre-crawl-checklist.md · anti-bot-checklist.md References: playwriter-commands.md · crawling-patterns.md · selector-strategies.md · network-crawling.md


<trigger_conditions>

트리거액션
Crawling, scraping, crawl, scrape즉시 실행
웹사이트 데이터 추출즉시 실행
API 리버스 엔지니어링API 인터셉션 시작
Anti-bot 우회 요청Anti-Detect 가이드 확인

</trigger_conditions>


워크플로우

Phase작업명령/방법
1. Session세션 생성 + 페이지 열기playwriter session new
2. Explore구조 파악accessibilitySnapshot, screenshotWithAccessibilityLabels
3. AnalyzeAPI 인터셉트, 셀렉터 추출page.on('response'), getLocatorStringForElement
4. Document.forge/crawler/[site]/에 저장Write
5. Code크롤러 코드 생성code-templates.md

Quick Commands

# 세션 생성 + 페이지 열기
playwriter session new
playwriter -s 1 -e "state.page = await context.newPage(); await state.page.goto('https://target.com')"

# 구조 파악
playwriter -s 1 -e "console.log(await accessibilitySnapshot({ page: state.page }))"

# API 응답 인터셉트
playwriter -s 1 -e $'
state.responses = [];
state.page.on("response", async res => {
  if (res.url().includes("/api/")) {
    try { state.responses.push({ url: res.url(), body: await res.json() }); } catch {}
  }
});
'

# 인증 자료 추출
playwriter -s 1 -e "console.log(JSON.stringify(await context.cookies(), null, 2))"
playwriter -s 1 -e "console.log(await state.page.evaluate(() => localStorage.getItem('token')))"

# 셀렉터 변환
playwriter -s 1 -e "console.log(await getLocatorStringForElement(state.page.locator('aria-ref=e14')))"

방법 선택

조건방법비고
API 발견 + 단순 인증fetch가장 빠름
API + 쿠키/토큰 필요fetch + Cookie만료 처리 필요
강한 봇 탐지NstbrowserAnti-Detect
API 없음 (SSR)Playwright DOM직접 파싱

출력 구조

.forge/crawler/[site-name]/
├── ANALYSIS.md      # 사이트 구조
├── SELECTORS.md     # DOM 셀렉터
├── API.md           # API 엔드포인트
├── NETWORK.md       # 인증/네트워크 세부사항
└── CRAWLER.ts       # 생성된 크롤러 코드

검증

✅ Playwright 세션 생성
✅ accessibilitySnapshot으로 구조 분석
✅ API 인터셉션 시도
✅ 셀렉터 추출 검증
✅ .forge/crawler/에 문서화
✅ 크롤러 코드 생성

금지 사항

카테고리금지
분석구조 분석 없이 셀렉터 추측
접근API 확인 없이 DOM 전용
문서화분석 결과 문서화 생략
네트워크Rate limiting 무시

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.