Article Extractor

Extracts the main content from web articles and blog posts, removes navigation, ads, newsletter signups, and other clutter, and saves the result as clean readable text.

Contents

• When to Use
• Priority Order
• Quick Examples
• Verification
• References

When to Use

Activate when the user:

• Provides an article/blog URL and wants the text content
• Says "download this article", "extract content from [URL]", or "save this blog post as text"
• Needs clean article text without distractions

Priority Order

1. Detect available tool: reader → trafilatura → fallback
2. Extract content with the best available tool
3. Extract title for filename
4. Clean filename for filesystem (strip illegal chars, cap length)
5. Save to [clean-title].txt
6. Show preview and verify (see Verification)

See references/installation.md for tool detection and install commands.

Quick Examples

reader (recommended):

reader "URL" > article.txt
TITLE=$(head -n 1 article.txt | sed 's/^# //')

trafilatura (best for news/non-English):

trafilatura --URL "URL" --output-format txt --no-comments > article.txt

fallback (no dependencies):

See references/method-fallback.md — uses curl + a small inline Python parser.

For the full auto-detect + filename-cleanup script, see references/full-workflow.md.

Output Contract

The saved file should contain:

• Article title (when the tool reports one)
• Author and publish date (when available — trafilatura JSON, sometimes reader)
• Main article text with section headings preserved

The saved file should not contain:

• Navigation menus, headers, footers
• Ads and promotional content
• Newsletter signup forms
• Related-articles sidebars
• Cookie / consent banners
• Social-media buttons
• Comment sections (suppress with trafilatura --no-comments)

If the output contains any of the above, the extractor either picked the wrong method or the site is hostile to extraction (heavy JS, anti-scraping). Retry with the next tool in the priority chain or surface the failure.

Verification

After extraction, confirm all of:

• Output file exists and is non-empty
• First 10 lines look like article prose, not nav/ads/cookie banners (matches the Output Contract above)
• Title was extracted (filename is meaningful, not Article.txt)
• Filename has no illegal characters (/, :, ?, ", <, >, |)
• Show the preview to the user (first 10–15 lines). The standard report is: extracted title, save path, file size, then preview.

If the site is a JavaScript SPA or requires authentication, all extractors will fail — tell the user explicitly rather than saving empty output.

References

• references/installation.md — tool detection and install
• references/method-reader.md — reader (Mozilla Readability)
• references/method-trafilatura.md — trafilatura flags and use cases
• references/method-fallback.md — curl + Python fallback parser
• references/full-workflow.md — complete end-to-end script
• references/troubleshooting.md — paywall, JS SPAs, noisy output, edge cases