social-content-parser
Skill3 starsChanged 5 months agoMITView on GitHub
Parse public short-video and social share URLs into local content packages through platform-specific APIs plus an aggregated fallback covering 20+ platforms including Douyin, Kuaishou, Xiaohongshu, Bilibili, X/Twitter, YouTube, TikTok, Xigua, Haokan, Weishi, Pear Video, AcFun, Zhihu, Oasis, Meipai, Quanmin, Huya, Instagram, Doubao, Jimeng AI, and WeChat Channels. Use for extracting or archiving public links, noisy share parameters, FXTwitter, fallback routes, or when the user wants a share-link video transcribed with the companion video-to-text skill.
In this repository8 other files it gives its agents.
- agent-pipeline-orchestrationSkill
- ai-news-digestSkill
- api-data-researchSkill
- grok-x-searchSkill
- markdown-platform-packSkill
- miniapp-figma-alignmentSkill
- video-to-textSkill
- weijincha-content-checkSkill
The file
Show its rules and text
We split this skill into the rules it gives an agent
15 rules. About 0 are common, 7 are distinctive, and 8 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
8
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Use this skill to turn a public short-video or social share URL into a local content package.
- Very specific: line 27
Use scripts/parse_social.mjs as the primary entrypoint.
- Distinctive: line 34
If the primary parser fails or returns an empty payload, try the platform backup parser when available.
- Distinctive: line 35
If both platform parsers fail, fall back to the aggregated short-video parser.
- Very specific: line 38
Write a human-readable report.md.
- Very specific: line 40
Only when --transcribe is explicitly supplied, select the first normalized video and invoke video-to-text.
- Very specific: line 128
Add --no-download when only the transcript is needed.
- Distinctive: line 140
Do not assume every fallback wants the same cleaned URL.
- Distinctive: line 141
When the primary parser misses content, the skill can fall back to the aggregated short-video parser to recover a usable result.
- Very specific: line 147
Do not use the official key=value sample as a real parameter.
- Distinctive: line 148
Do not invent engagement metrics.
- Distinctive: line 149
Do not assume every platform returns the same schema.
- Very specific: line 150
Do not download videos unless the user asks or --download-video is provided.
- Very specific: line 151
Do not transcribe unless the user asks or --transcribe is provided.
- Very specific: line 152
Treat --transcribe as authorization for one Tencent ASR task.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Social Content Parser
Use this skill to turn a public short-video or social share URL into a local content package. Treat the aggregated fallback as the broad coverage layer; use the platform-specific routes when the link clearly belongs to Xiaohongshu, Douyin, Bilibili, or X/Twitter.
Supported platforms:
- Xiaohongshu via
https://api.bugpk.com/api/xhsjx - Douyin works via
https://api.bugpk.com/api/douyin - Douyin profiles via
https://api.bugpk.com/api/dyzy - Bilibili via
https://api.bugpk.com/api/bilibili - X/Twitter via
https://api.fxtwitter.com/:screen_name/status/:id - Aggregated fallback via
https://api.bugpk.com/api/short_videosfor 20+ platforms
Fallback order:
- Platform-specific parser.
- Platform-specific backup parser when available.
- Aggregated short-video fallback.
Workflow
- Use
scripts/parse_social.mjsas the primary entrypoint. - Pass a public share URL with
--url. - Let the script auto-detect the platform from the URL.
- For Douyin, let the script choose profile or work parsing:
/user/...links or--kind profileuse the profile API and returnitems.- video, image, and live-photo share links use the work API and return
type=video|image|live.
- Strip tracking query parameters from share URLs before parsing.
- If the primary parser fails or returns an empty payload, try the platform backup parser when available.
- If both platform parsers fail, fall back to the aggregated short-video parser.
- Preserve the raw API response in
raw.json. - Normalize shared fields into
normalized.json. - Write a human-readable
report.md. - Download avatar, cover, images, live-photo media, and optional videos into
media/unless disabled. - Only when
--transcribeis explicitly supplied, select the first normalized video and invokevideo-to-text; write the result beside the other content files.
Output Contract
Default output root:
./social-notes/
Each parsed item creates one subfolder:
<platform>-<title>-<YYYY-MM-DD>-<id>/
Files:
report.mdnormalized.jsonraw.jsonmedia/transcript.txt|json|srt|mdwhen transcription is requested and succeeds
Normalized fields:
platformkindcanonicalUrlfallbackUsedsourceUrlcodemsgtypetitledescauthor.nameauthor.idauthor.avatarcoverprimaryVideovideoBackupsimageslivePhotomusicstatisticspartsitemspaginationtranscription.enabledtranscription.statustranscription.requestedModetranscription.usedModetranscription.formattranscription.output
Commands
node scripts/parse_social.mjs --url "https://v.douyin.com/..."
node scripts/parse_social.mjs --url "https://www.douyin.com/user/..." --count 10
node scripts/parse_social.mjs --url "https://b23.tv/..."
node scripts/parse_social.mjs --url "https://www.xiaohongshu.com/explore/..."
node scripts/parse_social.mjs --url "https://v.douyin.com/..." --transcribe
node scripts/parse_social.mjs --url "https://www.xiaohongshu.com/explore/..." --transcribe --transcript-format srtUseful options:
node scripts/parse_social.mjs --url "<share-url>" --out-dir ./social-notes
node scripts/parse_social.mjs --url "<share-url>" --no-download
node scripts/parse_social.mjs --url "<share-url>" --download-video
node scripts/parse_social.mjs --platform douyin --kind profile --id "<sec_uid>" --count 10
node scripts/parse_social.mjs --from-file raw.json --platform douyinTranscription options:
node scripts/parse_social.mjs --url "<share-url>" --transcribe
node scripts/parse_social.mjs --url "<share-url>" --no-download --transcribe
node scripts/parse_social.mjs --url "<share-url>" --transcribe --transcription-mode direct
node scripts/parse_social.mjs --url "<share-url>" --transcribe --transcription-mode local --speaker-diarization
node scripts/parse_social.mjs --url "<share-url>" --transcribe --transcript-format srt --sentence-max-length 20- Transcription is off by default because Tencent ASR is a paid external operation.
- Install the companion
video-to-textskill as a sibling before using--transcribe. directis the default. It sends the normalized media URL straight to Tencent without downloading or transcoding it locally.localdownloads the media first and never falls back to the remote URL. Use it when CDN URLs require request headers or expire quickly.autois an explicit recovery mode: it tries a temporary local download first and falls back to direct URL only when local preparation fails before ASR task submission.- Add
--no-downloadwhen only the transcript is needed. Direct transcription does not need a local video copy. - The workflow discovers
video-to-textas a sibling skill first, then project/user.agents/skillsand.skillsinstalls, or accepts--video-to-text-script/VIDEO_TO_TEXT_SCRIPT. - Douyin profile results contain multiple works and are not implicitly transcribed. Supply a single work URL instead.
- There is no COS upload bridge yet. If a protected CDN URL cannot be fetched by Tencent and the locally transcoded audio still exceeds Tencent's 5 MB inline limit, upload the media to COS or another public object store and run
video-to-text --urlon that URL.
Platform Notes
- Xiaohongshu:
descis the正文.video_backupmay be a string or list. - Douyin profile:
/user/...homepage links usedyzy; normalized output haskind=profile,type=profile, and anitemslist. - Douyin work: video, image, and live-photo share links use
douyin; normalized output haskind=postandtype=video|image|live.live_photoincludes paired image/video values. - X/Twitter: the parser uses FXTwitter first, keeps tweet text, author info, media, metrics, and long-form article content when available, then falls back to the aggregated parser if needed.
- Share links often contain tracking parameters; the parser keeps the content path and drops those extras before calling the API.
- Do not assume every fallback wants the same cleaned URL; some fallback parsers prefer the original share link.
- When the primary parser misses content, the skill can fall back to the aggregated short-video parser to recover a usable result.
- Bilibili: supports short videos and collections. Use
videosas parts when present. Direct video URLs may expire. - Fallback parser:
short_videoscovers 20+ short-video platforms and is useful when the platform-specific parser misses a share link.
Rules
- Do not use the official
key=valuesample as a real parameter. - Do not invent engagement metrics. Keep unsupported fields empty or absent.
- Do not assume every platform returns the same schema.
- Do not download videos unless the user asks or
--download-videois provided. - Do not transcribe unless the user asks or
--transcribeis provided. - Treat
--transcribeas authorization for one Tencent ASR task. Do not retry a submitted ASR task through another mode; automatic fallback is allowed only before task submission. - Preserve
raw.jsonexactly for debugging and future schema changes.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

