shiny-to-datapage
Skill12 starsChanged 19 days agoMITView on GitHub
Playbook for migrating a Shiny/Django data site to a static Quarto + Observable site backed by a versioned Redivis dataset (the "datapage" pattern). Use whenever a session involves converting, porting, or rebuilding a lab data site (shiny app, database-backed site) onto Redivis/Quarto/OJS, building lazy-loading in-browser visualizations over Redivis data, staging a legacy MySQL database (including multi-version history) to Redivis, or re-importing a dataset from an auth-walled upstream source. Covers architecture tiers (incl. hash-sharded DuckDB for high-cardinality lookups), OJS/Quarto gotchas, Redivis upload/release traps, legacy-DB staging at scale, upstream re-acquisition and parser validation, CI/deploy, preview-repo team QA, AI-derived tables, and companion R package migration (characterization tests, CRAN safety, the R binary-bake trap).
In this repository14 other files it gives its agents.
- adversarial-protocol-reviewSkill
- babyview-datasetSkill
- cdi-vocabulary-dataSkill
- cran-safe-releaseSkill
- gcp-large-stanSkill
- pre-submission-reviewSkill
- psis-loo-chunkedSkill
- quarto-bookSkill
- redivisSkill
- remote-vm-gotchasSkill
- reproducibility-auditSkill
- rmd-qmd-manuscript-collabSkill
- stan-mixing-cold-startSkill
- wordbankSkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 8
Read the full playbook FIRST — it encodes hard-won lessons from the wordbank and childes-db migrations and will save hours on each of these:
- Distinctive: line 36
Prefer plain JSON slices; DuckDB-WASM only for big raw tables needing arbitrary SQL splits.
- Very specific: line 57
Keep the playbook updated at the end of a migration session: append any new traps encountered to ~/Projects/shiny-to-observable-advice.md.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Shiny → Redivis datapage migration
Read the full playbook FIRST — it encodes hard-won lessons from the wordbank and childes-db migrations and will save hours on each of these:
~/Projects/shiny-to-observable-advice.md
Reference implementations (every pattern, working):
- https://github.com/langcog/wordbank-datapage (small-data patterns, AI tables; live: langcog.github.io/wordbank-datapage)
- langcog/childes-db-website branch
datapage(big-data patterns: hash-sharded DuckDB frequency/bigram lookups, preview-repo QA) with langcog/childes-dbredivis/+pipeline/(multi-version MySQL→Redivis staging; auth-walled upstream mirror + chatter parser + QA harness) - Companion packages: langcog/wordbankr branch
redivis(issues #47–52) and langcog/childesr branchredivis(characterization suite that caught NULL-in-IN and collation drift; binary-bake mitigation).
Process that worked (follow this order)
- Survey the old site/apps in parallel (what each page computes, where data lives, what model fitting exists — only true fitting blocks static migration; everything else runs in the browser).
- Extract → Redivis: tidy tables shaped like the existing API's return
values; keep source data verbatim (no NA flattening); validate derived
numbers against the old app's own cached outputs before uploading.
Set
upload_merge_strategy="replace"viatb$update()EVERY upload or re-releases double the data. - Site skeleton: one working lazy-loading viz end-to-end first, then the rest. Tier every table (embed / lazy JSON slice / Redivis-only). Prefer plain JSON slices; DuckDB-WASM only for big raw tables needing arbitrary SQL splits.
- Static pages + styling: replicate the original look (fetch its real rendered CSS/markup — declared stylesheets can be dead code; measure the live site). Plot/Table tabsets on every viz.
- CI: fetch small tables from Redivis, rebuild slices from committed compact artifacts, publish to gh-pages (branch must pre-exist; publish step needs the same secrets; LIBARROW_BINARY=true).
- R package (if any): characterization tests from the old package first — captured against the LIVE legacy backend before touching infra — then the lean Redivis backend against them.
- Team QA before cutover: preview repo with Pages for the site (never in-place on a Jekyll-from-master prod), installable branch for the package, and a written issue-triage list. External users on binary installs hit bugs dev machines can't (see the binary-bake trap).
- If the data are also being updated: stage all legacy versions to Redivis first (oldest→newest, one Redivis version each) so infra work never blocks on the new import; the re-import lands as just another version. Two-phase every release: pre-stage the draft, release by flag after sign-off.
Keep the playbook updated at the end of a migration session: append any new traps encountered to ~/Projects/shiny-to-observable-advice.md.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.