chronos-embedded-durability
Skill4 starsChanged yesterdayApache-2.0View on GitHub
Test AllSource Core embedded durability by running Rust integration tests that verify WAL recovery, crash recovery, and checkpoint correctness. Catches silent-no-op checkpoint bugs like issue #84. Triggers on "test embedded durability", "embedded crash recovery", "test wal recovery", "test embedded persistence", "verify embedded durability", "issue 84".
In this repository25 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- allsource-incident-statusSkill
- allsource-partnershipsSkill
- ritual-fullSkill
- ritualSkill
- semantic-isaSkill
- admin-healthSkill
- allsource-data-accessSkill
- allsource-debugSkill
- allsource-incident-statusSkill
- allsource-mcp-setupSkill
- allsource-onboardSkill
- chronos-data-flow-embeddedSkill
- chronos-data-flowSkill
- chronos-durabilitySkill
- chronos-releaseSkill
- fly-statusSkill
- mammoth-memorySkill
- ritual-fullSkill
- ritualSkill
- semantic-isaSkill
- seo-aeo-auditSkill
- allsource-customerSkill
- allsource-prime-memorySkill
The file
We split this skill into the rules it gives an agent
3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 25
Run the targeted Rust integration tests:
- Distinctive: line 32
If those pass, run the full embedded durability suite:
- Very specific: line 39
If the durability_status() API is available, run the #84 regression suite:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Chronos Embedded Durability Test
Tests the allsource-core Rust crate's durability guarantees directly — no Docker, no HTTP, no MCP. Exercises the exact code paths that run when EmbeddedCore::open is called on a directory with existing WAL/Parquet data.
Why this exists
Issue #84 revealed a class of bug where:
- WAL recovery loads events into memory correctly
flush_storage()returnsOk(())as a silent no-op (empty Parquet batch)- WAL is truncated unconditionally after "successful" checkpoint
- Events exist only in memory → process exit → data loss
The existing Docker durability test (chronos-durability) doesn't catch this because it tests the HTTP server path, not the embedded library path. The embedded data flow test uses clean shutdown, which works fine — the bug only manifests on unclean restart.
When invoked:
- Run the targeted Rust integration tests:
cd apps/core cargo test --test integration_tests test_wal_durability_and_recovery -- --nocapture cargo test --test embedded_core_api --features embedded events_survive -- --nocapture - If those pass, run the full embedded durability suite:
cd apps/core cargo test --test embedded_core_api --features embedded -- --nocapture cargo test --test integration_tests -- --nocapture - If the
durability_status()API is available, run the #84 regression suite:bash tooling/embedded-durability-test/test-embedded-durability.sh --status - For the full suite including all scenarios:
bash tooling/embedded-durability-test/test-embedded-durability.sh - Analyze the output and report:
- Which tests passed/failed
- Whether the WAL recovery checkpoint bug (#84 pattern) is present
- Whether events survive unclean shutdown (drop without
shutdown()) - Whether Parquet files are actually written during checkpoint-on-open
Test Matrix
The script and Rust tests cover these scenarios:
| Scenario | What's Tested | Bug #84 Relevant? |
| Clean shutdown + reopen | shutdown() → drop → open() → query | No (this path works) |
| Drop without shutdown + reopen | drop (no shutdown()) → open() → query | YES — this is the #84 path |
| WAL-only recovery | Write events, skip Parquet flush, reopen | YES |
| Parquet-only recovery | Flush to Parquet, delete WAL, reopen | No (different path) |
| WAL + Parquet recovery | Both exist on disk, reopen | Partially |
| Checkpoint verification | After recovery, assert Parquet files exist on disk | YES — catches silent no-op |
| durability_status() after ingest | memory > 0, wal > 0, durable=true | Validates fsync is working |
| durability_status() after recovery | parquet_pending_batch == memory_events | YES — the exact #84 invariant |
| durability_status() warns on memory-only | warnings.len() > 0 for dangerous state | Runtime detection of #84 |
| durability_status() unclean restart | Drop → reopen → durable=true, no warnings | End-to-end #84 regression |
| Large volume recovery | 1000+ events, drop, reopen, verify count | Stress variant of #84 |
| Concurrent write + kill | Spawn writer tasks, kill mid-write, reopen | Edge case |
Key Assertions (what distinguishes this from other durability tests)
- After unclean restart,
query()returns all events — not just the ones that were in Parquet before the crash - After recovery checkpoint, Parquet files exist on disk —
ls storage/is not empty - After recovery checkpoint, WAL can be safely truncated — events are in Parquet, not just memory
flush_storage()after WAL recovery writes > 0 bytes — catches the silent no-op
durability_status() API (#84 regression tests)
The other thread is adding EmbeddedCore::durability_status() which returns the exact internal state
that caused #84. The --status flag exercises this API through 5 invariant checks:
{
"memory_events": 63,
"wal_entries": 63,
"wal_bytes": 91000,
"wal_sequence": 63,
"parquet_files": 0,
"parquet_bytes": 0,
"parquet_pending_batch": 63,
"durable": false,
"warnings": ["63 events in memory but 0 in Parquet and 0 in WAL — data loss on restart"]
}| Test | Invariant Checked | #84 Signal |
durability_status_after_ingest | memory_events > 0 && wal_entries > 0 && durable == true | If wal_entries == 0, fsync isn't working |
durability_status_after_flush | parquet_files > 0 && parquet_pending_batch == 0 | Parquet actually wrote to disk |
durability_status_after_recovery | memory == wal && parquet_pending_batch == memory && durable == true | The #84 check — after WAL recovery, events must be in parquet_pending_batch before truncation |
durability_status_warns_on_memory_only | warnings.len() > 0 when events in memory but not WAL/Parquet | Would have flagged #84 at runtime |
durability_status_survives_unclean_restart | Drop (no shutdown) → reopen → durable == true && warnings.is_empty() | End-to-end #84 regression |
Run with:
bash tooling/embedded-durability-test/test-embedded-durability.sh --statusThese tests expect corresponding #[tokio::test] functions in apps/core/tests/embedded_core_api.rs
that call core.durability_status() and assert the invariants above. The Rust tests are the source of truth;
this shell script just runs them and reports results.
Common Failure Patterns
- "events lost after restart": The #84 bug — WAL events loaded into memory but not checkpointed to Parquet before WAL truncation
- "storage/ directory empty after recovery": Same root cause —
flush_storage()is a no-op becausecurrent_batchwas never populated from WAL recovery - "test_wal_durability_and_recovery fails": The EventStore-level recovery path has the bug
- "events_survive_store_restart_via_wal passes but unclean test fails": Clean shutdown works (events go through
ingest()→append_event()→current_batch), but recovery path doesn't populatecurrent_batch
Relationship to Other Skills
| Skill | Scope | Catches #84? |
chronos-durability | Docker container restart (HTTP path) | No |
chronos-data-flow | Docker stack connectivity | No |
chronos-data-flow-embedded | MCP embedded backend (clean shutdown) | No |
chronos-embedded-durability | Rust crate crash recovery (unclean shutdown) | Yes |
Running Manually
# Quick: just the #84-relevant tests
bash tooling/embedded-durability-test/test-embedded-durability.sh --quick
# durability_status() API regression tests only
bash tooling/embedded-durability-test/test-embedded-durability.sh --status
# Full: all embedded durability tests
bash tooling/embedded-durability-test/test-embedded-durability.sh
# Or run cargo tests directly:
cd apps/core
cargo test --test integration_tests test_wal_durability -- --nocapture
cargo test --test embedded_core_api --features embedded durability_status -- --nocapture
cargo test --test embedded_core_api --features embedded events_survive -- --nocaptureDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

