agentleFS
all-source-osall-source-os/all-source/.claude/skills/chronos-embedded-durability/SKILL.md

chronos-embedded-durability

Skill4 starsChanged yesterdayApache-2.0View on GitHub

Test AllSource Core embedded durability by running Rust integration tests that verify WAL recovery, crash recovery, and checkpoint correctness. Catches silent-no-op checkpoint bugs like issue #84. Triggers on "test embedded durability", "embedded crash recovery", "test wal recovery", "test embedded persistence", "verify embedded durability", "issue 84".

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 2 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run the targeted Rust integration tests:

    line 25
  2. Distinctive:

    If those pass, run the full embedded durability suite:

    line 32
  3. Very specific:

    If the durability_status() API is available, run the #84 regression suite:

    line 39

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Chronos Embedded Durability Test

Tests the allsource-core Rust crate's durability guarantees directly — no Docker, no HTTP, no MCP. Exercises the exact code paths that run when EmbeddedCore::open is called on a directory with existing WAL/Parquet data.

Why this exists

Issue #84 revealed a class of bug where:

  1. WAL recovery loads events into memory correctly
  2. flush_storage() returns Ok(()) as a silent no-op (empty Parquet batch)
  3. WAL is truncated unconditionally after "successful" checkpoint
  4. Events exist only in memory → process exit → data loss

The existing Docker durability test (chronos-durability) doesn't catch this because it tests the HTTP server path, not the embedded library path. The embedded data flow test uses clean shutdown, which works fine — the bug only manifests on unclean restart.

When invoked:

  1. Run the targeted Rust integration tests:
    cd apps/core
    cargo test --test integration_tests test_wal_durability_and_recovery -- --nocapture
    cargo test --test embedded_core_api --features embedded events_survive -- --nocapture
  2. If those pass, run the full embedded durability suite:
    cd apps/core
    cargo test --test embedded_core_api --features embedded -- --nocapture
    cargo test --test integration_tests -- --nocapture
  3. If the durability_status() API is available, run the #84 regression suite:
    bash tooling/embedded-durability-test/test-embedded-durability.sh --status
  4. For the full suite including all scenarios:
    bash tooling/embedded-durability-test/test-embedded-durability.sh
  5. Analyze the output and report:
    • Which tests passed/failed
    • Whether the WAL recovery checkpoint bug (#84 pattern) is present
    • Whether events survive unclean shutdown (drop without shutdown())
    • Whether Parquet files are actually written during checkpoint-on-open

Test Matrix

The script and Rust tests cover these scenarios:

ScenarioWhat's TestedBug #84 Relevant?
Clean shutdown + reopenshutdown() → drop → open() → queryNo (this path works)
Drop without shutdown + reopendrop (no shutdown()) → open() → queryYES — this is the #84 path
WAL-only recoveryWrite events, skip Parquet flush, reopenYES
Parquet-only recoveryFlush to Parquet, delete WAL, reopenNo (different path)
WAL + Parquet recoveryBoth exist on disk, reopenPartially
Checkpoint verificationAfter recovery, assert Parquet files exist on diskYES — catches silent no-op
durability_status() after ingestmemory > 0, wal > 0, durable=trueValidates fsync is working
durability_status() after recoveryparquet_pending_batch == memory_eventsYES — the exact #84 invariant
durability_status() warns on memory-onlywarnings.len() > 0 for dangerous stateRuntime detection of #84
durability_status() unclean restartDrop → reopen → durable=true, no warningsEnd-to-end #84 regression
Large volume recovery1000+ events, drop, reopen, verify countStress variant of #84
Concurrent write + killSpawn writer tasks, kill mid-write, reopenEdge case

Key Assertions (what distinguishes this from other durability tests)

  1. After unclean restart, query() returns all events — not just the ones that were in Parquet before the crash
  2. After recovery checkpoint, Parquet files exist on disk — ls storage/ is not empty
  3. After recovery checkpoint, WAL can be safely truncated — events are in Parquet, not just memory
  4. flush_storage() after WAL recovery writes > 0 bytes — catches the silent no-op

durability_status() API (#84 regression tests)

The other thread is adding EmbeddedCore::durability_status() which returns the exact internal state that caused #84. The --status flag exercises this API through 5 invariant checks:

{
  "memory_events": 63,
  "wal_entries": 63,
  "wal_bytes": 91000,
  "wal_sequence": 63,
  "parquet_files": 0,
  "parquet_bytes": 0,
  "parquet_pending_batch": 63,
  "durable": false,
  "warnings": ["63 events in memory but 0 in Parquet and 0 in WAL — data loss on restart"]
}
TestInvariant Checked#84 Signal
durability_status_after_ingestmemory_events > 0 && wal_entries > 0 && durable == trueIf wal_entries == 0, fsync isn't working
durability_status_after_flushparquet_files > 0 && parquet_pending_batch == 0Parquet actually wrote to disk
durability_status_after_recoverymemory == wal && parquet_pending_batch == memory && durable == trueThe #84 check — after WAL recovery, events must be in parquet_pending_batch before truncation
durability_status_warns_on_memory_onlywarnings.len() > 0 when events in memory but not WAL/ParquetWould have flagged #84 at runtime
durability_status_survives_unclean_restartDrop (no shutdown) → reopen → durable == true && warnings.is_empty()End-to-end #84 regression

Run with:

bash tooling/embedded-durability-test/test-embedded-durability.sh --status

These tests expect corresponding #[tokio::test] functions in apps/core/tests/embedded_core_api.rs that call core.durability_status() and assert the invariants above. The Rust tests are the source of truth; this shell script just runs them and reports results.

Common Failure Patterns

  • "events lost after restart": The #84 bug — WAL events loaded into memory but not checkpointed to Parquet before WAL truncation
  • "storage/ directory empty after recovery": Same root cause — flush_storage() is a no-op because current_batch was never populated from WAL recovery
  • "test_wal_durability_and_recovery fails": The EventStore-level recovery path has the bug
  • "events_survive_store_restart_via_wal passes but unclean test fails": Clean shutdown works (events go through ingest() → append_event() → current_batch), but recovery path doesn't populate current_batch

Relationship to Other Skills

SkillScopeCatches #84?
chronos-durabilityDocker container restart (HTTP path)No
chronos-data-flowDocker stack connectivityNo
chronos-data-flow-embeddedMCP embedded backend (clean shutdown)No
chronos-embedded-durabilityRust crate crash recovery (unclean shutdown)Yes

Running Manually

# Quick: just the #84-relevant tests
bash tooling/embedded-durability-test/test-embedded-durability.sh --quick

# durability_status() API regression tests only
bash tooling/embedded-durability-test/test-embedded-durability.sh --status

# Full: all embedded durability tests
bash tooling/embedded-durability-test/test-embedded-durability.sh

# Or run cargo tests directly:
cd apps/core
cargo test --test integration_tests test_wal_durability -- --nocapture
cargo test --test embedded_core_api --features embedded durability_status -- --nocapture
cargo test --test embedded_core_api --features embedded events_survive -- --nocapture

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.