SkyWalking AI Sessionizer

Conversation-level observability for long-lived AI agents. It assembles fragmented agent telemetry — transcripts, subagent streams, workflow journals — into one durable conversation structure.

Two boundaries carry the design: Conversation is the durable ownership boundary and its identity is supplied, never inferred; ExecutionStream is the ordered continuity boundary, within which model-message continuity is evaluated.

Apache-2.0. A subproject of Apache SkyWalking: github.com/apache/skywalking-ai-sessionizer.

Documentation

Every claim in the docs is backed by a measurement against a real corpus. When changing them, keep that standard: state what was measured, on what sample, and say unavailable rather than approximating.

Writing

Comments, docs and commit messages use plain, widely understood English. Someone reading this project is often not a native English speaker, and the code is the hard part — the prose should not be.

• No abbreviations or slang. Write "remove duplicate records", not "dedup". Write "configuration", not "config", outside identifiers.
• No unusual or figurative words where a common one works: use "essential" not "load-bearing", "hide" not "launder", "insert" not "splice", "block" not "veto", "cut out" not "amputate".
• Short sentences. One idea each. Prefer a period to a dash or a semicolon.
• Say why, not what. The code already says what it does. A comment earns its place by recording the reason, the measurement, or the failure that forced the choice.

Commits

• No AI attribution. Never add a Co-Authored-By trailer, a "Generated with" line, or any other mention of an AI tool to a commit message or a pull request description. The person who commits is the author. This is an Apache project and its history records people.

Structure

asz.yaml                        default configuration, every value written out; read when no -config is given
install/                        the install scripts: asz.sh and asz.ps1 for asz, claude-code-plugin.sh and .ps1 for the plugin; run from a version's tag
cmd/asz/                        CLI: sources · collect · index · show · parse · conversation · view · glossary · verify
pkg/sessiondata/                Session Data: the landed record, `.sd` — public, adapters produce these
pkg/model/                      the conversation vocabulary: node kinds, relations, qualification
pkg/sessionflow/                Session Flow: the round chain of conversation structure, `.sf`
pkg/sessionview/                asz.view: one conversation as one document, version 1.0; never a file
pkg/providerbody/               provider bodies: cuts a body against what its session holds, and rebuilds it byte for byte
pkg/skeleton/                   the skeleton `.sk` that takes a landed file's place when retention removes it
internal/index/                 derived lookup structure the assembler resolves against
internal/assemble/              the eight-stage pipeline, index in and structure out
internal/parse/                 one round: assemble, compare against the chain, publish the delta
internal/storage/               landing zone, atomic writes, cursors, session/index state, locks
internal/verify/                contiguity and digest checks over landed data
internal/view/                  reads a conversation and serves it as a page
internal/view/conversation-view/ Horizon's conversation renderer, built from the pinned commit in HORIZON_COMMIT; never edited here
internal/export/otlp/           sends landed files and rounds to an OpenTelemetry logs receiver, one record per file, and the metrics spool
internal/metrics/               derives the runtime's token metric from landed Session Data into the spool; collector side; testdata holds a redacted capture of the runtime's exporter
internal/repack/                re-cuts landed files into a new root under another budget
internal/retention/             the two levels for idle conversations: keeps only the metadata, a skeleton in each landed file's place, then retires the session whole (collector side)
internal/adapters/claudecode/   the claude-code-local adapter
internal/adapters/claudecodeprovider/ the claude-code-provider adapter: the request and response bodies Claude Code writes, landed into their sessions
internal/adapters/kiro/         the kiro-local adapter: the sessions Kiro's V3 engine writes, for kiro-cli and Kiro Crew
internal/adapters/mock/         the mock dialect: Session Data a scenario writes directly
internal/scenario/              scenarios: the model, the clock, the two writers, and each session's marker (collector side)
internal/scenario/expect/       expectation files evaluated over a root (server side)
internal/scenario/remove/       removes a sent scenario session by its marker's policy (collector side)
internal/scenario/run/          the runner that wires build, collect, parse and check; command and tests only
tests/scenarios/                the assembly tests, one scenario and one expectation file each
internal/config/                YAML configuration
tests/chain/                    what a scenario cannot express: crashes, lost state, races, the round budget
tests/boundary/                 the import rules between the two sides

pkg/ is public because third-party adapters and consumers need it. Built-in adapters stay in internal/.

Scenarios are the tests. A property of assembly is a scenario under tests/scenarios/ with an expectation file beside it, run in both formats by tests/scenario_test.go. Go tests remain only for what a scenario cannot express: crashes, races, captured raw shapes, and package units. Add a scenario before adding a Go test.

Two sides, one contract. The collector side (internal/adapters/...) lands Session Data. The server side (internal/assemble, internal/parse, internal/view, internal/verify, pkg/sessionflow) assembles and serves it. They share internal/storage, internal/index, pkg/sessiondata and pkg/model, and they meet only at the storage root. Neither imports the other; tests/boundary fails when one does. cmd/asz is the only place that wires them together, and internal/scenario/run is the one other wiring package, for the command and the tests only. This is what keeps a later split into a collector binary and a server binary, talking over a network, a packaging change rather than a refactor.

Phase 1 (implemented): pull collection from local Claude Code files, plus a derived index built while landing. No configuration of Claude Code required, and it works on history that already exists. Phase 2 (implemented): assembly into a conversation, published as an append-only chain of immutable rounds — see the assembly doc. Phase 3 (in progress): read and export. asz collect is the pipeline: every period it lands what is new, parses what moved, and sends what export.otlp asks for, in that order. asz view serves the conversations as a page and changes no collected data: a list at /, and one conversation at /c/{id} drawn by Horizon's conversation renderer, embedded from a pinned Horizon commit so asz and the SkyWalking UI draw a conversation identically; the page adds the landed record behind a step, which only asz has. It reads the folded chain on demand. asz server is the pipeline and the page in one process, which is what a person runs locally, and the list page shows the last and the next refresh. asz push sends a root that is already there, in one pass. A static export belongs here too; it adds measurement only, never structure.

Build

make build          # -> ./bin/asz
make test           # whole suite, -race
make test-e2e       # the scenarios, the chain tests and the boundary rules, verbose
make scenarios      # every scenario through the built command, as CI runs them
make check          # vet, lint, license headers, dependency licenses, tests — what CI runs
make docker         # the container image, as CI builds and publishes it
make help           # all targets

Go 1.27+. Dependencies: gopkg.in/yaml.v3, and for the OTLP push the protocol's own Go modules, go.opentelemetry.io/proto/otlp, google.golang.org/grpc and google.golang.org/protobuf. Nothing else; adding one needs a reason.

Every source file carries the Apache-2.0 header; make license-fix inserts missing ones. license-eye is pinned to v0.9.0 — v0.7.0 does not build under Go 1.26.

Invariants worth knowing before changing assembly code

• The conversation is the fold of every round. There is no other state. A round is immutable and is never rewritten; later evidence produces a new revision in a later round.
• Entity ids come from stable evidence, never from position. An id that shifts when an earlier source is backfilled cannot supersede its own earlier revision.
• A round is cut at a byte budget by narrowing its input window. A round covering one landed file may exceed the budget. Revisions that need no new evidence, as after an upgrade, go first in rounds over no new file, cut by their frames. Round.More says the chain has not reached the index yet, or has revisions left.
• A round carries no wall-clock time, so the same evidence reproduces the same bytes and digest. Anything mutable or temporal lives in conversation.state, outside every digest. Record times are evidence, not the clock: the header carries the window's first and last record time, and the session node the conversation's, both inside the digest.
• Absence in a round means unchanged, never deleted. Removal is an explicit tombstone, and a resolved reference is superseded with that state rather than dropped.
• Landed files plus the round chain are the whole conversation. Nothing else has to travel with them: the index rebuilds from the landed files, and the chain head is recovered by listing the rounds directory. A complete export of a conversation's files, such as a bundle, can be read, verified and re-parsed with no source files and no collector; a pull's archive is not one, since it holds only the files the pulling machine lacks. A skeleton stands in for a removed file's structure, never its text. After a removal, the session's provider/ store travels with them too: later provider bodies are built from the bytes it keeps.
• A skeleton keeps every field the index reads. The index rebuilt from a .sk must equal the index rebuilt from its .sd, byte for byte, or a conversation continues differently after a removal. A new field the index reads goes into skeleton.Of too; tests/chain compares every scenario with and without removals.
• A provider body is cut after a removal exactly as with none. providerbody.Session.Add takes a removed body's skeleton in its place, and reads removed tips and pieces from the session's store, kept by the removal and by the collector before a body that cites one lands. Anything the cutter reads to decide a cut must be kept in the skeleton or the store, or bodies land differently; pkg/skeleton and tests/chain compare them byte for byte, under a budget too.
• Assembly reads the index, never the payloads. Structure needs identifiers; text is read only when a conversation is rendered. The page reads .sd, .sk, .sf and the provider store only, never the index; a test in tests/boundary fails if it does.
• A prompt cycle id is not unique across streams. Anything keyed on it must be scoped by stream.
• Never use a stop reason to find a call's last fragment. A main transcript stamps it on every fragment. Line order is the only rule that works on both stream kinds.
• Cross-stream flow is a relation, never containment. A child's work stays under the child.

Invariants worth knowing before changing collector code

• Landed files are write-once. temp → fsync → chmod 0444 → rename → sync the directory. Never appended. A failed directory sync does not fail the write (a process crash keeps the rename, and a reader may have taken the file) and is reported once per directory; failing it made a disk that kept failing land the same records on every pass. Retention, when it removes, must make a replacement durable, its directory synced and the error checked, before it deletes what it replaces.
• Only retire_after removes a real session whole, and its id is never collected again. A scenario build's removal takes only a session the build marked, by the policy in its marker, and only once all of it was sent to the one receiver the pipeline sends to. A removal renames each directory away before it deletes it, so no reader sees half a session. storage.retention has two levels on one clock, the header time of the newest landed file, sent or not: off by default, metadata_only_after removes the landed files of an idle conversation and keeps its structure, a skeleton in each file's place and the index, cursors, state and chain untouched; retire_after, at least as long, removes everything and retires the session. Neither runs over a scenario root.
• A retired session is never written again. Its marker _retired/<session> is an empty file, written and synced before anything of the session is deleted, and never removed. Every writer asks storage.Retired before the first MkdirAll or lock that would make the session's or its chain's directory, since taking a lock makes it; a marker that cannot be read is an error, never taken for none. A retired session's unsent metrics requests are written again from their receipts before its directory goes, and keep its owner in push.state until they are sent.
• A landed file is cut once. A round addresses records by {seq, row} and binds to file digests, so a referenced file is never re-cut. A new budget applies to new files; an existing root is brought under it with asz repack into a new root, where the chains are built again.
• Platform-specific calls live in one file per platform. The session lock is flock on Unix and an exclusive open on Windows; a file's identity is an inode on Unix and a volume serial plus file index on Windows. Relative source paths are written with forward slashes everywhere.
• Payload bytes are the source bytes where a part carries the source's own JSON: byte for byte, apart from whitespace between tokens. json.RawMessage both directions, and the writer never re-encodes a part's data. Older files hold \u003c, \u003e and \u0026 for the same values. The one exception: JSON holding a byte that is not valid UTF-8 is left out, its size and SHA-256 kept and a dropped entry saying why, and a change or execution record is written again with U+FFFD in the byte's place. Every landed line is valid UTF-8, since an OTLP receiver refuses anything else.
• A landed file fits its budget. A collector cuts a file by what its records take once written (sessiondata.Cut), never by source bytes: an envelope made 4 KiB of short lines 19 KB. A single record over max_delta_bytes lands alone with its largest payloads taken out as too_large, size and SHA-256 kept, through the writer's Budget. A change or execution record is never taken out whole, and the provider cutter's own Budget never takes out a manifest. A collector without a cut and a budget is a bug. asz repack carries records whole.
• Land before committing the cursor. At-least-once is intentional; the reverse loses data.
• next_seq is monotonic across all streams in a session, so a session is processed single-threaded and under a lock. The filesystem is the authority on the counter, and a skeleton's name counts: a sequence is never used twice, removed or not.
• Each writable tree has one owner. A production pass holds that root's _root. Production passes leave _remote/ and _pull/ alone. A pull or sweep owns one _remote/<remote>/<conversation>/ under its _pull. Independent workers can own different conversations. No worker holds two conversation locks. Creation of _remote/ and its remote directories belongs to pull. History, publication intent and removal records use the conversation's outer _root. Lease-file creation, probing and removal use _admit only. That gate also protects snapshot admission and detachment. A snapshot's evidence never changes. Its judge reserves and publishes stages, regroups and work directories under the snapshot's _root. The run alone writes prompts, evaluator output and logs in its work directory after releasing _root. Inherited logs and navigation failure files stay within that chunk's work directory within the run. A reader lease covers all snapshot handles and judge work. The same judge work rule applies on a production root. One serving process holds _pull/owner/.lock for its lifetime of pull work. It owns that root's _pull/. Its .lock protects state transitions. Workers own private builds under their build locks and in-process slots. Requests hold active-work locks. The server's conversation _root protects history and removal records. No build or network write holds the global lock. The LangSmith receiver publishes each request to _langsmith/ through WriteExclusive under its in-process mutex. A production pass consumes requests under _langsmith's file lock. These are different locks. The consumer passes over an empty .request file, as the round reader passes over an empty reservation, so the last-resort install, which makes the name before renaming the request over it, cannot stop collection. A leftover empty reservation is harmless and left. These owners never edit each other's evidence in place.
• Locks are taken in one order: the server lifetime owner, _scenario, a local conversation's _pull or a server worker's in-process slot then build-owner lock, external _admit or server _pull/.lock, the root's in-process gate, _root, _export, _metrics, _provider, _langsmith, a session's, then a chain's. This is the order for every acquisition that can wait. A probe of capacity, another build owner, a lease or active work never waits; a judge run's .active is taken under _root without waiting and probed the same way. If a group of probes cannot finish, release its new locks and try later outside the transition lock. Holding a worker or lifetime lock does not forbid reentering the transition lock. Its holder is never waited for there. No holder of _root waits for _admit or the server lock. Release a judge's _root before releasing its lease. Capacity waits, expensive verification and network writes happen outside transition locks. A new lock takes its place in this order, or two commands can wait on each other for ever.
• A judge run is admitted in one decision, and holds .active until it has ended. Under _root: refuse a retired session, a removal intent, and another active run or regroup of the conversation; run the caller's selection; reserve the directory, take .active, write run.json. Retention probes .active before it removes anything at either level. After the work ends no evaluator call starts, and what finished is published under a cleanup context bounded to two minutes. asz server's policy writes _judge/ only under _root: a record in _judge/runs/ before the run's reservation, which the run's run.json names back, and automatic.lock taken without waiting. What is due is decided by each talk's view digest, never by a rule about which evidence affects which talk.
• asz never collects its own evaluators' sessions. kiro-local leaves out any session with a workspace inside the storage root, links resolved and unknown never taken for outside; an agent CLI runs without OTEL_LOG_RAW_API_BODIES. A new evaluator that keeps a session needs its own guard.
• A session is sent to one owner, the endpoint, the configured service name and the instance it was first sent under. push.state records the owner, and saves it, before the first request carrying anything of the session goes out, because a receiver holds a session whole. Never the service a session resolves to: that depends on reading its files, and a person cannot set it back.
• Never use bufio.Scanner on source records. Real lines reach ~1 MB, past its 64 KB default, where it fails silently.
• The index is derived and disposable. Delete it and it rebuilds from landed files; a schema bump discards rather than migrates. Nothing downstream may treat it as authoritative.
• The index stores roles, not runtime field names. A runtime's vocabulary stops at its adapter, so a rename there cannot reach the model. internal/index and the model docs must stay free of promptId, parentUuid, agentId and their kin.
• Both vocabularies survive. The model's words are the project's, but a reader can ask for the runtime's with -terms native. The adapter declares a glossary and a test fails if a name the model can emit has no entry - including entries that say the runtime has no word for it.
• A record type is not what a record is. A user record is usually a tool result, not a human — only ~7% are typed by a person. Check origin and the content blocks, never type alone.