Events, projection, and reconcile

cairn stores no state you cannot reconstruct. Every write is one event appended to a durable, append-only log; everything you query — the task graph, observations, handoffs, metadata, even the views you define — is a projection folded from those events. The log is the truth. The database is a cache of the log, and it can be thrown away and rebuilt. This is why a cairn store survives a crash mid-write, moves between machines as plain text, and resumes the same way every time: the answer to "what happened?" is always the events, replayed in the same order.

You do not call any of this. There is no commit, no flush, no migration step in the tool surface. Understanding the model tells you why cairn's writes are cheap, why two agents writing the same store do not corrupt it, and why the durable artifact on disk is a file you can read.

The event log is the source of truth

Each task owns a log: a single events.ndjson file, one JSON event per line. That file is canonical. Every observation, status change, link, fork, and handoff is appended to it as an event before anything else happens, and an event, once written, is never edited or deleted. To know the full history of a task, you read its log top to bottom.

The line format is deterministic. cairn encodes each event with its keys sorted and its data payload nested as a canonical object, so the same event always serializes to the same bytes — the encoder and decoder are exact inverses, and a line decoded on one machine folds into the same state on another. An event carries its identity and provenance: event_id is a portable ULID, the stable id that travels with the event; event_key is the de-duplication key; task_id, type, ts, and the session that wrote it ride along too. The local row number is the only field the line does not carry, because it is local.

Because the log is append-only NDJSON, it is durable and portable for free. There is no binary format to corrupt, no schema to migrate in place. Copy the directory and you have copied the history.

The projection is a fold over events

You never query the log directly. cairn folds the events into a SQLite projection — the materialized view your reads run against — and that projection holds the shapes you actually ask about: the tasks table, the edges between them, the full-text observations index, handoffs, task_metadata, and named_views. Every one of these is derived. Apply the events in order and the projection appears; the same events always produce the same projection.

Two consequences matter. First, the projection is disposable: it is a cache, and cairn can rebuild it from the logs at any time. Second, replay is portable, not local. cairn folds the log by (ts, event_id) — timestamp, then ULID as tiebreaker — not by the local row number a given database happened to assign. The local seq orders one database's storage; the portable order reproduces the same state on any store, regardless of the order rows landed locally. That distinction is what lets a store rebuilt on a fresh machine match the original exactly.

User-defined views are not special. When you call define! to name a query (see Built-in and user-defined views), cairn records a view-definition event and folds it into the named_views table. undefine! records the inverse. Replay reproduces your view vocabulary along with everything else, because a view is just more events in the log — there is no separate place your customizations live and could be lost.

Reconcile unions the log and the cache

The two stores can drift. A database row gets committed but the matching append to the log never lands — the crash gap between a write and its mirror. Or you copy fresh logs into a store whose cache predates them. cairn closes both gaps on store-open with reconcile, which unions the per-task logs (the truth) with the SQLite cache (rebuildable) and brings them back into agreement.

Reconcile runs in three moves:

Ingest log lines the cache lacks. cairn replays every log into the database, skipping events it already holds — de-duplication is keyed on event_key, so re-ingesting the same event is a no-op. If any new event enters the cache, cairn re-folds the whole projection in portable order, so the materialized state reflects the full log rather than a partial splice.

Export cache rows the logs lack. For the reverse gap — a committed row whose append never reached the file — cairn appends those events back to the right task's log. This is the only path that writes the log during open, and it writes only what is missing, comparing by event_key. After this step, every event in the cache is also on disk.

Refresh the watermarks. cairn stamps each log's current watermark so the next open sees an unchanged store and skips it.

That last move is the efficiency story. A watermark is a content hash and byte length of a log file, stored in the database's schema_meta. On open, cairn compares each log's current watermark against the stored one; a file that has not moved is skipped in both directions. An unchanged store reconciles to a no-op — reconcile is cheap precisely because most logs do not change between opens. A store with no backing file (an in-memory store) reconciles to a no-op as well.

Why the order is append-then-project

The durable append happens first, and the cache update rides only after it. cairn appends the event to the log, then folds it into the projection — never the other way around. During ingest, cairn deliberately suppresses the mirror-write step, because the events it is replaying are already on disk; mirroring them again would be redundant work against the file it is reading from.

This ordering is what makes the crash gap recoverable rather than fatal. If cairn crashes after the log append but before the cache catches up, the next open's ingest step replays the missing line and the projection heals. If it crashes after a cache write but before the mirror append — the rarer gap — the export step writes the line back. Whichever side is ahead, reconcile pulls the other forward, and because both directions key on event_key, neither can double-apply.

Where this model earns its keep, and where it does not

Event sourcing is not free. The projection is a second copy of state that must be kept in step, replay has a cost proportional to history, and reasoning about a fold is more work than reasoning about a row you overwrote. cairn takes that cost on purpose, because its whole job is continuity: an agent must be able to resume into the exact state a prior session left, on a machine that may not be the one that wrote it, after a context reset or a crash. A durable, portable, replayable log is the mechanism that delivers it, and the current-task pointer is the one piece of state deliberately kept outside this log, because it is per-session, not part of the shared history.

If your work is a single throwaway task in one session, you will never see reconcile do anything and the projection will never need rebuilding — the model is overkill, and that is fine; it stays out of your way. The model pays off the moment work outlives a session or is shared across more than one agent. That is the case cairn is built for.

Related

• The task graph — the typed-edge graph and observation index that the projection materializes
• Tasks, observations, handoffs — the three nouns that become events
• The current-task pointer — the per-session state kept outside the event log
• Views — how define! records a view event that replay reproduces