Your Agent Memory Should Outlive Your Harness

Your agent’s memory should survive your next choice of agent. Changing an editor, swapping a model, or moving a task into a background worker should not mean rebuilding everything the system learned about your work.

A harness owns execution: prompts, tool selection, context assembly, and the loop that turns a request into actions. That makes it a reasonable place for session state. Durable decisions, operational lessons, and shared knowledge have a different lifetime. Coupling those records to one harness makes the execution environment their accidental owner.

I want external agent memory to be a service with its own identity, authorization, versioning, and retrieval policy. The interesting payoff goes beyond persistence. Multiple clients can work from the same evidence, multiple agents can improve shared lessons, and retrieval can evolve without rebuilding every client.

One person, several clients, one body of knowledge

Cross-client AI memory starts with a mundane problem. A decision made in an editor is useful to the assistant reviewing a design document tomorrow. A deployment constraint discovered by a background agent matters when an interactive agent prepares the next release. Copying instructions between their private memory directories turns continuity into a synchronization job.

An external store gives those clients a common place to read and write. Their transcripts and temporary plans can remain local. The shared record contains the result worth keeping: what was decided, why, what evidence supports it, and where it applies.

This already helps a single tenant. It does not require multiple organizations, automatic learning, or a sophisticated agent swarm. Two authorized clients reading the same versioned decision are enough to demonstrate the value.

MCP can provide the interface. Its architecture separates hosts, clients, and servers, with servers able to run as remote services. A connection does not automatically upload a conversation or guarantee that the model retrieves anything. The host still decides what context to send and when to invoke memory operations.

The companion article on MCP memory and lifecycle hooks addresses when those operations run. Here the question is where the knowledge lives, who can reuse it, and how retrieval improves.

Shared learning needs an explicit meaning

When I say harnesses should learn from each other, I mean one agent can record a useful finding that another retrieves and evaluates later. I do not mean the agents update each other’s model weights. I also do not mean every generated conclusion deserves promotion into permanent instructions.

Consider a worker that discovers a migration cannot run until a particular compatibility check passes. The useful memory is a scoped lesson with its evidence and prerequisites. A second harness can retrieve it before preparing a similar migration, check whether those prerequisites still apply, and avoid repeating the investigation.

The record needs a correction path. Perhaps the next release removes the constraint. Overwriting the old sentence loses the reason earlier work followed it; keeping both sentences without status creates a contradiction. Versioned lessons with supersession let later readers distinguish current guidance from the history behind it.

AgentleFS implements this distinction today. Knowledge items include lessons, dead ends, workflows, and skills. Drafts are private to the identity that authored them. Publishing to a folder or document scope makes an item eligible for readers who satisfy its access rules. A later publication can supersede a previous version. Search excludes superseded knowledge items from its current candidates.

Publication is therefore a deliberate knowledge operation. A successful task does not automatically become a lesson, and an agent repeating a claim does not make that claim correct.

Multi-tenant memory is not a communal transcript

A service can host memory for many tenants while keeping their data separate. That is multi-tenant memory infrastructure. It does not grant one customer’s agents access to another customer’s experience.

Even within one organization, sharing must respect scope. Two agents may have the same human owner and different delegated permissions. A lesson derived from restricted material can disclose that material through a summary. Removing the original filename does not remove the information.

AgentleFS scopes published knowledge and supports source-lineage enforcement, but that enforcement is configurable and off by default. Do not assume a summary automatically inherits every source restriction. Set and verify the intended policy. Cross-organization folder sharing uses explicit offers and acceptance; public reuse has a separate registry workflow: propose content, have a human review and approve publication, then explicitly pull a package into an organization. Agents cannot approve their own public proposals.

That boundary matters more than the phrase “collective intelligence.” Shared infrastructure lets us operate one retrieval system. Shared knowledge requires a separate decision about exactly what may cross a boundary.

Owning retrieval is the second major benefit

A filesystem is a good storage interface. Files can be versioned, indexed, synchronized, and permissioned. A repository plus a capable search service can support excellent memory. The limitation appears when the effective retrieval policy is whatever one harness happens to do with the files it can currently see.

Suppose a saved note says “schema compatibility gate” and a later task asks why a migration must wait. A literal search for the task’s wording might miss the note. Semantic retrieval may find it, but could also rank an obsolete proposal above the current rule. Neither “just grep” nor “just embeddings” specifies a complete retrieval policy.

A service gives that policy an explicit home. You can change how documents become chunks, combine lexical and vector candidates, preserve exact identifier matches, filter by scope, and rerank candidates before spending the model’s context budget. These are retrieval improvements, not model weight fine-tuning.

AgentleFS already searches names and aliases, labels, text, and meaning, combining candidates into a ranked result. Its search implementation includes keyword retrieval, reciprocal rank fusion, and optional reranking. PostgreSQL’s full text search and pgvector provide underlying retrieval capabilities. These mechanisms complement one another; vector similarity is not a substitute for an exact document name.

The optional document reranker operates on candidates that passed access filtering. If it fails or times out, it preserves those candidates’ pre-rerank order. Configuration and provider availability affect which capabilities actually run. This is an implemented retrieval stack, not a claim that every deployment uses the same model or that users have a per-tenant tuning dashboard.

Concrete deployment controls include the embedding model, reranking candidate depth, passage length, and provider timeouts. The defaults use hash embeddings and no reranker; richer model capabilities require configuration. Changing the heading-aware chunker requires a code change and re-ingestion, rather than flipping a client preference. Owning the service makes these choices available to its operators, but does not make a change automatically safe or effective.

Measure changes against the questions people ask

The next step after exposing tuning knobs is resisting random knob turning. Build a set of representative questions with judged relevant documents, including exact identifiers, paraphrases, outdated guidance, and documents the caller must never receive.

AgentleFS includes a frozen retrieval evaluation corpus and relevance checks. That provides a repeatable baseline, not proof that retrieval works for every organization. Tenant-specific evaluation sets and systematic comparisons of chunking, candidate depth, and ranking would be further work.

I would track relevance alongside latency, context size, stale evidence, and access correctness. A ranking improvement that exposes a restricted candidate is a failure. So is a result that looks relevant but cites an obsolete decision. Freshness should be evaluated against the task; the newest document is not always the authoritative one.

Chunk boundaries deserve the same scrutiny as the embedding model. Splitting a requirement from its exception can create a confident wrong answer. Larger chunks retain context but consume more of the retrieval budget. The right choice depends on the material and should survive an evaluation, not just a convincing demonstration.

A concrete workflow, and the costs worth accepting

With AgentleFS today, start in one connected client and search for an existing lesson before creating another. Draft a finding with its evidence and limitations, then publish it into an appropriate writable scope. In a second client, authenticate with the intended identity, search, and read the published item. Confirm the version and applicability before using it. If the finding changes, publish a superseding version.

This workflow exercises existing knowledge and retrieval operations. It does not depend on an automatic capture hook, background transcript ingestion, or cross-tenant access. The connection guide and documentation are the practical starting points.

Externalizing memory adds a network dependency and an operational owner. Clients need deadlines and a policy for unavailable memory. Concurrent writers need conflict handling; derived indexes can lag committed records. Backups, deletion, and usable export matter because moving dependence from a harness to a service does not eliminate dependence.

For one client working on a small repository, local files may remain the best choice. I would externalize memory when its useful lifetime spans clients, when agents need shared evidence, or when retrieval deserves independent engineering. Keep the execution loop replaceable. Give the knowledge a home that can outlive it.