A four-layer memory architecture for agents that stay useful for months

Why a bigger context window isn't memory

The default fix when an AI forgets is to give it a bigger context window — the amount of text it can hold in view at once. The point here is that this doesn't solve forgetting, it delays it. Restart the session and everything is gone, and even mid-session the model reads the middle of a long context worst (the well-documented "lost in the middle" effect).

The broken starting setup is familiar: several disconnected chat windows, each with its own stale picture, agents contradicting each other because neither can see what the other knew, instructions from three weeks ago simply forgotten. The reframe is the whole piece: durable agent memory is an infrastructure problem, not a prompting problem.

In finance terms it's the difference between remembering a number because you're staring at it and writing it into the ledger. The first survives until you look away; the second survives the year. Memory isn't cramming more into the prompt. It's deciding what gets written down, and where.

Layer 1 — what's live this session

The first layer is what the agent reads the moment a session starts: an identity file (how it should behave, what it never does without asking) plus a memory index — a plain table of contents pointing to lots of small fact-files, read on demand.

The insistence is that the index stays "not a vector database, not embeddings." It's kept in markdown for one reason: "Because I can read it, edit it, and debug it. When an agent starts acting wrong, I open the index and find the bad instruction."

The detail worth stealing is file per fact, not one big document. Deleting a stale memory then never disturbs the others, and the index stays cheap to load and cheap to prune.

Layer 2 — what to carry to the next session

Markdown-only memory has one hole: it captures only what someone remembers to type. The valuable material is implicit — a decision made mid-run, a failure pattern you fixed, a preference the user corrected.

Layer 2 is a small local facts bank: a store the agent auto-writes curated facts into at the end of a session and queries at the start of the next one, before it answers. The safety rail is the important part. The path is fact → human review → index entry, so retention is automatic but nothing becomes permanent memory until a person approves it.

That gate is what separates a facts bank from an agent quietly teaching itself things you never sanctioned. Automatic capture, manual promotion.

Layer 3 — shared state, and the one rule that makes it work

Add a second agent and single-agent memory breaks. They drift; one thinks the project status is X, the other thinks Y, and within a week they contradict each other. The fix is one shared file both agents read before every reply and append to after every meaningful turn — a live, append-only log.

The rule that stops two writers corrupting it translates cleanly into three worlds. In software it's event sourcing: you never edit a past record, you append a new one that supersedes it, and the state is the replay of all entries. In accounting it's the oldest rule in the book, where you never erase a posting, you book a reversing entry, and the audit trail stays intact. In git it's just commits.

Each entry is signed (which agent, what kind, a one-line summary), kept append-only, and no agent may edit another's entry: "convergence is achieved by appending, never editing." Across hundreds of sessions that produced exactly one conflict, two agents writing in the same minute, resolved by reconciling on the next turn with no automated merge.

Layer 4 — what either agent can look up later

The first three layers are episodic: what happened, what was decided, what to carry forward. Layer 4 is the semantic one — a compiled wiki running as a search service (an MCP server, the standard socket that lets an agent query an external tool) over the shared knowledge base. It returns a ranked list of relevant pages with their sources instead of dumping everything into context.

The framing is the line to keep: L1 to L3 are what the agent carries; L4 is what it can look up. That's the difference between memory and recall.

The quiet lesson under all four layers is that they need different retention policies. Live state churns by the minute, shared state by the turn, carried facts by the session, and lookup knowledge is near-permanent. Treating them as one undifferentiated pile is the mistake that works for about six weeks and then rots.

The honest tradeoff, and where to start

The part that earns trust is that it isn't oversold: "It is not a magic always-on system. It is structured manual discipline plus automation at the seams." You have to write things down, maintain the files, and review what gets retained before it becomes permanent.

The reassuring number for a solo builder is that the full four-layer stack took six months to stabilise, but layers 1 and 3 took one weekend: an identity file plus an index, and one shared append-only log. That's the minimum viable version, and it carries most of the value.

The closing instruction is the one to borrow. Build the memory before you add more tools. The useful agents aren't the ones with the biggest context window, they're the ones that remember what matters and forget what doesn't.

Vocabulary

• context window — the amount of text an AI can hold in view at once
• memory index — a small always-loaded contents list linking to fact-files read on demand
• facts bank — a store an agent auto-fills with session facts, gated by human review
• append-only log — a record you only add to, never edit or delete, so history stays intact
• event sourcing — store every change as a new entry; current state is their replay
• MCP server — a standard socket letting an agent query an external tool or search index

If you're building — what to watch for

• Split memory by how long each fact needs to live, not by what kind of thing it is. Retention policy is the axis that decides where a fact belongs.
• Keep memory in files you can open and edit by hand. When an agent misbehaves you want to find the bad instruction and delete it, not retrain something.
• One fact per file, with a thin index on top. Deleting a stale memory should never disturb the ones around it.
• If two agents share state, make the shared log append-only and signed, and never let one edit another's entry. Reconcile on the next turn instead of automating a merge.
• Gate automatic capture behind human promotion: the agent may draft what it learned, but a person decides what becomes permanent.

Reading it critically

• It's an n=1 report — one builder, self-measured ("one conflict in hundreds of sessions"), with no independent benchmark. Read it as a working blueprint, not proof.
• An append-only log trades merge conflicts for staleness. It only grows, and a "read the last 30 lines" habit will miss an older entry that still matters.
• The facts bank and the signed-log ceremony may be over-engineering for one person with one agent. By its own numbers, layers 1 and 3 are the weekend-sized 80%; add the rest when a second agent actually forces it.
• Privacy is untouched. The shared state assumes every agent may read everything, which is exactly the assumption to constrain if any of your context is walled off or confidential.
• The named tools behind the layers are repos of unknown maturity. The pattern is the keeper; the specific backends need their own check before you depend on them.