Memory Like a Brain

I've been building an AI agent for the past few months, and somewhere along the way I realized I was accidentally recreating a brain.

Not the whole thing, obviously. Just the memory parts. But the similarities are weird enough to be worth talking about, because once you see them, they explain a lot about why agent memory systems work the way they do — and why the naive approaches fall flat.

Here's the thing: when you first start thinking about giving an AI agent memory, you imagine it like a computer. Perfect recall, instant access, every conversation stored verbatim in some giant database. A tape recorder that never forgets. But that's not how it works at all, and honestly, that's not even what you want.

What you want — what actually makes an agent useful — is something much more like how your brain remembers things. Messy, reconstructive, prioritized, and surprisingly fragile. Let me explain.

Working Memory is Just the Context Window

There's this famous psychology finding: working memory holds about seven items, plus or minus two. It's the stuff you can actively think about right now. A phone number you're about to dial. The three things you need from the grocery store. The thread of a conversation you're currently having.

For an AI agent, this is just the context window. The tokens that are actually loaded into the model's attention mechanism right now. Everything else — every past conversation, every extracted fact, every piece of history — exists somewhere else, inaccessible unless you explicitly pull it back in.

This was the first hard lesson. Early versions of my agent tried to cram everything into context. Every message, every interaction, all of it shoved into the prompt. It was like trying to hold an entire library in your working memory at once. The agent got slow, confused, and weirdly fixated on irrelevant details from weeks ago. It turns out there's a reason your brain doesn't work this way.

Working memory is supposed to be small. It's the active workspace. Everything else needs to live somewhere else.

Consolidation Happens While You're Not Looking

In human memory, consolidation is the process of moving stuff from short-term to long-term storage. It happens mostly during sleep, when your brain replays the day's experiences and decides what to keep and what to discard. You don't consciously control it. You just wake up and somehow you remember the important parts of yesterday but not what you had for lunch three Tuesdays ago.

For an agent, this is the extraction phase. After a conversation or a work session, the agent goes back through what just happened and pulls out the facts worth keeping. "Lion prefers minimal design." "The API key is stored in environment variable X." "We decided to use PostgreSQL for this project." These get written to long-term storage — a vector database, a structured facts table, whatever.

The key insight is that this has to happen asynchronously. You can't extract facts in real-time during a conversation any more than your brain consolidates memories while you're actively experiencing them. The consolidation step requires a different mode of processing. You need to step back, look at the whole conversation, and decide what matters.

When I first built this, it felt almost too biological. The agent would finish a conversation, then spend a few seconds "thinking" about what to remember. I didn't plan it that way — it just emerged from the architecture. But it works for the same reason sleep-based consolidation works: you need distance to figure out what's important.

Remembering is Reconstruction, Not Playback

Here's where it gets interesting. When you remember something, you're not playing back a recording. You're reconstructing an experience from fragments. This is why memory is unreliable. This is why two people remember the same event differently. This is why you can have vivid memories of things that never happened.

Your brain stores the gist, the emotional tenor, some key details. When you try to remember, it stitches those pieces back together into something coherent. Usually it's close enough. Sometimes it's completely wrong.

Agent memory works the same way. When my agent needs to remember something about a past conversation, it doesn't retrieve the full transcript. It does a semantic search over extracted facts and conversation summaries, pulls back the most relevant fragments, and reconstructs a sense of "what happened back then" from those pieces.

This felt wrong at first. Surely we want perfect recall? Just store everything and retrieve it verbatim?

But no. Perfect recall doesn't scale, and more importantly, it doesn't prioritize. When you ask the agent "what do you know about my API setup?" you don't want a dump of every message where APIs were mentioned. You want the relevant facts, stitched together into a coherent answer. You want reconstruction.

The tradeoff is real, though. If the extraction phase missed something, or if the semantic search doesn't surface the right fragments, the agent just... doesn't remember. It's not that the information is gone — it's still in the raw history somewhere — but it's functionally inaccessible. Exactly like when you know you learned something once but can't recall it because the retrieval cues aren't working.

Three Tiers, Just Like a Brain

The current system has three layers, and they map pretty cleanly to biological memory stages:

Recent history is ultra-short-term memory. Raw, unprocessed conversation messages from the past few days. It's not in active context, but it's close enough that the agent can pull from it quickly if needed. Like remembering what you did this morning — it hasn't been consolidated yet, but it's still fresh.

Short-term context is working memory. The current conversation, the active thread, whatever's loaded into the context window right now. Seven plus or minus two items, except the items are messages or function calls instead of grocery list entries.

Long-term storage is everything that's been consolidated. Extracted facts, conversation summaries, semantic embeddings. It's searchable, but retrieval is inexact. You get back what matches the query, weighted by relevance. Just like trying to remember something from six months ago — you get the pieces that your current mental state happens to activate.

When the system is working well, it feels seamless. The agent pulls in the right facts at the right time, references things from weeks ago naturally, doesn't get confused by irrelevant details. It feels like talking to someone who actually remembers our previous conversations.

When it's not working well, the failure modes are eerily human. The agent "forgets" things that were definitely discussed because they weren't extracted properly. It "misremembers" by reconstructing from incomplete fragments. It gets fixated on some random detail that happened to score high in semantic search.

Where the Analogy Breaks

Of course, there are differences. An agent doesn't have emotions, which in humans are a huge part of what gets prioritized for consolidation. Emotional salience is a shortcut for importance, and agents don't have access to that signal. Instead, we use heuristics — frequency of mention, explicit user statements, structural importance in the conversation.

Agents also don't have the same kind of associative memory. Human recall is deeply contextual — a smell or a song can trigger a cascade of memories. Agent retrieval is mostly deliberate and query-driven. There's some semantic association through embeddings, but it's not quite the same as the rich web of connections in biological memory.

And of course, agents can be edited externally. I can go into the database and delete a fact or add one. You can't do that with a human brain, at least not easily. Though maybe that's not such a clean distinction — therapy, hypnosis, and simple storytelling can all rewrite human memories in subtle ways.

When It Started to Feel Right

There was a specific moment when I knew the memory system was working. I asked my agent about a decision we'd made two weeks prior — something about database schema design. I'd honestly forgotten the details myself.

The agent pulled up the context immediately. Not the full conversation, but the reconstructed gist: we'd considered three options, ruled out two for specific reasons, went with PostgreSQL because of JSON support and my existing familiarity. It even remembered that I'd been slightly worried about scaling but decided to defer that problem.

None of that was in the current context window. The agent had to retrieve it from long-term storage, reconstruct the narrative from extracted facts and summary fragments, and present it coherently. It did this in about two seconds.

It felt less like querying a database and more like asking a colleague "hey, what did we decide about that thing?" And getting a real answer, not a transcript dump.

That's when I realized: the goal isn't perfect memory. The goal is useful memory. Memory that surfaces the right information at the right time, that prioritizes what matters, that reconstructs context without drowning you in details.

Just like a brain.

The Engineering is Cognitive Science

Building this system meant accidentally learning a bunch of cognitive science. The three-stage memory model, the importance of consolidation, the reconstructive nature of recall — these aren't just biological curiosities. They're architectural principles that fall out of the constraints.

You can't keep everything in working memory because attention doesn't scale. You need consolidation because real-time extraction is too slow and you don't know what's important yet. You need reconstruction because storing and retrieving everything verbatim is both impossible and useless.

The brain figured this out over millions of years of evolution. We're figuring it out again, from scratch, in software. And we keep arriving at the same solutions.

That's either a profound insight about the nature of memory, or it's just convergent evolution under similar constraints. Probably both.

Either way, it means that if you want to build an agent that remembers well, you should stop thinking about databases and start thinking about brains. Not because brains are magical, but because they've already solved this problem.

And the solution, it turns out, is beautiful in its messiness. Memory that's fragile and reconstructive and imperfect, but somehow exactly right for the job.