AI coding assistants forget too much.
Open a new agent session and the model usually starts cold. It does not know your project habits, the weird library you avoid, the reason a feature was built in a specific way, or even the basic preferences you have repeated across earlier chats.
Most tools handle this with one of two blunt approaches.
Some pull anything that looks relevant from a vector database and paste it into the prompt. Others rely on a huge CLAUDE.md, project instruction file, or running summary that loads at the start of every session.
Both approaches can work for a while. Then the bill shows up in tokens.
Before the agent writes code, it has already spent part of the context window reading memory scaffolding. Latency goes up. API costs go up. The model also has more unrelated text competing for attention.
MemPalace takes a stricter approach: load only enough memory for the agent to know where to look next.
The project's interesting claim is a startup cost of roughly 170 tokens. That number matters because it changes the shape of agent memory. Instead of treating the prompt as a storage unit, MemPalace treats startup memory as an index.
Long running agent memory gets expensive quickly.
Imagine six months of conversations totaling about 19.5 million tokens. You cannot paste that into a model at startup. Even heavily summarized memory can become large if the system keeps compressing every session into a global file.
The rough comparison looks like this:
Memory approach | Tokens loaded on boot | Estimated annual cost |
|---|
Naive paste | 19.5M tokens | Physically impractical |
Continuous LLM summaries | ~650,000 tokens | ~$507.00 |
MemPalace wake up | ~170 tokens | ~$0.70 |
The exact numbers depend on model pricing and usage patterns, so I would treat them as directional rather than universal. The point still holds: memory that loads by default has a permanent tax.
That tax is especially painful for coding agents. A developer agent already needs room for source files, diffs, errors, logs, tool results, and the current task. Spending thousands of tokens on background memory before the real work starts is a bad trade.
MemPalace avoids that by splitting memory into layers.
MemPalace uses a progressive retrieval stack. The first two layers are always loaded. The deeper layers stay on disk until the agent needs them.
+------------------------------------------------------------+
| L0: Identity (~50 tokens) |
+------------------------------------------------------------+
| L1: Essential stories (~120 tokens) |
+------------------------------------------------------------+
| L2: Room recall, loaded when a topic matches |
+------------------------------------------------------------+
| L3: Deep search, used for exact passages and fallback |
+------------------------------------------------------------+
This design shifts work away from startup. The agent wakes up with a tiny map, then retrieves richer memory only when the conversation gives it a reason.
Layer 0 is the small profile that should always be available.
It can include the assistant's role, the user's basic preferences, and a few stable facts that apply across sessions. In MemPalace, this is usually a compact text file such as ~/.mempalace/identity.txt.
This layer should be boring on purpose. It is not a diary. It is not a project archive. It is the minimum identity context the agent needs before doing anything else.
Layer 1 is where the token savings become more interesting.
Instead of loading full memories, MemPalace loads compact pointers. The outline describes these pointers with the AAAK format: Assertion, Assumption, Action, Knowledge.
A Layer 1 record might look like this:
Z:0|E:ALC,JOR|T:trust_building|Q:I never told anyone|W:0.95|EMO:vul,tru|F:ORIGIN,CORE
That line is not meant to be pretty prose. It is meant to tell the model what exists.
The record points to a memory ID, entities, topic, short quote, importance weight, emotional tags, and flags. The model gets an index of important memory without receiving the full memory body.
This is the useful part. Large language models are good at reading structured shorthand when the format is consistent. MemPalace uses that ability to keep Layer 1 small enough for startup while still giving the agent a sense of which memories may matter.
The agent wakes up with a map, not a pile of text.
Layer 2 handles topic based recall.
MemPalace groups memories into a hierarchy of wings and rooms. A wing might represent a project or broad area. A room narrows the context to something like authentication, billing, deployment, or a specific relationship.
When the active conversation matches a topic, the system can query only that room:
where = build_where_filter(wing="project_alpha", room="auth")
That means an auth question does not need billing memories, old personal notes, or unrelated project context. The retrieval stays scoped.
This is where the design starts to feel practical for daily agent use. Most requests do not need the whole memory store. They need the right slice.
Layer 3 is the fallback.
If the identity layer, essential story pointers, and room recall are not enough, MemPalace can run a deeper semantic search against the underlying store, such as ChromaDB. This is where the agent can pull exact conversation turns or raw passages.
That deeper search is useful, but it is not free. It adds retrieval time and extra prompt tokens. MemPalace keeps it out of the startup path for that reason.
The system only pays for deep memory when the user asks something that actually needs it.
The storage design is only half of the trick.
MemPalace also uses a behavioral rule in the system prompt, described in the outline as the PALACE PROTOCOL:
"BEFORE RESPONDING about any person, project, or past event: call mempalace_kg_query or mempalace_search FIRST."
That rule changes the agent's behavior. The model is not expected to pretend that the startup prompt contains everything. It is told to query memory before answering about history.
This is a useful pattern for agent design in general. If you keep memory out of the prompt, you need strong habits around retrieval. Otherwise the model will answer from whatever partial context it has and sound more certain than it should.
The trade is simple: MemPalace saves context space by accepting occasional retrieval latency.
The first time you mention an obscure person, project, or decision, the system may need to query SQLite, vector tables, or a knowledge graph before it can answer well. That can add a small delay.
For many workflows, that is a good trade.
Most prompts do not need six months of history. They need the current task, a few stable preferences, and a way to fetch memory when the topic calls for it. Keeping the prompt clean gives the model more room for code, logs, plans, and reasoning about the actual problem.
The interesting lesson from MemPalace is not the palace metaphor. It is the restraint.
As context windows get larger, it becomes tempting to paste more into them. That works until the prompt turns into a junk drawer. More context is not the same as better context.
MemPalace shows a cleaner pattern:
Keep startup memory tiny.
Store durable memories outside the prompt.
Load index pointers before full text.
Scope retrieval by topic.
Use deep search only when the lighter layers are not enough.
That is a better default for long running agents.
The 170-token startup number is useful because it forces the design question: what does the model really need to know before the user asks anything?
Usually, the answer is not "everything."
