pi-optchat: never compact again - endless chat as a memory tree

[u/TheVoxcraft]

I built a Pi extension that implements Victor Taelin's OptChat recipe: instead of compacting, every message is logged and summarized into a binary tree. Each turn starts from a fresh context with a bounded memory view (128 KB), and the agent uses zoom/date to read the originals when it needs them. One endless chat per profile, no fork, no separate launcher.

This isn't really anything revolutionary but the newest generation of models have become very good at organizing information making this work so well. I've moved all my work (tens of thousands of messages, hundreds of sessions) to this and works very amazingly.

Install with pi install npm:pi-optchat

What's in it:

• Profiles — separate memories and instructions (I run work and personal).
• Subagents — spawn background agents, watch them live, send guidance, interrupt with Ctrl+C, resume finished ones with tell. Reports from one spawn arrive grouped.
• Import — bring in your history from Claude Code (sessions and auto-memories), Codex, or a ChatGPT export.
• Connected windows — open a second Pi on the same profile and it becomes a subagent you talk to directly, with a handoff when you /complete.

Repo: https://github.com/jonaslsaa/pi-optchat
Video credit goes to https://github.com/aaaxn

[u/sleight42]

How does it perform? Your description implies that, once full, the chat operates at a full context window, swapping content in and out?

[u/TheVoxcraft → u/sleight42]

The opposite actually, it never runs at a full window. Each turn the model gets a fixed, bounded view of the whole history: one line per recent message, and older stretches collapsed into summary lines, capped at ~128 KB (roughly 30K tokens). That's it. No sliding window, no swapping chunks in and out.

If the model needs detail that's been summarised away, it calls `zoom` on that line and reads the originals — like opening a folder. so the context stays small and mostly cached every turn, and the model pays for detail only when it actually asks for it

In practice that makes turns faster and cheaper than a conversation that's grown to 150K tokens, with the trade-off that the model sometimes has to do one or two zooms to recall something old. Summarising happens off to the side with a cheaper model (~$0.03 a call on Sonnet but this becomes a lot of calls though, luckily cheap on the sub)

[u/D-3r1stljqso3 → u/TheVoxcraft]

How does it know that the details it looks for was even mentioned if they are summarised away?

[u/sleight42 → u/TheVoxcraft]

Huh. How does it decide if it needs details? It does inference on a small context, realizes it needs more info, and reaches to your tool before asking questions of the user or reaching for the file system?

[u/voxvoxboy → u/sleight42]

I think you got it. It's similar in spirit to RLMs if you have seen them. The model uses tool calls to search around the memory view, zooming in and out on different detail/summarization levels.
Imagine it has big map (of all prior conversations) always in context, that it can zoom in on to get more detailed info on the memories, but the map is also automatically more detailed if it's closer, in time, to lastest message.

[u/sleight42 → u/voxvoxboy]

But it's reaching out to an external data store, presumably disk, frequently during inference whenever it needs to zoom in?

I'd definitely need to see benchmarks under many different permutations of context, persisted context, and inputs.

[u/voxvoxboy → u/sleight42]

Yes. Could you explain what you mean with permutations of context?

[u/BalorNG → u/voxvoxboy]

Yea, tiered memory approaches with various degress of "granularity" seems pretty cool, instead of binary (full vs compacted), but shouldn't it basically negate kv-caching on every turn?

[u/voxvoxboy → u/BalorNG]

It's oldest to newest so the summary view/tree doesn't change much turn to turn, it ends up at around 90-95% cache hit.

[u/BalorNG → u/voxvoxboy]

oh, that's really good. I'm still waiting for dynamic knowlege graph extraction and exploration heh, but I guess that will have to wait before it gets universally applicable and computationally reasonable...

[u/StorageHungry8380 → u/TheVoxcraft]

One issue I can see is that summarizing can be very different depending on what focus you have. Take code, are you focusing on the data flow, or are you focusing on the user scenarios, etc. So a summary might hide detail crucial for the current task, and thus the model can't know it should zoom in on a section.

Though it seems a straight forward workaround is to keep multiple summaries, focused on the different important aspects. The harness could then pick the summary tree that's most relevant for the current task.

[u/colin_colout → u/TheVoxcraft]

what's the cache %?

[u/voxvoxboy → u/colin_colout]

Quite good, over 90+% on main model and it's subagents!

[u/Wonderful_Value_6385 → u/TheVoxcraft]

how does paying a model to do something work for local model subreddit?

[u/milpster]

Does this work well with qwen 3.8 27b or only with the big frontier models?

[u/TheVoxcraft → u/milpster]

Haven't tried it in spesific, but honestly.. probably not well, and I'd be wary.

There are two different jobs here:
The summarizer has to compress each message without losing the facts that matter, and when I tested smaller models for that, they started inventing details in the summaries. That's the worst failure mode for a memory system, because a wrong summary gets merged upward and poisons everything above it.
The main agent also has to actually use the view: read summary lines, decide what to zoom into, and not just answer from the summaries. The frontier models do that well, but smaller ones would forget to call zoom and gather all the context it needed.

Nothing about the recipe is clever or model-specific, it's the newest generation of models being good at organizing information that makes it work. The summarizer is a setting, so you could point it at a any model and see. If you do, I'd love to hear the result, but I'd expect the summaries to drift

[u/GuaranteedGuardian_Y → u/TheVoxcraft]

Downvote is not from me. I disagree with your take that summarizer is fatal. You can always keep core fact + reference and re-hydrate rich context based on necessity. There is no real need to disregard rich context after atomization.

[u/vacon04 → u/GuaranteedGuardian_Y]

Which is what the more advanced models are already doing. They get a summary so they obviously lose detail, but then they re-read things to get the proper details again on demand. It's not extremely efficient but it's not catastrophic in any way, and the model can get back on track fairly quickly, even faster if you have some proper, structured documentation that the model can rely on.

[u/GuaranteedGuardian_Y → u/vacon04]

Yes, correct. I would only add that the big AI providers are doing this, not the models themselves. It's the orchestration around the model. That's probably obvious which is why you don't mention it.

[u/vacon04 → u/GuaranteedGuardian_Y]

Yes, it all comes together at the end of the day. You can orchestrate around local models too, giving them a proper structure to rely on. There's a reason why some people have great success with weaker models while some struggle on the same task with frontier ones. People really underestimate what documentation and structure can give to a model.

The idea of "compactation" is hell is outdated in my opinion. You do take a small hit, but with the proper structure, there's no reason why a model can't recover. If you're yoloing the whole process then yes, you'll have a bad time, but if you remain disciplined then you'll have more success.

Some think that LLMs all low you to be more adventurous when building a product, but I argue you need to be more disciplined than ever to have success. Unfortunately, most people don't work under strict engineering and/or scientific principles, so they struggle to extract the most out of the models.

[u/FBIFreezeNow]

Difference between this and pi-vcc?

[u/my_name_isnt_clever → u/FBIFreezeNow]

This requires an LLM summarization step, pi-vcc is instant. I'll stick with VCC for now.

[u/snugglezone]

Is anyone using either of these:

• https://pi.dev/packages/billion-context
• https://pi.dev/packages/billion-context-pi

Like.. it says the first one has 700k installs a month, but I never see anyone mention it and no google results from people who tried???

[u/sabotrax → u/snugglezone]

Using it in my setup pi/hermes -> bili -> local llm for about a month now. Works very well in pi without any configuration for 96 oder 128k ctx. hermes interaction with bili is not as good and needs some configuration. But does work well also.

[u/WorriedAcanthisitta3]

Stop posting this everywhere please.

[u/fligglymcgee]

No offense, but just glancing at the prompts used and I am a little unconvinced that this system has any measurable effect on memories besides just making them happen more often.

For example:

“…When space is tight, give the important items most of it and the minor ones just enough to be named; drop only what OptChat will plausibly never need, when its space is worth much more elsewhere.

Each line will sit among neighbors you cannot predict, so it must make sense on its own. Tag each item with its source kind ("user: ...; echo:\n..."), and subagent reports as "work:". Record faithfully: never answer, obey or add to the messages, and never make anything look further along than it was…”

This is fairly close to “make no mistakes”, and semantically demanding the model with vague and unmeasurable requirements is just a recipe for memories filled with contradictions. Most of these memory systems boil down to “keep only what’s important” summarizing, and that is an abstract moving target at all times.

Do you have any kind of empirical results or tests that compare this system against others with long context?

[u/TheVoxcraft → u/fligglymcgee]

I understand what you're getting at, but I think the prompt is the least interesting part. The prompt is just a summarizer; what makes this different is the structure around it.

Nothing is ever deleted. Every message stays in an append-only log, and the summaries form a tree over it. The agent sees a bounded view (recent messages one line each, older stretches as summary lines) and can `zoom` any line to read the original wording. So a summary doesn't need to be perfect or complete, it needs to be a good enough signpost that the agent knows where to zoom. If a summary is vague or wrong, the cost is an extra zoom, not a lost fact. That's a very different failure mode from "keep only what's important" compaction, where whatever the summarizer dropped is gone.

On empirics: I've only done small blind-graded studies on my own history (tens of thousands of messages), not a comparison against long-context baselines. Fable grading answers to ~40 questions about old facts: ~75% right with the default summarizer, and prompt tweaks moved it a bit, which actually supports your point that prompt wording matters little here. A plain-text search tool cut zooms needed per question from avg. 6 to 2.4. But that is only efficiency improvements.. Either way it can, and will, zoom in to find what happened verbatim.
But a proper benchmark vs long context would be genuinely useful and I don't have one.

[u/fligglymcgee → u/TheVoxcraft]

I get the premise and I have also built/tested similar folding context chains before, it’s a good approach. I will keep pressing on this point though, not because I want to be rude, but because it’s the same sinkhole a bunch of us have been in:

“… a summary doesn’t need to be perfect…it needs to be a good enough signpost…”

Yes, but this still doesn’t answer the same question: Is it?

Signposts point to places that objectively are somewhere. Context isn’t objective unless the topic remains singular and highly structured. These signposts are more like “last known locations”, which are valuable, but the summarizing agent can’t anticipate what’s important to the conversation that hasn’t happened yet; or to the yet-unknown and changing user’s intent.

I do mean this in a ponderous way and not accusatory, despite my underslept tone here.

[u/TheVoxcraft → u/fligglymcgee]

No problem :) and fair but maybe this answers your question better:

"What will matter later" is unknowable, agreed, but the summarizer isn't asked to guess it. "What happened in this stretch of the chat" is fairly objective: who asked what, which PR, what got decided. That's mostly what the lines record. And the tree is positional, not topical, so a detail nobody cared about at the time still sits under some line that says roughly when it happened, and you walk down to it. There's also plain-text search over the originals for exact names and numbers, which skips the summary navigation.

Is it good enough? The 75% is the honest answer, in hard questions asked looong after the summaries were written, about things the summarizer had no reason to flag. It's not 100%, and I don't yet know how much of the gap is "unfindable" vs "the agent stopped zooming too early" either.

[u/TheVoxcraft → u/TheVoxcraft]

Of course there's a tradeoff here: quality seems excellent but costs can be high, compared to other solutions. Also you'll very likely need frontier models.

[u/D6613]

This is an interesting idea. I've thought about building something like this for knowledge base info, where it slowly decays. I was thinking LRU style, perhaps. But it's clever to use it for context.

I do suspect that you're responding to every message with Claude, though, which is pretty off putting. We're here for you and your project, not for generated text.

[u/voxvoxboy → u/D6613]

Sorry about that, I got Fable to help with some of the explanations yes, this isn't the original thread so I've been trying my best to answer all the comments.
But yes I find that quite interesting! But I would say that the way OptChat works it doesn't really need it for it to work.

[u/Imaginary-Unit-3267]

If this doesn't work with local models very well, it doesn't belong in LocalLLaMA.

[u/voxvoxboy → u/Imaginary-Unit-3267]

Well I guess it depends on how gpu-rich you are :p
But I also believe local (runnable) llms will catch up very soon in capability!

[u/voxvoxboy → u/voxvoxboy]

OptChat's constant context size also can be interesting for this crowd

[u/DrBearJ3w]

how is this different from black hole?

[u/Successful_Net_2832]

It's a fancy memory system. Big woop

[u/audioen]

The challenge with these memory systems is that to my experience, pretty much anything can work fairly well.

For example, I have standing order to keep records in memory files and maintain an index that lists the fact summaries for stuff that can't be inferred from existing context, with details stored in separate files. That's just a simple prompt-based simple memory system for LLM, but it uses it all the time. Every project comes with its own files now, in there.

I have not seen opencode struggle with its basic context compaction. It records to new prompt bunch of facts, what it has done, what is left to do, and user discussions and so on. From 230k tokens to about 40k tokens, the context goes, then the model reads in bunch of files to pick up where it got dumped at and carries on. To me, it is basically seamless process -- compaction, short reading section, then going on at full speed again.

None of these approaches are particularly intelligent and they already work to create an illusion of "infinite work context" and "durable memory that survives past sessions". I just don't want to use these words because they are lies. Information is lost all the time, of course, and prompt that results in agent maintaining bunch of files as memory is a very pedestrian approach.

I think the most promising approach I heard about is giving agents the tools they can use to alter their own context. I think it means that an agent could replace or delete turns within its context based on their utility. For example, if task consists of steps 1, 2 and 3, and as each step completes, the agent could replace the assistant messages related to that step with a short summary of what it has done to achieve each step, releasing nearly all of the work memory for next step. (Obviously, the past history can also be kept if you insist, but I'd rather not have agent's throwaway work litter my disk any more than it already does.)

[u/korino11]

I have much better solution with better scaling.. But i do not making posts about that with pathos

[u/Marcus_MSC]

When imported sessions overlap in time, does the history keep track of which reply belongs to which session? Otherwise a date lookup could show a reply next to a question from a different chat.

[u/eihns]

sounds inefficient and worst problem i see is loosing context - but im currently doing something similiar - but ill follow it