LLM session context
TL;DR - managing LLM context into outputs
Motivation here is to be able to somehow corral the sprawling context out and be able to safely do a meeting summary in effect to close off threads, flag new threads to open up, but ultimately make something durable and compacting of information.
Problem
This started with me trying to parallelise a few more things out to LLMs and spending way too long working out how to do so efficiently (still not 100% sure tbh) but it led to an observation that I had a bunch of Yorick skulls. Basically:
- Half a dozen console windows so i could jump back into a thread when I wanted to pick it up again.
- Session reuse and ever expanding context windows as I kept poking (and possibly chasing unrelated threads) at ideas.
These are pretty normal things - the first is the context-switching problem, and the second is just brainstorming. The difference is that with an LLM in the loop, the main place both are stored is the session’s context, and that’s a bad place to keep anything.
Session as storage (but..)
So, I can read the transcript as a how-i-got here, but refinement means what I want is to hack into the KV cache to eviscerate things and refine and that’s not really an option. The model’s internal workings are opaque and can’t be edited, just influenced. Compaction is lossy, and the model picks what survives, not me.
Which then leads to a simpler problem mapped onto this: “why did we have this meeting again?”. If this happened in meatspace i’d want to, at the end of a session make sure we agreed on next steps/actions and also summarize what we learnt/discarded etc. That agreement then can stand in for the meeting transcript, but importantly it can’t be a single-person view, it’s gotta be agreed or else there’s no real proof that all participants got the same thing out.
Aaand that maps nicely to writing out a handoff for an LLM.
Handoff
At some point, the operator has to decide when the meeting has run its course and do handoff. It’s slightly different from a summary, and more pointedly for an LLM workflow it’s got to be a testable summary. Me reading it and nodding my head is marginally useful, but agents start from zero. So here’s the test pattern:
targeted compression
Summary is a dump. These headings are not, and the discussion and model have to use in-mem context to fit under these headings which is the refinement i want:
- Understood: what we now think is true, and why.
- Decided: including the negatives, like “don’t do X” and “Y is ruled out because…”.
- Threads: what looked interesting, and why it mattered.
- Open: what we couldn’t answer.
- Next: often “explore this thread”, not a task.
- Anything else: a catchall for anything that is especially notable.
Empty headings get “none”, not padding. The negatives also get called out as they’re routes we don’t want to revisit.
There’s also a “how we got here” section. It’s a short log of the pivots: what we believed, what changed it, and what we believe now. It has to stand on its own, because the transcript behind it is low value and should get thrown away.
That gives three tiers:
- The pointed summary, which everyone reads.
- The pathway, which you read if the summary feels off.
- The raw log, which you only probe with a specific question if you disagree with the pathway, and otherwise delete.
Handoff testing
A handoff is still a form of a summary, so the test stage is to have the generating agent make the handoff. Ideally the operator (you) have some test questions in mind as the independent 3rd party. The validation question(s) change but pending the type of work:
- Exploration: something like “what approaches have been ruled out, and why? What approaches are promising?”
- Build: “what’s the next step, and what must you not break?”
- Coordinating session: “what’s in flight? What steps are next?”
Responses to this can be generated by the session doing the handoff, and a subagent getting the handoff and the same question but not the answers can do the same. In effect, if two systems joined only by the handoff get to the same rough conclusions, we have a functioning handoff.
The human can spot check where this matches/floats etc, where stuff is missing is where the handoff wasn’t good, or where the human might have had an unvoiced assumption. This can be auto-checked as well but i’d spot check this to avoid drift.
What I’ve ended up with
I’ve ended up with one skill, invoked by hand at the point of handoff. That way the instructions are fresh when they’re needed, instead of rotting at the top of a long context. Putting the handoff rules in the always-loaded agent instructions was a mistake. The first test session raced to “planning is done, shall I export?”, because the handoff had become a goal. That’s not necessarily a bad thing but it isn’t intentional yet.