Claude :: Week10 :: Special Series :: AI Task Delegation Research :: Lazy-Loading / Deferred-Context Startup Design for Long-Running AI Agents

Jun 9

Written By Richard Ketelsen

Deep Research request. Be thorough, cite sources, prioritize reliable information from the last ~24 months.

OBJECTIVE: Determine the smartest "load only what's needed now, defer the rest" startup design for an agent

that currently boots by reading large history/log files in full just to recover recent context.

Research and answer ALL of the following up front (do not pause to ask me; state any assumptions you make):

1. What is the minimal read-set at boot that still preserves correctness?

2. How should the system decide what to defer vs. load eagerly?

3. How do you DETECT when deferred context is actually needed mid-task (triggers / signals)?

4. How do you make a cheap boot degrade gracefully for a non-expert user?

5. What is the cheapest reliable way to recover just the "recent tail" of prior context?

6. What current patterns/tools exist (RAG-at-boot, rolling summaries, checkpoints, memory stores)? Cite them.

SOURCES & RECENCY: Favor credible, recent sources; cite non-obvious claims; separate established vs. emerging; flag uncertainty.

OUTPUT: Per technique use the 6-part format (idea / how it works / why non-obvious / example / failure modes /

adoption cost). End with a Source map + "State of the art as of today."

CONSTRAINT: Generic framing only; proceed without asking for private details.

TL;DR

Boot from a checkpoint plus the log tail, not the full history. The minimal correctness-preserving read-set is: the most recent durable snapshot/checkpoint, the append-only log entries after that snapshot's offset, and lightweight manifests/indexes/pointers to everything else — then lazily fault in deferred context on demand. This is the LLM-agent version of database WAL+checkpoint recovery and OS demand paging, and Anthropic, LangGraph, Temporal, and MemGPT/Letta all converge on it.
Defer by default; load eagerly only the working set. Keep hot, recent, high-recompute-cost state resident; demote warm/cold state to external stores reachable by reference (file paths, queries, vector IDs). Trigger on-demand loads via "page-fault" signals: context/cache misses, low RAG relevance scores, tool-call failures, unknown-entity references, and the model explicitly requesting more.
Degrade gracefully for non-experts with progressive loading, skeleton/optimistic UI, sensible defaults, and transparent fallbacks — but never let a cheap boot silently drop correctness-critical state. The cheapest reliable "recent tail" recovery is reading the last N lines/bytes of an append-only log from a snapshot offset (a ring-buffer/sliding-window over events), exactly what tail, Kafka offsets, and event-sourcing snapshot+replay already do.

Key Findings

There is a clean mapping from classic systems recovery to agent context management. A database does not re-read its entire write-ahead log (WAL) on restart; it loads the last checkpoint and replays only the log suffix after the checkpoint's log-sequence-number. An OS does not load a whole process image; it demand-pages and relies on locality of reference. The right agent boot does the same: restore a snapshot of agent state, replay/read only the recent log tail, and defer the rest behind references.
The single biggest correctness rule: boot must recover all state that, if missing, would cause incorrect (not merely less-informed) behavior — open commitments, in-flight tool calls, idempotency/dedupe keys, the current plan/goal, and any "exactly-once" side-effect markers. Everything else (background knowledge, old conversation detail) can be deferred and fetched lazily, because its absence degrades quality but not correctness.
Industry has standardized the "just-in-time" pattern. Anthropic's context-engineering guidance explicitly recommends maintaining "lightweight identifiers (file paths, stored queries, web links)" and loading data into context at runtime, citing Claude Code's use of head/tail/grep over large data instead of loading it. MemGPT/Letta implements OS-style virtual memory paging for context. LangGraph/Temporal/DBOS provide checkpoint+replay durable execution. These are now established, not speculative.
The newest work questions blanket checkpointing. The 2026 "Crab" study finds over 75% of agent turns produce no recovery-relevant state, so checkpoint selectively, aligned to semantic boundaries — the agent analog of only checkpointing dirty pages.

Details

I treat the agent generically (LLM-based agent or long-running stateful service). I assume large append-only/semi-structured logs, that "recent context" is needed for correctness but full history rarely is, and that a cost/latency-sensitive boot is desired. Where a claim is forward-looking, vendor-reported, or contested, I flag it.

PART 1 — MINIMAL READ-SET AT BOOT (what to read to preserve correctness)

The minimal correctness-preserving boot read-set is: (a) the most recent durable snapshot/checkpoint, (b) the append-only log tail after that snapshot's offset, (c) manifests/indexes/TOC and pointers to summaries, read metadata-only. Everything else is deferred.

Established technique — Snapshot + log-tail replay (WAL/checkpoint recovery).

Idea: Restore the last consistent snapshot, then replay only log records newer than the snapshot.
How it works: Databases assign each log record a monotonic Log Sequence Number (LSN); a checkpoint records the LSN up to which data pages are durable. On recovery the engine starts at the checkpoint LSN and replays forward (ARIES: analysis → redo → undo), avoiding a scan "from the very beginning of time." Event-sourcing systems do the same: load the latest snapshot, then apply only events after the snapshot version.
Why non-obvious: The naive instinct is "read the whole history to be safe." The insight is that a snapshot is a materialized fold of all prior events, so prior events are redundant for state recovery — you only need the suffix.
Concrete example: A bank account with 18,615 transactions is restored from a snapshot at event 18,000 plus 615 replayed events, not 18,615. LangGraph restores a thread's state from its last checkpoint and resumes at the next super-step rather than re-running the whole graph.
Failure modes: Snapshot/log version skew ("split-brain" between replaying events vs. trusting a snapshot); partially written snapshots (mitigated by atomic rename + checksums + manifests); a corrupt tail record (mitigated by per-record CRC — the partial record fails checksum and is discarded). Note LangGraph caveat: on resume, "nodes after the checkpoint re-execute, including any LLM calls, API requests, or interrupts," so non-deterministic or side-effecting steps must be wrapped behind durable boundaries or they double-fire.
Adoption cost: Low–moderate. Free if you adopt LangGraph checkpointers / Temporal / an event store; moderate if hand-rolled (you must define snapshot cadence, log format, offset tracking).

Established technique — Manifests / indexes / table-of-contents + metadata-only reads.

Idea: Read a small index that describes the corpus, not the corpus itself.
How it works: LSM-tree databases keep a MANIFEST file tracking the versioned hierarchy of SSTables and key-range mappings; the engine reads the manifest to know what exists without reading data blocks. Agents do the analog: read a directory listing, a CLAUDE.md / NOTES.md, a memory-file index, or a vector-store metadata table at boot, and defer the bodies.
Why non-obvious: Metadata (filenames, timestamps, sizes, folder hierarchy) is itself high-signal. Anthropic notes that test_utils.py in tests/ vs. src/core_logic/ "implies a different purpose," so the agent can prioritize without reading contents.
Concrete example: Claude Code drops CLAUDE.md into context up front but uses glob/grep to fetch files just-in-time, "bypassing the issues of stale indexing."
Failure modes: Stale indexes pointing to moved/deleted data; metadata that misleads (misnamed files); index becomes a bottleneck if it itself grows unbounded.
Adoption cost: Low. A manifest is cheap to maintain on write.

Correctness guarantees to enforce at boot. Recover, eagerly and verifiably: (i) the current goal/plan and open subtasks; (ii) in-flight/durably-pending tool calls and their results (LangGraph's per-task checkpoint_writes let completed nodes' writes survive a sibling's failure without re-running); (iii) idempotency keys / dedupe markers so replay does not resend an email or re-charge a card; (iv) any "exactly-once" side-effect ledger. Quality-only context (old turns, general knowledge) is explicitly not in the minimal set.

PART 2 — DEFER vs. EAGER DECISION (caching theory applied)

Decision rule: load eagerly the estimated working set (recent + high-recompute-cost + high-access-frequency state); defer everything else behind references, and let demand signals pull it in. This is the working-set model (Denning, 1968/1970) applied to context.

Established — Working-set estimation + demand paging.

Idea: Keep resident only the pages referenced in the most recent window Δ; fault the rest on demand.
How it works: Locality of reference means a process touches a small fraction of its pages in any phase. The working set W(t,Δ) is the distinct pages used in the last Δ references; OSes keep the working set resident to minimize page faults and avoid thrashing (excessive faulting when the working set exceeds memory). For agents: the "pages" are memory blocks/summaries/files; Δ is the recent task window.
Why non-obvious: Bigger is not better. Adding context can raise error rates — Chroma's 2025 research report "Context Rot: How Increasing Input Tokens Impacts LLM Performance" (research.trychroma.com) tested 18 frontier models (including GPT-4.1, Claude 4, Gemini 2.5, Qwen3) and found that "as needle-question similarity decreases, model performance degrades more significantly with increasing input length," with degradation appearing well before the window is full — an attention-budget analog of thrashing. Anthropic frames context as a finite resource with diminishing returns.
Concrete example: A coding agent keeps the 5 most recently accessed files resident (Claude Code's compaction keeps "the five most recently accessed files") and defers the rest.
Failure modes: Wrong Δ — too small misses the active locality; too large pulls cold data. Locality shifts (task switch) cause a burst of faults.
Adoption cost: Low conceptually; moderate to tune Δ and recompute-vs-fetch thresholds.

Established — Eviction/admission policies (LRU, LFU, ARC, TinyLFU) and hot/warm/cold tiering.

Idea: When resident context is full, evict by recency (LRU), frequency (LFU), or a hybrid; tier data hot→warm→cold.
How it works: LRU assumes recent⇒future; LFU keeps consistently popular items; ARC/2Q/Windowed-TinyLFU adaptively balance recency and frequency (TinyLFU is "state of the art for in-memory caches"). MemGPT/Letta maps this directly to three tiers: main context (RAM), recall storage (disk), archival storage (cold vector store), moving data between tiers via function calls.
Why non-obvious: The cost function for agents is not just hit-rate; it is cost-of-recompute vs. cost-of-fetch vs. token cost. A summary that is expensive to regenerate but cheap to store should be cached even if rarely accessed; a cheap-to-fetch document should be deferred even if recently used.
Concrete example: Snapshots "make sense when replay time > 100ms for hot aggregates" — a recompute-cost threshold for what to materialize vs. recompute.
Failure modes: LRU evicts a hot asset during a one-off burst (the "breaking-news 15MB video" pathology); LFU keeps stale once-popular items (needs aging/decay); similarity-based admission grows "confidently wrong" as the pool grows (see Part 3).
Adoption cost: Low if using an existing cache/memory store; moderate to encode the recompute-vs-fetch cost model.

Emerging — Predictive prefetching / ML-assisted eviction.

Idea: Predict the next-needed context and prefetch it before the fault (prepaging).
How it works: History-based or content-based prediction; ML/RL policies (LeCaR, Cold-RL, DEAP) learn reuse distance. UI-layer analog: link prefetch-on-intent (hover/focus preloads next page).
Why non-obvious: Prefetching hides latency only if predictions are good; bad prefetch wastes the very token/latency budget you're trying to save. Reported gains exist (Cold-RL claims +146% hit ratio under high pressure on adversarial workloads) but are workload-specific and not yet standard in agents — treat as emerging/uncertain.
Concrete example: OpenAI's "dreaming" background process curates memories from chat history asynchronously so they're ready next session (vendor-reported; see Part 6).
Failure modes: Misprediction cost; added system complexity; cold-start with no history.
Adoption cost: High (training data, infra, eval).

PART 3 — DETECTING MID-TASK THAT DEFERRED CONTEXT IS NEEDED ("page faults" for agents)

The core idea: make "I'm missing context" an observable, trappable event, just like a hardware page fault, and load on demand.

Established — Retrieval relevance threshold + fallback (RAG page-fault).

Idea: If the best retrieval score is below a threshold, the agent is "faulting" — trigger a broader/deeper fetch or abstain.
How it works: Compute similarity for top-k; if max score < τ (e.g., 0.6), fall back to web search, sub-query expansion, or human-in-the-loop. Corrective RAG (Yan et al., 2024) routes on a retrieval evaluator into correct/ambiguous/incorrect → refine / web-search / discard. Self-RAG trains the model to emit relevance/support tokens.
Why non-obvious: Confidence is a treacherous trigger. As the memory pool grows, mean similarity rises while actual relevance falls — the agent gets "confidently wrong," and a monitor that alerts on low confidence "will never fire." So thresholds must be calibrated against retrieval precision, not raw similarity.
Concrete example: A hybrid retriever issues a web search when local FAISS scores fall below 0.6; in one study this fired on a large share of queries.
Failure modes: Miscalibrated τ; silent confidence inflation; over-fetching (thrashing) if τ too high.
Adoption cost: Low–moderate.

Established — Tool-call failures, unknown-entity references, and explicit model requests as triggers.

Idea: Treat an error, a reference to an unknown entity, or the model asking for more as a demand-load signal.
How it works: A failed tool call / "not found" → fetch the missing definition or doc just-in-time (Anthropic's progressive tool discovery: explore a filesystem of tool modules and read only what's needed). MemGPT is explicitly event-triggered: a system alert about limited context, or the model calling archival_memory_search, drives paging.
Why non-obvious: The agent itself is the best fault handler — modern guidance is to let the model decide when to retrieve rather than pre-stuffing everything; "smarter models require less prescriptive engineering."
Concrete example: MemGPT receives a "system alert about limited context space" and writes/reads persistent memory via function calls — an OS page-fault interrupt analog.
Failure modes: The model fails to recognize it's missing context (false confidence); infinite retrieval loops; latency from runtime exploration ("runtime exploration is slower than retrieving pre-computed data").
Adoption cost: Low if your framework supports tool-driven memory; moderate to add fault detection.

Emerging — Uncertainty/confidence calibration as a fault signal. Verbal-confidence calibration for RAG (NAACL-style "parametric fallback": when no helpful passage is retrieved, answer from internal knowledge) is active research; reliable uncertainty estimation for LLMs remains unsolved — flag as uncertain.

PART 4 — GRACEFUL DEGRADATION FOR NON-EXPERT USERS

Principle: a cheap boot may degrade quality gracefully, but must never silently violate correctness. Hide latency, set expectations, and make fallbacks transparent.

Established — Progressive loading + skeleton screens + optimistic UI.

Idea: Show structure and immediate feedback while real context loads behind the scenes.
How it works: Skeleton screens preview layout; optimistic UI reflects an action immediately and reverts only on failure; render the cheap-boot answer first, then enrich.
Why non-obvious: Perceived performance ≠ actual performance, and the thresholds are well established. Jakob Nielsen's response-time limits (Nielsen Norman Group, from Usability Engineering, 1993) hold that "0.1 second is about the limit for having the user feel that the system is reacting instantaneously," ~1.0 s keeps the "flow of thought" uninterrupted, and 10 s is the attention limit — so feedback should appear within ~100 ms of an action. Skeleton screens don't reduce load time but improve perceived speed: Mejtoft, Långström & Söderström (ECCE '18, "The effect of skeleton screens") found skeleton pages "scored higher on average on both perceived speed and ease of navigation," though they note the differences were not statistically significant and spinner users actually rated first-visit article loading as faster — so treat the skeleton-screen benefit as a real but modest perceptual edge, not a guarantee.
Concrete example: Allbirds slides in a mini-cart optimistically before the API confirms; the animation "buys time" for the round-trip. For an agent: answer from the recent tail immediately, with a subtle "still loading older context" affordance.
Failure modes: Optimistic UI that must revert confuses users ("user confusion" / "queue jumping" when a later task finishes first, eroding trust); skeletons used for critical data hide that the answer is incomplete.
Adoption cost: Low–moderate (frontend work).

Design rules for non-experts (decision-ready):

Sensible defaults: boot with safe, conservative defaults; never require the user to understand checkpoints/tiers.
Transparent fallback, not silent: if operating on partial context, say so plainly ("Answering from your recent activity; still loading older history") and let correctness-critical actions wait or confirm.
Never optimistic on irreversible/unknown-ID actions: UX guidance is to wait for confirmation when creating records or when IDs/fields are unknown — for agents, that means gating irreversible side effects until correctness-critical state is loaded.
Latency hiding via background work: do enrichment (dreaming/consolidation) off the critical path.

PART 5 — CHEAPEST RELIABLE "RECENT TAIL" RECOVERY

Bottom line: read the last N lines/bytes of an append-only log starting from the last snapshot offset. That is the cheapest reliable recovery of recent context, and it's what tail, Kafka consumer offsets, ring buffers, and event-sourcing snapshot+replay all implement.

Established — Log tailing / append-only offsets.

Idea: Seek to a known offset (snapshot LSN or "now minus N") and read forward; never read the head.
How it works: Append-only logs give cheap sequential reads and stable offsets; Kafka retains an ordered, replayable log and consumers track offsets; tail/head read bounded byte/line ranges. Anthropic cites Claude Code using tail to analyze large data "without ever loading the full data objects into context."
Why non-obvious: You don't need random access or a full index for recency — a single offset + sequential scan is O(tail), not O(history).
Concrete example: Event-sourced read: snapshot = Snapshot.for(id); new_events = EventStore.after(id, snapshot.version); state = apply(snapshot.state, new_events).
Failure modes: If the tail window is too short you miss a still-open commitment older than N (so: bound by semantic boundaries, not just count); log gaps if compaction removed a needed record.
Adoption cost: Very low.

Established — Rolling/sliding-window summaries + ring buffers + log compaction.

Idea: Keep the last K turns verbatim (sliding window / ring buffer) plus a rolling summary of everything older; compact the log so replay stays cheap.
How it works: Sliding-window/buffer memory keeps the last K exchanges and discards older; ConversationSummaryBufferMemory keeps recent turns raw and a running summary so distant "aims" survive. Recursive summarization updates a summary = f(old summary, new turns). Log compaction creates a snapshot and removes superseded entries; Kafka compacted topics retain the latest snapshot per key. A ring buffer uses memory proportional to rate, not window width.
Why non-obvious: Combining a window (recent, high-fidelity) with a summary (distant, compressed) is the only single structure that preserves both recent detail and long-range goals cheaply — pure windows forget goals; pure summaries lose detail.
Concrete example: Anthropic's compaction in Claude Code: summarize the conversation near the limit, reinitialize with the summary + 5 most-recent files. Lightest-touch variant: tool-result clearing.
Failure modes: Over-aggressive compaction "can result in the loss of subtle but critical context whose importance only becomes apparent later"; summaries drift/hallucinate; last-write-wins snapshots lose intermediate states.
Adoption cost: Low (windows) to moderate (good summarization prompts, tuned for recall-then-precision).

PART 6 — CURRENT PATTERNS & TOOLS (established vs. emerging)

Established

RAG-at-boot / retrieval-augmented loading. Load a small index/manifest at boot; retrieve bodies on demand. Cursor is a canonical production example: it chunks code locally, stores embeddings + file paths/line ranges in a vector DB (Turbopuffer), and at query time does nearest-neighbor search, returns obfuscated paths, then reads the actual code locally — RAG with the codebase index as the retrieval mechanism. The index is kept fresh with a Merkle-tree sync engine; The Pragmatic Engineer's "Real-world engineering challenges: building Cursor" reports the sync engine "runs every 3 minutes" (secondary write-ups cite ~5–10 min, so treat the exact interval as approximate). Failure mode: stale indexes; "confidently wrong" as corpus grows. Cost: moderate (embedding pipeline + vector store).

Checkpointing / durable execution. LangGraph checkpointers snapshot graph state per super-step into threads (InMemorySaver for dev; Postgres/DynamoDB/Couchbase for prod), enabling resume, human-in-the-loop, and time-travel. Durability modes trade safety vs. speed: exit (fastest, no mid-execution recovery), async (small risk of a lost checkpoint on crash), sync (every checkpoint written before continuing, at a performance cost). Temporal/DBOS/Restate provide stronger durable execution: Temporal's Event History is "a complete and durable log of everything that has happened," replayed to rebuild state after a crash, skipping already-completed activities. Important distinction (flag): one vendor argues "checkpoints are not durable execution" — LangGraph/CrewAI/ADK give you a save point but leave failure detection, automatic recovery, and distributed coordination (e.g., two processes resuming the same thread_id) to you. Treat framework checkpointing as a building block, not turnkey guaranteed completion.

Memory stores / hierarchical memory.

MemGPT/Letta (Packer et al., 2023): OS-inspired virtual context management — main context (RAM), recall storage, archival storage; the model pages data via function calls. Established academic foundation; Letta is the productized framework.
Mem0 (ECAI 2025, arXiv:2504.19413): extract salient memories from turns, apply add/update/delete; vector-first with optional graph.
Zep (Rasmussen et al., 2025): temporal knowledge graph (Graphiti) modeling when facts were true.
LangMem (LangChain): semantic/episodic/procedural memory with background ("subconscious") consolidation; native LangGraph fit.
Benchmark caveat (flag explicitly): the LoCoMo/LongMemEval "benchmark wars" are real and unresolved — vendors dispute each other's setups (e.g., Zep's 84% claim vs. Mem0's recomputed 58.44%, vs. Zep's rebuttal of 75.14%). Do not take any single vendor's SOTA claim at face value; LoCoMo itself has been criticized as too short/easy (~16k–26k tokens).

Rolling/hierarchical summaries. Conversation summary memory, recursive summarization, and hierarchical memory trees are established for conversational continuity (see Part 5).

Anthropic context management (public beta). Two primitives launched with Sonnet 4.5 (Sept 2025): context editing (auto-clears stale tool calls/results as limits approach) and the memory tool (file-based /memory directory the agent CRUDs, client-side, persisting across sessions). Per Anthropic's "Managing context on the Claude Developer Platform" (claude.com/blog/context-management): "combining the memory tool with context editing improved performance by 39% over baseline. Context editing alone delivered a 29% improvement. In a 100-turn web search evaluation, context editing enabled agents to complete workflows that would otherwise fail due to context exhaustion—while reducing token consumption by 84%." (Vendor-reported on an internal agentic-search eval — flag.)

Code execution with MCP / progressive tool discovery (Nov 2025). Present MCP servers as code APIs in a filesystem; the agent reads only the tool modules it needs and processes data in the runtime. Anthropic reports a workflow dropping from ~~150,000 to ~2,000 tokens (~~98.7% reduction) for one scenario (vendor-reported). Related: the Tool Search Tool discovers tools on-demand, reportedly preserving ~191,300 vs. 122,800 tokens of context. This is the tool-loading analog of demand paging.

OpenAI / ChatGPT memory. Saved memories (Apr 2024) + "reference chat history" (Apr 2025) + "dreaming" background curation; a 2026 rollout makes dreaming the core architecture with a user-visible memory summary page and edit controls. OpenAI-reported evals: factual recall 67.9%→82.8%, preference adherence 55.3%→71.3%, time-sensitive accuracy 52.2%→75.1% (2025→2026), with ~5× compute reduction for serving free-tier dreaming (all vendor-reported — flag).

CLAUDE.md-style files / hybrid eager+lazy. Claude Code drops CLAUDE.md in eagerly for stable, "never-forgotten" project facts, then lazily greps/globs files. Anthropic recommends this hybrid for less-dynamic domains (legal/finance).

Emerging

Semantics-aware selective checkpointing — Crab (2026). "Crab: A Semantics-Aware Checkpoint/Restore Runtime for Agent Sandboxes," Wu, Chang, Cao, Gao & Wang, arXiv:2604.28138 (cs.OS/cs.AI, submitted Apr 30, 2026; 15 pages). The abstract states "over 75% of agent turns produce no recovery-relevant state, so most checkpoints are unnecessary," diagnoses an "agent-OS semantic gap" (agent frameworks see tool calls but not their OS effects; the OS sees state changes but lacks turn-level context), and uses an eBPF-based inspector to classify each turn's OS-visible effects and choose checkpoint granularity. Reported results on shell-intensive/code-repair workloads: "Crab raises recovery correctness from 8% (chat-only) to 100%, cuts checkpoint traffic by up to 87%, and stays within 1.9% of fault-free execution time" (vs. chat-only recovery succeeding on only 8–13% of Terminal-Bench tasks and every-turn full checkpointing slowing execution up to 3.78×). Preprint, authors' own results, not yet peer-reviewed — flag as emerging. The principle (use lightweight OS-visible signals for adaptive state management) is the agent analog of only flushing dirty pages.
Agentic/Corrective/Self-RAG, adaptive retrieval routing — maturing, not yet universal.
ML/RL prefetching & eviction (Cold-RL, DEAP, LeCaR) — research-grade.
"Claude Dreaming" / background memory consolidation — newly described; availability varies.

Recommendations (staged, with thresholds)

Stage 0 — Instrument first (week 1). Log, per boot, how many bytes/tokens you read and how much was actually used in the first task. If you're reading the full history but using a small recent fraction, you have a working-set mismatch — proceed. Threshold to act: if <20–30% data-preserve-html-node="true" of loaded context is referenced before the first useful action, lazy-load.

Stage 1 — Snapshot + tail (weeks 1–3, biggest win). Introduce a periodic snapshot of correctness-critical agent state (goal/plan, open commitments, idempotency keys, pending tool calls) and an append-only event log with offsets. Boot = restore latest snapshot + replay log suffix. If on LangGraph/Temporal/DBOS, adopt their checkpointers rather than hand-rolling. Choose durability mode by blast radius: use sync-equivalent when a missed checkpoint means a duplicated/irreversible side effect; async when loss is cosmetic. Wrap all non-deterministic/side-effecting steps behind durable boundaries so replay doesn't double-fire.

Stage 2 — Defer the rest behind references (weeks 2–4). Replace eager history loads with manifests/indexes + lightweight identifiers (file paths, queries, vector IDs). Keep resident only the working set: recent K turns (sliding window/ring buffer) + a rolling summary + the few highest-recompute-cost artifacts. Adopt compaction/tool-result-clearing near context limits.

Stage 3 — On-demand fault handling (weeks 3–6). Wire "page-fault" triggers: retrieval score < calibrated τ → broaden/fallback; tool-call failure / unknown entity → fetch definition; explicit model request → search memory. Calibrate τ against retrieval precision, not raw similarity, and add a guard against confidence inflation as the memory pool grows. Benchmark before/after on your tasks — ignore vendor LoCoMo claims.

Stage 4 — UX polish for non-experts (parallel). Answer from the recent tail immediately; show feedback within Nielsen's ~100 ms instantaneous threshold via skeleton/optimistic UI; transparent "still loading older context" affordance; gate irreversible actions until correctness-critical state is confirmed loaded. Never be optimistic about unknown-ID/irreversible operations.

Stage 5 — Optimize only if measured (later). Consider predictive prefetching / semantics-aware selective checkpointing (Crab-style) only after Stages 1–3, and only if checkpoint traffic or boot latency is still a measured bottleneck. Threshold: if profiling shows most checkpoints capture no recovery-relevant state, move to selective checkpointing.

What would change these recommendations: if your agent has no correctness-critical mutable state (pure stateless Q&A), skip checkpointing entirely and just RAG-at-boot. If full history genuinely is needed every task (rare), lazy-loading hurts — but the working-set data will tell you. If you operate in a regulated domain, favor the eager-hybrid (CLAUDE.md-style) and synchronous durability.

Caveats

Vendor-reported numbers (Anthropic +39%/84%/98.7%, OpenAI recall/preference gains, Cursor mechanics, Cold-RL +146%) are self-reported and benchmark-specific; treat as directional, not independently verified.
Memory-system benchmarks are contested (LoCoMo/LongMemEval "benchmark wars"); no clean SOTA ranking exists, and LoCoMo may be too easy. Validate on your own workload.
Checkpointing ≠ durable execution: framework checkpointers leave failure detection, automatic recovery, and distributed thread_id coordination to you.
Replay re-executes side-effecting steps unless wrapped behind durable boundaries — a correctness hazard if ignored.
Confidence/uncertainty signals are unreliable as fault triggers; calibrate against precision and beware confidence inflation as memory grows.
Crab (2026) is an unreviewed preprint with authors' own results; the skeleton-screen perceptual benefit (Mejtoft et al.) was not statistically significant.
"Context rot" (Chroma, 2025) means more context is not strictly better; lazy-loading has a quality upside, not just cost — but over-aggressive compaction can drop subtly critical state.

Source map

Anthropic context engineering / tools: "Effective context engineering for AI agents" (Sept 29, 2025); "Managing context on the Claude Developer Platform" (context-management news); Memory tool docs (platform.claude.com); "Code execution with MCP" (Nov 2025); "Introducing advanced tool use" (Tool Search Tool).
Checkpointing / durable execution: LangChain LangGraph persistence docs; AWS DynamoDBSaver blog; Temporal Event History docs & "Understanding Temporal"; Diagrid "Checkpoints Are Not Durable Execution"; Resonate/TheAgentStack on replay & DBOS/Restate.
Agent memory architectures: MemGPT paper (arXiv:2310.08560) & Letta; Mem0 (arXiv:2504.19413, ECAI 2025); Zep/Graphiti (Rasmussen et al., 2025) & Zep-vs-Mem0 benchmark disputes; LangMem docs; recursive summarization (arXiv:2308.15022); hierarchical memory surveys (arXiv:2603.07670).
Systems foundations: Denning working-set model & demand paging (UCSD/UIC OS notes); WAL/ARIES recovery (Sookocheff; LSN/checkpoint explainers); LSM-tree MANIFEST/WAL (freeCodeCamp; RocksDB papers); event sourcing snapshots & log compaction (Kurrent, Conduktor/Kafka).
Caching/eviction/prefetching: CacheLib eviction docs; ACM survey on ML-based caching/tiering (3803857); Cold-RL (arXiv:2508.12485); DEAP/LeCaR (arXiv:2009.09206); TinyLFU/ARC overviews.
RAG triggers / confidence: Corrective RAG & Self-RAG overviews; Microsoft "Confidence-Aware RAG"; "Your RAG Gets Confidently Wrong as Memory Grows" (Towards Data Science); NAACL verbal-confidence calibration (arXiv:2601.11004).
UX / progressive loading: Nielsen Norman Group "Response Time Limits" (Nielsen, Usability Engineering, 1993); Mejtoft, Långström & Söderström, "The effect of skeleton screens" (ECCE '18); LogRocket & Clay skeleton-screen guides; Smart Interface Design Patterns (loading UX); Remix pending/optimistic UI; Simon Hearne optimistic UI patterns.
Context rot / Cursor / OpenAI memory: Chroma "Context Rot" report (research.trychroma.com, 2025); The Pragmatic Engineer "Real-world engineering challenges: building Cursor" & Engineer's Codex on Cursor indexing; Cursor docs (semantic & agentic search); OpenAI "Memory and new controls" + Memory FAQ; 2026 dreaming rollout coverage.
Emerging: Crab (arXiv:2604.28138).

State of the art as of today (June 9, 2026)

The field has firmly converged on "just-in-time" / deferred context: boot small, keep a tight working set, and demand-load the rest behind lightweight references — explicitly mapping OS demand paging, WAL+checkpoint recovery, log compaction, and cache tiering onto LLM context management. Established and safe to build on now: snapshot+log-tail recovery (LangGraph/Temporal/event-sourcing), manifest/index-driven metadata-first boot, sliding-window+rolling-summary memory, compaction/context-editing, RAG-at-boot with on-demand retrieval (Cursor-style), and OS-inspired hierarchical memory (MemGPT/Letta). Maturing: managed memory services (Mem0/Zep/LangMem) — useful but with contested benchmarks; agentic/corrective RAG with relevance-threshold fault handling; tool-loading-as-paging (code execution with MCP, Tool Search). Emerging / watch: semantics-aware selective checkpointing (Crab, 2026) that checkpoints only recovery-relevant turns; background memory consolidation ("dreaming"); and ML/RL prefetching/eviction. The throughline matches Denning and ARIES from decades ago and Anthropic's 2025 guidance alike: find the smallest set of high-signal tokens, recover only what's needed for correctness, and fault in the rest on demand. The principal open risks are confidence/relevance miscalibration (agents that are "confidently wrong"), replay double-firing side effects, and over-aggressive compaction — all manageable with the staged plan above.

Ketelsen.ai 2.0ClaudeKetelsen.ai 2.0 Week 10

Richard Ketelsen