Perplexity :: Week10 :: Special Series :: AI Token Compression and Task Delegation Research :: Shrinking AI Agent Instruction Files & Tiered Context Loading at Boot

• THE ORIGINAL DEEP RESEARCH PROMPT 1

  Deep Research request. Be thorough, cite sources, prioritize reliable information from the last ~24 months.

  OBJECTIVE: Identify the most effective and most NON-OBVIOUS techniques for shrinking a large

  instruction/configuration file that an AI agent re-reads at the start of every single turn (so its size is

  paid repeatedly), WITHOUT losing any behavioral fidelity.

  Research and answer ALL of the following up front (do not pause to ask me; state any assumptions you make):

  1. What categories of content can be moved out and loaded on demand vs. what must stay inline to preserve behavior?

  2. Which concrete techniques cut size the most — across structure, references/pointers, externalization, summarization, and encoding/compression?

  3. How do practitioners PROVE nothing was lost after compression (regression/parity/eval methods)?

  4. Which "compressions" look safe but silently change model behavior (the traps)?

  5. What current tools, papers, or established patterns address this (e.g. prompt/context compression, context distillation, instruction caching)? Name them with sources.

  SOURCES & RECENCY: Favor credible, recent (≈last 24 months) sources; cite every non-obvious claim with a link;

  distinguish established practice from experimental; flag thin/contested evidence.

  OUTPUT: For each distinct technique, use: (1) Idea one-liner (2) How it works (3) Why non-obvious (4) Concrete

  worked example (5) Failure modes (6) Adoption cost (low/med/high + why). End with a Source map and a

  "State of the art as of today" paragraph.

  CONSTRAINT: Answer generically (universal AI-agent design). Do not ask me for proprietary details.

W1·Q1 — Instruction File Compression Without Behavioral Loss

W1·Q2 — Tiered / Lazy Context Loading at Session Start

---

Executive Summary

Two compounding costs quietly dominate every turn of a production AI agent loop: (1) the sheer size of the instruction/configuration file that is re-read at the start of each turn, and (2) the habit of bulk-loading large history or log files at boot even when only a small tail of that state is needed. Both are solvable, but most solutions that appear safe harbor subtle behavioral traps.

This report synthesizes the current state of the art as of June 2026, drawing on peer-reviewed papers (EMNLP 2023, ICLR 2024, arXiv 2024–2026), Anthropic's engineering blog (September 2025), controlled A/B experiments (May 2026), and production system disclosures from Claude Code, LangGraph, Zep, and others. Each technique is presented in the six-part format requested: idea one-liner → how it works → why non-obvious → worked example → failure modes → adoption cost.

---

Part 1 — Shrinking the Instruction/Configuration File

1.1 What Can Be Moved Out vs. What Must Stay Inline

The Taxonomy of Context Content

Content in an agent's system/configuration file falls into three behavioral tiers:

Tier	Content Examples	Can Externalize?	Rationale
Core-inline required	Safety guardrails, negation-heavy constraints, identity rules, critical "never do X" rules	No	RLHF tension means these lose compliance at medium compression; must be verbatim and reinforced [1][2]
Deferrable on-demand	Step-by-step workflow detail, tool usage guides, domain-specific examples, sub-task procedures	Yes — fetch on trigger	Needed only for specific sub-workflows; loading them all turns cold-start into worst-case [3]
Fully externalizable	Reference tables, code templates, large few-shot examples, project-specific knowledge	Yes — RAG/tool call	Pure lookup; zero cost to defer until the task context makes them relevant [4]
Cache-eligible	Static preambles, persona descriptions, organizational boilerplate	Yes — prefix cache	Identical across turns; KV cache reuse yields 90% cost reduction on reprocessing [5][6]

Key finding: Safety rules, constraint lists, and negation-heavy behavioral boundaries are the one content class that must not be compressed aggressively or deferred. A/B testing of a real agent stack found that collapsing 8 verbose safety bullets into 3 dense sentences caused correct permission-asking to drop from 100% to 33% across repeated identical prompts. Redundancy in safety instructions is not waste — it is enforcement.[7]

---

1.2 Concrete Compression Techniques

Technique 1: Just-in-Time Section Loading (Bootstrap + Directory Stub)

Idea one-liner: Replace the full instruction file with a 20–30-line table of contents; deliver detail sections only when their trigger condition fires.

How it works: The always-loaded system prompt is reduced to a directory — each section name, its one-line trigger condition, and a pointer (e.g., Detail: get_custom_data("strategy", "post_task")). When the agent's reasoning path encounters a trigger condition, it makes a single tool call to fetch that section from an MCP server or knowledge store. Full sections are shipped with the server code, not in a database, so retrieval latency is negligible.[3]

Why non-obvious: The natural instinct is to give the agent everything it might need upfront "so it doesn't miss anything." The non-obvious insight is that a well-structured directory is itself sufficient context for routing decisions — the agent only needs the detail once a branch has been entered. This mirrors how humans use documentation: we don't memorize the entire employee handbook, we know where to look.[4]

Concrete example: Engrams MCP reduced a 400-line YAML strategy file (10,000 tokens) to a 30-line bootstrap stub (800 tokens). Per-session total, including on-demand loads of triggered sections, averaged 1,000–3,000 tokens — an 80–90% reduction with no measurable behavioral degradation.[3]

Failure modes:

• Circular dependency: a trigger condition that itself requires deferred context to evaluate
• Cold-start latency: the extra tool-call round trip adds ~100–300ms per unique section first invoked
• Trigger mis-detection: if the stub is too terse, the model may fail to recognize that a complex workflow was triggered and skip the detail fetch

Adoption cost: Medium. Requires refactoring the instruction file into modules, implementing a retrieval endpoint (trivially done with MCP), and writing the stub directory. One-time investment, very low per-session cost thereafter.

---

Technique 2: Semantic Compression (Terse Rewrite by LLM)

Idea one-liner: Use an LLM to rewrite verbose prose instructions in the most information-dense natural-language form an LLM can reliably follow.

How it works: LLMs understand terse, imperative instructions just as well as verbose narrative ones — because they were trained on both forms. The compression exploits the fact that LLMs have internalized common procedural knowledge. A 50-line YAML block describing "scan the workspace root for manifest files to detect project type" can be reduced to a 12-line flat lookup table with identical behavioral outcomes. A two-pass LLM prompt — first merging redundant rules, then compressing per content type — achieves 47–54% reduction empirically.[7][3]

Why non-obvious: The instinct is to use mechanical string replacement or regex (removing markdown formatting, blank lines, redundant headers). This only achieves 2–15% reduction in already-lean files. LLM-based compression achieves 20–34% on the same files because the model understands which words carry semantic payload and which are scaffolding. The analogy: a recipe that says "in a large, clean, dry mixing bowl, add the flour" vs. "Add flour" — a trained chef (or LLM) needs only the latter.[7]

Concrete example: On a 37.8KB steering stack (10 Markdown files), regex compression averaged 2.7% reduction; LLM compression averaged 24% (range 19–34% per file). The obsidian-integration.md (5,634 → 4,287 tokens, 24% reduction) and cli-tools.md (5,448 → 3,603 tokens, 34% reduction) files showed the largest gains.[7]

Failure modes:

• Safety content boundary violation: a two-pass LLM compressor that merges safety rules crosses the behavioral cliff (see Section 1.4)
• Hallucinated compression: the rewriting LLM may silently introduce new constraints or remove edge cases it deemed redundant
• Model-specificity: a compression optimized for GPT-4o may lose fidelity when the target agent runs on Claude or Mistral, because tokenization and attention patterns differ[8]

Adoption cost: Low (one-time, ~30 min per file). Automate with open-source tools like context-compress llm (vidanov/context-compress). Ongoing cost: re-run when instructions change.[7]

---

Technique 3: Structural Deflation (Format Normalization)

Idea one-liner: Convert hierarchically-nested YAML/JSON instruction blocks to flat, colon-delimited tables; eliminate empty template keys and redundant nesting.

How it works: YAML structure amplifies token count for data that is logically flat. A detection-rule table with indicator, suggests, and project_type subkeys per entry consumes ~4× the tokens of a single-line colon-separated mapping. Equally, YAML placeholder keys (empty thinking_preamble: | blocks, unfilled examples: [] arrays) consume tokens while contributing zero information[3]. Converting the representation from deeply nested objects to flat tables, and stripping all zero-information structure, typically yields 30–40% savings on instruction files originally authored in configuration-file style.

Why non-obvious: The format was chosen for human readability and tooling compatibility. The non-obvious insight is that the agent consumes the file as tokens, not as a schema. A table in compressed Markdown is semantically equivalent to a nested YAML object — and several times smaller.

Concrete example:

# Original YAML (50 lines, ~400 tokens):
post_task_setup_questionnaire:
  description: |
    Runs during first-time Engrams setup to configure verification checks...
  steps:
    - step: 1
      action: "Auto-detect project type by scanning workspace root..."
      detection_rules:
        - indicator: "package.json"
          suggests: ["npm test", "npm run lint"]
          project_type: "Node.js/TypeScript"
        ... (5 more language blocks, each similarly expanded)

# Compressed Markdown (12 lines, ~100 tokens):
POST_TASK_SETUP (first-time only):
  1. Detect type from manifest:
     package.json→Node, pyproject.toml/requirements.txt→Python,
     Cargo.toml→Rust, go.mod→Go, pom.xml→Java, Gemfile→Ruby
  2. Suggest test/lint/build checks for detected type
  3. Store → log_custom_data("post_task_checks", "verification_commands")

Failure modes:

• Loss of implicit hierarchical meaning (e.g., subkeys that encode priority or ordering)
• Human maintainability regression: flat tables are harder for contributors to read and extend

Adoption cost: Low. Mechanical transformation; can be scripted.

---

Technique 4: Single Source + Thin Delta Architecture

Idea one-liner: If one instruction file serves multiple tool variants, maintain one canonical core with per-tool delta overrides — never N full copies.

How it works: When an agent system targets multiple runtimes (Cursor, Windsurf, Claude Code, Roo Code, etc.), 90–95% of the instruction content is identical across all variants. The differences are cosmetic: header comments, workspace path variable names, one or two tool-specific function names. Maintaining N full copies means compression effort must be applied N times and kept synchronized. Moving to a _core.yaml plus _delta_<tool>.yaml reduces maintenance surface to 1× and makes structural changes propagate automatically. This does not reduce per-user per-turn cost — the generated output file is identical in size — but it creates a compounding leverage point for Techniques 1–3.[3]

Why non-obvious: The benefit is not immediate token reduction but the elimination of maintenance friction that prevents compression. Teams with 7 copies of a 400-line file tend not to compress because the work multiplies. With one source, compression is applied once.

Adoption cost: Medium. Requires a build step (merge core + delta at install time or CI).

---

Technique 5: Token-Oriented Object Notation (TOON) for Structured Payloads

Idea one-liner: For tool-output or API-response sections of the instruction/context, declare field names once and use positional encoding for subsequent records.

How it works: Standard JSON repeats field names on every record. For uniform arrays (resource inventories, API response lists, tool schemas), declaring the schema once and encoding rows as positional arrays achieves 30–60% token reduction. This is analogous to CSV vs. JSON-with-keys. The technique is especially valuable for inline tool schemas embedded in system prompts (e.g., the full OpenAPI-style descriptions of every available MCP tool).[7]

Concrete example: Claude Code's MCP Tool Search (released January 14, 2026) replaced loading full tool definitions (67,000 tokens for 7 servers with 50+ tools) with a lightweight search index (8,500 tokens — 87% reduction). Full tool schemas are fetched on-demand only when a tool is about to be invoked.[9][10]

Why non-obvious: Tool schema verbosity is rarely examined by instruction file authors because it's injected automatically by the framework, not written by hand. It is among the largest single contributors to instruction overhead in any MCP-enabled setup.

Failure modes:

• Schema decode complexity: the consuming model must be able to reconstruct the full tool semantics from the compact form — not all models handle pure positional encoding without explicit column names
• Tool selection errors: if the summary in the search index is too brief, the agent may fail to identify the right tool for a task

Adoption cost: Low for MCP-enabled stacks (built-in via enable_tool_search). Medium for custom tool schema formats.

---

Technique 6: Prefix Caching (KV Cache Reuse)

Idea one-liner: Structure the system prompt so its immutable portion is always identical and placed at the very beginning, enabling the provider's KV cache to serve it without recomputation.

How it works: Modern LLM providers (Anthropic, OpenAI, Google, AWS Bedrock) cache the computed key-value attention states for prompt prefixes that appear identically across requests. Anthropic's prompt caching offers up to 90% cost reduction and 85% latency reduction for cached prefix tokens. The rules are strict: the prefix must be character-for-character identical including whitespace; even a single character difference invalidates the cache. Dynamic elements (timestamps, session IDs, user names) must be placed after the stable content, not injected into the beginning.[5][6]

Why non-obvious: Many teams place dynamic context (session timestamps, current user) at the top of the system prompt for readability. This silently disables caching on every single turn, converting what should be a 10-cent repeated computation into a full reprocessing charge. The instruction file is not being shortened — but its effective per-turn cost approaches zero for the cached portion.

Concrete example: A 15,000-token system prompt with a stable 12,000-token instruction block and a 3,000-token dynamic tail: with prefix caching, only the 3,000-token tail is computed on every turn. The 12,000-token block costs only 10% of normal input price per call once cached. Break-even vs. paying full price: turn 2 of every session.[11]

Failure modes:

• Non-deterministic serialization (e.g., Python json.dumps without sort_keys=True) randomizes key order, causing cache misses even when data is logically identical
• Anthropic/OpenAI caches have a ~5-minute sliding window timeout; idle sessions lose the cache between turns[12]
• Cache hit rate is a per-workload metric; do not assume a global number applies

Adoption cost: Low. Purely an ordering change in the system prompt construction code.

---

Technique 7: LLMLingua / RECOMP Token Pruning (for Reference/RAG Sections)

Idea one-liner: For long retrieved-document sections or examples embedded in the instruction file, apply learned token pruning to remove low-entropy tokens while preserving semantic fidelity.

How it works: LLMLingua uses a small language model (GPT-2, LLaMA-7B) to compute per-token perplexity and removes tokens below an information threshold, with a budget controller that preserves different compression rates across instruction vs. demonstration vs. question segments. LLMLingua-2 (Microsoft, 2024) reformulates this as a binary classification task (preserve/discard per token), training a bidirectional BERT-style encoder on GPT-4-distilled data — achieving 2–5× compression at 3–6× faster inference speed. RECOMP (ICLR 2024) applies a similar approach specifically to retrieved documents prepended to prompts, achieving compression rates as low as 6% with minimal loss on QA tasks.[13][14][15][16][8]

Why non-obvious: These tools are primarily marketed for RAG pipelines, but they apply equally well to the large example blocks and reference tables that often occupy 30–50% of a well-constructed instruction file. An instruction file that includes 20 worked examples for few-shot prompting can have those examples compressed 4–5× while preserving most of the behavioral signal.

Failure modes:

• Task-agnostic pruning (LLMLingua-2) sacrifices some performance on domain-specific tasks vs. task-aware methods[14]
• Token-level pruning can break syntactic structure; compressed output is human-unreadable (though LLM-readable)
• Cross-model family transfer: a compressor aligned to GPT-3.5 may lose fidelity on Claude — alignment fine-tuning is required for cross-family transfer[8]

Adoption cost: High for full integration (requires running a small LM as a preprocessing step). Medium for teams already using LlamaIndex (LLMLingua is natively integrated).[13]

---

Technique 8: Context Distillation / Internalization (Fine-Tuning Path)

Idea one-liner: Fine-tune the agent model to internalize the system instructions as parametric knowledge, making the runtime instruction file unnecessary.

How it works: Context distillation (Anthropic original technique, now extended by Generative Context Distillation, arXiv 2411.15927, Nov 2024) fine-tunes a model to produce the same outputs conditioned on a minimal instruction stub that it would produce conditioned on the full prompt. The "teacher" model sees the full prompt + scratchpad; the "student" is fine-tuned to predict the final answer directly from minimal input. In production agent settings, this means the model is trained to exhibit the correct behavioral rules without needing them re-stated every turn.[17][18]

Why non-obvious: This transforms a per-turn token cost into a one-time training cost. The non-obvious implication is that the full instruction file can be reduced to a 1–2 sentence system message confirming identity/role, with all behavioral rules expressed through weights rather than tokens.

Failure modes:

• Catastrophic forgetting: fine-tuning on behavioral distillation can degrade base capabilities
• Update cost: any change to instructions requires re-distillation
• Black-box opacity: behaviors are hidden in weights, making auditing, compliance review, and debugging harder
• Not applicable to API-only workflows where model weights are inaccessible

Adoption cost: High. Requires fine-tuning infrastructure, evaluation harness, and ongoing re-distillation on instruction updates. Established pattern for very high-volume, stable-instruction deployments.

---

1.3 Proving Nothing Was Lost: Regression / Parity Methods

The Golden Test Harness Pattern

The standard evaluation method for instruction compression is a golden test set — a set of input prompts with expected behavioral outputs, run against both the original and compressed instruction file, with automated scoring.[19][20]

Minimum viable test set structure:

1. Happy-path behavior tests — prompts that should invoke each major workflow; verify correct workflow execution
2. Safety / constraint compliance tests — prompts that test each "never do X" rule explicitly; verify refusal or confirmation-seeking behavior
3. Edge-case and negation tests — prompts designed to trip up over-compressed negation statements ("always ask before" → "ask before" → silent omission)
4. Regression delta tests — compare behavioral outputs between original and compressed using an LLM judge with near-perfect inter-rater agreement (Fleiss' κ ≥ 0.90 achievable for constraint compliance per CDCT benchmark)[1][2]

The CDCT Framework (Dec 2025): The Compression-Decay Comprehension Test independently measures constraint compliance (CC) and semantic accuracy (SA) across compression levels. Key finding: these dimensions are statistically orthogonal (r=0.193, p=0.084) — a compressed instruction can maintain semantic accuracy while silently violating constraints. Testing only for semantic similarity (e.g., cosine similarity of outputs) is insufficient. Both dimensions must be tested independently.[2][1]

Concrete procedure for A/B testing:

1. Record a baseline run of 20–50 diverse prompts against the original instructions in --no-interactive / batch mode
2. Apply compression
3. Replay identical prompts against compressed version
4. Score with an LLM judge on: (a) task completion, (b) constraint compliance per each stated rule, (c) style/preference adherence
5. Human review of any behavioral delta flagged by the judge
6. Gate deployment on zero constraint-compliance regressions[7]

Parity threshold guidance:

• Safety/constraint rules: 100% required — any probabilistic compliance drop is a regression
• Style preferences: ≥90% compliance acceptable
• Knowledge/domain accuracy: depends on task criticality

---

1.4 The Traps: Compressions That Silently Change Behavior

Trap 1: The RLHF Helpfulness Collision

The U-curve in constraint compliance (CDCT, arXiv 2512.17920): constraint violations peak at medium compression (c=0.5, ~27 words per instruction), not at extreme compression. Counterintuitively, models perform better at extreme brevity than at mid-length. The mechanism: RLHF-trained "helpfulness" behaviors compete with explicit constraints, and at medium compression the instruction is long enough to activate helpfulness training but too brief to override it. Removing RLHF helpfulness signals improved constraint compliance by 598% (71/72 trials). Constraint effects were 2.9× larger than semantic effects.[1]

Implication: Don't stop at 50% compression. Either stay verbose (100%) on safety rules, or go all the way to imperative one-liners. The danger zone is the middle.

Trap 2: Negation Fragility

Negation statements are disproportionately sensitive to reformulation. "NEVER execute commands without explicit user approval" reduced to "! exec w/o approval" caused the agent to skip asking for approval in 2 out of 3 runs of an identical prompt — shifting from deterministic to probabilistic compliance. Negation requires syntactically explicit negative phrasing; symbol-encoded negation is not reliably parsed.[7]

Trap 3: Redundancy as Enforcement

Repeating the same safety constraint with different phrasings is not waste. It is reinforcement. A 54% reduction from merging 8 safety bullets into 3 semantically-equivalent sentences produced probabilistic rather than deterministic compliance. Behavioral redundancy in safety content functions like unit test coverage: reducing it reduces reliability.[7]

Trap 4: Semantic Compression Hallucination

LLM-based rewriting can silently introduce new constraints or quietly drop edge cases the compressor LLM judged as obvious. This is especially dangerous for security templates, IAM policies, and OIDC conditions — content that must be pinned verbatim.[7]

Trap 5: The Middle-of-Context Attention Hole

Even when an instruction file is correctly sized and placed in the system prompt, content in the middle of a long context suffers the "lost-in-the-middle" effect — models attend well to the beginning and end but poorly to the interior. Instruction files should front-load the most critical behavioral rules, not bury them in the middle. Ironically, adding a long preamble of verbose examples or reference material pushes critical safety rules into the attention hole.[21]

Trap 6: Prefix Cache Timestamp Invalidation

Inserting dynamic elements (timestamps, user names, session IDs) early in the system prompt breaks the prefix cache on every single turn, converting a 10%-cost prefix into a 100%-cost recomputation — silently negating all caching benefits.[5][12]

---

1.5 Current Tools and Papers

Tool / Paper	Type	What It Does	Source
LLMLingua (Microsoft, EMNLP 2023)	Established	Up to 20× token pruning via small-LM perplexity scoring	[13][22]
LLMLingua-2 (Microsoft/Tsinghua, 2024)	Established	Task-agnostic token classification; 2–5× compression, 3–6× faster	[14][8]
LongLLMLingua (Microsoft, 2024)	Established	Query-aware long-context compression; position bias correction	[23]
RECOMP (ICLR 2024)	Established	Extractive + abstractive compressors for retrieved docs; 6% compression floor	[15][16]
Selective Context (2023)	Established	Self-information token scoring; 32% latency, 50% cost reduction	[8][24]
AutoCompressors (2024)	Experimental	Soft-prompt compression into summary vectors; requires model fine-tuning	[25][26]
GRACE (2024)	Experimental	Gated refinement + adaptive compression for prompt optimization	[27]
CDCT Benchmark (arXiv 2512.17920, Dec 2025)	Established	First benchmark separating constraint compliance from semantic accuracy	[1][2]
context-compress CLI (vidanov, May 2026)	Emerging	Open-source CLI for LLM + regex compression, dedup, token stats	[7]
MCP Tool Search (Anthropic, Jan 2026)	Established	Lazy-load tool schemas; 85–95% reduction in tool definition token overhead	[28][9][10]
Anthropic Prompt Caching (2024–2025)	Established	KV cache reuse; 90% cost / 85% latency reduction for stable prefixes	[5][6]
Generative Context Distillation (arXiv 2411.15927, Nov 2024)	Experimental	Fine-tune model to generate/internalize system prompt	[17]
AGENTS.md / CLAUDE.md files	Established practice	Persistent project-level instruction files; studied at scale (2,303 files, 1,925 repos)	[29][30]

---

---

Part 2 — Tiered / Lazy Context Loading at Session Boot

2.1 The Minimal Read-Set at Boot

What Correctness Requires

A cold boot that reads everything in a history/log file to recover recent context is almost always unnecessary. Research converges on a four-item minimal read-set that preserves correctness:

1. System identity stub — who the agent is, what it is authorized to do (small, stable, prefix-cached)
2. Progress narrative — a structured file (e.g., claude-progress.txt, MEMORY.md) authored by the previous session encoding: what was recently worked on, what is currently incomplete, what the next session should prioritize, and any environmental gotchas[21]
3. Recent tail of history — the last N turns of conversation (typically 5–10) or the last N minutes of a rolling-time window; not the full log
4. Priority queue — the explicit ordered next-step list, updated by the previous session at close

What is explicitly not in the minimal read-set: raw tool outputs, intermediate search results, full conversation transcripts beyond the recent tail, and verbose reference tables not needed by the first task.[4][21]

Anthropic's multi-session agent research found that agents encountering broken state from prior sessions would "spend substantial time trying to get the basic app working again" — but agents supplied with a structured progress narrative recovered immediately and continued correctly.[21]

---

2.2 Tiered / Lazy Context Loading Techniques

Technique 1: Rolling Summary Compaction with Observation Masking

Idea one-liner: Summarize full conversation history into a compact narrative, then at each subsequent boot load only the summary plus the N most recent turns.

How it works: Compaction takes a conversation approaching the context limit, passes it to the model to summarize into a narrative preserving architectural decisions, unresolved bugs, implementation details, and next steps — discarding raw tool outputs and redundant messages. The agent then resumes with this compressed summary plus the 5 most recently accessed files as the full boot context. Anthropic's JetBrains benchmark (250-turn trajectories on SWE-bench Verified) found that observation masking — replacing older environment observations with placeholders while preserving reasoning and actions — achieved 52% cost reduction while matching or exceeding LLM-based summarization in solve rate. With Qwen3-Coder 480B, masking achieved 2.6% higher solve rates than LLM summarization while being 52% cheaper. The reason: LLM summarization inadvertently extended agent trajectories by 13–15% by obscuring natural stopping signals.[4][21]

Why non-obvious: The instinct is that LLM summarization preserves more semantic content than masking. The research finding — that masking often outperforms summarization — is counterintuitive. Masking preserves the exact wording of the agent's reasoning steps while removing bulky environment state. Summarization inadvertently rewrites reasoning in ways that shift subsequent behavior.

Concrete example: Claude Code's /compact and auto-compact system: when the conversation approaches the auto_compact_limit, the model summarizes its history. The boot context for the next turn is: [compact summary] + [last 5 accessed files] + [current system prompt]. Users get continuity without context window starvation.[4][21]

Failure modes:

• Overly aggressive compaction loses subtle context ("the user said they wanted this approach specifically, not the obvious one")
• Compaction prompt tuning is an art — poor compaction prompts produce summaries that are accurate but lose non-obvious behavioral constraints
• Automatic compaction triggered at the wrong moment may mid-task compact partially executed workflows

Adoption cost: Low. Most frameworks (Claude Code, Codex CLI, LangGraph) ship this natively. Custom implementations require a carefully tuned compaction prompt.

---

Technique 2: Structured Progress File (Narrative Bridge Pattern)

Idea one-liner: Each session writes a machine-readable narrative at close; the next session reads only that file at boot, not the full log.

How it works: At session end, the agent writes a structured handoff document encoding five layers:[21]

1. State snapshot — current values of key tracked variables
2. Narrative context — 3–5 sentences on why the state looks as it does
3. Decision log — what was decided, what was explicitly deferred and why
4. Priority queue — ordered next steps for the next session
5. Warnings and gotchas — environment issues, rate limits, in-progress side effects

At boot, the agent reads only this file (typically 500–2,000 tokens) rather than parsing the full git history or conversation log. This is the pattern Anthropic formalized in their multi-session harness research.[21]

Why non-obvious: The difference between persistence (storing data) and handoff (communicating between sessions) is widely misunderstood. Most teams persist raw data (JSON state, conversation history). A handoff tells a story — it encodes not just what is true but what matters right now, which requires the ending session to author it deliberately. This human-readable narrative bridge is vastly more boot-efficient than querying a vector store for "recent context."

Concrete example: Anthropic's claude-progress.txt pattern: each session begins with pwd, reads progress log + recent git history (git log --oneline -10), runs init.sh to start the dev server, runs end-to-end tests to detect undocumented bugs, then selects the next incomplete feature from the JSON feature list. Git serves as a recovery mechanism when the prior session's changes were incomplete.[21]

Failure modes:

• Stale handoffs: if the previous session crashed without writing the handoff, the current session has no narrative bridge
• Premature completion: a session that reads "90% done" may prematurely declare the task complete without verifying
• Handoff rot: fields written in early sessions become irrelevant but are still loaded and consume attention budget

Adoption cost: Low. Requires only that agents write a structured file at close and read it at open. No framework dependency.

---

Technique 3: RAG-at-Boot (Semantic Retrieval of Recent Context)

Idea one-liner: Index the full history into a vector store; at boot, retrieve only the semantically relevant subset for the current session's likely task.

How it works: All prior context (conversation history, tool outputs, decisions) is chunked and embedded into a vector store (Mem0, Zep, Qdrant, Weaviate). At boot, a lightweight session-opening prompt ("What are we working on today?") seeds an embedding query that retrieves the most contextually relevant N chunks. Zep's Graphiti system goes further: it builds a temporal knowledge graph from conversation history, enabling hybrid retrieval combining vector similarity with graph traversal (e.g., "what was the last decision about authentication?" surfaces the specific fact plus the relationships that make it relevant).[31][32][33]

Zep's benchmark: their graph-based system scored 18% higher than a "dump everything in context" approach while using 1/10 the processing time and 1/100 the context tokens.[33]

Why non-obvious: The non-obvious benefit is not just token reduction but precision. Dumping full conversation history (even within context window limits) performs worse than selective retrieval because of context rot — Chroma's 2025 study found significant degradation at every increment of context growth, with 30%+ accuracy drops for information buried in the middle of a conversation. Less context, precisely selected, outperforms more context, bulk-loaded.[21]

Concrete example: Mem0 hybrid store: vector + key-value + graph. At session boot, a query embedding of the session's stated goal retrieves the top-K relevant memories. A coding agent starting a new session on "fix authentication bug" retrieves prior notes on authentication architecture, last decision on JWT vs. session cookies, and the last known error trace — not the full history of every tool call ever made.[33]

Failure modes:

• Embedding drift: if the embedding model is updated, older embeddings may not match semantically identical queries
• Recency bias: vector search on a broad query may retrieve highly similar but old content, missing the fact that the most important thing is "the last commit broke the login flow"
• Bootstrap bootstrapping: a purely RAG boot with no anchor context may retrieve irrelevant content if the session's opening query is too generic

Adoption cost: Medium-High. Requires a running vector/graph store, embedding pipeline, and session startup code. Tools like Mem0 and Zep provide managed APIs. Open-source: Mem0 (22,000+ GitHub stars), Zep community edition.[33]

---

Technique 4: Tiered Eager/Deferred Loading with Content-Type Classification

Idea one-liner: Classify context content into tiers at write-time; boot loads only tier 1, defers tier 2 and 3 until a task signals need.

How it works: The Claude Code GitHub issue #11364 (filed Nov 2025, resolved Jan 2026 via MCP Tool Search) formalized a three-tier model for tool schemas that generalizes to all context content:[34]

Tier	Content	Load	Format
Tier 1 (Always)	Server name, tool name, one-line description	Session start	Lightweight index
Tier 2 (On-Signal)	Parameter types, examples, detailed description	When tool is selected	Full schema fetched
Tier 3 (On-Use)	Response templates, error recovery guides	When tool call is about to be made	Retrieved from MCP

The same model applies to context other than tools: a coding agent's "always-loaded" tier might include the project summary (100 tokens), while full architectural docs (5,000 tokens) are loaded only when architectural decisions are being made.

Why non-obvious: The three-tier model with automatic promotion is a pattern from database design (primary key index → page → column) applied to prompt engineering. The key non-obvious insight: the agent doesn't need to know the details of a tool to select it. A one-line description is sufficient for routing. Details are only needed at invocation time — which may be never in a given session.

Concrete example: Claude Code MCP Tool Search: at session start with 7 MCP servers, tool definitions dropped from 67,000 tokens (33.7% of 200K context) to 8,500 tokens (4.25%) — an 87% reduction in initial context consumption. The system switches automatically when tool descriptions exceed 10% of available context.[28][10][34]

Failure modes:

• Tool selection errors: if tier 1 descriptions are too terse, the agent selects the wrong tool
• Latency spikes: on-demand schema loading adds round-trip latency on first use of any tool
• Thundering herd: in a complex task requiring many tools simultaneously, multiple schema fetches compound

Adoption cost: Low for MCP environments (Anthropic ships enable_tool_search as native). Medium for custom tool frameworks requiring a retrieval endpoint.

---

Technique 5: Semantic Checkpointing with Sparse Restore (Crab Pattern)

Idea one-liner: Checkpoint agent state at turn boundaries but only for turns that produce recovery-relevant OS-side effects, skipping the 75%+ of turns that don't.

How it works: Crab (arXiv 2604.28138, April 2026) bridges the "agent-OS semantic gap": agent frameworks see tool calls but not their OS effects; the OS sees state changes but lacks turn-level context to judge relevance. An eBPF-based inspector classifies each turn's OS-visible effects (file writes, process spawns, network calls) to decide checkpoint granularity. A coordinator aligns checkpoints with turn boundaries and overlaps C/R (checkpoint/restore) with LLM wait time.[35][36]

Results: recovery correctness from 8% (chat-only) to 100%, checkpoint traffic reduced by up to 87%, and execution time within 1.9% of fault-free operation. The key finding: over 75% of agent turns produce no recovery-relevant state — so most checkpoints are unnecessary.[35]

Why non-obvious: The standard approach (checkpoint every turn) is both over-complete (captures irrelevant state) and under-complete (misses OS-side effects not visible to the agent framework). Crab's insight is that checkpoint relevance requires semantic understanding of what constitutes "recovery-relevant" state — which the OS cannot determine alone and the agent framework cannot determine alone, but which an eBPF inspector positioned between them can determine.

Failure modes:

• eBPF-based inspection adds infrastructure complexity not appropriate for all environments
• Classification errors (marking a turn as recovery-irrelevant when it wasn't) cause silent recovery failures
• Not applicable to API-only (cloud) agent deployments without OS-level access

Adoption cost: High. Requires host-side runtime infrastructure (eBPF-capable kernel, modified sandbox runtime). Currently in research/early experimental stage.

---

Technique 6: Context Rotation with Two-Stage Monitoring (Production Pattern)

Idea one-liner: Monitor context fill percentage in real-time; trigger memory sync at ~64% fill and session rotation at ~80% fill — never wait for the limit.

How it works: Context rot begins well before the token limit — Chroma's 2025 study documented significant performance degradation at every increment of context growth, with the practical threshold for proactive management at well under 50% of nominal capacity. Two-stage monitoring prevents reactive (lossy) truncation:[21]

• Stage 1 (Early warning, ~64% full): Write current state to external memory while the agent still has sufficient context to do so coherently
• Stage 2 (Rotation threshold, ~80% full): Initiate graceful session rotation — compact current context, write handoff, open new session with compact summary

The gap between stages is critical: it gives the memory sync time to complete before the rotation is required. Triggering both simultaneously creates a race condition.[21]

Why non-obvious: Most teams set the compaction trigger at 90–95% of context capacity — at which point context rot has already degraded performance significantly, and the compaction prompt itself may be operating under degraded conditions. The counterintuitive move is to compact earlier and more aggressively, at a point when the model still has full attention capacity for high-quality summarization.

Failure modes:

• Over-rotation: excessive compaction discards genuinely needed mid-task context
• Cooldown omission: monitoring loops that fire every 30 seconds can trigger multiple concurrent syncs if a cooldown guard is absent
• Degraded compact quality: if the compaction prompt is invoked when context is already degraded (>80%), the resulting summary may be low quality

Adoption cost: Low. Monitoring hooks are available in all major agent frameworks. LangGraph checkpoints at every node execution natively.[32][21]

---

2.3 How to Detect When Deferred Context Is Actually Needed

Detection of a deferred context need ("lazy load trigger") requires three signal types working in concert:[37][4]

1. Keyword / topic triggers: The agent's current reasoning window mentions a topic for which a deferred section exists (e.g., "authentication" triggers load of the auth architecture notes). This is the primary signal and is cheapest to implement.

2. Tool call intent signals: The agent selects or calls a tool that implies a workflow — this triggers loading the detailed procedure for that workflow. Implemented in MCP Tool Search as the primary tier-promotion mechanism.[10][28]

3. Self-reported uncertainty: The agent, when generating a response, produces tokens indicative of uncertainty ("I'm not sure how to handle...", "I should check..."). This can be monitored programmatically as a trigger to inject relevant deferred context before the response is completed. This requires streaming output monitoring.

Graceful degradation for non-expert users: The system should degrade gracefully if a trigger fires but the retrieval fails. The recommended pattern: if deferred context cannot be fetched, the agent proceeds with an explicit uncertainty acknowledgment ("I don't have the detailed procedure for this step — proceeding with best judgment") rather than silently executing with degraded knowledge.[38][37]

---

2.4 Recovering the "Recent Tail" Cheaply

The cheapest reliable method for recovering just the recent tail of prior context is the structured git + progress-file pattern:

1. git log --oneline -10 — recovers the last 10 commits (typically 200–500 tokens), giving an authoritative record of what changes were made and in what order
2. Read claude-progress.txt (or equivalent) — recovers the narrative bridge authored by the previous session (500–2,000 tokens)
3. Read any explicitly referenced files that are named in the progress file — on-demand, targeted, not a bulk directory scan

Total boot cost: typically 800–3,000 tokens vs. 15,000–67,000 tokens for bulk context loading.[3][21]

For non-git environments, the equivalent is a rolling timestamped log with only the last N entries read on boot, combined with a structured JSON state snapshot written at each session close.

---

2.5 Source Map

Source	Type	Coverage
LLMLingua (arXiv:2310.05736, EMNLP 2023)	Peer-reviewed	Token pruning, 20× compression
LLMLingua-2 (Microsoft/Tsinghua, 2024)	Peer-reviewed	Task-agnostic token classification
LongLLMLingua (arXiv:2310.06839)	Peer-reviewed	Long-context query-aware compression
RECOMP (ICLR 2024)	Peer-reviewed	Extractive/abstractive doc compressors
Prompt Compression Survey (arXiv:2410.12388, Oct 2024)	Peer-reviewed	Hard/soft prompt method taxonomy
ICML 2024 Characterization Study	Peer-reviewed	Extractive vs. abstractive vs. token pruning comparison
CDCT Benchmark (arXiv:2512.17920, Dec 2025)	Preprint (under TMLR review)	Constraint compliance vs. semantic accuracy under compression
Generative Context Distillation (arXiv:2411.15927, Nov 2024)	Preprint	Fine-tuning to internalize system prompts
AGENTS.md Empirical Study (arXiv:2601.20404, Jan 2026)	Preprint	Agent context file runtime & token impact
Agent Context Files Study (arXiv:2511.12884, Nov 2025)	Preprint	2,303 file analysis of structure and maintenance
Crab (arXiv:2604.28138, Apr 2026)	Preprint	Semantic checkpoint/restore for agent sandboxes
Anthropic "Effective Context Engineering" (Sep 2025)	Engineering blog	Compaction, note-taking, sub-agents; official Anthropic practice
Anthropic Multi-Session Harness Research	Engineering blog	Progress file pattern, session lifecycle, warm start
Engrams "Cut AI Agent Overhead 90%" (Feb 2026)	Practitioner	JIT loading, semantic compression, delta architecture
Vidanov A/B Test (dev.to/aws-builders, May 2026)	Practitioner	Behavioral cliff, safety rule compression limits, LLM vs. regex
MCP Tool Search (Anthropic, Jan 2026)	Production deployment	85–95% tool schema token reduction
BentoML Prefix Caching Handbook	Technical reference	KV cache mechanics, provider comparison
Zep/Mem0/Letta/Graphiti memory comparison	Product documentation	Agent memory framework capabilities
LangGraph checkpointing + Platform GA	Product documentation	Framework-level checkpoint/restore
Zylos "Context Window Management" (Mar 2026)	Technical analysis	Session lifecycle, tiered memory, context rot
LangChain "Context Engineering for Agents" (Apr 2026)	Blog	Context engineering taxonomy
Selective Context (GitHub:liyucheng09, 2023)	Open source	Self-information token scoring
context-compress CLI (vidanov, 2026)	Open source	LLM + regex compression automation

---

State of the Art as of June 2026

The field has moved rapidly from treating system prompt size as a fixed cost to treating it as an engineered variable. On the compression side, the most important practical advances are: (1) the Anthropic/industry convergence on prefix KV caching as a near-zero-cost 90% savings mechanism for stable content — widely deployed but still widely misconfigured; (2) the MCP Tool Search lazy-loading pattern (Jan 2026, 85–95% tool schema reduction) as a production model for JIT instruction loading that the broader agent ecosystem is now replicating; (3) the CDCT benchmark (Dec 2025), which for the first time cleanly separates semantic accuracy from constraint compliance under compression and identifies the medium-compression "danger zone" that makes half-measures more dangerous than either extreme. The critical behavioral insight — that RLHF helpfulness training is the dominant cause of constraint violations at medium compression, not semantic information loss — is poorly understood in practitioner communities but has major design implications.

On the lazy loading / boot efficiency side, the JetBrains observation masking finding (2025) — that masking older environment observations outperforms LLM summarization in both cost and solve rate — is the most practically significant recent result, directly contradicting the intuition that "smarter summarization is better." The Anthropic multi-session harness work formalized the narrative progress-file pattern as a minimal, high-reliability boot recovery mechanism that sidesteps the need for complex vector retrieval at session start. The Crab checkpoint paper (April 2026) represents the emerging frontier for OS-aware agent state management, showing that 75%+ of checkpoints are unnecessary overhead and that correctness recovery can reach 100% with semantic-aware sparse checkpointing — though this remains in research and is not yet production-ready for most teams.

The dominant unsolved problem is compression verification at scale: while the CDCT benchmark provides methodology, automated constraint compliance testing across all behavioral rules is still a largely manual process for most practitioners. The gap between "this looks the same" (semantic similarity) and "this behaves identically" (constraint compliance) remains a significant operational risk in any agent system that compresses its instructions.

Assumption log: This report assumes a single-LLM agent (not multi-agent hierarchy) with a file-based instruction set, API-access only (no fine-tuning access), and instruction content that is primarily natural language (not executable code). Techniques 7 and 8 in Part 1 have caveats for API-only environments. All claims about percentage improvements are from cited sources and may not generalize across all models, task types, or instruction styles.