Domain 5: Context Management & Reliability (15%)

5.1 Preserving critical info across long interactions

Knowledge:
- Progressive summarization risk: numerical values, %, dates, customer-stated expectations get condensed into vague summaries and lost
- "Lost in the middle": models process start/end of long inputs reliably but may drop mid-section findings
- Tool results accumulate in context disproportionately to relevance (40+ fields from an order lookup when only 5 matter)
- Full conversation history must be passed on subsequent API requests for coherence

Skills:
- Extract transactional facts (amounts, dates, order #s, statuses) into a persistent "case facts" block included in every prompt, outside summarized history
- Extract/persist structured issue data into a separate context layer for multi-issue sessions
- Trim verbose tool outputs to only relevant fields before they accumulate
- Place key-findings summaries at the start of aggregated inputs; use explicit section headers (mitigates lost-in-the-middle)
- Require subagents to include metadata (dates, source locations, methodology) in structured outputs
- Upstream agents return structured data (facts, citations, relevance scores) instead of verbose reasoning chains when downstream context budget is limited

5.2 Escalation and ambiguity resolution patterns

Knowledge:
- Valid escalation triggers: explicit customer request for a human, policy exceptions/gaps (not just "complex"), inability to make progress
- Escalate immediately on explicit demand; offer-to-resolve first when straightforward and customer hasn't insisted
- Sentiment-based escalation and self-reported LLM confidence are unreliable proxies for real complexity
- Multiple customer matches → ask for more identifiers, don't guess heuristically

Skills:
- Explicit escalation criteria + few-shot examples in the system prompt — Sample Q3's answer (not confidence scores, not a separate classifier, not sentiment)
- Honor explicit "get me a human" requests immediately, no investigation first
- Acknowledge frustration + offer resolution when within capability; escalate only if customer reiterates
- Escalate on ambiguous/silent policy (e.g., competitor price matching when policy only covers own-site adjustments)
- Ask for additional identifiers on multi-match results instead of heuristic selection

5.3 Error propagation across multi-agent systems

Knowledge:
- Structured error context (failure type, attempted query, partial results, alternatives) enables intelligent coordinator recovery — Sample Q8's answer
- Access failures (timeout, needs retry decision) ≠ valid empty results (successful query, no matches)
- Generic error statuses hide valuable context
- Anti-patterns: silently suppressing errors (empty result marked "success"), or terminating the whole workflow on one failure

Skills:
- Return structured error context: failure type + what was attempted + partial results + alternatives
- Distinguish access failures from valid empty results in reporting
- Local recovery in subagents for transient failures; only propagate unrecoverable errors (with partial results)
- Coverage annotations in synthesis output: which findings are well-supported vs. which have gaps due to unavailable sources

5.4 Managing context in large codebase exploration

Knowledge:
- Context degradation in extended sessions: inconsistent answers, references to "typical patterns" instead of specific classes discovered earlier
- Scratchpad files persist key findings across context boundaries
- Subagent delegation isolates verbose exploration output; main agent keeps high-level coordination
- Structured state persistence for crash recovery: each agent exports state to a known location; coordinator loads a manifest on resume

Skills:
- Spawn subagents for specific questions ("find all test files," "trace refund flow deps") while main agent stays high-level
- Maintain scratchpad files of key findings, referenced for later questions (counters degradation)
- Summarize findings from one exploration phase before spawning subagents for the next; inject summaries into initial context
- Crash recovery via structured agent-state manifests the coordinator loads and re-injects
- /compact to reduce context usage mid-session when verbose discovery fills the window

5.5 Human review workflows and confidence calibration

Knowledge:
- Aggregate accuracy (e.g., 97% overall) can mask poor performance on specific doc types/fields
- Stratified random sampling measures error rates in high-confidence extractions, detects novel error patterns
- Field-level confidence scores calibrated against labeled validation sets route review attention
- Must validate accuracy by document type/field segment before automating away human review

Skills:
- Stratified random sampling of high-confidence extractions for ongoing error-rate measurement
- Analyze accuracy by document type + field before reducing human review
- Field-level confidence scores + threshold calibration using labeled validation sets
- Route low-confidence or ambiguous/contradictory-source extractions to human review, prioritizing limited reviewer capacity

5.6 Provenance and uncertainty in multi-source synthesis

Knowledge:
- Source attribution is lost when summarization compresses findings without preserving claim-source mappings
- Structured claim-source mappings must be preserved/merged through synthesis
- Conflicting stats from credible sources: annotate the conflict with source attribution, don't arbitrarily pick one
- Temporal data: require publication/collection dates so date differences aren't misread as contradictions

Skills:
- Subagents output structured claim-source mappings (URLs, doc names, excerpts) that downstream agents preserve
- Structure reports with explicit sections: well-established vs. contested findings, preserving original characterizations/methodology
- Complete analysis with conflicting values explicitly annotated; let the coordinator decide reconciliation before synthesis
- Require publication/collection dates in structured subagent outputs
- Render content types appropriately in synthesis (financial → tables, news → prose, technical → structured lists) — not one uniform format

Discussion Prompts / Things to Drill