Domain 4: Prompt Engineering & Structured Output (20%)

4.1 Explicit criteria to improve precision / reduce false positives

Knowledge:
- Explicit criteria beats vague instructions ("flag comments only when claimed behavior contradicts actual code behavior" vs. "check that comments are accurate")
- Generic instructions like "be conservative" / "only report high-confidence findings" do NOT improve precision vs. specific categorical criteria
- High false-positive categories erode trust even in accurate categories

Skills:
- Write specific criteria for what to report (bugs, security) vs. skip (style, local patterns) — not confidence-based filtering
- Temporarily disable high-FP categories to restore trust while improving those prompts
- Define explicit severity criteria with concrete code examples per level

4.2 Few-shot prompting for consistency

Knowledge:
- Few-shot > detailed instructions alone for consistently formatted, actionable output
- Demonstrates ambiguous-case handling (tool selection, branch-level coverage gaps)
- Enables generalization to novel patterns (not just memorizing pre-specified cases)
- Reduces hallucination in extraction (informal measurements, varied doc structures)

Skills:
- 2-4 targeted examples for ambiguous scenarios, showing reasoning for the chosen action over alternatives
- Examples demonstrating desired output format (location, issue, severity, fix)
- Examples distinguishing acceptable patterns from genuine issues (reduces FP, keeps generalization)
- Examples for varied document structures (inline citations vs. bibliographies)
- Examples for correct extraction from varied formats to fix empty/null required fields

4.3 Structured output via tool_use + JSON schemas

Knowledge:
- tool_use with JSON schema = most reliable path to guaranteed schema-compliant output (eliminates syntax errors)
- tool_choice: "auto" (may return text) / "any" (must call a tool, any) / forced named tool
- Schema compliance ≠ semantic correctness (e.g., line items not summing to total) — tool_use does NOT catch semantic errors
- Schema design: required vs optional, enum + "other"+detail pattern for extensibility

Skills:
- Define extraction tools with JSON schema params; extract from the tool_use response
- tool_choice: "any" when multiple extraction schemas exist and doc type is unknown
- Force a specific tool ({"type":"tool","name":"extract_metadata"}) to run before enrichment steps
- Nullable/optional fields when info may be absent — prevents fabrication to satisfy "required"
- Enum values like "unclear" / "other" + detail field for extensibility
- Include format-normalization rules in the prompt alongside the strict schema

4.4 Validation, retry, feedback loops for extraction quality

Knowledge:
- Retry-with-error-feedback: append specific validation errors to the prompt on retry
- Retry is ineffective when info is simply absent from source (vs. format/structural errors, where retry helps)
- Feedback loop: detected_pattern field tracks which code constructs trigger findings → analyze dismissal patterns

Skills:
- Follow-up requests include: original doc + failed extraction + specific validation errors
- Identify retry-will-fail cases (info in an external doc not provided) vs. retry-will-succeed cases (format mismatch, structural errors)
- Add detected_pattern fields to findings for FP pattern analysis
- Self-correction validation: extract calculated_total alongside stated_total to flag discrepancies; add conflict_detected booleans

4.5 Batch processing strategies

Knowledge:
- Message Batches API: 50% cost savings, up to 24h window, no guaranteed latency SLA
- Good for: non-blocking, latency-tolerant (overnight reports, weekly audits, nightly test gen)
- Bad for: blocking workflows (pre-merge checks) — Sample Q11's key distinction
- Batch API does not support multi-turn tool calling within a single request
- custom_id correlates request/response pairs

Skills:
- Match API to latency need: sync API for blocking pre-merge checks, batch for overnight/weekly
- Calculate submission frequency from SLA constraints (e.g., 4h windows to guarantee a 30h SLA against 24h batch processing)
- Handle failures: resubmit only failed docs (by custom_id), e.g., chunk docs that exceeded context limits
- Refine prompts on a sample set before batch-processing large volumes (maximize first-pass success, reduce resubmission cost)

4.6 Multi-instance and multi-pass review architectures

Knowledge:
- Self-review limitation: model retains generation reasoning context → less likely to question its own decisions in the same session
- Independent review instances (no prior reasoning context) catch subtle issues better than self-review instructions or extended thinking
- Multi-pass review: per-file local analysis + separate cross-file integration pass → avoids attention dilution and contradictions

Skills:
- Second independent Claude instance reviews generated code without the generator's context
- Split large multi-file reviews into per-file passes + integration pass — Sample Q12's answer
- Verification passes with self-reported confidence per finding to enable calibrated routing

Discussion Prompts / Things to Drill