The failure mode nobody handles well
In the previous experiments, I focused on what happens when tools fail — timeouts, degraded APIs, missing data. Circuit breakers and fallback chains handled that well.
But there's a different kind of failure that's harder to catch: the model returns something, it looks like a valid response, and it's wrong.
Not hallucination in the dramatic sense. More like:
- The response ignores half the context you passed in
- It answers a different question than what was asked
- It produces valid JSON with nonsensical values
- It gives a confident answer when the context clearly doesn't support one
These are structural failures. The output has the right shape but the wrong substance. And they're the hardest to catch because, from the outside, everything looks fine — you got a 200, the response parsed, the format matched.
I wanted the system to catch these itself.
The idea
Instead of treating generation as a single call, I split it into three phases:
- Generate — produce the initial response
- Evaluate — ask a second pass to assess the output against explicit criteria
- Heal — if evaluation fails, retry with a modified prompt that includes the diagnosis
The important constraint: the evaluator isn't a separate model. It's the same model, but with a different prompt — one focused entirely on critique, not generation.
The flow looks like this:
Drawing the diagram…
The evaluation schema
The first thing I needed was a structured way to express what "good" looks like. Vague quality checks don't work — the evaluator needs specific, verifiable criteria.
I defined evaluations as a schema:
from dataclasses import dataclass
@dataclass
class EvalCriterion:
name: str
description: str
weight: float # 0.0 to 1.0
@dataclass
class EvalResult:
criterion: str
passed: bool
reasoning: str
confidence: float
@dataclass
class EvalReport:
results: list[EvalResult]
overall_pass: bool
score: float
failures: list[str]And a default set of criteria that apply to most tool-augmented responses:
DEFAULT_CRITERIA = [
EvalCriterion(
name="context_grounding",
description="The response uses information from the provided context, not external knowledge. Claims are traceable to specific context entries.",
weight=0.3,
),
EvalCriterion(
name="query_alignment",
description="The response directly addresses the user's question. It does not answer a related but different question.",
weight=0.3,
),
EvalCriterion(
name="completeness",
description="The response covers the key aspects of the query. It does not ignore significant parts of the question or context.",
weight=0.2,
),
EvalCriterion(
name="confidence_calibration",
description="The response's confidence matches the evidence. It does not state uncertain things with certainty, or hedge on well-supported claims.",
weight=0.2,
),
]These aren't generic quality rubrics. They're specific to the failure modes I kept seeing in tool-augmented systems — where the model has rich context but sometimes ignores it, or answers confidently when the context is thin.
The evaluator
The evaluator takes the original query, the context, and the generated response, then scores it against the criteria.
import json
def build_eval_prompt(
query: str,
context: str,
response: str,
criteria: list[EvalCriterion],
) -> str:
criteria_block = "\n".join(
f"- **{c.name}** (weight: {c.weight}): {c.description}"
for c in criteria
)
return f"""You are a strict evaluator. Your job is to assess whether a response meets specific quality criteria.
You are not generating a response. You are auditing one.
## Original Query
{query}
## Context Provided to the Model
{context}
## Response to Evaluate
{response}
## Criteria
{criteria_block}
For each criterion, return a JSON object with:
- "criterion": the criterion name
- "passed": true or false
- "reasoning": one sentence explaining your judgment
- "confidence": 0.0 to 1.0
Return a JSON array of these objects. No preamble, no markdown fences. Just the array."""Parsing the result:
def evaluate_response(
query: str,
context: str,
response: str,
criteria: list[EvalCriterion] = DEFAULT_CRITERIA,
) -> EvalReport:
prompt = build_eval_prompt(query, context, response, criteria)
raw = call_llm(prompt, temperature=0.1)
try:
results_data = json.loads(raw)
except json.JSONDecodeError:
return EvalReport(
results=[],
overall_pass=True,
score=1.0,
failures=["eval_parse_failure"],
)
results = [
EvalResult(
criterion=r["criterion"],
passed=r["passed"],
reasoning=r["reasoning"],
confidence=r.get("confidence", 0.5),
)
for r in results_data
]
weights = {c.name: c.weight for c in criteria}
score = sum(
weights.get(r.criterion, 0.0) * (1.0 if r.passed else 0.0)
for r in results
)
failures = [r.criterion for r in results if not r.passed]
return EvalReport(
results=results,
overall_pass=len(failures) == 0,
score=score,
failures=failures,
)Two deliberate choices here. First, temperature=0.1 on the eval call — the evaluator should be as deterministic as possible. Second, if the eval output fails to parse, the system treats it as a pass. The alternative — blocking a valid response because the evaluator broke — is worse.
The healing loop
When evaluation fails, the system doesn't just retry blindly. It feeds the failure diagnosis back into the prompt.
def build_healing_prompt(
query: str,
context: str,
previous_response: str,
eval_report: EvalReport,
) -> str:
failure_descriptions = []
for r in eval_report.results:
if not r.passed:
failure_descriptions.append(
f"- **{r.criterion}**: {r.reasoning}"
)
failures_block = "\n".join(failure_descriptions)
return f"""You are regenerating a response that failed quality checks.
## Original Query
{query}
## Context
{context}
## Previous Response (failed evaluation)
{previous_response}
## Why It Failed
{failures_block}
Generate an improved response that specifically addresses the identified failures. Do not repeat the same mistakes.
Focus on:
1. Using the provided context, not external knowledge
2. Answering the exact question asked
3. Matching your confidence to the available evidence"""This is the key mechanism. The model gets told what went wrong and why. It's not a generic "try again" — it's a targeted correction with the specific failure modes laid out.
The full pipeline
Putting it all together:
@dataclass
class PipelineResult:
response: str
attempts: int
final_score: float
healed: bool
eval_history: list[EvalReport]
def generate_with_healing(
query: str,
context: str,
criteria: list[EvalCriterion] = DEFAULT_CRITERIA,
max_retries: int = 2,
score_threshold: float = 0.7,
) -> PipelineResult:
response = call_llm(
build_generation_prompt(query, context),
temperature=0.7,
)
eval_history = []
for attempt in range(max_retries + 1):
report = evaluate_response(query, context, response, criteria)
eval_history.append(report)
if report.overall_pass or report.score >= score_threshold:
return PipelineResult(
response=response,
attempts=attempt + 1,
final_score=report.score,
healed=attempt > 0,
eval_history=eval_history,
)
if attempt < max_retries:
response = call_llm(
build_healing_prompt(query, context, response, report),
temperature=0.5,
)
# max retries exhausted — return best attempt
best_idx = max(range(len(eval_history)), key=lambda i: eval_history[i].score)
return PipelineResult(
response=response if best_idx == len(eval_history) - 1 else "[best attempt returned]",
attempts=max_retries + 1,
final_score=eval_history[best_idx].score,
healed=True,
eval_history=eval_history,
)A subtle detail: the generation temperature drops on retries (0.7 → 0.5). The initial attempt gets some creative freedom. The healing attempt should be more conservative — it has a specific target to hit.
And score_threshold=0.7 means the system accepts imperfect responses. This is intentional. A response that grounds 70% of its claims in context is usable. Demanding perfection means more retries, more latency, and often no improvement.
The generation prompt
For completeness — the initial generation prompt that starts the pipeline:
def build_generation_prompt(query: str, context: str) -> str:
return f"""Answer the following question using only the provided context.
If the context doesn't contain enough information, say so explicitly rather than guessing.
## Context
{context}
## Question
{query}"""Nothing clever here. The constraint ("using only the provided context") is what makes the context_grounding evaluation criterion meaningful. Without it, the evaluator has nothing to check against.
What I observed
Most responses pass on the first attempt. Around 75-80% of responses cleared all criteria without healing. The system adds no overhead for those — the eval call is the only cost.
Healing works best for context grounding failures. When the model ignores context and relies on parametric knowledge, the healing prompt reliably pulls it back. The failure description makes the issue concrete enough for the model to correct.
Query alignment failures are harder to heal. When the model answers the wrong question, it sometimes does it again — just more carefully. The failure description helps, but the model occasionally doubles down on its interpretation of the query rather than reconsidering it.
Two retries is the sweet spot. One retry catches most fixable issues. The second catches edge cases. Beyond that, if the model hasn't figured it out, it won't. I never saw meaningful improvement past attempt three.
Eval parsing is more robust than I expected. With temperature=0.1 and a clear schema, the evaluator returned valid JSON in roughly 95% of calls. The 5% failure rate is handled by the pass-on-parse-failure fallback.
Where this gets dangerous
Self-evaluation has a fundamental limitation: the model is grading its own work.
If the model has a systematic blind spot — say it consistently misinterprets a certain type of query — the evaluator will have the same blind spot. The eval might pass a response that a human would flag, because the evaluator shares the same biases as the generator.
This is not hypothetical. I saw cases where the model confidently answered a question, the evaluator agreed it was well-grounded, and the answer was subtly wrong because both passes relied on the same flawed reasoning.
Mitigations I considered:
# Use a different model for evaluation
def evaluate_response_cross_model(query, context, response, criteria):
prompt = build_eval_prompt(query, context, response, criteria)
return call_llm(prompt, model="claude-3-5-haiku-20241022", temperature=0.1)
# Use multiple eval runs and take majority vote
def evaluate_with_consensus(query, context, response, criteria, n=3):
reports = [
evaluate_response(query, context, response, criteria)
for _ in range(n)
]
avg_score = sum(r.score for r in reports) / n
all_failures = set()
for r in reports:
all_failures.update(r.failures)
majority_failures = [
f for f in all_failures
if sum(1 for r in reports if f in r.failures) > n // 2
]
return EvalReport(
results=reports[0].results,
overall_pass=len(majority_failures) == 0,
score=avg_score,
failures=majority_failures,
)Cross-model evaluation helps but adds latency and cost. Consensus voting is more robust but multiplies eval calls. Neither fully solves the fundamental problem — an LLMLLMLarge language model, the kind of model behind ChatGPT and Claude. evaluating LLM output is inherently circular.
The honest answer: self-healing prompts catch structural failures well. They don't catch reasoning failures reliably. Know which failure mode you're targeting.
Cost and latency implications
This isn't free. Every evaluation is an additional LLM call. Every retry is two more (generation + evaluation). In the worst case — two retries — you're making five LLM calls for a single query.
# Rough cost model
# Best case (pass on first try): 2 calls (generate + eval)
# One retry: 4 calls (gen + eval + heal + eval)
# Two retries: 6 calls (gen + eval + heal + eval + heal + eval)
# Weighted average (~80% pass rate, ~15% one retry, ~5% two retries):
# 0.80 * 2 + 0.15 * 4 + 0.05 * 6 = 2.5 calls per queryAt 2.5x the baseline cost on average, this only makes sense for high-value queries — cases where a wrong answer is significantly more expensive than an extra LLM call. For low-stakes completions, it's overkill.
I ended up making the healing pipeline opt-in per query type:
def route_query(query: str, context: str) -> PipelineResult:
classification = classify_query(query)
if classification in ("factual", "high_stakes", "data_grounded"):
return generate_with_healing(query, context)
return PipelineResult(
response=call_llm(build_generation_prompt(query, context)),
attempts=1,
final_score=1.0,
healed=False,
eval_history=[],
)Not every query needs self-healing. The ones that do are exactly the ones where you'd notice a bad answer.
How this connects to the Go layer
In the broader system, the Python service handles all of this internally. The Go orchestrator doesn't know or care that self-healing is happening — it sends context and a query, and gets a response back.
But there's useful metadata to surface. I extended the Python response to include healing information:
@app.post("/generate")
def generate(payload: dict):
context = payload["context"]
query = payload["query"]
result = generate_with_healing(query, context)
return {
"response": result.response,
"metadata": {
"attempts": result.attempts,
"healed": result.healed,
"final_score": result.final_score,
},
}The Go layer can log this, surface it in dashboards, or use it for downstream decisions. A response that required two healing attempts might warrant different caching behaviour than one that passed immediately.
The takeaway
LLM pipelines fail silently. The output looks valid, parses correctly, and is confidently wrong.
Self-healing prompts don't solve this completely — they can't catch reasoning failures that the model itself is blind to. But they catch structural failures reliably: ignored context, misaligned answers, miscalibrated confidence.
The pattern is simple: generate, evaluate against explicit criteria, and retry with the diagnosis. The hard part is knowing when the evaluator shares the same blind spot as the generator.
For tool-augmented systems where context grounding matters, this is worth the cost. For everything else, a single well-crafted prompt is still the better investment.