The problem that prompted this
In my previous experiment, I moved tool orchestration into Go and got parallel execution, timeouts, and structured aggregation working. The system handled partial failures — if one tool timed out, the rest still contributed to the response.
But "handled" is generous. What actually happened was: a tool failed, the system logged the error, and the model received a [error] tag in its context. Every failure was treated identically.
That's fine when failures are rare. It breaks when they're not.
The moment an external API starts degrading — not fully down, just slow or flaky — the system keeps calling it. Every request pays the latency cost. Timeouts fire repeatedly. The model keeps getting [error] tags for the same source, polluting the context with noise.
Worse: some tools have natural fallbacks. If a semantic search endpoint is down, a keyword search over the same corpus might return something useful. But without a fallback mechanism, the system doesn't know that. It just reports the failure and moves on.
This experiment was about making the system smarter about failure — not at the model level, but at the infrastructure level.
What I borrowed
Two patterns from microservices resilience stood out as directly applicable:
Circuit breakers — stop calling a service that's consistently failing, and periodically probe it to check if it's recovered. This prevents cascading latency when a dependency degrades.
Fallback chains — when a primary tool fails, automatically try an alternative that serves a similar purpose, potentially at lower quality but higher reliability.
Together, they turn "this tool failed" into "this tool is unreliable right now, here's the next best option."
The flow looks like this:
Drawing the diagram…
The circuit breaker
The implementation is intentionally simple. No external libraries — just a struct that tracks failures and decides whether a tool should be called.
type CircuitBreaker struct {
mu sync.Mutex
failureCount int
lastFailure time.Time
state string // "closed", "open", "half-open"
threshold int
resetTimeout time.Duration
}
func NewCircuitBreaker(threshold int, resetTimeout time.Duration) *CircuitBreaker {
return &CircuitBreaker{
state: "closed",
threshold: threshold,
resetTimeout: resetTimeout,
}
}The state transitions are straightforward:
func (cb *CircuitBreaker) Allow() bool {
cb.mu.Lock()
defer cb.mu.Unlock()
switch cb.state {
case "closed":
return true
case "open":
if time.Since(cb.lastFailure) > cb.resetTimeout {
cb.state = "half-open"
return true
}
return false
case "half-open":
return true
}
return false
}
func (cb *CircuitBreaker) RecordSuccess() {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.failureCount = 0
cb.state = "closed"
}
func (cb *CircuitBreaker) RecordFailure() {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.failureCount++
cb.lastFailure = time.Now()
if cb.failureCount >= cb.threshold {
cb.state = "open"
}
}Three failures within the window trips the breaker. After the reset timeout, the system sends a single probe request. If it succeeds, the circuit closes. If not, it stays open.
Nothing clever here. That's the point — the pattern works because it's simple enough to trust.
Fallback chains
Fallback chains define what to try when a primary tool is unavailable. Each tool can have zero or more fallbacks, ordered by preference.
type ToolChain struct {
Primary string
Fallbacks []string
}
var toolChains = map[string]ToolChain{
"semantic_search": {
Primary: "semantic_search",
Fallbacks: []string{"keyword_search", "cached_search"},
},
"live_api": {
Primary: "live_api",
Fallbacks: []string{"cached_api", "static_snapshot"},
},
"db_query": {
Primary: "db_query",
Fallbacks: []string{},
},
}The key decision: fallbacks are defined at the infrastructure level, not by the model. The model asks for semantic_search. The system decides whether to actually call it, or silently substitute a fallback.
This is deliberate. The model shouldn't need to know about infrastructure health. It should describe intent, and the system should figure out the best way to fulfil it.
Wiring it together
The executor now checks the circuit breaker before each call, and walks the fallback chain if needed.
var breakers = map[string]*CircuitBreaker{}
func init() {
tools := []string{
"semantic_search", "keyword_search", "cached_search",
"live_api", "cached_api", "static_snapshot",
"db_query",
}
for _, t := range tools {
breakers[t] = NewCircuitBreaker(3, 30*time.Second)
}
}
func ExecuteWithResilience(toolName string, query string) ToolResult {
chain, exists := toolChains[toolName]
if !exists {
chain = ToolChain{Primary: toolName}
}
candidates := append([]string{chain.Primary}, chain.Fallbacks...)
for _, candidate := range candidates {
cb := breakers[candidate]
if !cb.Allow() {
continue
}
output, err := ExecuteToolWithTimeout(candidate, query, 2*time.Second)
if err != nil {
cb.RecordFailure()
continue
}
cb.RecordSuccess()
return ToolResult{
Name: candidate,
Output: output,
Err: nil,
Original: toolName,
}
}
return ToolResult{
Name: toolName,
Output: "",
Err: fmt.Errorf("all candidates exhausted for %s", toolName),
}
}I added an Original field to ToolResult so the aggregation layer knows what was requested vs. what actually ran. This matters for the model — seeing [keyword_search][ok][fallback_for:semantic_search] in context tells it the result might be lower fidelity.
Updating the aggregation layer
The aggregation now surfaces fallback information so the model can adjust its behaviour.
type ToolResult struct {
Name string
Output string
Err error
Original string
}
func Aggregate(results []ToolResult) string {
var builder strings.Builder
for _, r := range results {
status := "ok"
if r.Err != nil {
status = "error"
}
source := r.Name
if r.Original != "" && r.Original != r.Name {
source = fmt.Sprintf("%s (fallback for %s)", r.Name, r.Original)
}
builder.WriteString(fmt.Sprintf(
"[%s][%s]: %s\n",
source,
status,
r.Output,
))
}
return builder.String()
}The model now sees context like:
[semantic_search][ok]: Results about distributed caching patterns...
[cached_api (fallback for live_api)][ok]: Last known pricing data from 2h ago...
[db_query][ok]: 47 matching records found...This is more useful than a raw error tag. The model can reason about freshness and reliability, because the infrastructure tells it what actually happened.
What I observed
Latency stabilised under degraded conditions. Before circuit breakers, a flaky API meant every request paid the timeout penalty. After, the system skipped it entirely once the breaker tripped. The difference was noticeable — p95 latency dropped significantly when one of three tools was degrading.
Answer quality held up better than expected. Fallback results are, by definition, worse than primary results. But a keyword search result is still better than no result. The model consistently produced more useful answers with a fallback result than with an error tag.
The model adapted to fallback annotations. When the context included (fallback for ...), the model naturally hedged its responses — less definitive phrasing, more caveats. I didn't prompt for this. It just responded to the signal in the context.
Probe requests caught recoveries faster than I expected. The half-open state means the system detects recovery within one reset window (30 seconds in my config). Without circuit breakers, you'd either keep hammering the failed service or need manual intervention.
What didn't work perfectly
Tuning thresholds is fiddly. Three failures before tripping? Five? It depends on the tool, the expected error rate, and how expensive a false trip is. I ended up wanting per-tool configuration, which added complexity.
Fallback quality varies wildly. A cached search result from two hours ago might be fine for general knowledge but useless for time-sensitive queries. The system doesn't understand this distinction — it just knows the fallback succeeded. Smarter fallback selection (based on query type, not just tool availability) would be a natural next step.
Observability becomes critical. With circuit breakers silently rerouting traffic, you need dashboards showing breaker states, fallback rates, and recovery times. Without visibility, the system looks healthy while quietly degrading.
Cold start after outages is abrupt. When a circuit closes after an outage, all pending requests suddenly hit the recovered service. A gradual ramp-up (canary-style) would be more production-friendly, but I haven't implemented that yet.
What this reinforced
Resilience in AI systems isn't about preventing failures. It's about deciding what happens when they occur.
The model can't do this. It doesn't know that an API is flaky, that a cache is stale, or that a search index is lagging. That's infrastructure knowledge.
What the model can do is reason about the quality of what it receives — if you tell it. Structured context with fallback annotations gives the model enough information to adjust its confidence without needing to understand the underlying system.
This split keeps getting clearer with each experiment: the model handles reasoning, the system handles reality.
What I'd try next
The natural evolution is adaptive fallback selection — choosing fallbacks not just by availability, but by query characteristics. A time-sensitive query shouldn't fall back to a stale cache. A broad exploratory query probably can.
That starts to look like a small routing layer between the planner and the executor — another piece of infrastructure that makes the model's life easier without the model needing to know it exists.
The takeaway
Circuit breakers and fallback chains aren't new ideas. They've been standard in microservices for years.
What's different is applying them to AI tool-calling systems, where the consumer of the result isn't a user or another service — it's a language model that can actually reason about the quality of what it receives.
That creates an interesting feedback loop: better infrastructure annotations lead to better model reasoning, which leads to better answers, even when the system is partially broken.
The system doesn't need to be perfect. It needs to be honest about its imperfections.