Why I went down this route
Most AI systems I’ve built followed a familiar pattern: Python handles everything. Retrieval, embeddings, tool calls, orchestration, generation — all sitting in one place. It works well enough, especially early on, because the entire system is easy to reason about.
That starts to break the moment you introduce tools in a serious way.
Once you move into MCPMCPModel Context Protocol. A standard way for a model to discover and call external tools and data sources.-style systems — where the model decides to call tools, chain them, or depend on multiple external sources — the architecture stops being linear. You’re no longer passing input → model → output. You’re coordinating execution across multiple components with different latencies, failure modes, and response structures.
At that point, the problem shifts. It’s no longer “how good is the model?” but “how well does the system execute what the model wants to do?”
That’s what led me to experiment with Go — not to replace Python, but to take over the part of the system that behaves like a distributed workflow engine.
What I actually built
I kept the model layer in Python and moved orchestration into Go. The idea was to let each layer specialise: Python for anything model-related, Go for execution control.
A simplified version of the flow looked like this:
Drawing the diagram…
The interesting part here is that the system is no longer sequential. The orchestrator is effectively building a small execution graph per request.
A basic handler in Go looked something like this:
func HandleQuery(query string) string {
plan := PlanTools(query)
results := ExecutePlan(plan, query)
context := Aggregate(results)
return CallLLM(context, query)
}Even this simple structure made a difference because it forced a separation between decision-making (what tools to use) and execution (how they run).
Concurrency stopped being something I had to think about
The first noticeable shift was how natural concurrency felt.
In Python, parallel tool execution is possible, but it usually requires structuring everything around async — which becomes fragile when you mix I/O, retries, and conditional logic.
In Go, the same idea translates into something much closer to how you’d describe it conceptually.
type ToolResult struct {
Name string
Output string
Err error
}
func ExecutePlan(tools []string, query string) []ToolResult {
resultsCh := make(chan ToolResult, len(tools))
for _, tool := range tools {
go func(t string) {
output, err := ExecuteToolWithTimeout(t, query, 2*time.Second)
resultsCh <- ToolResult{Name: t, Output: output, Err: err}
}(tool)
}
var results []ToolResult
for i := 0; i < len(tools); i++ {
results = append(results, <-resultsCh)
}
return results
}This isn’t just syntactic convenience. It changes how you design the system. You stop thinking in terms of “which step comes next” and start thinking in terms of “which steps can happen independently.”
That alone reduced latency in my pipeline, because I wasn’t accidentally serialising things that didn’t need to be.
Timeouts and failure handling became first-class
Once multiple tools are running concurrently, failure handling becomes unavoidable. One slow or failing tool can block the entire system if you’re not careful.
Go made it straightforward to enforce boundaries.
func ExecuteToolWithTimeout(tool string, query string, timeout time.Duration) (string, error) {
ch := make(chan string, 1)
go func() {
ch <- ExecuteTool(tool, query)
}()
select {
case res := <-ch:
return res, nil
case <-time.After(timeout):
return "", fmt.Errorf("%s timed out", tool)
}
}What this enabled was not just handling failures, but designing for them. Instead of assuming all tools succeed, the system expects partial results and continues anyway.
That required changing how aggregation worked.
Aggregation became more important than retrieval
Once results come back from multiple tools — some successful, some failed, some partial — you need a consistent way to represent them before passing them to the model.
I ended up standardising tool outputs into a structured format:
type NormalizedResult struct {
Source string
Content string
Status string
}And then aggregating them into a single context:
func Aggregate(results []ToolResult) string {
var builder strings.Builder
for _, r := range results {
status := "ok"
if r.Err != nil {
status = "error"
}
builder.WriteString(fmt.Sprintf(
"[%s][%s]: %s\n",
r.Name,
status,
r.Output,
))
}
return builder.String()
}This forced every tool to behave predictably. Instead of returning arbitrary formats, everything was normalised before reaching the model.
Interestingly, this improved answer quality more than expected. The model performed better when the context was structured and consistent, even if some parts were missing.
The Python boundary became cleaner
Rather than mixing orchestration and model logic, the Go service simply called a Python endpoint for generation.
func CallLLM(context string, query string) string {
payload := map[string]string{
"context": context,
"query": query,
}
body, _ := json.Marshal(payload)
resp, err := http.Post("http://llm-service/generate", "application/json", bytes.NewBuffer(body))
if err != nil {
return "LLM service unavailable"
}
defer resp.Body.Close()
responseBody, _ := io.ReadAll(resp.Body)
return string(responseBody)
}On the Python side, things stayed simple:
from fastapi import FastAPI
app = FastAPI()
@app.post("/generate")
def generate(payload: dict):
context = payload["context"]
query = payload["query"]
prompt = f"""
Context:
{context}
Question:
{query}
"""
return llm.generate(prompt)This separation ended up being cleaner than expected. Each layer had a clear responsibility, and neither needed to know about the internal complexity of the other.
What didn’t go as smoothly
Go is not designed for ML tasks, and that boundary shows quickly.
Anything involving embeddings, tokenisation, or model interaction was still significantly easier in Python. Trying to replicate that in Go added friction without real benefit.
There’s also unavoidable overhead in splitting the system:
- More services to deploy
- Network latency between components
- More points of failure
Debugging becomes more distributed as well. When something goes wrong, you need visibility across both layers.
And while Go simplifies concurrency, it doesn’t remove the need for discipline. Poorly structured goroutines can still lead to subtle bugs if not managed carefully.
What this changed for me
The biggest shift wasn’t performance — it was how I thought about AI systems.
Before this, tool usage felt like an extension of prompting.
After this, it felt like building a scheduler.
The model decides what should happen. The system decides how it actually happens. That separation made the system easier to reason about, especially as complexity increased.
What the future looks like
As AI systems move toward tool use, agents, and multi-step execution, orchestration becomes the dominant challenge.
Not model quality.
Not embeddings.
Execution.
Systems need to handle:
- Multiple concurrent operations
- Partial failures
- Dynamic execution paths
- Real-time constraints
That starts to look less like ML and more like distributed systems engineering.
Go fits naturally into that space. Not because it’s an “AI language,” but because it’s good at concurrency, coordination, and reliability.
The takeaway
This experiment didn’t replace Python. It clarified where Python should stop.
If you’re building simple pipelines, keep everything in one place.
But if your system involves multiple tools, dynamic execution, and real-world constraints, you’re no longer just building an AI feature.
You’re building a system that happens to use AI.
And in that system, Go starts to make a lot more sense.