Building RLGPT: What I Learned Shipping an LLM to 9,000 People

The demo worked on day one

The prototype was convincing. We had a basic Retrieval-Augmented Generation (RAGRAGRetrieval-augmented generation. The model answers from documents fetched at query time, not only from its training.) pipeline, a clean chat interface, and a model that could answer questions about internal processes in plain English. Leadership loved it, and the green light came quickly.

At that stage, the system architecture was intentionally simple. A user query would be embedded, matched against a vector database, and the top results passed into a prompt for generation. It was a linear pipeline with very few moving parts:

This worked because the constraints were minimal. There was no real concurrency, no strict latency expectations, and no meaningful consequences for being slightly wrong. The demo succeeded because it optimised for clarity, not robustness.

That distinction becomes critical later.

What "production" actually means for an LLM system

Moving from a demo to a production system exposed a fundamental shift in requirements. At 9,000 users distributed across time zones, the system needed to behave predictably under load, maintain low latency, and—most importantly—preserve user trust.

Latency, in particular, stopped being an abstract metric and became a user experience constraint. In a typical pipeline, total latency is the sum of multiple components: embedding generation, retrieval, prompt construction, model inference, and network overhead.

Total Latency = Embed + Retrieval + Prompt + Generation + Network

Individually, each component might seem acceptable. In aggregate, they become problematic. A 150ms embedding call, a 200ms vector search, and a 2.5-second generation step quickly accumulate into a three-second response time. That is tolerable in a demo setting but feels broken in a real workflow.

The solution was not a single optimisation but a series of systemic improvements. We introduced asynchronous execution patterns so that parts of the pipeline could run in parallel, and implemented streaming responses so users received tokens as they were generated rather than waiting for the full output.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
 
app = FastAPI()
 
async def stream_llm_response(prompt):
    async for token in llm.stream(prompt):
        yield token
 
@app.post("/chat")
async def chat(query: str):
    prompt = build_prompt(query)
    return StreamingResponse(stream_llm_response(prompt))

Caching also became essential. Embeddings for repeated or semantically similar queries were stored in Redis, reducing redundant computation and shaving meaningful time off the critical path.

def get_embedding(text: str):
    if cache.exists(text):
        return cache.get(text)
    
    embedding = model.embed(text)
    cache.set(text, embedding)
    return embedding

Another structural optimisation was model tiering. Instead of sending every query to a large, expensive model, we introduced smaller models for routing and classification tasks. This reduced both latency and cost while preserving quality where it mattered.

Hallucination is a trust problem, not just an accuracy problem

In production, hallucinations are not just incorrect outputs—they are system-level failures that erode trust. A single confidently incorrect answer to a policy-related query can permanently damage user confidence.

We approached this by reframing generation as a constrained process rather than a creative one. The system was designed to answer only when sufficient, high-confidence context was available. If retrieval failed to meet a defined threshold, the model would abstain.

def generate_answer(query, docs):
    if not docs or confidence_score(docs) < 0.75:
        return "I don't have enough reliable information to answer that."
    
    prompt = build_prompt(query, docs)
    return llm.generate(prompt)

Confidence scoring itself was not trivial. It combined embedding similarity, metadata alignment, and document weighting to produce a more nuanced signal than raw cosine similarity.

def confidence_score(docs):
    scores = [doc.similarity * doc.metadata_weight for doc in docs]
    return sum(scores) / len(scores)

Prompt design reinforced this constraint. The model was explicitly instructed to operate only within the provided context and to admit uncertainty when necessary. This shifted the system from being generative-first to retrieval-first, which proved critical for maintaining trust.

Scale is a Kubernetes problem

At scale, infrastructure decisions become first-order concerns. Serverless architectures, while attractive for rapid prototyping, introduced unacceptable cold-start latency for LLMLLMLarge language model, the kind of model behind ChatGPT and Claude. workloads. Model inference requires warm instances, and even small delays compound under concurrent load.

We migrated to Kubernetes and implemented horizontal pod autoscaling. Crucially, scaling decisions were tied to request queue depth rather than CPU utilisation.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: queue_length
      target:
        type: AverageValue
        averageValue: 30

This distinction matters because CPU usage does not reflect user experience. A system can appear underutilised while requests are queuing and users are waiting. Queue length, on the other hand, directly captures demand pressure and allows scaling decisions to align with real-world usage patterns.

The retrieval layer was everything

While initial attention tends to focus on the LLM itself, production performance is overwhelmingly determined by the quality of the retrieval layer. The model can only reason over the context it is given.

The ingestion pipeline had to normalise documents from a wide range of sources—PDFs, DOCX files, HTML pages, and image-derived metadata—each with inconsistent structure and formatting. Parsing alone required careful handling to extract meaningful text without introducing noise.

from unstructured.partition.auto import partition
 
def parse_document(file_path):
    elements = partition(filename=file_path)
    return [el.text for el in elements if el.text]

Chunking strategy emerged as a particularly sensitive lever. Smaller chunks improved retrieval precision but often lost contextual coherence, while larger chunks preserved meaning at the cost of specificity. The solution was a hybrid approach that adapted based on document structure.

def chunk_text(text):
    if is_structured(text):
        return fixed_chunks(text, size=500, overlap=50)
    else:
        return semantic_chunks(text)

Equally important was the use of metadata-enriched embeddings. By attaching attributes such as department, region, and document type to each chunk, we enabled pre-filtering before semantic search. This significantly reduced the candidate space and improved both latency and relevance.

def retrieve(query_embedding, filters):
    return vector_db.search(
        embedding=query_embedding,
        top_k=5,
        filters=filters
    )

This shift—from pure semantic search to hybrid filtering—was one of the highest-leverage improvements in the system.

The human problems were harder than the technical ones

Despite the technical complexity, the most difficult challenges were organisational. The system depended on internal knowledge, which is inherently tied to ownership, incentives, and trust.

Teams were often reluctant to contribute or maintain content, either because they perceived the system as a threat or because they were concerned about misuse of sensitive information. Addressing this required more than technical safeguards—it required deliberate change management.

Role-based access control became a foundational requirement rather than an enhancement. Documents were filtered at retrieval time based on user permissions, ensuring that only authorised information could be surfaced.

def filter_docs_by_access(docs, user):
    return [
        doc for doc in docs
        if doc.metadata["department"] in user.allowed_departments
    ]

Critically, this filtering occurred before prompt construction. Any leakage at this stage would propagate directly into model outputs, making post-generation controls ineffective.

What I'd do differently

Looking back, several decisions would have been made earlier with the benefit of hindsight. Access control, in particular, should have been treated as a first-class design constraint. Retrofitting RBAC required reworking the ingestion pipeline, re-indexing documents, and revisiting core architectural assumptions.

Evaluation infrastructure is another area where earlier investment would have paid significant dividends. Automated evaluation of retrieval quality and answer accuracy allows issues to be detected before they reach users.

def evaluate(query, expected_answer):
    docs = retrieve(query)
    response = generate_answer(query, docs)
    
    return {
        "retrieval_score": relevance(docs),
        "answer_score": similarity(response, expected_answer)
    }

Finally, feedback loops should exist from the beginning. Every negative user interaction is a valuable signal that can inform both retrieval and generation improvements.

@app.post("/feedback")
async def feedback(query_id: str, rating: int):
    db.save_feedback(query_id, rating)

Closing that loop enables continuous system improvement rather than reactive fixes.

The outcome

RLGPT became Ralph Lauren's primary internal AI interface globally. The system succeeded not because of the model alone, but because of the infrastructure, retrieval design, and organisational alignment that supported it.

The central lesson is straightforward but often overlooked: production LLM systems are not primarily machine learning problems. They are systems engineering, data engineering, and change management problems, with the model acting as just one component in a much larger system.