What I thought mattered vs what actually mattered
When I started building production AI systems at Ralph Lauren, I thought the hard problems were technical: model selection, prompt engineering, latency optimisation. Those are real problems. They're also the problems with the most documentation, the most tooling, and the most community knowledge behind them.
The problems that actually determined whether projects succeeded were different.
Adoption is the product
RLGPT was technically excellent before it was widely adopted. The gap between "the system works" and "9,000 people use it and trust it daily" was filled by communication, onboarding, feedback loops, and visible responsiveness to user concerns — not engineering.
Every AI system has two products: the technical system and the change management process that gets people to use it. Most engineering teams only build the first one and are surprised when adoption is slow.
The best model is the one your team can operate
There's a tendency in AI engineering to reach for the most powerful model available. Power isn't always the right variable to optimise for. Operational complexity, cost at scale, latency profile, and your team's ability to debug failures when they happen matter more in production than benchmark performance.
We made deliberate choices to use models that were slightly less capable but much more operationally predictable. In an enterprise context, predictable failure modes are worth a meaningful capability delta.
Evaluations are infrastructure, not an afterthought
You cannot improve what you cannot measure. We added evaluation infrastructure late in the RLGPT build and spent the first months of production flying somewhat blind on quality metrics. Building evals from day one — even simple ones — gives you a baseline to defend and a signal to improve against.
The human in the loop is a feature, not a fallback
AI systems that route uncertain cases to human review aren't failing — they're working correctly. The instinct to maximise automation is understandable but wrong in contexts where error cost is high. Design explicit human-in-the-loop paths for low-confidence decisions. Your users will trust the system more, not less, for knowing when it asks for help.
What I'm still figuring out
How to build AI systems that improve from production usage automatically, without manual intervention, without overfitting to recent data, and without introducing bias from skewed feedback distributions. The feedback loop problem is genuinely unsolved in most production deployments, including mine.