Lawrence Jones from Incident.io @ AIE Europe: building an AI SRE
Lawrence Jones on the AI SRE incident.io is building: how it root-causes incidents across thousands of investigations a day, why dumping your logs into an LLM doesn't work, and the time it read Chinese-language telecom docs to catch a bug their own engineers missed.
Recorded live at AI Engineer Europe in London, this one's with Lawrence Jones, who works on the AI SRE at incident.io — a tool built on the idea that as teams ship more AI-generated software, they need AI to help them manage what that software does in production.
We get into how the tool actually performs, why pasting your logs into an LLM doesn't get you very far, and the moment it caught a bug buried in Chinese-language telecom documentation that had the team convinced it was a firewall issue.
The Q&A below is experimental — extracted from the episode transcript with AI. If you spot a mistake, let us know at jack@scalingdevtools.com.
You're giving a talk here about fighting AI with AI — what does that mean for what you're building at incident.io?
We're building an AI SRE on the basis that as people ship a lot more software with AI, they need AI to help them manage that in production — the more software you put out, the more problems you end up hitting. If AI can help you debug those issues faster, or tackle them before they even land, that helps manage the complexity of shipping all this software. Internally, we're running investigations to root-cause incidents across hundreds or thousands of customer accounts, thousands of investigations a day — actually understanding how that system is performing isn't something a human can do, so we lean on AI internally too, to see if it's getting better, if it's getting worse, and why. I think this is just going to be a consistent theme — fighting fire with fire, AI with AI.
How does it actually perform? Has it been helpful?
We have a ton of tools to tell us how the AI SRE is performing. We're working with several design partners now, anywhere from really big companies through to just running it on our own account. When you've got the system configured well, we're seeing 85 to 90% accuracy on the incidents we run root-cause analysis against, which is mind-blowing and not something we really thought we'd get to. The hard part is that it's such a nondeterministic system — we're plugged into logs, metrics, traces, and sometimes your logging system might just be down. We might run the same investigation one day and the next day it doesn't do quite as well. Maybe that's us, maybe that's variability in the AI, but potentially it's that your logs aren't responding or your application is misconfigured. Digging through all of that to figure out what's actually going on is what all of this tooling is built to do.
Why can't I just chuck my error logs into an LLM and ask it what's wrong?
There's a bunch of reasons. Naively, telemetry data and logs can be gigabytes — you've immediately blown the context window. So we have very specific ways of formatting, structuring, and summarizing that data so the AI gets a summary and then digs into the interesting parts. But the biggest reason is that if you just open up an LLM and say "figure out what this problem is," you're essentially taking a really smart engineer from a different company and asking them to debug your production incident cold. We know that doesn't work, because we don't hire engineers, put them on the team, and go "good luck, here's a Sev0." The best people in your organization are extremely smart, but they also know how to leverage the context in the organization and have memory of how it works. That's exactly what our tool does — it's continually building an understanding of your organization and your production infrastructure, and feeding that to the AI so we can ground everything in what we've actually seen happen before, with structures in place to grade our confidence.
What have you been excited about at AI Engineer?
Honestly, just speaking with so many people here, including a ton of customers like Braintrust — watching everyone try to solve problems that are quite novel given the age of AI, and seeing the really cool stuff they're building that simply wasn't possible before. That's what's been energizing about this event for me. One thing I want to take away: Braintrust's custom tracing solution does more ambient analysis of everything going on in the system, trying to identify patterns you may not have previously thought to look for. Our internal tools right now are pretty targeted — we know what we're grading and what we're looking for. I'd love something that's better at finding the thing we didn't realize was there.
What are you most excited about at incident.io right now?
Getting the AI SRE out to everyone — we're properly launching general access very soon. Watching our design partners, who are a total range of companies, come in with feedback has been amazing. Someone messaged us the other day saying they were "quite shook" that it figured something out; others have said a fix would have taken four of their engineers days and they're still not sure they'd have found it. We had it find something recently where our engineers were debugging an issue sending pages in China, and they assumed it was the usual global firewall problem. Instead, the AI SRE started scanning the documentation for our telecom provider in China — which is in Chinese — found a documented 750-millisecond timeout, correlated it with a trace, and posted it in the channel. The engineers were sure it was a firewall issue, and it just wasn't. That's the kind of thing that isn't possible without these tools.