Charity Majors on AI, Observability, and the Future of Software
Charity Majors on why the cost of generating code dropping toward zero makes monitoring the new source of truth, why code review was never really about finding bugs, and the writer's-block disaster that made her throw out months of her new book.
Charity Majors is the founder and CTO of Honeycomb, and one of the most prolific voices in observability. We talk about what changes when the cost of generating code drops toward zero — why she thinks monitoring becomes the substrate everything else depends on, why code review was never really about catching bugs, and why great engineering practices that nobody would fund for humans are suddenly getting funded for agents.
We also get into how she's built such a visible personal brand while still running engineering at Honeycomb, and the near-disaster that hit her while rewriting a chapter of the second edition of Observability Engineering.
The Q&A below is experimental — extracted from the episode transcript with AI. If you spot a mistake, let us know at jack@scalingdevtools.com.
Now that the cost of generating lines of code is basically zero, what are the new bottlenecks for building software?
My friend Boris Tane wrote a piece about how this is the end of the software development life cycle as we knew it — all the middle stages are being compressed, an agent doesn't really know what stage it's in. The only stage that remains is monitoring, and monitoring needs to evolve, because it's actually the substrate that everything else leads up to. It's the validation, it's the source of truth, it's the source of meaning.
You've said you're shocked that people are shocked they have to understand code from first principles now. Where does that come from?
It goes back to my experience at Parse, which was a lot like a Heroku-style mobile backend business. We had over a million mobile apps running on it by the time I left, and we never got to talk to those developers — we had to understand their code from first principles and operate it from first principles, because we had no other choice. That experience seeded Honeycomb. So when I see everyone shocked that they suddenly have to understand lines of code from first principles again now that AI is writing so much of it, I'm like — that's not a new transition. We went through it at Parse years ago.
There's a debate about whether code review is dead now. What's your take?
My friend David Poll, who's now senior director of engineering at GitHub and was also at Parse, wrote a great piece basically arguing that code review was never really about finding bugs. He and his team would spend hours debating whether an API made sense — not for bugs, but for the mental model. Do we let this into our product or not? Can people reason about it? Is it coherent? One of my favorite lines from his piece was that the word "code" in "code review" gives people the wrong assumption — it's not about the code, it's about the product. We've been overloading code review to do way too many jobs at once. It absolutely needs to evolve, but not by just getting rid of it — we've got to find new ways of reviewing that don't rely on humans catching bugs with their eyeballs, because honestly, we were never that good at that part anyway.
If models keep getting better at generating code, what does a human actually bring that the LLM will not?
Taste. Judgment. I think about it through the lens of durable code versus disposable code. There's no question we're witnessing a genuinely powerful new method of development, but it's far less clear to me that it works for all types of code. There's an old line I've used for fifteen years — the closer you get to laying bits down on disk, the more paranoid you have to be: operating systems, databases, devices. A friend of mine put it better — it's not really about disk, it's about dependability, about how many things depend on you being predictable and not changing too fast. Nobody wants to use a Slack that's subtly changing from day to day. Disposable code is great for rapid iteration and trying things, but the way we know how to build maintainable, reliable software people love is by taking what's known to be good and iterating on it in small, frequent changes. How you cross from one mode to the other is a genuinely open question — I don't think anyone who claims to know exactly where this goes in two or three years is being honest.
You've talked about validation being the new "test in prod." What does that mean in practice?
It's all about declaring intent. Maybe you generate the code, maybe you write it — either way, you test the intent, you deploy it, and you validate your intent. I've spent the last ten-plus years publicly telling people: instrument your code as you write it, use wide structured events, use feature flags, progressive deployments, canary testing, automated rollbacks — look at your data after you ship and check whether it's doing what you meant it to do, and whether it's doing anything else that looks weird. Close the loop. I've come to think of that as the eat-your-vegetables lecture, and for years I asked myself why our opinions traveled so much further than our product ever did.
So why does that eat-your-vegetables advice suddenly feel like it's landing differently now?
Because for years, people would read that advice and go "yeah, that would be great," and then not do it — not because they're stupid, but because it's genuinely harder and requires more of you, and some of the best engineers in the world work at companies with two- or six-month deploy pipelines anyway. Nobody was funding fast feedback loops for humans. What I've realized is that AI is our chance to bring those engineering values to the mainstream, to make them accessible — not because we're telling people to do it, but because of competitive pressure. AI can generate change at 10x, 100x, 1000x human speed, and our systems have only ever been held together at human speed through a bunch of human tricks: intuition, the person who's been there fifteen years, documentation that's out of date but someone knows where to look. When that rate of change goes up, all of that duct tape flies off. I hear this constantly from platform engineers and SREs: I've spent my whole career advocating for guardrails and nobody would fund them — until it was for the agents instead of for humans. You can be sour about that, or you can see it as an opportunity.
Should software engineers be stressed about whether their skills will still be needed?
No — but I still meet engineers who are averse to framing what they do in terms of return on investment or value-add, and while I deeply understand that purity streak, because I used to be one of those engineers, I think you need to get over it. It doesn't serve us for engineering to be treated as untouchable artists. This shows up most clearly around engineers hating for their output to be measured at all — and while it's true that value gets created in different ways, and true that plenty of companies have misused metrics, we still have to be willing to look at our output and talk about it in the language of the business, which at the lowest common denominator is time and money. Engineers don't need to worry about their skills becoming useless — it's the nature of technology that skills constantly need to evolve — but more than ever, we need to get comfortable talking and thinking about our work in business terms.
You've built one of the biggest personal brands among DevTools founders while also being CTO. How have you actually managed both?
Honestly, the first answer is that I stepped off X for the last couple of years and I've realized that was a mistake — I'm trying to ramp back up because the company needs me to. But the real answer is: ask anyone at Honeycomb, I'm not really present in day-to-day engineering. I've only been able to do the external stuff because I completely trust that the team has it handled. I was CEO for the first three or four years, and our entire go-to-market strategy was: any place that invites me to speak, I say yes. I was flying constantly, constantly on Twitter, while Emily Nakashima, our first front-end engineer, stepped into running engineering — I couldn't do what I do without her, or without engineers who genuinely care about the craft, the culture, and the users, and who were never precious about marketing or sales being beneath them. There's also a dynamic I've noticed: anytime you want people inside the company to really pay attention to something, say it outside the company. I can post something internally and some people will engage, but if I say the same thing publicly, it carries more currency — colleagues' friends message them asking "do you guys really do this?" and then they have to go represent it themselves.
You mentioned nearly derailing the second edition of your book. What happened?
We planned a new part six for the book — observability governance, since observability engineering teams didn't really exist when we wrote the first edition. I was supposed to write the last chapter, about buying software, and realized I'd never actually bought software myself, so I put out a call asking people who had to email me their advice. The internet had a lot of opinions. I immediately hit the worst writer's block of my life — six weeks of blocking out eight hours a day and getting nothing done. Then, the week before Thanksgiving, I was up at midnight working through it with Claude and realized everything I'd written needed to be thrown out — I'd written the wrong thing for the wrong people. The stories I kept getting described a completely broken buying process with zero alignment between CTOs, VPs, and staff engineers, most of whom hadn't been hands-on in ten or fifteen years but still think they are. So I rewrote it from scratch as an open letter to CTOs arguing that everything they want to do over the next few years is probably blocked on organizational learning speed — the ability to understand and make decisions about code, not just generate it. Which is what observability actually is. As for whether other DevTools founders should write a book: it's worth it if it's an O'Reilly book, for the branding and credibility that carries in the industry. But more broadly, everyone should write about their work and commit to doing it in public, being willing to be wrong in public — writing is thinking on paper, and it forces you to work out the kinks in your thinking faster than you'd otherwise manage.