Matt Aitken from Trigger.dev @ AIE
Matt Aitken on why Trigger.dev snapshots the entire VM — CPU, memory, and file system — to give agents real state, the shift from CRIU to Firecracker microVMs, and why nobody's fully solved the sandbox-permissions problem yet.
Recorded on the show floor at AI Engineer, this is a quick one with Matt Aitken, cofounder and CEO of Trigger.dev. Later that day his colleague Eric was giving a talk on durable agents, so we used the time to get into what that actually means: why LLM APIs quietly make you responsible for state, the different mechanisms teams use to handle it, and where Trigger.dev is focused next as it moves onto Firecracker microVMs.
The Q&A below is experimental — extracted from the episode transcript with AI. If you spot a mistake, let us know at jack@scalingdevtools.com.
What are you talking about here at AI Engineer?
We're sponsoring for the first time, which is cool — we have a booth with a really fun game where you try to prove you're a human to an AI, a reverse Turing test. Later today, Eric's going to be talking about durable agents.
What's a durable agent?
It's basically: how do you build a reliable agent that's stateful? Computers have been stateless for a really long time — you make a request to a server, you get a response, and the server itself has no real memory of what happened. You use databases and stuff to add that memory. We had long-running servers, then we invented serverless, which is an even more extreme version of stateless. Agents are different because you really want them to remember what just happened. One of the beauties of LLM APIs is they're really simple — every time you call them, you include the full conversation history, at least for a while, until that history becomes massive and the context gets bloated and problematic. So the APIs are simple, but it shifts the responsibility onto you as a developer to figure out how to make the thing stateful. A couple of years ago people were building really basic chat apps, and that was fine because you just had the message history in the browser and resent it each time. Now people are building much more interesting things — background agents running constantly, more sophisticated chat apps — and at some point you have to deal with the problem of state.
What are the trade-offs between the different ways of handling that state?
One type is what we call replay — it's what we used to do in Trigger.dev v2. You divide your code into steps, each with an ID, and the next time the function executes it caches the ones it's already done and only runs the new stuff. That gets you a serverless function that has state, and lets you build long-running agents. But there are issues: anything outside a step has to be deterministic, because it'll happen twice, and agent runs get really long now — some people are building agents with thousands, even millions of turns. If every call has to replay everything, it gets slow and doesn't scale well — think about a chat agent with a conversation that's thousands of messages long. More recently, Anthropic announced a managed agents product that takes a different approach: just take the LLM history — the messages, the tool calls — and store that as an append log you can replay. That scales much better, but the downside is that the machine your code runs on also has state: files, RAM, whatever's in memory if you're doing images or video, and that state is gone if you move to a different server. Our approach is that you need both — the context state and the execution state. For execution state, unlike everyone else, we actually snapshot the VM it's running in: the CPU, the memory, and the file system. You can turn that server off, and when you turn it back on, it resumes exactly how it was before. That's also how we handle a lot of human-in-the-loop cases — a lot of the time with agents you're waiting, not just on the LLM but on feedback from another agent, an external event, or a human approval, and you don't want to pay for those compute cycles. So we snapshot the machine, turn it off, and boot it back up when the next message comes in.
What's the focus right now?
The way we've been doing the snapshotting until now has been using a Linux feature called CRIU, which no one's really heard of, and we're about to start using Firecracker microVMs, which are pretty popular at the moment because of sandboxes. That'll let us do much faster start times, snapshots, and resumes, and we're going to be open-sourcing a lot of it. Firecracker itself is open source — it's what powers Lambda — but people don't really open-source all the extra stuff you need to get it to actually work. We're open-sourcing a ton of that, basically a Docker-compatible CLI but for Firecracker, which I think will be really successful as an independent open-source product on its own. We'll also release a managed sandbox product that's also open source, so people can self-host it.
The sandbox space feels like it's exploding right now — what do you think matters, and who wins?
I think people treat sandboxes as one market, and everyone's obsessed with start times. That's important for some use cases, but there are others where start time matters less and it's more about persistent file systems, or being able to execute for a long period of time. Networking and access are also very tricky — it's all well and good having a VM that's secure, but if you give it all your credentials, it can do crazy stuff. There are a lot of companies working on that: how do you not hand over API keys directly, how do you use a proxy layer — there are a lot of interesting challenges there that aren't fully solved. Part of the problem is that a lot of very popular APIs don't have fine-grained access controls — you want read-only access in a lot of cases, or the ability to draft an email but not send it. There are two schools of thought right now: people who are just yoloing everything and being like whatever happens is fine, and people who are so terrified they're not doing anything. I think in the next six months a lot of these problems get solved and these things get much, much safer to use.
What else are you focused on?
A lot on observability. You're dealing with non-deterministic code, stuff's running and you don't really know what's happening, so being able to see what actually went on — on a per-execution basis — is really useful. But when you've got a product in market doing thousands or millions of executions, you're not going to look through every single one. So you need to be able to write queries, which we just launched and open-sourced — it's a way of writing SQL queries on the underlying ClickHouse data we have, inspired by HogQL from PostHog. There's a bunch of metrics features too, and later this year we're launching a hybrid deployment model where customers can bring their own data store, so the data isn't stored on our side — same for compute.
Are you hiring?
Yeah — backend and product engineers, and an AI engineer, someone with experience building agents to production, based in Europe. We're also hiring a growth person: someone who's really good at creating content that cuts through — there's more content than ever out there, especially about AI and agents — and who can actually promote it on Twitter and LinkedIn and make it visible. You don't need 50,000 followers, but you need to show you can grow an audience.