Andrew Filev, founder of Zencoder: AI Software Engineering agents
Zencoder's coding agent went from solving 4% of real-world SWE-bench issues to topping the leaderboard at 70% in about a year — but Andrew Filev says professional engineering teams still aren't seeing their 10x, while self-taught 'vibe coders' with better prompts already are.
Andrew Filev is the founder of Zencoder, which builds AI software engineering agents, and was previously the founder of Wrike, the project-management tool he grew into a multibillion-dollar business.
We get into the generations of coding agents so far — from code completion, to agents working inside your IDE, to autonomous agents dropped straight into CI/CD pipelines — why professional engineers on large, messy codebases haven't gotten the 10x speedup that vibe coders building from scratch already have, and why a self-taught 'vibe coder' with a strong prompting setup can end up out-engineering a senior engineer within a couple of weeks.
The Q&A below is experimental — extracted from the episode transcript with AI. If you spot a mistake, let us know at jack@scalingdevtools.com.
You've talked about two generational changes in coding agents happening in about a twelve-month period. What are the generations?
It started with AI code completion, and the market-maker there was GitHub Copilot — not the first product, but the household name. It was nice and helpful, but not the game-changer some people claimed. Those first products were single-shot LLM calls, and a lot of the market's thinking was 'how do we make the model itself into a mega-brain.' We thought the more efficient approach was to focus on what happens before and after the LLM call: giving it the right context, since it's stateless and knows nothing about your specific repository unless you give it to it, and giving it tools rather than trying to teach the model to be a compiler or a calculator itself. Then Anthropic shipped Claude 3.5 in October last year, the first truly agentic model that could reliably use tools and explore an engineering environment, and that unlocked the second generation: capable agents working inside your IDE, human in control, which is the dominant mode today. The next surface is agents running autonomously in your CI/CD pipeline — pre-commit or post-commit hooks, a code review agent that runs on every commit. These generations don't replace each other; you'll keep using code completion, IDE agents, and pipeline agents side by side.
Do you have any sense of why Claude 3.5 was the one that could finally do reliable tool calls?
The frontier labs don't disclose what actually happens, so you can only guess, or get retrovision when open source gets similar results later. I think two things mattered: accumulation of the right data — companies like Scale made billions generating exactly this, senior engineers producing trajectories that teach agents how to do things — and reinforcement learning, labs figuring out how to build training environments where an agent attempts a real engineering problem multiple times and you can feed the correct solution back into training. A big part of the industry's progress in the last twelve months traces to one benchmark, SWE-bench, which used real, documented issues from open-source repositories with a golden-standard solution and unit tests, instead of Olympiad-style coding problems that don't transfer to real engineering. Models could originally solve about 4% of those tasks. In May, we submitted an agent using a mixture model, primarily Claude 3.7, and came in as the number one result — solving 70% of the issues.
Is it actually profitable to run agents doing that kind of work?
Yes — when AI works, it is absolutely more cost-efficient than human labor. I'm obviously a proud user of our own product, but I also pay for others: I have the top plan on Anthropic and on OpenAI, and the more I pay, the more I save, because I can offload tasks that would otherwise need an executive assistant. That's the caveat — when it works. When it does, the more you use it, the better it gets, and yes, the more you typically pay because the compute is expensive. But my salary is much more than $200 a month, and so is most people's, so the saving is definitely there.
Professional engineers using agents in their IDE today — are they seeing the same 10x that vibe coders building from scratch are seeing?
Not yet, and there's a good reason. There are two very different audiences here. Vibe coders — often not engineers at all — build simple prototypes from scratch, and for them it's close to infinite x, because they might not have been able to build the thing at all otherwise. Professional engineers working in a large monorepo, or 200 smaller repos and a bunch of microservices, are dealing with codebases that are hard even for experienced humans to navigate, let alone a model with limited context. There's proprietary APIs the model was never trained on, and older applications built five or ten years ago in languages like Java, using IDEs like JetBrains that the models weren't as tuned for. You can still get a lot of value, but you need to know how to wrangle the agent — writing good custom AI instructions and sharing them across your team, rather than everyone rediscovering the same fixes alone.
What does sharing that kind of institutional knowledge actually look like in practice?
A real example from our own company: our coding agent does noticeably better on VS Code than on JetBrains, because JetBrains isn't one IDE — it's a dozen products with different APIs across different versions. We had to figure out what context and instructions to give the agent so the code it produces for JetBrains is still correct and doesn't hallucinate APIs, then package that up once so the rest of the team benefits instead of everyone hitting the same wall. Same pattern with our testing agents: to test a billing page you need to mock the credit card checks first, and once someone's done that setup, the agent can generate a whole suite of test scenarios — but that setup knowledge needs to be captured and shared, not rediscovered by every engineer.
If AI can already deliver something like a 2x increase in release velocity, why aren't teams seeing it?
When I talk to VPs of engineering today, we're not yet seeing that 2x, even though I think today's technology could already deliver it. A lot of it isn't just the coding — you write code, throw it over the wall to QA, they're busy with something else so it takes days to get to it, they find something that was really your job to catch, throw it back, and now you're context-switching back into old work. All of that accumulates into delay more than actual coding time does. If the coding agent writes the code and a different agent writes tests that catch the obvious issues immediately, you save those cycles and stay in the zone. There's still an important human role at every stage — with more code being generated, architecture matters more, and I wouldn't trust hard architectural decisions completely to a model today. Same with testing: yes, AI should write more tests, but someone still has to build the frameworks and make sure it all actually works.
You made a pretty provocative point about senior engineers versus vibe coders. What was that?
Companies hire senior engineers for good reason — they're skilled and experienced. When their agent fails, that engineer does what he's paid to do: he steps in and manually solves it. A vibe coder, without that experience, doesn't have that option — so when the agent fails, a good vibe coder with technical chops will instead try to solve it with the agent, writing better AI instructions, iterating on prompts. Fast-forward a week or two, and the senior engineer has the same level of AI capability he started with, while the vibe coder has a much more advanced harness, tooling, and prompts — and ends up being the better AI engineer. I've seen this work for senior engineers too: one company had a huge migration with nowhere near enough people to do it manually, so they were forced to build hierarchical prompt templates and a multi-agent workflow just to make it possible. Sometimes senior engineers need to create that same artificial constraint deliberately, and then they get the best of both worlds — their AI chops and their years of experience.
What did you make of Devin's approach when it launched?
I'm pretty opinionated here — I think they alienated a lot of people. They promised things that weren't working, and even today their agents are slow and not best in class. I also don't like the positioning: I'm not in this business to replace engineers, I'm in this business to help everybody ship more software. In my previous company, at its peak with several hundred people in R&D, maybe 5% of my product ideas ever got built — there's never enough time or engineers, and a lot of the work isn't glamorous: bug fixes, migrations, internationalization, accessibility, security. As an engineer, why would I spend four hours babysitting Devin if I don't get credit for the PR? That works if you're a solo founder who's the owner and the engineer, but in a commercial setting you have to give credit where it's due.
If someone's building something like a feature-flagging API, how should they think about open source given where AI is heading?
If what you're building isn't core to your business and doesn't differentiate you against competitors, there's more and more incentive not just to consume open source but to contribute to it — because the next drop of the frontier model is going to train on your open-source code and documentation, so you get your code baked into the model for free, no GPUs required. Previously, when I started in SaaS, there were fewer players, so building your own SSO or SAML in-house felt defensible. Today, every B2B SaaS product has SSO one way or another — there's no reason to keep something like that proprietary anymore. The more software companies and software reuse there is, the less value there is in keeping the plumbing to yourself, and the more value shifts to speed and understanding your customers.
Any practical tips for people trying to get more out of these tools day to day?
If you're facing a complex problem and AI isn't working well for you, use AI to help generate a more extensive prompt — a 'mega prompt.' Tell it what you're trying to accomplish and where you're failing, and ask it to help you build a prompt that alleviates that in the future. It works even better if you give it access to relevant tools — I've done this by giving an agent access to our own code files so it could extract useful context, but you could give it access to your Gmail or your calendar to pick up patterns from past information. It'll generate a good prompt you can reuse going forward.
Links & references
This episode is brought to you by WorkOS.