This week we have two episodes (Black Friday deal woo!)
- Samuel Colvin - the creator of Pydantic - the most popular data validation library for Python. Used by basically everyone (Anthropic, OpenAI, Meta, NVIDIA, even the NSA). He shares the story behind his startup Logfire which just raised $12.5m from Sequoia. Full episode here
- Ankur Goyal - the founder of Braintrust, a year old LLM eval platform that is already used by Figma, Vercel and Stripe and just raised $36m from a16z. A rocketship. Full episode here
Samuel Colvin - creator of Pydantic/Logfire
This is one of the most popular episodes we've ever recorded. Samuel has proven he knows how to build extremely successful open source projects.
Key takeaways:
- You can just build a different product to your open source project and leverage your brand. And that brand has a lot of value.
- Quality of product matters a LOT (if you can build a popular open source project, can probably build a quality paid product)
- Really helps to be part of a movement. Hard to predict but Pydantic benefited from two (types and LLMs)
- GitHub stars are a vanity metric compared to download numbers
I filmed in person with SamuelWhy GitHub stars are a vanity metric
8,000 stars means 8,000 people said “oh that looks nice, star!”
Because even if:
- you install and it’s useless
- documentation lacking
- not what you need
You won’t un-star.
So stars just aren’t a good metric for whether someone found a project useful.
Samuel's "30-3-300" theory for successful open source libraries:
- Understandable in 30 seconds
- Useful in 3 minutes
- Still valuable after 300 hours
(Lots of GitHub stars just means that you hit the first point).
Ankur Goyal - founder of Braintrust
Ankur Goyal wrote down a list of 50 leading, target companies (with help from legendary investor Elad Gil). Then he reached out to all of them - getting their feedback, learning how they were using GenAI and the challenges they had
And since they were also the companies leading the way in GenAI, he could use that to drive product development, knowing others would follow their path
His first users were Zapier, Airtable and Coda and now everyone on his list uses Braintrust
They now have probably the best set of logos I've ever seen for a one year old DevTool: Vercel, Stripe, Instacart.
And $36m Series A from Martin Casado at a16z
Strong User Research:
- Spent significant time preparing structured interview questions
- Asked about programming languages, cloud services, workflow anxieties
- Discovered evals were a common bottleneck for companies
Product Development
- Strategy: If they could make the frontrunners happy, others would follow
- Built first prototype in 3-4 days
- TypeScript-first approach (contrary to Python dominance in AI)
- Evolved from pure evals to include logging and monitoring features
Working with Early Customers
- Early relationship with Vercel started through Guillermo Rauch
- Zapier were early advocates who reached out wanting the product immediately
- Embraced critical feedback as a sign of genuine product usage
Great investors
- Elad Gil sounds sounds almost like a cofounder. Helped plot out strategy for Braintrust
- Martin Casado sends bug reports constantly - heavy user of Braintrust.
- Alana Goyal - Ankur's wife is a DevTools investor!
Advice for DevTool Founders
- Focus on solving real business problems, not just technical solutions
- Ensure you can have conversations purely about the problem (not just solutions)
- Don't get overly attached to specific technical methods (e.g., fine-tuning)