Experimental Marketing with Natwar Maheshwari
Natwar Maheshwari once copied a Stripe documentation trick from seven years ago straight into his own newsletter's signup box — a text field that's auto-selected on page load. He'd never run an experiment to justify it. He just noticed his own cursor land there, thought 'that's cool,' and shipped it.
Natwar Maheshwari is developer marketing lead at Algolia, previously worked on APIs at Mailchimp, and started his career as an automation engineer. He also runs Engineering Brew, a curated newsletter of engineering blog posts.
We get into why he thinks marketing experimentation has to work on a completely different timeline than engineering debugging, why analysis paralysis is the real risk once you've wired up every analytics tool available, why trust and brand matter even for developer tools that feel purely technical, and why he has no shame about copying a good idea, like Stripe's auto-filled staging API keys, whenever he sees one.
The Q&A below is experimental — extracted from the episode transcript with AI. If you spot a mistake, let us know at jack@scalingdevtools.com.
How should someone think about developer marketing when they're just getting started?
At the very beginning, just look at what you think is cool and put it out there. A lot of developers build amazing things and assume it's not that great, but for someone else in this world it's freaking cool. Don't worry that something isn't finished or that others might not like it — the goal, especially early on with a limited audience, is to just be you. Also be careful about advice from traditional marketing sources — it's not that it's wrong, it's that there's a lot of unstated skill behind a case study like 'took a blog from zero to 1.5 million visitors in 18 months.' Do experiments instead, because in the beginning experiments are the only thing that tells you what actually works for you, even though a lot of them will fail. Coming from development, you're used to trying something and seeing it work or not almost immediately — in marketing you do something and then you just have to wait, which is why running multiple experiments in parallel matters. And put your team's cool work out there regardless of platform — if GitHub is your place, use GitHub; if you can contribute to an open-source project, do that. Document your process too — I'm bad at this myself, but companies like Amazon do it very well, and it's genuinely helpful even to people who hit issues with your exact steps.
Are there specific things a team can do to build that experimental culture?
It's hard, because in development you see results right away — an API either works or it breaks something else in your codebase immediately. In marketing that mindset has to change: you try something, it takes time, and that's okay, and you need to tell yourself and your team that building a brand and trust in an ecosystem takes time. Be clear on what good and bad look like for an experiment, and define upfront when you'll stop. We have every tool imaginable to capture clicks and scrolls, and it's genuinely fun to spend hours driving insights from tools like Mixpanel or Segment, but don't fall into analysis paralysis — the 80/20 rule works really well early on: 20% of your work gives you 80% of your insight, so get that and move on. You also need basic domain knowledge about what takes time and what doesn't — an SEO experiment based on a 2,000-word blog post needs time to rank on Google, while a 301 redirect doesn't. Having other developers or founders to talk to about these experiments is very helpful, because someone who's already run similar tests can tell you upfront that something is going to take time. And be comfortable cutting your losses and revisiting an experiment later — your domain authority today might be 20 and next year might be 57, and the same experiment that failed before might work then.
It sounds like a key point is being willing to try things you don't have much expertise in, while still building enough baseline knowledge to interpret the results correctly.
Exactly, you put that better than I did. Doing quick one-on-ones on basic marketing concepts is very helpful. Brand marketing, for example, is very hard and takes a ton of time, so you might read about it a little and decide you're not ready for it yet. SEO you might read about for four to eight hours and decide it's worth doing because it delivers sustainable, consistent traffic over time. Partnerships are another channel — partnering with people who have a bigger audience than yours because you've got a better or cooler product. Marketing has essentially a hundred thousand possible tactics, so you have to pick the ones that fit your own skill set and goals.
So there's no universal playbook — you just have to figure out what works specifically for you.
One hundred percent. If you find good developer founders or marketers, they'll tell you honestly that they don't know if something will work for you, because there are so many reasons something succeeds or fails. I struggle with things like preconceived notions about a brand — maybe you've seen my product's logo before, but I don't know in what context, whether it was on an open-source project or a website you hated. There are so many invisible reasons things work or don't. All I'd say is work with someone who's done things and is comfortable saying 'I don't know,' because you don't know either. Having a peer group, not consultants necessarily, is genuinely valuable.
Algolia has a strong brand. Does experimentation play into brand building as well?
Definitely, and it's a very important piece of the puzzle, especially building a product for developers. Brand is trust — trust that you're not going away tomorrow, that things will work the way you say they do, that you're not going to sell someone's data, that if you give them a library and it breaks, there's support behind it. If I'm not consistent, if we as a company don't consistently stand behind what we say, it becomes very hard for someone to pick and use our product, because ripping something like that back out of their systems is a lot of work. There's an old saying, 'nobody gets fired for buying an IBM' — IBM's brand was so strong that even if the implementation failed, you weren't at risk. That's what companies competing with IBM were up against. It's not always directly relevant to developer tools, but someone bringing your product into their own ecosystem is putting their own reputation on the line, and that's why trust matters so much. Being honest and being yourself is very helpful, because you don't have to remember a lot of things — you're just you. Easier said than done, of course.
Is experimentation also about working within constraints? How do you think about that process?
It's different at different stages and for different products. For Engineering Brew, my small curated newsletter, my experiments are very simple, partly because I have a full-time job and partly because I wanted to keep it clean — I'm not trying to make money from it. On something like Algolia, we use a proper A/B testing product, but A/B tests only work with a ton of traffic to be statistically significant — that's a real constraint. When I ran my own startup with about 125 visitors a day, my A/B tests were basically useless because I'd never get decent data even after a month. Understanding what you have and don't have, and being brutally honest about it, matters. For Engineering Brew, the email signup text box is automatically pre-selected on page load — I saw that on someone else's site, thought 'I'm going to subscribe,' and noticed my cursor was already sitting there. It's a tiny thing, and I didn't run an experiment for it — I saw it, thought it was cool, and used it. If you see something great in the world, like Stripe's documentation auto-filling your staging API keys when you're logged in, something they built maybe seven or eight years ago, there's nothing wrong with copying it. Even now, 90% of API documentation doesn't do that, so people remember you for it — that's trust, that's brand building.