The startup behind ChatGPT voice - Russ d'Sa from LiveKit
LiveKit founder Russ D’Sa explains the real-time infrastructure behind ChatGPT voice, the improbable email that tied his company to generative AI, and why teams should discover scaling limits through real demand instead of designing forever for imagined traffic.
Russ D’Sa founded LiveKit as open-source infrastructure for real-time audio and video. Its network and agent framework now carry the live media behind products including ChatGPT voice and Character AI.
After a live ChatGPT lesson on lightning, Russ explains how the system works, how OpenAI found a tiny team through a personal account, what happened when voice mode reached free users, and where natural interfaces may take human-computer collaboration next.
The Q&A below is experimental — extracted from the episode transcript with AI. If you spot a mistake, let us know at jack@scalingdevtools.com.
What role does LiveKit play in ChatGPT voice?
The phone and an agent on OpenAI’s backend both connect to LiveKit’s cloud network. A user’s live audio travels over that network to the agent, which relays it into the model; the generated response returns through the same path. If the model is the brain, LiveKit is the nervous system carrying audio and video to and from it.
Why is real-time media infrastructure technically difficult?
It is not a file-transfer problem. Netflix can download and buffer chunks of a stored video, but conversation is generated in the moment and must stream continuously in both directions without an intermediary file. LiveKit connects users to nearby servers, gets their media onto the internet backbone with low latency and transports it to wherever the customer runs compute.
How did LiveKit begin working with OpenAI?
In August 2023, a large company offered roughly $20 million to acquire LiveKit and suggested it could be worth ten times more if it had a generative-AI story. I could not see the connection and left worried that the company might crush us. On the drive home, an email arrived from OpenAI. Its team had quietly built a voice-mode proof of concept on LiveKit Cloud using a personal Gmail address and wanted to discuss working together.
How did that change your view of LiveKit’s market?
Video conferencing infrastructure turned out to be the eyes, ears and mouth of a much larger class of machines. A capable computer should not require people to translate every intention into mouse and keyboard input; it can see, listen and speak. I expect real-time traffic to shift from mostly human-to-human connections toward humans interacting with cloud models, personal devices and eventually robots.
Why did older voice assistants lose users’ trust?
Systems such as Alexa behaved like enormous trees of predefined branches. Ask for something outside those branches and the system fails, so people retreat to the handful of commands they know will work. An LLM almost always offers a response and lets the user rephrase or continue. Even hallucination can improve that narrow interaction loop by avoiding the repeated dead end of “I can’t do that,” though correctness still matters for the task.
How might voice change creative and engineering work?
The interaction could resemble delegating to a colleague: explain an idea naturally, let the system produce a first draft or proof of concept, then meet again to critique and refine it. Humans still provide product judgment and oversight because they are building for other humans, but more of the mechanical work can happen asynchronously. Product design may become more important than manually operating every implementation tool.
What happened when OpenAI expanded voice mode to free users?
The team reviewed the architecture and refined likely bottlenecks in advance, while Character AI was also ramping on LiveKit. OpenAI then gave us about 15 minutes’ notice before opening voice mode to free users. Engineers checked the dashboards, said the system was ready and watched it scale without trouble. Character AI later exposed a different limit, prompting a rapid component rewrite — exactly the kind of unknown that only production demand reveals.
What would you tell a startup onboarding a vastly larger customer?
Embrace the chaos. Twitter repeatedly hit technical limits yet succeeded because it nailed the product’s essential value; rewriting a system can be a privilege earned by finding demand. Infrastructure does carry a higher reliability obligation — LiveKit supports critical services including some 911 traffic — so we stress-test, simulate failures and monitor carefully. The point is to prepare for visible risks and react well, not delay the market for every imaginable future failure.
How can developers use LiveKit?
The software is fully open source, so a team can deploy and operate it themselves. LiveKit Cloud runs the infrastructure for teams that would rather spend their time on the application. In both cases the interesting work is what developers build on top: real-time experiences linking people to people and, increasingly, people to intelligent machines.