Big Ideas in App Architecture · Cockroach Labs

Innovation and Disruption: How Materialize Pioneered a New Era in Data Streaming

·1 hr 15 min·5 clips

The Spanner paper emphatically proved you could have strong consistency and good performance at massive scales, changing Arjun's trajectory.

This episode features an interview with Arjun Narayan, CEO and co-founder of Materialize, discussing his journey and the company's streaming database technology. Hosts explore his transition from liberal arts to a computer science PhD and his early career at Cockroach Labs. The conversation traces how foundational research papers and industry experiences shaped Materialize's creation. Arjun studied at Williams College before pursuing a PhD at the University of Pennsylvania, focusing on distributed systems. He highlights the 2012 Google Spanner paper as a pivotal moment that challenged prevailing "big data" assumptions about weak consistency. After his PhD, he joined the early team at Cockroach Labs, a company then struggling to keep its database running for 24 hours. He later connected with Frank McSherry, a Microsoft Research scientist behind the 2013 Naiad paper on stream processing. McSherry initially showed little interest in commercializing Naiad but later built an open-source version called Timely Dataflow. Arjun credits Cockroach Labs' co-founders, particularly CEO Spencer Kimball, for mentoring him on startup transparency and operations. A key insight is that distributed systems research in the early 2010s often rediscovered concepts database communities had established decades prior. Arjun argues the extreme complexity of existing stream processing limits its adoption to only tech giants like Netflix or Uber. Materialize aims to democratize this by offering a SQL interface, framing its product as "materialized views that stay always up to date." The architectural discussion contrasts centralized scheduler models, like early Apache Spark, with dataflow models used by Naiad and Apache Flink that avoid scheduler bottlenecks. A memorable technical challenge cited is performing a streaming join, which requires maintaining indexed state from both input streams. The tone is conversational and educational, blending personal anecdote with deep technical history. It adopts a mentor-interview style, with the host sharing his own experiences in India's education system and at Cockroach Labs. Listeners interested in database history, distributed systems research, or founder origin stories will appreciate this episode. Those seeking a purely technical deep dive into Materialize's current architecture might find the early personal and historical context less immediately relevant.

As heard by us

A clear look at how research became a usable streaming-data product.

This episode follows Arjun Narayan, the CEO and co-founder of Materialize, as he moves from researcher to founder and uses that path to make the company feel concrete.

Read the full review in PlayNext →

Why you'd press play

If streaming data feels abstract, this makes the architecture feel concrete.

Read the full recommendation in PlayNext →

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from Big Ideas in App Architecture