Software Misadventures · Ronak Nathani, Guang Yang

Life as a Distinguished Engineer | Joakim Recht (Uber)

·1 hr 16 min·5 clips

Uber put databases in containers when Docker was new and Kubernetes was still in alpha.

1. Software Misadventures uses Joakim Recht's distinguished-engineer career at Uber and founding role at BeyondWork to examine senior individual-contributor work. 2. The episode features Joakim Recht as the Uber distinguished engineer and BeyondWork founding engineer, with Jasper Lundqvist and Jens Nielsen conducting the conversation. 3. The central question is how a distinguished engineer makes company-wide technical decisions while remaining close enough to code, teams, and production failures to retain trust. 4. Recht describes Uber's informal "reverse sausage" idea as accepting that software systems are messy, imperfect, and shaped by real-world constraints. 5. He says teams gain more value by releasing a workable system and observing production behavior than by pursuing perfection in isolation. 6. At Uber, Recht spent most of his time building deployment platforms for stateful workloads rather than only stateless services. 7. The team put databases in containers when Docker was new and Kubernetes was still in alpha, despite skepticism from other engineers. 8. The resulting platform eventually ran Uber's non-stateless workloads, including systems built around data that could not simply be discarded. 9. Recht says the platform had to assume that hardware could fail, power could be removed, and every possible sequence of events could not be modeled in advance. 10. The migration began around 2015 and continued until roughly 2024 as the team tried to bring other groups and their existing tools onto one platform. 11. The team used Schemeless, an Uber-built key-value store based on MySQL, as an initial platform workload before expanding to other teams. 12. Recht identifies Kafka and HDFS as examples of systems that resist cloud-like movement because they expect stable environments and long uninterrupted runs. 13. Uber targeted re-provisioning every server every 30 days, a policy that created difficult data-movement and replication costs for large stateful systems. 14. Recht defines the distinguished-engineer level primarily through scope, describing Uber's roughly eight distinguished engineers among about 4,500 engineers. 15. He says senior engineers must balance technical quality against revenue, reliability, politics, staffing, and other projects because the theoretically optimal solution may never be achievable. 16. Recht argues that communication becomes increasingly important because engineers must explain decisions, receive corrections, and persuade others to use their systems. 17. He describes writing as an iterative outline process: start with bullets, establish a single message and flow, then expand each bullet into a section rather than repeatedly polishing the finished draft. 18. He says senior engineers should continue writing, reviewing code, fixing bugs, refactoring, and improving developer tooling so they do not drift away from the teams whose trust they need. 19. Software engineers interested in technical leadership, infrastructure, and pragmatic architecture. 20. Listeners seeking product launches, entertainment, or nontechnical career stories might skip it.

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from Software Misadventures