Software Misadventures · Ronak Nathani, Guang Yang

Oliver Leaver-Smith - On how "just a monitoring change" took down the entire site and resilience engineering - #5

February 19, 2021·1 hr 1 min·5 clips

Oliver Leaver-Smith reveals how a simple monitoring change can crash an entire site.

Oliver Leaver-Smith, a DevOps engineer at Sky Betting and Gaming, joins hosts Ronak and Austin to discuss software resilience and a major outage caused by a monitoring change. The episode explores how teams can prioritize reliability alongside new features. Leaver-Smith shares his long-standing interest in how technology fails. The conversation centers on a specific incident where a routine update to monitoring configurations unexpectedly caused a complete site outage. Leaver-Smith details the principles of chaos and resilience engineering practiced by his team. His team regularly conducts "fire drills," which are controlled chaos engineering exercises. These exercises intentionally test both software systems and the human response procedures during simulated failures. A recent fire drill example involved deliberately introducing a failure to observe the system's and team's reactions. A key insight is that system resilience depends as much on prepared people and processes as on technical architecture. The outage story illustrates how a change in one area, like monitoring, can have catastrophic, unforeseen consequences in another. Leaver-Smith argues that without dedicated effort, teams naturally gravitate toward building new features over reinforcing existing systems. He suggests that engineers must proactively design for failure because outages are inevitable. The fire drills aim to build institutional muscle memory for handling crises. The episode presents the perspective that major failures provide the deepest learning opportunities about a system's true behavior. A surprising claim is that a "simple" change can be the most dangerous, as it receives less scrutiny. The discussion reveals that gaming and betting platforms treat resilience as a core business requirement due to their high-traffic nature. The hosts and guest agree that post-mortem analyses are invaluable but preventing repetition requires proactive chaos testing. The tone is conversational and educational, blending a war story narrative with practical engineering philosophy. The style is technical yet accessible, focusing on real-world anecdotes and team practices. Software engineers, DevOps practitioners, and technical leads focused on system reliability would find this episode highly relevant. Listeners solely interested in feature development or theoretical programming might find the operational focus less engaging.

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from Software Misadventures