Software Misadventures · Ronak Nathani, Guang Yang

Tammy Bryant Butow - On failure injection, chaos engineering, extreme sports and being curious - #6

March 7, 2021·1 hr 4 min·5 clips

Tammy Bryant Butow explains the three key areas to focus on for implementing chaos engineering in organizations.

Tammy Bryant Butow, a principal SRE at Gremlin, joins hosts Ronak and Guang to discuss chaos engineering and building reliable systems. She brings experience from leading SRE teams at Dropbox and working at Digital Ocean. The conversation explores practical strategies for implementing failure injection and fostering a reliability culture. Bryant Butow outlines a three-step focus, starting with educating teams on chaos engineering and SRE fundamentals. She emphasizes the need to establish a culture where injecting failure is standard practice. A key decision involves choosing between a centralized team for chaos engineering or a self-service model for broader access. The hosts share their perspective that production failures, while stressful, provide critical learning opportunities about system behavior. Bryant Butow recounts an early-career outage where a core switch failure took down half a data center. She also details her experience running a disaster recovery test, which she says taught her to be a better person. Her career path into infrastructure engineering was driven by personal curiosity. The episode frames these "misadventures" as essential for developing resilience principles and practical debugging skills for complex systems. One insight is that foundational education on "why" must precede any technical failure injection work. The discussion reveals that organizational structure—centralized versus distributed—is a pivotal consideration for chaos engineering programs. The anecdote about the data center switch highlights how single points of failure can have massive, unexpected impacts. The disaster recovery test is presented as a profound learning experience beyond just technical procedures. The tone is conversational and experience-based, featuring stories from engineering trenches. The style is educational, aiming to distill lessons from past failures and outages. Listeners interested in site reliability engineering, resilience patterns, and post-mortem analysis would appreciate this episode. Those seeking highly technical, tutorial-level content on specific chaos engineering tools might find it less directly applicable.

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from Software Misadventures