Accelerate Your Machine Learning Experimentation With Automatic Checkpoints Using FLOR

·47 min·4 clips

Rolando Garcia built FLOR to let machine learning researchers recover lost training data without rerunning week-long jobs.

The episode begins with host Tobias Macy introducing guest Rolando Garcia, a PhD candidate at UC Berkeley advised by Joe Hellerstein, who works on machine learning model management. Garcia explains his background, learning Python through AI MOOCs like Berkeley's Pacman project. He describes FLOR (Fast Low Overhead Recovery), a project from the RISE Lab that sprouted from the Ground data context service. FLOR aims to solve two core problems: the tension between engineering best practices and rapid ML experimentation, and the high cost of rerunning training jobs due to missing logs. Garcia recounts a lab horror story where a week-long training job had to be rerun because of a missing logging statement. He contrasts FLOR with manual checkpointing, noting FLOR enables parallel recovery. The implementation is entirely in Python, using Fork for background materialization to avoid GIL overhead and adaptive checkpointing targeting 6% logging overhead. Key challenges included side-effect estimation for memoization and maintaining under 2x overhead. FLOR offers two modes: a hands-free command-line tool and a manual API for experts. It stores metadata and serialized data using Cloud Pickle, integrating with PyTorch and TensorFlow serialization. Garcia discusses FLOR's agnostic design toward ML frameworks and its focus on the training phase. He argues FLOR can reduce the 'foggy' nature of AI by enabling post-hoc analysis of failures. Long-term, FLOR aims to aid explainability across the ML lifecycle, including deployment. Future work includes versioning code automatically and supporting distributed training. Garcia shares maintenance challenges as a solo researcher and an unexpected use case where a student used FLOR to generate traces for code recommendation models. He notes FLOR is less suitable for single-epoch training. The episode concludes with picks, including the show Severance and GitHub Codespaces.

As heard by us

A focused review of Flora's checkpointed replay approach, its side effect model, and the overhead limits that shape faster ML iteration.

Machine-learning teams often pay for the same work twice when training runs have to be repeated, and this episode follows one attempt to avoid that waste.

Read the full review in PlayNext →

Why you'd press play

If repeated model training is slowing your experiments, you will find a concrete case for Flora's checkpointed replay workflow.

Read the full recommendation in PlayNext →

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from The Python Podcast.__init__