Software Engineering Institute (SEI) Podcast Series · Members of Technical Staff at the Software Engineering Institute

Understanding Container Reproducibility Challenges: Stopping the Next Solar Winds

·25 min·1 clip

SunSpot changed source code only while the SolarWinds build was running, then removed the changes afterward.

1. The Software Engineering Institute podcast episode focuses on container reproducibility and the open-source tool Vessel. 2. Grace Lewis hosts the conversation as a principal researcher at Carnegie Mellon University Software Engineering Institute, and guests Kevin Pistick and Lihan Zhang are SEI software engineers working on Vessel. 3. The episode asks how developers can detect why two builds from the same Docker file differ and whether those differences signal risk. 4. Kevin Pistick defines reproducible builds as getting the same output from the same input, then names Docker files, build context, Git repositories, build arguments, and environment variables as inputs. 5. He lists timestamps, random numbers, randomly generated UUIDs, and changing package-manager dependencies as reasons container images often differ from one build to the next. 6. Kevin says those checksum differences make it hard to tell trivial noise from a malicious change in a container image. 7. Grace Lewis ties the discussion to the 2020 SolarWinds hack and asks how reproducible builds could have helped. 8. Kevin says the attackers installed SunSpot on a build server, watched for build commands, changed source code files during the build, and removed the changes immediately afterward. 9. He says source-code checks and signed binaries would not have detected that attack because the compromise happened only while the build was running. 10. Kevin says comparing two independent builds would have produced a red flag if one build was infected and the other was not. 11. Lihan Zhang says Vessel began with no existing tool built specifically for comparing container images. 12. He says the team built on Diffoscope for files, archives, and data, then wrote a customized parser to convert Diffoscope output into Vessel format. 13. Lihan also names Hadolint as a Dockerfile linter and says the team uses it to find reproducibility issues before containers are built. 14. He says Diff-OCI can compare images on GitHub, but Vessel also compares finalized file systems and container metadata. 15. Lihan walks through a workflow where a user compares two images from the same Docker file, accepts default flags for trivial issues like timestamps, and then adds custom flags for Java-specific paths or regex matches. 16. He says Vessel can be embedded into a CI pipeline so each container build can report the number of reproducibility issues. 17. Kevin describes two years of work that started with a literature review, moved to prototype diff tooling, and then used representative Docker files from GitHub for testing. 18. He says the team saw long diff times because more representative containers contained many files and many differences, so they are exploring faster modes and other optimizations. 19. This episode uses a conversational interview style, with Grace Lewis guiding the structure and both guests answering in detail about tools, workflow, and research process. 20. Software developers, CI engineers, and DoD teams interested in supply-chain assurance should listen; people looking for a brief product overview may skip it.

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from Software Engineering Institute (SEI) Podcast Series