Collective Intelligence Community Podcast · The AI Collective

OpenPipe Co-Founder on Reinforcement Learning - David Corbitt | Ep. 22

·56 min·9 clips

David opens by naming OpenPipe and what it is trying to do. He frames the company as a way to help small models win against much stronger closed-source systems on specific tasks, especially when lower cost, lower latency, or better quality is the goal. That is the basic premise. The conversation then splits the work into two main paths: supervised fine-tuning and reinforcement learning, both aimed at nudging a model toward outputs that fit a particular use case. Supervised fine-tuning is described as teaching the model to imitate a dataset of correct outputs. Reinforcement learning is described as adjusting weights over time so the model moves toward a more desirable pattern of token generation. The practical end point stays the same. The goal is an open-source fine-tuned model that can outperform a closed-source model on the task that matters. The host keeps pulling the discussion back to why these methods matter. That keeps the exchange focused on concrete questions about how much model improvement comes from data, how much comes from optimization, and how much comes from choosing the right training setup. The episode then turns to the mechanics of reinforcement learning itself. PPO is treated as a single-output approach where a model gets a high or low reward, while DPO works with pairs of outputs and can push away from a given behavior. GRPO is presented as the most recent technique, described as having been pioneered by DeepSeq in 2024 or at least published then, and it creates several outputs for one input so each can be rewarded differently. That makes the comparison clearer. Instead of only rewarding one good answer and one bad answer, the method can compare four, eight, or even many more outputs and move weights toward the strongest ones. The conversation uses a simulation example to show why that matters. If a model only does the right thing once in a hundred tries, the training signal can still push hard toward repeating that rare behavior more often. The host also asks where humans fit into all of this. That keeps the episode grounded in the practical boundary between human feedback, automated reward, and the different techniques people group together under reinforcement learning. The closing advice is straightforward: learn quickly, check claims with other people, and if you want to build something, just build it.

As heard by us

A practical look at how smaller models can outperform larger ones on the right task.

David Corbitt treats reinforcement learning as a product decision, asking when a smaller model becomes the better choice because it is cheaper, faster, or stronger on a specific task.

Read the full review in PlayNext →

Why you'd press play

You want a crisp map of how OpenPipe helps small models win on your task.

Read the full recommendation in PlayNext →

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from Collective Intelligence Community Podcast