eBay spent billions on search ads and then found many of them were largely ineffective.
1. HBR On Strategy examines how data-driven decision making can go wrong when leaders read numbers without testing how they apply to the decision at hand.
2. Curt Nikish hosts Michael Luca of Johns Hopkins Carey Business School and Amy Edmondson of Harvard Business School, who wrote the HBR article “Where Data-Driven Decision Making Can Go Wrong.”
3. The episode asks how teams can use internal and external data to make better business choices without treating any result as self-evident.
4. Edmondson says the problem is not too much data, but that leaders and teams are often “not using it well.”
5. Luca says the decision challenge is how to map analysis from a company, a news story, or a research paper onto the problem in front of you.
6. The discussion separates internal validity from external validity, using the warehouse-pay example to show how one causal result may or may not transfer to another setting.
7. Luca also points to Google, Amazon, and other large tech companies as examples of firms with abundant internal data but still difficult metric choices.
8. The guests argue that internal and external data need to be combined rather than treated as a single answer.
9. The eBay search-ad experiment becomes the clearest case study, because billions of dollars in ads initially appeared to work through correlation.
10. Luca explains that targeted ads made it look as if advertising increased sales, when the better causal test showed many of the ads were largely ineffective.
11. The conversation then shifts to the difference between what is measured and what matters, especially in platform design experiments that show short-term traffic but not long-term retention.
12. Edmondson says managers need to “go slow to go fast” by pausing to unpack assumptions before acting on a metric.
13. Luca highlights sample size and confidence intervals as tools for judging whether an apparent 5% lift is meaningful or just within the margin of error.
14. The guests note that a recent paper found many advertising experiments lacked the statistical power to determine whether ROI was positive or negative.
15. They also warn against overweighting a single salient result, especially when confirmation bias makes the finding seem more persuasive than it is.
16. Another trap is overgeneralizing from one context, such as assuming that a Google finding about grades would transfer exactly to another company’s hiring process.
17. Edmondson pushes the culture side of the problem, describing data discussions as a learning and problem-solving opportunity rather than a demand for instant answers.
18. The interview stays conversational and practical, with Curt Nikish pressing for examples while Luca and Edmondson answer in a measured, explanatory style.
19. Listeners who manage teams, run experiments, or read business research will get the most from it.
20. Listeners seeking entertainment or a narrative arc may skip it.
