STAT AI: An Emergency Medicine Podcast · Ethan Abbott and DJ Apakama

EP6 Nick Genes, MD, PhD, Emergency Medicine NYU Langone

·32 min·2 clips

A new study found large language models fail to match physicians in diagnosing common abdominal emergencies like appendicitis.

This episode features emergency physicians DJ Abhakama and Ethan Abbott analyzing a recent AI study and interviewing Dr. Nick Genes, an informatics director at NYU Langone. The hosts begin by dissecting a July 2024 Nature Medicine paper by Paul Hager et al. that tested large language models (LLMs) on abdominal diagnoses. The study used 2,400 real MIMIC cases of appendicitis, cholecystitis, diverticulitis, and pancreatitis to evaluate models like LLaMA2 and Clinical Camel. It found LLMs performed significantly worse than physicians, with diagnostic accuracy for cholecystitis dropping as low as 33.8% and for diverticulitis to 34.2%. The models often failed to adhere to clinical guidelines and misinterpreted lab results, leading the authors to conclude they are not suitable for autonomous clinical decision-making. Dr. Genes discusses his background in clinical informatics, defining it as ensuring clinicians are better off with a computer. He details a project using a HIPAA-compliant GPT to read radiology reports and extract incidental findings for patient summaries, aiming to improve follow-up on "incidentalomas." Another initiative involves using GPT to create handoff summaries for clinicians, with early feedback showing attendings found it useful while residents were more skeptical. Genes notes they have yet to find a hallucination in these summaries when the model's creativity parameter is set to zero. He also envisions AI generating tailored discharge instructions and even acting as a post-discharge digital assistant for patient questions. The conversation is technical and educational, blending a journal club review with an expert interview focused on practical implementations. Emergency physicians and medical informaticists interested in the near-future clinical applications of AI will find this episode highly relevant. Listeners seeking a broad, non-technical overview of AI in medicine might find the detailed study critique and implementation hurdles less engaging.

As heard by us

AI framed as CDS, follow-up, and the realities of an ED shift.

The episode treats AI as a workflow question first, which suits the way emergency medicine is framed here: a field of practical choices shaped by clinical decision support.

Read the full review in PlayNext →

Why you'd press play

You want AI talk that still remembers the emergency department has to function, which is rare enough.

Read the full recommendation in PlayNext →

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from STAT AI: An Emergency Medicine Podcast