Building Trust in AI: Chatbots, LLMs, Decentralized Finance, and JPMorgan Chase

·30 min·2 clips

James Massa reveals he interviews LLMs like job candidates, checking their training and data handling for trust.

Luke Malks, host of The Brave Technologist, interviews James Massa, Senior Director of Engineering and Architecture at JPMorgan Chase. Massa holds six patents and a master's degree from Harvard, focusing his work on AI data quality and model self-healing. He argues that responsible AI fundamentally boils down to building systems you can trust, similar to hiring and vetting a human employee. Massa has co-authored research with Kennesaw State University on establishing trust scores for decentralized finance projects. His team used four LLMs to analyze smart contracts from perspectives like code vulnerabilities, suspicious transactions, anomalous price changes, and social media scam sentiment. He also explains LLM distillation, a process for creating smaller, cheaper local models from larger hyperscaler LLMs like OpenAI's. JPMorgan is involved in blockchain through its Onyx project, and Massa notes the space is "ground zero for trust issues" due to past malfeasance. A key challenge is ensuring chatbots only give approved answers, which Massa says is addressed through Retrieval Augmented Generation (RAG) and LLM guardrails. RAG works by comparing a user's query to pre-approved document chunks in a vector database to generate responses. He distinguishes between AI precision, where every result must be correct, and recall, where the goal is to miss nothing, noting his compliance work prioritizes perfect recall. Massa emphasizes that AI models require ongoing "model monitoring" because data can drift or the underlying LLM can change, necessitating retraining. He believes the future of work involves humans acting as managers of teams of LLM agents, particularly in software development and customer service chatbots. For organizations adopting AI, he identifies a tension between aggressive, grassroots innovation and a controlled, top-down approach focused on cost-benefit analysis. Massa advises starting with U.S.-only deployments to navigate unclear global regulations and minimizing the number of production models to reduce regulatory risk and duplication. The conversation is educational and conversational, exploring practical implementation challenges within a large financial institution. It balances technical explanations of RAG and model distillation with broader discussions on corporate governance and ethics. Listeners interested in enterprise AI adoption, especially within regulated sectors like finance, will find concrete strategies and use cases. Those seeking highly technical deep dives or debates on AI's societal impacts might find the discussion too high-level or institutionally focused.

As heard by us

A sober look at AI trust, compliance, and decentralized finance.

The episode keeps its focus on AI trust where it matters most: finance, compliance, and the cost of being wrong. It uses precision, recall, false positives, false negatives, and the confusion matrix to make that case concrete, then widens out to blockchain and decentralized…

Read the full review in PlayNext →

Why you'd press play

For a measured look at trust in AI across finance, blockchain, and DeFi.

Read the full recommendation in PlayNext →

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from The Brave Technologist