EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation
Milos Vukadinovic, Xiu Tang, Neal Yuan, Paul Cheng, Debiao Li, Susan Cheng, Bryan He, David Ouyang
2024-10-16
Summary
This paper introduces EchoPrime, a new AI model designed to interpret echocardiography videos, which are used to assess heart health, by analyzing multiple video views for better accuracy.
What's the problem?
Echocardiography is a common method for checking heart function, but many existing AI models only look at one view of the heart at a time. This limits their ability to provide a complete picture and can lead to less accurate assessments because they miss important information that can be seen from different angles during a full exam.
What's the solution?
EchoPrime addresses this issue by using a multi-view approach, meaning it analyzes videos from different angles all at once. It was trained on over 12 million pairs of videos and reports, allowing it to learn how to connect what it sees in the videos with the relevant heart structures. The model uses advanced techniques like contrastive learning and anatomical attention to focus on the most important parts of the video for accurate interpretations. This way, it can provide a comprehensive assessment of the heart's condition.
Why it matters?
This research is important because it could significantly improve how echocardiograms are interpreted. By integrating information from multiple views, EchoPrime can help doctors make better and faster diagnoses, ultimately leading to improved patient care in cardiology. This model represents a step forward in using AI to enhance medical imaging and diagnostics.
Abstract
Echocardiography is the most widely used cardiac imaging modality, capturing ultrasound video data to assess cardiac structure and function. Artificial intelligence (AI) in echocardiography has the potential to streamline manual tasks and improve reproducibility and precision. However, most echocardiography AI models are single-view, single-task systems that do not synthesize complementary information from multiple views captured during a full exam, and thus lead to limited performance and scope of applications. To address this problem, we introduce EchoPrime, a multi-view, view-informed, video-based vision-language foundation model trained on over 12 million video-report pairs. EchoPrime uses contrastive learning to train a unified embedding model for all standard views in a comprehensive echocardiogram study with representation of both rare and common diseases and diagnoses. EchoPrime then utilizes view-classification and a view-informed anatomic attention model to weight video-specific interpretations that accurately maps the relationship between echocardiographic views and anatomical structures. With retrieval-augmented interpretation, EchoPrime integrates information from all echocardiogram videos in a comprehensive study and performs holistic comprehensive clinical echocardiography interpretation. In datasets from two independent healthcare systems, EchoPrime achieves state-of-the art performance on 23 diverse benchmarks of cardiac form and function, surpassing the performance of both task-specific approaches and prior foundation models. Following rigorous clinical evaluation, EchoPrime can assist physicians in the automated preliminary assessment of comprehensive echocardiography.