ORID: Organ-Regional Information Driven Framework for Radiology Report Generation
Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai
2024-11-21
Summary
This paper introduces ORID, a new framework designed to automatically generate detailed and accurate radiology reports from medical images, helping to ease the workload for radiologists.
What's the problem?
Radiologists often have to write reports analyzing medical images, which can be time-consuming and tedious. Current AI methods for generating these reports mainly focus on improving the structure of the models used but often struggle with integrating information effectively and filtering out irrelevant details from unrelated organs in the images.
What's the solution?
The ORID framework enhances report generation by combining multiple types of information (like image data and descriptive text) while minimizing noise from unrelated organs. It uses a new instruction dataset to train the model better and includes an organ-based cross-modal fusion module that merges information effectively. Additionally, it employs a Graph Neural Network (GNN) to analyze the importance of different organs, ensuring that only relevant information influences the report generation.
Why it matters?
This research is important because it improves the accuracy and efficiency of radiology report generation, which can save time for healthcare professionals and lead to better patient care. By focusing on relevant organ information, ORID can help produce clearer and more useful reports, ultimately benefiting both doctors and patients.
Abstract
The objective of Radiology Report Generation (RRG) is to automatically generate coherent textual analyses of diseases based on radiological images, thereby alleviating the workload of radiologists. Current AI-based methods for RRG primarily focus on modifications to the encoder-decoder model architecture. To advance these approaches, this paper introduces an Organ-Regional Information Driven (ORID) framework which can effectively integrate multi-modal information and reduce the influence of noise from unrelated organs. Specifically, based on the LLaVA-Med, we first construct an RRG-related instruction dataset to improve organ-regional diagnosis description ability and get the LLaVA-Med-RRG. After that, we propose an organ-based cross-modal fusion module to effectively combine the information from the organ-regional diagnosis description and radiology image. To further reduce the influence of noise from unrelated organs on the radiology report generation, we introduce an organ importance coefficient analysis module, which leverages Graph Neural Network (GNN) to examine the interconnections of the cross-modal information of each organ region. Extensive experiments an1d comparisons with state-of-the-art methods across various evaluation metrics demonstrate the superior performance of our proposed method.