Harvard Researchers Unveil ReXrank: An Open-Source Leaderboard for AI-Powered Radiology Report Generation from Chest X-ray Images

Harvard researchers have recently unveiled ReXrank, an open-source leaderboard dedicated to AI-powered radiology report generation. This significant development is poised to revolutionize the field of healthcare AI, particularly in interpreting chest x-ray images. The introduction of ReXrank aims to set new standards by providing a comprehensive and objective evaluation framework for cutting-edge models. This initiative fosters healthy competition and collaboration among researchers, clinicians, and AI enthusiasts, accelerating progress in this critical domain.

ReXrank leverages diverse datasets such as MIMIC-CXR, IU-Xray, and CheXpert Plus to offer a robust benchmarking system that evolves with clinical needs and technological advancements. The leaderboard showcases top-performing models that drive innovation and could transform patient care and streamline medical workflows. By encouraging the development and submission of models, ReXrank aims to push the boundaries of what is possible in medical imaging and report generation.

The leaderboard is structured to provide clear and transparent evaluation criteria. Researchers can access the evaluation script and a sample prediction file to run their assessments. The evaluation script on the ReXrank GitHub repository allows researchers to test their models on the provided datasets and submit their results for official scoring. This process ensures that all submissions are evaluated consistently and fairly.

One of the key datasets used in ReXrank is the MIMIC-CXR dataset, which contains over 377,000 images corresponding to more than 227,000 radiographic studies conducted at the Beth Israel Deaconess Medical Center in Boston, MA. This dataset provides a substantial foundation for model training and evaluation. The leaderboard for MIMIC-CXR ranks models based on various metrics, including FineRadScore, RadCliQ, BLEU, BertScore, SembScore, and RadGraph. Top-performing models, such as MedVersa, CheXpertPlus-mimic, and RaDialog, are highlighted, showcasing their superior performance in generating accurate and clinically relevant radiology reports.

The IU X-ray dataset, another cornerstone of ReXrank, includes 7,470 pairs of radiology reports and chest X-rays from Indiana University. The leaderboard for this dataset follows the split given by R2Gen and ranks models based on their performance across multiple metrics. Leading models in this category include MedVersa, RGRG, and RadFM, which have demonstrated exceptional capabilities in report generation.

CheXpert Plus, a dataset containing 223,228 unique pairs of radiology reports and chest X-rays from over 64,000 patients, is also utilized in ReXrank. The leaderboard for CheXpert Plus ranks models based on their performance on the valid set. Models such as MedVersa, RaDialog, and CheXpertPlus-mimic have been recognized for their outstanding results in generating high-quality radiology reports.

To participate in ReXrank, researchers are encouraged to develop their models, run the evaluation script, and submit their predictions for official scoring. A tutorial on the ReXrank GitHub repository streamlines the submission process, ensuring researchers can efficiently navigate it and receive their scores.

In conclusion, Harvard’s introduction provides a transparent, objective, and comprehensive evaluation framework; ReXrank is set to drive innovation and collaboration in the field. Researchers, clinicians, and AI enthusiasts are invited to join this initiative, develop their models, and contribute to the evolution of medical imaging and report generation. 

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter..

Don’t Forget to join our 47k+ ML SubReddit

Find Upcoming AI Webinars here