Radiology reporting and AI: how NLP transforms reports

As radiologists, our primary job is to create the radiological report. It's the most important thing we produce and is widely considered the gold standard for how we communicate our findings. Think of it as a vital legal document, which is why we have to be incredibly precise. Everything important about a patient’s condition needs to be described in detail. Essentially, this report is the bridge between a physician’s question and our interpretation of the images we’ve analyzed. It’s what makes the whole process work.

Over the past few decades, our field has changed dramatically. The introduction of digital systems like PACS and RIS completely transformed how we work. Suddenly, it was much easier for doctors to get their hands on our reports. The addition of speech recognition technology sped things up even more, allowing us to dictate directly into the system. Despite all this technological progress, the core of our work—the report and the process of creating it—has largely stayed the same.

AI and the future of our work

Now, AI is here, and it's one of the biggest game-changers we've ever seen. We’re already using it for image analysis, like with radiomics. There are so many new tools that can help us spot all kinds of issues—from fractures to tiny pulmonary nodules or even brain hemorrhages. These tools are designed to make us more accurate and more efficient.

The next frontier is AI that understands human language. This is called Natural Language Processing (NLP). The recent progress in this field is incredible, and it's already creating new ways for us to improve the reports we write, ultimately making them even better.

2.0 or improving reporting

Before we dive into all the possibilities of NLP, let's focus on our main goal: creating better reports. The reporting process is incredibly complex, but honestly, not much has changed since radiologists started doing this work. Most of the reports are still just free-text documents. The entire process begins with a clinical question from a doctor. After the patient has their scan, it's our job to answer that question in the report, and a great writing boils down to two things: the content has to be accurate, and the structure has to be easy to read.

That’s why there are so many guidelines out there for reporting. There’s a constant push for standardization, and lately, a big move toward structured reporting. Our professional societies are really encouraging this shift, as moving to a more standardized format can only make the reports more accurate and easier for everyone to understand.

Standardized vs. structured reporting

To truly understand how Natural Language Processing (NLP) can revolutionize radiology, we first need to clear up some confusion about standardized and structured reporting. In many ways, these terms are used interchangeably in the literature, which makes it hard to distinguish between them.

The key difference is simple:

• Standardized reporting is about the content itself. It means everyone uses the same vocabulary and grading scales—for example, the RADS family of scores. It's about making sure the medical language is consistent.
• Structured reporting is about the format and layout of the report. It's about how that medical content is organized and entered. Structured reporting can be as simple as a basic template (a Level 1 solution) or a more advanced, IT-supported system with dropdown menus and pick-lists (a Level 2 solution).

The real power of a Level 2 solution isn't just a cleaner-looking report; it’s in how the data is stored. Unlike free text, which is unstructured and hard to analyze, data from a truly structured system is stored in a way that makes it "FAIR"—Findable, Accessible, Interoperable, and Reusable. This means the information is directly usable for data mining, a task that is incredibly difficult with traditional free-text reports.

The potential of data with NLP

Radiology reports, along with a massive amount of other patient information, are stored in electronic health records (EHRs). While this data is a goldmine for research, education, and quality assurance, most of it remains unused because it’s saved as unstructured free text. Retrieving and analyzing this information is a time-consuming and labor-intensive process. This is where data mining comes in. Data mining uses powerful computer programs to analyze and search through vast amounts of information to find specific data points or hidden correlations. With the rise of AI, we can now mine enormous datasets with unprecedented speed and efficiency.

NLP is a sub-type of AI that focuses specifically on understanding and processing human language, which makes it the perfect tool for analyzing radiology reports. While structured reporting was once seen as the next big thing, its implementation hasn't lived up to expectations. NLP, however, has the potential to be a true game-changer because NLP can be integrated at multiple points in the workflow. It can help create the initial report, analyze existing reports, assist with diagnoses, and even pull relevant patient information from their EHR. The potential is huge, and to get it right, we need to understand exactly how and where to apply it most effectively in the reporting process.

From simple subdivisions to powerful transformers

Just a decade ago, the field of Natural Language Processing (NLP) was fairly straightforward. We generally saw it as having two distinct parts: Natural Language Generation (NLG), which was all about generating text, and Natural Language Understanding (NLU), which focused on comprehending it. The rule of thumb for both was "garbage in, garbage out," meaning the more organized and structured the input data was, the better the output would be.

Everything changed about five years ago, around 2020, with the arrival of transformers. This new technology completely shifted the way we think about NLP. We started using transformer encoder algorithms to handle understanding text (much like the old NLU) and transformer decoder algorithms for generating it (like NLG).

The two biggest names you’ve probably heard are Bidirectional Encoder Representations from Transformers (BERT) and Generative Pretrained Transformer (GPT). You can think of BERT as a powerful encoder and GPT as a powerful decoder. The introduction of transfer learning was also a huge leap forward. This allowed us to apply knowledge learned from one task to another. For example, an AI model that learned to recognize names or negations from a general dataset could then apply that same knowledge to medical information. This innovation made it much less critical to manually label data for every single task, rapidly boosting the capabilities of models like BERT and GPT.

As these algorithms grew more sophisticated and were trained on vast amounts of data, the lines between understanding and generating text began to blur. A prime example is ChatGPT, which was originally designed as a decoding tool but is now fully capable of both generating and understanding text. These powerful systems are now called Large Language Models (LLMs) because they’re trained on a staggering amount of data from books, articles, and the internet.

Some of these LLMs are even becoming multimodal (M-LLMs), meaning they can handle more than just text—they can work with images and other types of data, too. This ability to both understand and generate language opens up a world of possibilities for how we can use them in our work.

Right now, NLP in healthcare is mostly a research tool. You've probably heard all the buzz around using things like ChatGPT in medicine. While it’s fairly easy to apply these powerful models in a research setting—for example, when you need to find a few standardized pieces of information—it gets much harder when you need to be highly accurate across a large group of patients. Still, even with these limitations, this kind of "shallow" NLP is already proving useful in many areas of healthcare research.

In radiology specifically, we're seeing some great examples of NLP's ability to "read" our reports. We're using it for things like:

• Finding cases: Quickly pulling up all reports that mention a specific finding, which is great for research or building patient groups.
• Diagnostic and clinical support: Some tools can even alert a doctor to an urgent finding like appendicitis or pneumonia, or recommend follow-up imaging for a specific type of fracture.
• Quality control: We can use NLP to track things like follow-up appointments for cancer patients or ensure a report is complete.

While text generation tools are still playing catch-up, their potential is enormous, thanks to advancements in models like GPT. We’re already seeing research that uses them for things like quality control and even creating patient-friendly versions of our reports. Although none of these are in widespread clinical use yet, that's likely to change very soon.

The challenges of implementing NLP in healthcare

To be honest, implementing NLP in medicine is a big challenge. Compared to other fields, we have a very low tolerance for errors—a mistake could seriously harm a patient. Because of this, any NLP tool used for decision-making or reporting has to be incredibly accurate, or at the very least, closely supervised. The lack of readily available, high-quality medical data and the demand for near-perfect accuracy are our two biggest hurdles right now.

But there's more to it than just the AI itself. We need to create a better experience for radiologists. We use so many different tools and systems that it can be a mess. We need a central hub—a Graphical User Interface (GUI)—that brings everything together and lets us easily check and supervise what the AI is suggesting. This platform should be vendor-independent, so we're not locked into one company's software, and it should integrate everything, from our PACS and RIS to our EHR.

There's also the big debate between free text and structured reports. Free text gives us flexibility, but structured reports are more reusable for data analysis. For NLP to work best, the data still needs to be clean and well-organized. It’s a mistake to think that AI will suddenly make standardization and structuring unnecessary. The principles of FAIRdata—Findable, Accessible, Interoperable, Reusable—are still incredibly important. The best approach is to start with good data rather than trying to make bad data good later on. While we’re seeing models like ChatGPT challenge these assumptions, the medical field is always more cautious, which means adoption will probably happen more slowly here than in other industries.

Resources:

• Yim, W.W., Yetisgen, M., Harris, W.P., & Kwan, S.W. (2016). Natural language processing in oncology: A review. JAMA Oncology, 2(6), 797–804. https://doi.org/10.1001/jamaoncol.2016.0213
• Lee, S.J., Weinberg, B.D., Gore, A., & Banerjee, I. (2020). A scalable natural language processing approach for inferring BT-RADS categorization from unstructured brain MRI reports. Journal of Digital Imaging, 33(6), 1393–1400. https://doi.org/10.1007/s10278-020-00350-0
• Zeng, Z., Espino, S., Roy, A., Li, X., Khan, S.A., Clare, S.E., et al. (2018). Using natural language processing and machine learning to identify breast cancer local recurrence. BMC Bioinformatics, 19(Suppl 17), 498. https://doi.org/10.1186/s12859-018-2466-x
• Lou, R., Lalevic, D., Chambers, C., Zafar, H.M., & Cook, T.S. (2020). Automated detection of radiology reports that require follow-up imaging using natural language processing feature engineering and machine learning classification. Journal of Digital Imaging, 33(1), 131–136. https://doi.org/10.1007/s10278-019-00271-7