It’s not an exaggeration to say that machine learning has taken the world by storm in the last ten years.
Researchers are rapidly adopting machine learning in clinics for various medical applications, including diagnosis, prognosis and therapy. Machine learning (particularly deep learning) is well suited to automate many of the repetitive tasks in Radiation Oncology.
Dr. Michael Douglass, PhD, CMPS, MACPSEM
Medical Physicist & Research Fellow
Royal Adelaide Hospital,
University of Adelaide,
Australian Bragg Centre for Proton Therapy and Research
Part 1. Clinical Applications and basics
One task rapidly being replaced by automated models is organ contouring. 3D contouring the cancer target for treatment and surrounding normal organs (referred to as organs at risk, OaR) is usually time-consuming for the radiation oncologist (Figure 1(a)). It’s particularly labour-intensive when the doctor has to contour a CT scan image data set for a complex site, such as when treating the head and neck region (Figure 1(b)).
Before planning a patient’s treatment may begin, the radiation oncologist is responsible for outlining multiple CT scan image planes, the cancer target to eradicate and organs at risk that must receive less than a prescribed dose limit. The successful outcome of the patient’s radiation therapy is highly dependent on the correct outlining of these volume descriptions and the accuracy of the treatment delivery that follows.
The volume outlining process is limited by a certain degree of subjectivity and variability between radiation oncologists. This is due to their individual training, medical practices and experience. Differences in outline practices between radiation oncology departments and cancer site protocols can also lead to further contour variability. It’s become particularly problematic when the images and patient record data are shared between referral centres.
Many centres have adopted an anatomical atlas-based contouring procedure to overcome the difficulties in volume outlining and shorten the task time. This approach has partly addressed these issues, but we still need a better automated method to assist the radiation oncologist.
Atlas-based contouring (or atlas-based segmentation) uses a representative “atlas” of multiple, curated contours created from previous patients. When the atlas is used to outline images for a new patient, the model pulls a contour set from a patient in the atlas, which is similar to the CT, and then uses deformable image registration techniques to adapt the contours. While the atlas-based segmentation method may improve the efficiency of contouring tasks in the treatment planning workflow, it remains a relatively simple approach and lacks sufficient robustness and accuracy. The radiation oncologist still needs to spend time refining the contours generated using an atlas approach to be of a clinically acceptable level.
Utilising machine learning and artificial intelligence
Nicholas Hindley previously published an introduction to machine learning on this website:
See:
An Introduction to Unsupervised Learning
Deep Learning and the Era of Artificial Intelligence
In simple terms, machine learning models enable a computer to learn patterns in data without being explicitly programmed to do so. It can then process previously unseen data according to those same rules. Image processing tasks typically require more sophisticated machine learning models. They are called deep learning networks and are often likened to the architecture of neurons in the brain. The deep learning networks are generally trained by using large amounts of matched-pair data:–
(i) for the input image which needs to be processed (a picture of a cat, for example); and
(ii) the expected output for that corresponding image (an output indicating the photo contains a cat).
The matched pairs are sent through the untrained deep learning model (usually in batches). The model parameters are optimised and updated to minimise the difference between the “true” output and the model’s predicted output.
In essence, the model learns the patterns that link the input to the output to predict the output on a previously unseen data point. For example, a deep learning model may be used to predict the type of animal present in a photo. The deep learning network takes a photo as an input and produces an output that indicates the probability that a given animal appears in that image.
A Revolution in Deep Learning Segmentation
A watershed moment occurred in 2015 when Ronneberger et al. [1] published a revolutionary deep learning architecture called “U-Net”. The U-Net model takes an image as an input and produces a label map, designating what pixel belongs to a particular object in the image. U-Net is generally designed to create a label map with the exact dimensions of the input image (Figure 2).
Reference: arXiv:1505.04597v1 [cs.CV] 18 May 2015
This model is generally quicker to train than the alternatives and requires fewer matched training pairs to produce the same segmentation accuracy. Since the U-Net article was published, countless papers have used this model and variants thereof for segmentation applications in Radiation Oncology. We have reached the point where these deep learning models can contour with the same accuracy as the inter-observer variability of the data on which it was trained.
These models are only as good as the manually contoured CT slices in which they were trained, highlighting the need for high quality, curated training data. This is still a popular area of research, but at the very least, deep learning segmentation models have been demonstrated to dramatically improve the efficiency of a treatment planning workflow. It’s achieved by significantly reducing contouring time while achieving an accuracy close to a human. But, importantly, it also reduces the variability in the quality of contours produced by radiation oncologists, thus improving the quality of care for patients.
Since the accuracy of these models is limited by how good the data is that it’s trained on, large amounts of quality training data needs to be collected. In the case of auto-segmentation, the training data is generally produced by a radiation oncologist or other expert, manually contouring each organ at risk, slice by slice with a level of detail better than what would be considered “acceptable clinical quality”. As many reading this article would know, this is time-consuming and needs to be collected from hundreds of patients to get sufficient data to train a model with acceptable accuracy. Even with this much data, if a single oncologist curated the data, it doesn’t necessarily reflect the different contouring styles of other oncologists.
Synthetic Data Approach
A new approach used in other research fields is to generate “synthetic data”. This involves generating “fake” data, similar to the actual data intended for training the deep learning model. This technique has been utilised in other research areas, such as automotive machine learning models (Figures 3 and 4). Simulated scenes of roads with other cars and pedestrians are generated to help train autonomous vehicles
An important advantage of this approach is that the software can also create a corresponding, high-quality target output that trains the model when generating a fake training image. This means an expert doesn’t have to manually contour or label every image in the training data. In the automotive example, for every synthesised image of the road with cars, pedestrians, trees, animals, buildings, etc., a pixel-wise mask corresponding to each of these objects can be generated automatically at the same time as the image with an accuracy limited only by the resolution of the render engine.
In principle, this approach would be ideal for improving segmentation models’ accuracy in radiation oncology. It removes the radiation oncologists’ subjectivity and variability of contouring styles. However, it is difficult to use this technique to train deep learning auto-contouring models in radiation oncology.
While this approach has not been directly applied in radiation oncology auto-contouring, one method used is to generate synthesised MRI or CT datasets to overcome the limited training data problem. This approach often uses a machine learning architecture called Generative Adversarial Networks (GANs), which can generate randomised examples of images similar in context to those shown to it during training. The machine learning model is essentially two networks in one.
One part of the network tries to generate a fake image, similar to that of the training data it is shown. The second network tries to determine if the image produced by the first network is genuine or a synthesised fake. The two networks work against each other to become better at their respective tasks. After training, the generator network becomes extremely good at producing high-quality fake images indistinguishable from the actual data. The second network reaches the point where it can only tell a fake from a real picture 50% of the time. Essentially, it is randomly guessing.
A simple example of this can be seen on the website “This person does not exist“, which generates synthesised portraits of people [2]. This machine learning model was trained on thousands of real photos and, once trained, can generate synthesised pictures of people using only a random noise input to control the appearance.
A practical use of this in radiation oncology is that, given a small amount of MRI/CT data from a small group of patients, a GAN can generate more training data by allowing the network to “imagine” new data. This data can then be manually contoured (or perhaps automated) and used to train an auto-contouring network with more data.
Another application of this type of network is converting MRI data to CT data. This could offer the possibility of using the MRI data only for radiation dose calculations by the computer treatment planning system (TPS). Current TPS dose calculations are based on the planning CT scan’s electron density/physical density information.
MRI datasets are sometimes combined with CT scan data to assist the organ and target contouring task. Deep learning models can now learn the relationships between a large set of registered CT and MRI scans and synthesise a CT scan suitable for the patient’s TPS dose calculations using MRI scan data. In some cases, the need for the planning CT scan data can be eliminated using this approach. Other examples of this approach include generating high dose CT scans from low dose CT scans and upscaling low-resolution CT data to high-resolution scans. One benefit of this technique is that high-quality CT scans can be obtained in some cases with the same imaging dose as a low dose CT, therefore reducing the radiation exposure risk to patients.
Conclusion
Deep learning models require large amounts of example data to learn the patterns which relate the input to the desired output of the model. Obtaining large quantities of medical data to train these models can be time-consuming with other issues such as ethics approvals and expert curation. Data synthesis is a valuable method of increasing the amount of available training data and may help improve the accuracy of deep learning models for various tasks.
In Part 2, machine learning to replace or automate many of the manual QC tasks that radiation oncology staff carry out will be described. Some QC examples are provided to show how machine learning can improve the efficiency, accuracy, and consistency in the staff’s radiation oncology workflow.
References
[1] U-Net. Retrieved from https://doi.org/10.48550/arXiv.1505.04597
[2] Website: (https://this-person-does-not-exist.com/en)
Michael Douglass PhD, 30 April 2022
