By Prof Jurgen Futterer & Prof Henkjan Huisman, RadboudUMC (RUN)
Professors Jurgen Futterer and Henkjan Huisman are both affiliated with Radboudumc, Nijmegen, The Netherlands. They have a long-standing collaboration and are part of the EU H2020 project ProCAncer-I team, which aims to develop an AI platform integrating imaging data and models, supporting precision care through prostate cancer’s continuum.
Jurgen chairs the Minimally Invasive Image-Guided Intervention Center (MAGIC, https://magicnijmegen.nl/). His group has its roots in MRI-guided prostate treatment that has expanded to CT-, US-, and MR- image-guided oncologic interventions, for example, biopsy of prostate, bone, and liver; (non-) thermal ablations of pancreas, liver, kidney, prostate and vascular malformations.
Henkjan chairs a group within the Radboudumc AI unit (DIAG, diagnijmegen.nl) on deep learning AI applications for prostate and pancreas cancer and adhesions. His group focuses on diagnostic and interventional AI to improve workflow and accuracy of healthcare. He is the project lead of the EU project PANCAIM and co-lead of the Dutch project FastMRI.
Radboudumc has been pioneering challenges in medical image analysis that became popular after the Grand Challenges for Medical Image Analysis organization at the MICCAI conference in 2007. Radboudumc and Mevis Fraunhofer created Grand-Challenge.org (GC) in 2010 to make it easy for organizers of challenges to set up a website for a particular challenge. It aimed to bring all information on challenges in the domain of biomedical image analysis available in a single place. GC initially allowed Challenge organizers to upload the code that computes the score for submission in the form of a Docker container. This system has been operational since 2017.
GC recently added the option to upload an AI algorithm. Algorithms allow using confidential test data. Whether an algorithm is designed as a submission to a Challenge or as its own entity, physicians and clinical researchers can upload their data, have the algorithm process these, and download the results. Over 70,000 user GC accounts have been created on Grand Challenge from countries all across the globe. Radboudumc processes around 2,000 submissions per month, totaling almost 80,000 evaluated submissions that have been placed on a leaderboard.
The GC team has extended the platform to support various medical viewers that run in the browser and to set up Reader Studies. In a Reader Study, a user is presented with images and questions. Questions can include annotations, for example, “Segment the prostate”. The organizers of the Reader Study can download the results via the website or the API. With Reader Studies, researchers can carry out observer studies or set up annotation efforts that are usually needed to run a challenge. You can even set up training programs for physicians; providing a ground truth can provide immediate feedback after a question has been answered.
PI-CAI (Prostate Imaging: Cancer AI) is an all-new grand challenge, with over 10,000 carefully-curated prostate MRI exams to validate modern AI algorithms and estimate radiologists’ performance at csPCa detection and diagnosis. Key aspects of the study design have been established in conjunction with an international, multi-disciplinary scientific advisory board (16 experts in prostate AI, radiology, and urology) – to unify and standardize present-day guidelines and to ensure meaningful validation of prostate-AI towards clinical translation (Reinke et al., 2021). The 2022 edition of PI-CAI will focus on validating AI at automated 3D detection and diagnosis of csPCa in bpMRI. The PI-CAI challenge realizes some of the goals of ProCAncer-I and more. It involves ProCAncer-I researchers Anindo Saha, and Jasper Twilt, but includes many more from other projects and institutions.
PI-CAI consists of two sub-studies to estimate the performance of the average radiologist at the detection and diagnosis of csPCa in MRI and compare that with AI. Firstly an AI Study (Grand Challenge): An annotated multi-center, multi-vendor dataset of 1500 bpMRI exams (including their basic clinical and acquisition variables) is made publicly available for all participating teams and the research community at large. Teams can use this dataset to develop AI models and submit their trained algorithms (in Docker containers) for evaluation. At the end of this open development phase, all algorithms are ranked, based on their performance on a hidden testing cohort of 1000 unseen scans. In the closed testing phase, organizers retrain the top-ranking 5 AI algorithms using a larger dataset of 7500–9500 bpMRI scans (including additional training scans from a private dataset). Finally, their performance is re-evaluated on the hidden testing cohort (with rigorous statistical analyses), to determine the top 3 AI algorithms for automated 3D detection and diagnosis of csPCa in bpMRI (i.e. the winners of the grand challenge).
Secondly, a Reader Study: 64 international prostate radiologists perform a reader study using a subset of 600 scans from the hidden testing cohort.
