WP7: Validation and Dissemination
Test and validation studies, scientific publications and dissemination are all included and a strategic part of the DoMore! project.
Validation
When working with machine learning and deep learning, a proper predefined experimental design is crucial to ensure reliable results that are replicable in other labs. The experimental design for most of our projects is based on three main steps; learning, testing and validation, where each step usually employ independent data. Independence is of particular importance in the final validation studies. Ideally, these should be performed in an external lab, but this is in most cases not practically possible, and there are unfortunately very few replication studies published in science.
The Histotyping method (WP2) has been tested on a dataset produced externally, to ensure that the method is invariant to technical artefacts from the preparation of images. We have used multiple scanners to ensure invariance to technical artefacts in any given scanner. The final validation of Histotyping in colorectal cancers will be performed on tissue sections from more than 1000 patients with either stage II or stage III disease from a well described international clinical trial (Quasar 2).
For the methods developed in WP3, tumour delineation and nuclear segmentation, a design with an independent validation study is challenging because an objective ground truth is not available. It is in practice only possible to objectively assess to what extent the automatic segmentation equals the subjective segmentation defined by trained personnel. The final validation of each method in WP3 will therefore be empirically based on the validation studies where the methods are used, i.e., if Histotyping is successfully validated, so is the tumour delineation, as it was used to define the tumour areas to be analysed in Histotyping.
In WP4 we have several very different methods and employ slightly different experimental designs, but again use the three-step setup learning-testing-validation. For ploidy/stroma and Nucleotyping, the validation is explained and discussed in detail in the published papers (see next page).
The automated Gleason grading method, which is based on the established grading system developed by Gleason, is evaluated by comparing with results from several specialised uropathologists. The main challenge here is that there is a relatively high inter-observer variation. Our method do very well when we compare to those cases where the pathologists are in agreement, but one would have to argue that those cases might be the easier ones to grade. The final validation study will therefore be based on the actual outcome of the patients in a new cohort. The challenge in doing so is partly the tumour heterogeneity and partly that the clinical outcome of a prostate cancer patient is not known until 10-15 years after treatment. The validation cohort for this project is yet not defined.
For mitotic index, we have used a public material for learning, and the trained model has been independently tested in a cohort of uterine sarcomas, where we also have manual mitotic counting for comparison. The test is however on the outcome of the patient. We are currently seeking an external material for the final validation, which might be a breast cancer cohort, as this is a cancer type where mitotic index is already implemented in clinical routine.
Scientific publications
We have so far published 9 papers from the DoMore! project and another 8 related papers are published by DoMore! partners during these first half of the project. Our studies have been well received by biomedical journals with a mean impact of 12.4, which we find very satisfactory.
A further 5 manuscripts have recently been submitted for publication.
Dissemination
The overall strategic goal for the DoMore! project’s communication has been to increase awareness about AI among both professionals as the general public.
Images of DoMore! project featured in different media.
Through active promotion efforts towards journalists and editors in the top Norwegian newsrooms, the unit has reached some of the country’s most important media. The results are among others a front-page report and a corresponding TV feature in VG, two front-page reports in Norwegian daily newspaper Dagbladet, a story on the Norwegian public broadcaster NRK’s website, and a mentioning on the cover of Aftenposten’s weekly A-Magasinet.
Raising the level of public knowledge
The amount of interest we have seen so far indicates that we have been successful with our strategic communication goals. More than close to 40 presentations and more 50 news reports in national and international media over the past two and a half years have helped to position DoMore! at the public forefront of Norwegian AI-based cancer research.
ICGI’s Unit for dissemination and visualization provides the communication strategy. The team of communicators consists of a designer, a 3D designer, a web developer, and a writer and are contributing to the dissemination of DoMore! through various platforms and media channels. In addition to the website, we have so far shared popularizations of our research on the Institute’s YouTube channel with close to 45,000 subscribers, on Facebook, Twitter and more recently also on Instagram.
Media monitoring
To increase the efficiency of the outreach efforts, have we used press release tools such as the publication site Mynewsdesk and the media monitoring service Meltwater.
In the next years, we will continue to emphasize national and international media outreach, in addition to maintaining the activities we have described. Simultaneously, we will continue working towards increasing interest among national and international academic communities. We will also continue the work to establish and level an understanding for DoMore! in our own the hospital organisation, with future implementation in mind.
Oslo, 15th February 2020
Navigate to the other Work Package descriptions: