crowdsourcing for medical image analysis · 2018-05-31 · crowdsourcing for medical image...
TRANSCRIPT
![Page 1: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/1.jpg)
Crowdsourcing for medical image analysis
Veronika Cheplygina @vcheplygina
http://www.veronikach.com
![Page 2: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/2.jpg)
Case courtesy of A.Prof Frank Gaillard, Radiopaedia.org, rID: 8095
Does this person have COPD?
Where is the emphysema?
![Page 3: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/3.jpg)
Case courtesy of Radswiki, Radiopaedia.org, rID: 11384
![Page 4: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/4.jpg)
Case courtesy of Radswiki, Radiopaedia.org, rID: 11384
![Page 5: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/5.jpg)
Learning curve
Size labeled data
Performance
![Page 6: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/6.jpg)
Learning curve
Labeled data
Performance>10M images
<100 - 5K
CheXNet: 100K
![Page 7: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/7.jpg)
Learn from other data/labels
![Page 8: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/8.jpg)
Learn from other data/labels
https://arxiv.org/abs/1804.06353
![Page 9: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/9.jpg)
Learn from other tasks
Case courtesy of A.Prof Frank Gaillard, Radiopaedia.org, rID: 51158
![Page 10: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/10.jpg)
Which datasets to use? (Most similar? Most different?)
Case courtesy of A.Prof Frank Gaillard, Radiopaedia.org, rID: 51158
![Page 11: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/11.jpg)
Crowdsourcing
![Page 12: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/12.jpg)
You do it all the time!
![Page 13: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/13.jpg)
13Von Ahn, L., & Dabbish, L. (2004).
2004 - The ESP Game
![Page 14: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/14.jpg)
14Von Ahn, L., & Dabbish, L. (2004).
2004 - The ESP Game
![Page 15: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/15.jpg)
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on (pp. 248-255). IEEE.
2009: ImageNet
![Page 16: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/16.jpg)
https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/
![Page 17: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/17.jpg)
![Page 18: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/18.jpg)
![Page 19: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/19.jpg)
![Page 20: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/20.jpg)
20
![Page 21: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/21.jpg)
Crowd: easy taskExpert: difficult task
![Page 22: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/22.jpg)
Case courtesy of A.Prof Frank Gaillard, Radiopaedia.org, rID: 8095
How large are the airways?
![Page 23: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/23.jpg)
Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H. A., & de Bruijne, M. (2016). Early Experiences with Crowdsourcing Airway Annotations in Chest CT. In Large-Scale Annotation of Biomedical Data and Expert Label Synthesis (MICCAI LABELS), pp. 209-218
![Page 24: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/24.jpg)
Work by Dylan Dophemont
![Page 25: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/25.jpg)
https://challenge.kitware.com/#challenge/n/ISIC_2017%3A_Skin_Lesion_Analysis_Towards_Melanoma_Detection
Melanoma classification
![Page 26: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/26.jpg)
26
Human experts: “ABCDE”
• A – Asymmetry
• B - Border
• C – Color
• D – Diameter
• E - Evolving
http://www.skincancer.org/skin-cancer-information/melanoma/melanoma-warning-signs-and-images/do-you-know-your-abcdes#panel1-2
![Page 27: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/27.jpg)
Image analysis project for 1st year students
1. Measure features with algorithms
2. Measure features yourself
3. Evaluate
![Page 28: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/28.jpg)
![Page 29: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/29.jpg)
Principal component analysis of crowd features• Blue = healthy, red = melanoma
![Page 30: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/30.jpg)
Next
• How do different annotators compare?• Is disagreement informative?• How do different features (asymmetry, etc) compare?• …
• Train network to predict crowd features (easy), then melanoma (difficult)
![Page 31: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/31.jpg)
100 images
• ID• Feature_Group_Annotator• Group 7
– 5 features– 6 annotators
• Keratosis, Melanoma, Age, Sex
• Dropbox with images: – https://goo.gl/UPhMpy
![Page 32: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/32.jpg)
@vcheplygina
http://www.veronikach.com
• Need more labeled data• Crowdsourcing successful, but challenging for medical• Transfer learning with easy task?
Thanks to:
IMAG/e, Eindhoven University of TechnologyBIGR, Erasmus MC RotterdamPRLab, Delft University of Technology
?
![Page 33: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/33.jpg)
![Page 34: Crowdsourcing for medical image analysis · 2018-05-31 · Crowdsourcing for medical image analysis. Veronika Cheplygina. @vcheplygina. ... Kuo, W., Tiddens, H. A., & de Bruijne,](https://reader034.vdocument.in/reader034/viewer/2022050522/5fa5cfd048b3b60ce37d07ba/html5/thumbnails/34.jpg)
https://pixabay.com/en/psychics-crystal-ball-fortune-teller-1026092/
https://pixabay.com/en/anatomy-bone-bones-check-doctor-3003099/
34