sketch-based interactive segmentation and segmentation editing for oncological therapy monitoring...
TRANSCRIPT
Sketch-Based Interactive Segmentation and
Segmentation Editing for Oncological Therapy Monitoring
Frank HeckelMarch 17, 2015
BVM-Award 2015– PhD Thesis –
2 / 22
Medical Background
Change in tumor size is an important criterion for assessing the success of a chemotherapy RECIST1 1.1: Sum of maximum diameters of target lesions
Relative change
Volume is a more accurate measure Many tumors grow/shrink irregularly in 3D Requires appropriate segmentation
Oncological Therapy Response Monitoring
1 RECIST: Response Evaluation Criteria In Solid Tumors
Complete Response
Partial Response
Stable Disease
Progressive Disease
Disappearance
< -30% -30% … 20%
> 20%
3 / 22
The Segmentation Problem
Ultimate Goal: Automatic segmentation Reproducible results with no effort for the user Solutions for specific purposes Might fail (low contrast, noise, biological variability) Unsolved or insufficient for many real-world problems
Solutions: Manual segmentation Interactive tools Automatic segmentation + manual correction
Drawbacks: Higher effort Lower reproducibility
4 / 22
Interactive Segmentation
Based on common 2D user interaction: drawing contours Segmentation as an object reconstruction problem
Energy-minimizing surface reconstruction from a point cloud based on RBFs
3D surface based on contours from a few slices in arbitrary orientations
Variational Interpolation
𝑓 (�⃗� 𝑖 )=𝑃 (�⃗� 𝑖 )+∑𝑗=1
𝑘
𝑤 𝑗𝜙 (�⃗�𝑖− �⃗� 𝑗 )=h𝑖
5 / 22
Interactive Segmentation
Computation time optimization Shape preserving constraint reduction Parallelization
Robustness improvement Approximation instead of interpolation for resolving
contradictions Detection and consideration of self-intersection points
Main Challenges
6 / 22
Interactive Segmentation
Computation time: Speedup ≈80
Evaluation: Data: 15 liver metastases, 1 liver Participants: 2 experienced radiology technicians
Results
Before1
After2
Metastasis
57,53 s
0.7 s
Liver 629,1 s
8.3 s
1 CLAPACK, 1 thread, no reduction 2 MKL, 4/8 threads, reduction by ≈80%
Manual RBF-based Interpolation
Metastasis
111 s
21 contours
64 s 7 contours Overlap: 75%
Liver 1272 s
106 contours
665 s
22 contours
Overlap: 94%
7 / 22
Segmentation Editing
Segmentation Algorithm
Start
Semi-automatic
AutomaticSegmentation
ResultSatisfying?
Initial Algorithm allows
modification?
SegmentationEditing Algorithmno no
Stop
yes yes
Segmentation Algorithm
InteractiveSegmentation
ResultSatisfying? Stop
yes
no
Most existing methods are low-level and unintuitive in 3D High-level correction has not received much attention in
research
8 / 22
Segmentation EditingSketch-Based Editing in 2D
add
remove
add + remove
replace
9 / 22
Segmentation Editing
Estimate 3D size of the error by the „diameter“ of the edited region in
The Correction Depth
𝑪𝒔𝒖
𝑪𝒔𝒆𝒔
10 / 22
Segmentation Editing
Sample user contour into reference points Move reference points to next slice using a block matching Connect seed points using a shortest-path algorithm
Image-Based 3D Extrapolation
11 / 22
Segmentation Editing
Utilizes the RBF-based interpolation approach Reconstruct the new segmentation with contours in the
edited slice and a start / end slice given by the correction depth
Restrict the new segmentation to the edited region
Image-Independent 3D Extrapolation
12 / 22
Evaluation of Editing Tools
131 representative tumor segmentations in CT (lung nodules, liver metastases, lymph nodes)
5 radiologists with different level of experience
Editing rating score:
Qualitative Evaluation
𝑟 edit=1𝑁
¿
13 / 22
Evaluation of Editing ToolsQuantitative Evaluation
Analyze quality over time Editing quality score:
14 / 22
Evaluation of Editing Tools
Problem: High effort and bad reproducibility of user studies Idea: Replace user by a simulation Benefits:
Objective and reproducible validation Objective comparison Improved regression testing Better parameter tuning
Simulation-Based Evaluation
IntermediateSegmentation
Target Segmentation
Segmentation Editing
Satisfying?
User
Validationno
yes
Stop
Start
Control flow
Data flow
User Input
Previous Inputs
IntermediateSegmentation
Reference Segmentation
Segmentation Editing
Satisfying?
Simulation
Validationno
yes
Stop
Start
Control flow
Data flow
User Input
Previous Inputs
15 / 22
Evaluation of Editing Tools
Step 1: Find most probably corrected 3D error Step 2: Select slice and view where the error is most
probably corrected Step 3: Generate user-input for sketching Step 4: Apply editing algorithm
Simulation-Based Evaluation
16 / 22
Evaluation of Editing ToolsSimulation-Based Evaluation
17 / 22
Partial Volume Correction
Smoothing effect caused by limited spatial resolution (of CT)
Ill-defined border between tumor and healthy tissue, making segmentation an ill-defined problem
Could cause significant differences in size measurements
The Partial Volume Effect
28.4 ml(-27.5%)
39.2 ml 56.8 ml(+44.9%)
18 / 22
Partial Volume Correction
Spatial subdivision into spherical sectors to cover different tissues
Define reference tissue values inside and outside of the object ( and to) per sector
For each sector : compute the weight w of each partial volume voxel
Method
1.0
0.0
0.5
0.75
0.25
𝑤 (𝑉 )=𝑡𝑜 𝑠−𝑣
𝑡𝑜 𝑠− 𝑡𝑖 𝑠
,𝑉∈𝑃 𝑖𝑠∪𝑃𝑜𝑠
𝑉𝑜𝑙𝐿=∑𝑉 ∈𝐿
𝑤 (𝑉 )𝑉𝑜𝑙𝑉71.1 ml70.8 ml
19 / 22
Partial Volume CorrectionSoftware Phantom Results
20 / 22
Partial Volume CorrectionHardware Phantom Results
21 / 22
Partial Volume CorrectionMulti-Reader Data Results
22 / 22
Summary
Contributions: General image-independent interactive segmentation method Efficient and intuitive segmentation editing tools +
methodologies for their evaluation Fast algorithm for compensation of partial volume effects
Future Work: Improve algorithms for irregular and large objects Combine image-based and image-independent editing Make editing simulation more realistic HCI aspects in editing 4D and multi-label segmentations Establish volumetric measurements in clinical routine
Acknowledgement
Thanks to all colleagues at (Fraunhofer) MEVIS, particularly Dr. Jan Moltz, Lars Bornemann, Dr. Hans Meine, Dr. Stefan Braunewell, Dr. Markus Lang, Michael Schwier, Dr. Volker Dicken, Dr. Benjamin Geisler, Olaf Konrad, Wolf Spindler and Prof. Horst Hahn. Special thanks to Dr. Christian Tietjen, Dr. Grzegorz Soza, Andreas Wimmer, Dr. Ola Friman, Prof. Bernhard Preim, Prof. Andreas Nüchter, all clinical partners and the Visual Computing in Biology and Medicine community.An finally, my wife and my children!
Thank [email protected]
Bei Herausforderungen geht es nicht ums Gewinnen, sondern darum, herauszufinden, was für ein Mensch man ist.