a study of x-ray image perception for pneumoconiosis detection ms thesis presentation varun jampani...
TRANSCRIPT
A study of x-ray image perception for pneumoconiosis detection
MS Thesis Presentation
Varun Jampani (200502027)[email protected]
Adviser: Prof. Jayanthi Sivaswamy
Center for Visual Information TechnologyIIIT-Hyderabad
introduction
Medical Imaging Technologies• Forms one of the most effective
diagnostic tools in medicine• Used for planning treatment and
surgery • Several imaging technologies:
PET, MRI, X-ray, Nuclear medicine, Ultrasound etc…
• X-ray is still ubiquitous in clinical practice and will likely remain so for quite some time
Perception in medical imaging• Information in medical images itself not sufficient• Information has to be interpreted in accurate and timely
manner• Sever factors affect reading medical images
• Observer independent factors such as image quality and viewing settings
• Observer dependent perceptual and cognitive factors
The present work deals with understanding of some perceptual and cognitive factors involved in the diagnostic
assessment of Pneumoconiosis.
Pneumoconiosis• Inflammation of lungs• Caused by prolonged inhalation of industrial dust [Mason
and Broaddus 2005]• Formation of scar tissue making lungs less flexible and
porous• Symptoms
• Shortness of breath, cough, restless sleep, chest discomfort.
• Mainly diagnosed through chest x-rays• Effective way to prevent progress of this disease is to get
regular check ups• Deemed to be the most common and serious
occupational lung disease in developing countries like China and India [Wang and Christiani 2003]
Diagnosis of Pneumoconiosis• Complex process and requires certain level of expertize [Morgan et al.
1973]• International labor organization (ILO) classification scheme [IRPAINIR
Committee 1980]• Hierarchy of readers• Each lung is divided into 3 zones – Total 6 zones• Profusion level (concentration of small opacities) is assessed for each
zone
Importance of Perception Research• Radiologist’s interpretation of medical images is highly
subjective• Inter and Intra observer variations [Krupinski 2000]• At least half of errors made in clinical practice are
perceptual in nature [Krupinski et al. 1998]• The ultimate aim of all perception research is to improve
diagnostic accuracy by reducing errors due to perceptual and cognitive factors
• Understanding perceptual factors helps in development of • Better image acquisition and viewing systems• Computer aided diagnostic (CAD) tools• Better training regimens for resident radiologists
Objectives of present work• We are interested in answering the following research
questions1. What is the role of expertize and contralateral symmetry
information present in chest x-rays on the diagnostic error, time and eye movements of the observer?
2. Does the distribution of eye fixations change with observer error and observer assessment of pneumoconiosis?
3. What is the inter observer and intra observer variability of eye fixations?
4. What is the role of anatomical features in attracting the gaze of the observers?
5. What is the role of bottom up image features in attracting the gaze of the observers?
6. How do the visual strategies of the observers of different expertize levels change with time?
methodology
Methodology• Eye tracking experiment is chosen method of study• Experiments were conducted in a room dedicated to eye
tracking experiments• Experimentally manipulated variables
• Expertize• Contralateral symmetry• Disease level
• Recorded variables• Profusion categorization for each lung zone• Time of analysis• Eye fixations
Experimental Images• Good quality PA chest x-ray images of pneumoconiosis• Single and double lung images to study the role of
contralateral symmetry• Different disease stages
Disease StageDouble Lung
ImagesSingle Lung
Images
Stage 1 3 2
Stage 2 4 6
Stage 3 10 8
Total: 17 Total: 16
Participant (observer) details• Expertize varying from novices to staff radiologists• Total number of observers: 23
Observer Category Number
Novices 3
1st year medical students 3
2nd year medical students 3
3rd year medical students 3
4th year medical students 3
Resident Radiologists 4
Staff Radiologists 4
Eye tracker settings• Remote head free eye
tracker• Model: SR Research
Eyelink 1000• Mean spatial accuracy of
eye tracker is 0.50 visual angle and sampling rate was 500 Hz.
• 17 inch LCD monitor• Approximate distance
between the observer and the screen was around 60 cm
Experimental Procedure• With-in subject design• Steps in experiment
1. Consent form
2. Training session
3. Cover story
4. 9-point camera calibration
5. All 33 experimental images were shown one after other to each observer
• Unlimited viewing time• After viewing is done, observer has to note down profusion level for
each lung zone in the report form given to him/her
• Eye-movement data, response times and profusion levels were recorded for each observer and for each image
Eye movement terminology
Term Meaning
Fixations Points on the image where observers look (fixate)
Saccades Straight line paths between different fixations
Saliency Likelihood of an image location to be fixated
Saliency map Image with each pixel value representing saliency at that location
Heat map Saliency map overlaid onto the original image
Sample saccade maps
Role of expertize and contralateral symmetry
Research Questions
1. What is the role of
expertize and contralateral symmetry information
on
the diagnostic error, time and eye movements of the observer?
2. Does the
distribution of eye fixations
change with
observer error and observer assessment of pneumoconiosis?
Some existing results on pneumoconiosis diagnosis
• Image quality plays serious and significant role [Pearson et al. 1965, Reger et al. 1972]• Marked tendency to award higher readings to under-penetrated
films while the opposite is true for over-penetrated films• Experienced radiologists are more able to adjust for unsatisfactory
film quality
• Substantial inter-reader and some intra-reader variation in assessment of pneumoconiosis [Reger et al. 1972, Kruger et al. 1974]• Experienced radiologists have more consistency
• No eye tracking experiments on pneumoconiosis
Contralateral symmetry in chest x-rays• Detection of symmetry is one of the characteristics of human visual
perception• Important role in face perception [Chen et al. 2007] and
attractiveness [Grammer et al. 1994]• Contralateral subtraction technique
• Proved to be useful for highlighting tumor regions in chest x-rays [Tsukuda et al. 2002]
• Also used in computer aided analysis of tumors in chest x-rays [Li et al. 2000]
• No empirical studies on the role of contralateral symmetry (CS) in diagnosing lung diseases.
Source: Tsukuda et al. 2002
Analysis of observer performance• Sum of absolute differences
• Observer error for each observer is obtained by averaging over all images
Average sum of absolute differences
Analysis• Observer error for double lung images is seen to vary
significantly with expertise, which was confirmed by Kruskal-wallis test (χ2(6) = 13.38, p = .038)
• Thus, there is decrease in error with increase in expertize• The observer error for single lung images (Mdn = 0.813)
is significantly higher than that of double lung images (Mdn = 0.620). (wilcoxon signed rank test: Z = 3.13, p < .001)
• Significant difference between doctors (Mdn = 0.38) and non-doctors (Mdn = 0.18) when considering the difference of observer error between single and double lung images. (Mann-Whitney test: U = 28, p = .038) Doctors: residents and staff
Non-Doctors: Other groups
Penalize Over and Penalize Under• Penalize Over: Number of times an observer has given a
profusion rating higher than that of ground truth.• Penalize Under: Number of times an observer has given
a profusion rating lower than that of ground truth.• Over-estimation and Under-estimation
Penalize over• More penalize over in double lung images (Mdn = 0.31)
than in single lung images (Mdn = 0.25) (Wilcoxon signed rank test: Z = 2.13, p = .033)
Penalize Under• More penalize under in single lung images (Mdn = 0.34)
than in double lung images (Mdn = 0.28) (Z = 3.89, p < .001)
• Thus CS information is helping in not under-estimating profusion values
Inferences• CS plays a significant role in the diagnosis of
pneumoconiosis and its role is more important in doctors than in the case of non-doctors.
• A previous study [Rockoff and Schwartz 1988] on underestimation of asbestosis (a variant of pneumoconiosis)
• Thus, CS information helps by reducing the tendency of giving less profusion ratings
• More experiments required to study at what level (image/zonal/local) this CS information is being used
Time Analysis• Doctors (Mdn = 16.69s) took less time than non-doctors
(Mdn = 33.24s) (Mann-Whitney test: U = 21, p = .011)• Time taken for double lung images (Mdn = 30.84s) is less
than double the time taken for single lung images (Mdn = 38.38s) (Wilcoxon signed rank test: Z = 4.19, p < .001)
Eye fixation analysis• Average saccade velocity of doctors is significantly higher
than that of non-doctors (Mann-Whitney test: U = 20, p = .01)
• Average saccade amplitude is also significantly higher for doctors than that of non-doctors (U = 24, p = .022)
• Doctors seem to be moving eyes more quickly and over more distances compared to non-doctors.
Fixations vs. observer ratings• Average percentage fixation time in lung zones with observer
ratings of 1 and 2 is significantly higher than in the lung zones with observer ratings of 0 and 3, in both single and double lung images (Mann-Whitney test: p < .001).
• Zones considered by the observer as definite normal (profusion rating – 0) and definite abnormal (3), are less viewed when compared to that of other zones
Fixations vs. observer error• Significantly less time is spent on those zones with
absolute error of 3 when compared to that of zones with absolute errors of 0,1 and 2.
• This shows the importance of careful analysis of each lung zone
Inferences• Doctors are quick and efficient where as non-doctors are
slow and inefficient• For good diagnostic results, all zones should be looked
carefully.• X-rays should not be speed read
• Some of these results may not be applicable to x-rays of localized lung diseases such as lung cancer etc.
What attracts observer’s eyes while reading chest x-rays of pneumoconiosis?
Objectives• To get some insights into the factors guiding the attention
of the observers with different expertize levels• We mainly concentrate on the study of the role of
anatomical features and bottom-up saliency in guiding the fixations of the observers
• Long term goal: Develop a system which predicts fixations of observers on a given chest x-ray
• Can be done by analyzing the image features underlying the fixation points of radiologists.
Visual attention• Visual Attention: Process of selectively attending to an
area of visual field while ignoring the surrounding visual areas
• Mostly done by actively moving eyes over the visual scene
• The eye movement control is mostly unconscious• In general, where radiologists attend to in medical images
differs from what they think they have attended to
Top down and bottom up influences• Bottom up influences
• Dependent on the features of visual stimulus
• Independent of the observer• Stimulus driven or exogenous
attention
• Top down influences• Image independent factors
such as given task or goal and knowledge of the observer
• Goal-driven or endogenous attention
Eye movement recordings of an observer over a picture while performing different tasks
(source: Yarbus 1967)
Computational models of visual attention• Provides computational details of the process of visual
attention• Many existing models are biologically motivated• Output of any computational visual attention system is
saliency map• Many computational models have been proposed• Most are bottom-up models
Itti-Koch Model
Source: Itti and Koch 2000
Eye tracking research on chest x-rays• Most work done on chest x-rays of localized lung diseases
like tumors• Large areas in chest x-rays are not sampled by fixated
[Kundel 2000]• Radiologists move eyes in a pattern that is neither random
nor the same as that of a layman [Kundel and Wright 1969]
• Evolution of fixation pattern from that of an untrained person to that of a radiologist [Kundel and La Follete 1972]
Global focal model of visual search• 3 main components
• Overall pattern recognition• Focal attention to image detail• Decision making
• Initial global response involving entire retina followed by a series of checking fixations
Source: Nodine et al. 1987
Data Analysis in present study• Only first 80 fixations are considered for each observer• Non-parametric statistical tests are used as most data
didn’t pass the Normality test• P-values less than 0.05 are considered significant• Two tailed p-values are considered whenever two groups
are compared
Expertize Group Mean number of fixations
Novices 113.02
Medical Students 87.99
Resident Radiologists 62.26
Staff Radiologists 47.66
Total Mean : 79.77
ROC analysis for comparing saliency maps with fixations• An ROC metric is used to evaluate the performance of
saliency maps to predict eye fixations• Saliency map from the fixation locations of one observer
is treated as a binary classifier on every pixel in the image• Saliency maps are thresholded such that given
percentage of image pixels are classified as fixated and the rest are classified as not fixated.
• The fixations from remaining observers are treated as ground truth
• Standard approach used in eye tracking studies
Thresholded saliency maps
Top: A saliency map
Right: Corresponding saliency map thresholded to different percentage of pixels
10% salient
20% 80% 90% 30% 70%
40%
50%
60%
ROC analysis• ROC curve is drawn by varying the threshold• Area under ROC curve indicates how well the saliency
map of one observer can predict the ground truth fixations (fixations of remaining observers)
• The more the ROC area, the better is the predicting capability of the observer saliency map• For perfect classifier: ROC area – 100• For random classifier: ROC area - 50
An example ROC curve• 33% fixations in top 10% salient regions• 58% fixations in top 20% salient regions• 74% fixations in top 30% salient regions• 85% fixations in top 40% salient regions• 92% fixations in top 50% salient regions• 96% fixations in top 60% salient regions• 98% fixations in top 70% salient regions• 100% fixations in top 80% salient regions
Average AUC = 79.28 (fair accuracy)
Inter observer fixation consistency• One of the aims is to automatically detect the areas of
interest for the radiologists• A basic assumption behind this is that all the observers
would look at similar locations in a given image• This assumption should be validated
Does different observers fixate at same locations, in a given chest x-ray?
Previous research• It has been shown [Kundel 2008] that, while detecting lung nodules,
even though different observers have different scan paths, the distribution of their eye fixations is similar
• [Judd et al. 2009] found the good consistency between the eye fixations of different observers while free viewing the natural images
Source: Kundel 2008
Human Saliency Maps• Fixation maps are convolved with Gaussian to get human
saliency maps• Fixation points with more duration are more emphasized
Human saliency map of an observer: Fixation map convolved with Gaussians; and saliency map overlayed on the original chest x-ray
Results: Inter-observer consistency• Median AUC for all observers: 79 (reasonably good
accuracy)• More agreement in fixations among the observers of lower
expertize groups than that of higher expertize groups• Thus more common
factors guiding the fixations of lower expertize groups
Intra-observer fixation consistency• Since all images are PA chest x-rays, we can expect
some intra observer fixation consistency also• What is the consistency of eye fixations of an observer
while diagnosing pneumoconiosis?
Does an observer fixate at same locations, in different x-ray images?
Results: Intra-observer consistency• Median AUC for all the observers is 80.1• AUCs seems to be decreasing with increasing expertize
Inter vs. Intra-observer fixation consistency• AUCs corresponding to intra-
observer analysis (Mdn = 80.1) are significantly higher than those corresponding to inter-observer analysis (Mdn = 79) (Wilcoxon signed rank test: Z = 29.5, p < .001)
• Observer reading style and non-image specific features such as anatomical features seem to be playing more role in guiding the fixations of observers
Remarks• Reasonably good consistency of both inter and intra
observer eye fixations• Thus eye fixations of an observer can indicate important
regions in an image and helps in predicting the eye fixations of other observers
• Higher expertize groups seem to have divergent image-specific visual strategies compared to that of lower expertize groups
Role of image features in predicting fixations• Aim is to find image features which can be used to predict
fixations of observers• Image features
• Low-level: pixel intensity, color, orientation etc.• Mid-level: blobs, holes etc.• High-level: anatomical structures such as ribs, heart etc.
• Features studied in present study• High-level anatomical features• Low-level bottom-up image features
Role of anatomical features• Aim is to determine the role of anatomical features in
attracting the gaze of the observers• Anatomical features considered
• Lung, rib and inter-rib regions
• Top down knowledge
Regions of Interest• Lung region and inter-rib regions are generally considered
regions of interest for radiologists• Not yet studied whether observers really look at these
regions more
Segmentation of lungs and ribs• We used Euler number based thresholding [Wong and
Ewe 2005] to get roughly segmented lung regions, and then we used an active contour method, similar to the approach in [Annangi et al. 2010], to finely segment lung regions
• Ribs are manually segmented as existing methods are not good enough
Anatomical distribution of fixations
Fixation density in different anatomical regions for different expertize groups
• Different regions have different areas• Thus, normalize fixation percentages with respect to area
Results and Observations• Fixation density in lung regions is significantly well above the
fixation density of entire image• No significant relation between expertize levels and fixation
density of different anatomical regions• Both inter-rib and rib regions are fixated almost equally• Anatomical left regions have more fixation density compared to
anatomical right regions
Fixation density in different anatomical regions for different expertize groups
Remarks• Thus, lung regions are given more importance as
expected• Contrary to popular belief of importance to inter-rib
regions, rib regions are also given same importance• Reason for left anatomical region dominance is yet to be
studied
ROC analysis using anatomical saliency maps• Anatomical saliency maps
• Saliency maps with more saliency over the corresponding anatomical regions
• ROC analysis considering anatomical saliency maps as classifiers and observer fixations as ground truth
• Useful for comparisons with inter and intra observer fixation consistencies
Saliency over Inter-rib regions
Saliency over Rib regions
Saliency maps overlayed onto the original x-ray images
Thresholded anatomical saliency maps
• Rib saliency map: More saliency on ribs• Inter-rib saliency map: More saliency on inter-rib regions• Random lung saliency map: random saliency inside lung
regions
A sample x-ray and corresponding rib, inter-rib and random lung saliency maps thresholded to different percentage of pixels
Results• Wilcoxon signed rank test showed no significant
difference between AUCs corresponding to rib, inter-rib and random lung saliency maps
• In addition, there is a good correlation between AUCs related to rib, inter-rib and random lung saliency maps
• Thus ribs and inter-rib regions are given equal importance and same importance as any other random point inside lung region
Comparison with fixation consistencies
• Thus, much of fixation consistency can be explained by the importance given to lung regions
• Still significant difference between AUCs of random lung saliency maps and that of inter observer fixation consistency
• Median AUC of 74.3 for random lung saliency maps shows good role of lung regions in attracting gaze
• AUC for inter-observer fixation consistency = 79.0
Role of bottom-up saliency• Bottom-up saliency: Saliency due to image dependent
and observer independent factors• Several computational bottom-up saliency models in
literature• We used 10 state-of-art models in present study
What is the role of bottom-up saliency in attracting the gaze of the observers?
Previous Research• Several studies have shown the importance of bottom-up
saliency in guiding the visual attention of observers while viewing natural images.
• A recent study [Matsumoto et al. 2011] on brain CT images shows the importance of bottom-up saliency in attracting the gaze of neurologists
Saliency models considered1. Itti-Koch Saliency model (IK) [Itti and Koch 2000]2. Graph based visual saliency (GBVS) [Harel et al. 2006]3. Image Signature (SIG) [Hou et al. 2012]4. Spectral residual approach (SR) [Hou et al. 2007]5. Dynamic visual attention (DVA) [Hou et al. 2008]6. Adaptive whitening saliency (AWS) [Garcia-Diaz et al. 2009]7. Attention based on information maximization (AIM) [Bruce
and Tsotsos 2009]8. Saliency based on local self-resemblance (SDSRL) [Seo and
Milanfar 2009]9. Saliency based on global self-resemblance (SDSRG) [Seo
and Milanfar 2009]10. Context-aware saliency (CA) [Goferman et al. 2010]
BU saliency maps of a chest x-ray
Original image and corresponding bottom-up saliency maps (thresholded to different percentage of pixels)
ROC Analysis• With saliency maps as classifiers and observer fixations
as ground truth• GBVS and SIG models significantly outperform other
saliency models• AUCs related to GBVS
(Mdn = 77.1) are significantly higher than those of SIG saliency maps (Mdn = 73.8) (Wilcoxon signed rank test: Z = 2.0, p < .001)
Comparisons
Results• AUCs for GBVS saliency maps are significantly higher
than those of random lung saliency maps (Wilcoxon signed rank test: Z = 13.0, p < .001)
• The difference between GBVS and inter-observer saliency map AUCs is not significant (p = .0162)
• Thus, GBVS explains most of the observed inter-observer fixation consistency
• We can say that bottom-up saliency is an important factor in guiding the eye fixations of the observers.
Effects of Time
How do the inter-observer fixation consistency, intra-observer fixation consistency, role of lung regions and role of bottom-up saliency (GBVS) change with time (number of
fixations)?
•Would give more insights into the visual strategies used by the observers
Effects of Time
Inter-Observer Fixation Consistency Intra-Observer Fixation Consistency
Role of Random lung saliency maps Role of bottom up saliency
Concluding Remarks• First few fixations seems to be playing important role in
choosing the visual strategy, appropriate for the given image
• Experience seems to be helping observers to develop new visual strategies based on the image content so that they can quickly and efficiently assess the disease level
• Bottom-up saliency (GBVS) is shown to play an important role in attracting the gaze
• Lung regions attract most of the attention• Whereas, the role of bottom-up influences is more during
the initial few fixations, the role of top-down influence seems to be more during the latter part of the viewing.
Towards a new saliency model
Objective• To develop a gaze prediction system for chest x-rays of
Pneumoconiosis• Useful for the development of
• CAD systems• Training tools for radiologists
• Based on experimental results we developed a new saliency model by extending GBVS saliency model
A new saliency model• Bottom-up saliency map
• Extracted using GBVS saliency model
• Top-Down saliency map• Importance given to lung regions
• Both maps are combined with simple multiplication as in [Peters and Itti 2007]
• We call resulting saliency model as ‘Extended Graph based Visual Saliecy’ (EGBVS)
Extended graph based Visual Saliency
Sample EGBVS saliency maps
Assessment of proposed model• Used same ROC analysis as earlier
Results• From the above chart, it can be seen that the median
ROC areas for EGBVS model are higher than those of GBVS model, for all the participants
• Wilcoxon signed rank test showed that the ROC areas for EGBVS (Mdn=81.3) are significantly higher (Z=2.0, p<0.001) than those of GBVS (Mdn=77.1), for all the observers
• No significant differences of EGBVS AUCs across different expertize groups
• In other words, AGBVS model performs significantly better than GBVS model in predicting the eye fixations of the observers
Comparisons with fixation consistency
Results• AUCs related to EGBVS (Mdn = 81.3) are significantly
higher than those related to inter-observer fixation consistency (Mdn = 79) (Wilcoxon signed rank test: Z = 18.3, p < .001)
• No significant difference between AUCs related to EGBVS and those related to intra-observer fixation consistency
• Thus, EGBVS saliency maps performs significantly better than (2.9% increase in AUC) human saliency maps (of other observers) in predicting the eye fixations of the observers
Concluding Remarks• Simply modifying GBVS saliency maps with importance to
lung regions resulted in significant improvement in accuracy
• EGBVS saliency model seems to explain the observed inter-observer fixation consistency completely
• Need to incorporate other top-down influences such as influences specific to expertize etc…
Conclusion and future directions
Main conclusions in present work• Expertize and CS seems to play an important role in
diagnosis of pneumoconiosis• CS seems to be helping in reducing the general tendency
of giving less profusion ratings• Despite being specialized task, the bottom-up saliency
seems to be playing important role in attracting fixations• Lung regions attract most the attention• Lower expertize groups seem to be using same visual
strategies independent of image content• Higher expertize groups are able to develop different
visual strategies depending on the image content, so that they can quickly and efficiently assess the disease level
Future Directions• Understanding the level at which the CS information is
helping observers• Studying the role of other top-down influences• Finding the relative roles of bottom-up and top-down
influences• Study if the present results extend to diagnosing localized
lung diseases such tumors• Incorporating these results in developing CAD tool for
pneumoconiosis• Long term goals would be to develop
• an automated diagnostic system for pneumoconiosis• A assistive system for radiologists based on eye tracking
Related Publications / Conferences• V. Jampani, Ujjwal & J. Sivaswamy, Assessment of Computational Visual
Attention Models on Medical Images. Indian Conference on Vision, Graphics and Image Processing, Mumbai, India, Dec. 2012. (to be published)
• V. Jampani, V. Vaidya, J. Sivaswamy and L. T. Kishore. Role of expertize and contralateral symmetry in the diagnosis of pneumoconiosis: an experimental study, Proc. of SPIE 2011, Vol. 7966, March 2011
• V. Jampani, V. Vaidya, J. Sivaswamy, P. Ajemba and L. T. Kishore, Effect of expertise and contralateral symmetry on the eye movements of observers while diagnosing pneumoconiosis, Medical Image Perception Society Conference, Dublin, August 2011 (MIPS Student Scholar)
Acknowledgments• Special Thanks to
• Prof. Jayanthi Sivaswamy (my adivsor)• Prof. Bipin Indurkhya (played important part in my research)• Vivek Vaidya (helped in data analysis)• Peter Ajemba (helped in data analysis)• GE Global Research, Bangalore (for funding)• Dr. Kishore Taurani (for helpful discussions)• Radiologists at CARE Hospital Hyderabad (for participating in eye
tracking experiments)• Students of Osmania Medical College, Hyderabad (for participating
in eye tracking experiments)• To my friends, parents and brother
