faithful and customizable explanations of black box models · faithful and customizable...
TRANSCRIPT
Faithful and Customizable Explanations of Black BoxModels
Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3
1Harvard University 2Microsoft Research 3Stanford UniversityAIES 2019
Presenter : Derrick Blakelyhttps://qdata.github.io/deep2Read
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20191 / 33
Outline
1 Background
2 Related Work
3 The MUSE Framework
4 Results
5 Conclusion
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20192 / 33
Outline
1 Background
2 Related Work
3 The MUSE Framework
4 Results
5 Conclusion
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20193 / 33
ML for High Stakes Decisions
Medical diagnoses
Recidivism prediction
Air quality/pollution models
Inspiring Big Data stock photos
Finance
Etc etc etc
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20194 / 33
ML for High Stakes Decisions
Medical diagnoses
Recidivism prediction
Air quality/pollution models
Inspiring Big Data stock photos
Finance
Etc etc etc
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20194 / 33
ML for High Stakes Decisions
Medical diagnoses
Recidivism prediction
Air quality/pollution models
Inspiring Big Data stock photos
Finance
Etc etc etc
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20194 / 33
ML for High Stakes Decisions
Medical diagnoses
Recidivism prediction
Air quality/pollution models
Inspiring Big Data stock photos
Finance
Etc etc etc
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20194 / 33
ML for High Stakes Decisions
Medical diagnoses
Recidivism prediction
Air quality/pollution models
Inspiring Big Data stock photos
Finance
Etc etc etc
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20194 / 33
ML for High Stakes Decisions
Medical diagnoses
Recidivism prediction
Air quality/pollution models
Inspiring Big Data stock photos
Finance
Etc etc etc
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20194 / 33
Issues
Proprietary black box models
Uninterpretable models (often deep learning)
Often want explanations of model decisions
Models should be trustworthy
Want easy understanding and validation
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20195 / 33
Issues
Proprietary black box models
Uninterpretable models (often deep learning)
Often want explanations of model decisions
Models should be trustworthy
Want easy understanding and validation
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20195 / 33
Issues
Proprietary black box models
Uninterpretable models (often deep learning)
Often want explanations of model decisions
Models should be trustworthy
Want easy understanding and validation
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20195 / 33
Issues
Proprietary black box models
Uninterpretable models (often deep learning)
Often want explanations of model decisions
Models should be trustworthy
Want easy understanding and validation
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20195 / 33
Issues
Proprietary black box models
Uninterpretable models (often deep learning)
Often want explanations of model decisions
Models should be trustworthy
Want easy understanding and validation
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20195 / 33
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20196 / 33
Problems in Explainable AI
Fidelity
Unambiguity
Interpretability
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20197 / 33
Problems in Explainable AI
Fidelity
Unambiguity
Interpretability
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20197 / 33
Problems in Explainable AI
Fidelity
Unambiguity
Interpretability
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20197 / 33
Outline
1 Background
2 Related Work
3 The MUSE Framework
4 Results
5 Conclusion
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20198 / 33
Related Work
Logic-based approximations of black box
Decision trees
Decision lists
Decision sets
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20199 / 33
Related Work
Logic-based approximations of black box
Decision trees
Decision lists
Decision sets
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20199 / 33
Related Work
Logic-based approximations of black box
Decision trees
Decision lists
Decision sets
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20199 / 33
Related Work
Logic-based approximations of black box
Decision trees
Decision lists
Decision sets
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 20199 / 33
Related Work Shortcomings
Explanations are often ambiguous
Explanations too complicated
Not customizable
Don’t target an important use-case:“How does the model’s decision vary based on patient age?”
Feature subspaces matter
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201910 / 33
Related Work Shortcomings
Explanations are often ambiguous
Explanations too complicated
Not customizable
Don’t target an important use-case:“How does the model’s decision vary based on patient age?”
Feature subspaces matter
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201910 / 33
Related Work Shortcomings
Explanations are often ambiguous
Explanations too complicated
Not customizable
Don’t target an important use-case:“How does the model’s decision vary based on patient age?”
Feature subspaces matter
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201910 / 33
Related Work Shortcomings
Explanations are often ambiguous
Explanations too complicated
Not customizable
Don’t target an important use-case:“How does the model’s decision vary based on patient age?”
Feature subspaces matter
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201910 / 33
Related Work Shortcomings
Explanations are often ambiguous
Explanations too complicated
Not customizable
Don’t target an important use-case:“How does the model’s decision vary based on patient age?”
Feature subspaces matter
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201910 / 33
Related Work Shortcomings
Explanations are often ambiguous
Explanations too complicated
Not customizable
Don’t target an important use-case:“How does the model’s decision vary based on patient age?”
Feature subspaces matter
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201910 / 33
A bigger shortcoming
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201911 / 33
Outline
1 Background
2 Related Work
3 The MUSE Framework
4 Results
5 Conclusion
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201912 / 33
MUSE
Model Understanding through Subspace Explanations
“Differentiable explanation”: how does model vary across featuresubspaces?
Uses 2-level decision sets
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201913 / 33
MUSE
Model Understanding through Subspace Explanations
“Differentiable explanation”: how does model vary across featuresubspaces?
Uses 2-level decision sets
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201913 / 33
MUSE
Model Understanding through Subspace Explanations
“Differentiable explanation”: how does model vary across featuresubspaces?
Uses 2-level decision sets
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201913 / 33
MUSE Output
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201914 / 33
MUSE Output
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201915 / 33
MUSE Framework
Dataset D = {x1, x2, ..., xN}, black box B, class labels CGoal: create 2-level decision set R = {(q1, s1, c1), ..., (qM , sM , cM)}
qi : subspace description; a conjunctionsi : inner logic; a conjunctionci : label assigned by R
ND: candidate set of conjunctions; for the generating desctiprs
DL: same as ND but for inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201916 / 33
MUSE Framework
Dataset D = {x1, x2, ..., xN}, black box B, class labels CGoal: create 2-level decision set R = {(q1, s1, c1), ..., (qM , sM , cM)}
qi : subspace description; a conjunctionsi : inner logic; a conjunctionci : label assigned by R
ND: candidate set of conjunctions; for the generating desctiprs
DL: same as ND but for inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201916 / 33
MUSE Framework
Dataset D = {x1, x2, ..., xN}, black box B, class labels CGoal: create 2-level decision set R = {(q1, s1, c1), ..., (qM , sM , cM)}
qi : subspace description; a conjunctionsi : inner logic; a conjunctionci : label assigned by R
ND: candidate set of conjunctions; for the generating desctiprs
DL: same as ND but for inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201916 / 33
MUSE Framework
Dataset D = {x1, x2, ..., xN}, black box B, class labels CGoal: create 2-level decision set R = {(q1, s1, c1), ..., (qM , sM , cM)}
qi : subspace description; a conjunctionsi : inner logic; a conjunctionci : label assigned by R
ND: candidate set of conjunctions; for the generating desctiprs
DL: same as ND but for inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201916 / 33
MUSE Framework
Dataset D = {x1, x2, ..., xN}, black box B, class labels CGoal: create 2-level decision set R = {(q1, s1, c1), ..., (qM , sM , cM)}
qi : subspace description; a conjunctionsi : inner logic; a conjunctionci : label assigned by R
ND: candidate set of conjunctions; for the generating desctiprs
DL: same as ND but for inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201916 / 33
MUSE Framework
Dataset D = {x1, x2, ..., xN}, black box B, class labels CGoal: create 2-level decision set R = {(q1, s1, c1), ..., (qM , sM , cM)}
qi : subspace description; a conjunctionsi : inner logic; a conjunctionci : label assigned by R
ND: candidate set of conjunctions; for the generating desctiprs
DL: same as ND but for inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201916 / 33
MUSE Framework
Dataset D = {x1, x2, ..., xN}, black box B, class labels CGoal: create 2-level decision set R = {(q1, s1, c1), ..., (qM , sM , cM)}
qi : subspace description; a conjunctionsi : inner logic; a conjunctionci : label assigned by R
ND: candidate set of conjunctions; for the generating desctiprs
DL: same as ND but for inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201916 / 33
Label Assignment
x satisfies some qi ∧ si ∈ R → label = ci
x doesn’t satisfy any rules in R → assigned with default function
x satisfies multiple rules → one rule is picked via a tie-breaker
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201917 / 33
Label Assignment
x satisfies some qi ∧ si ∈ R → label = ci
x doesn’t satisfy any rules in R → assigned with default function
x satisfies multiple rules → one rule is picked via a tie-breaker
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201917 / 33
Label Assignment
x satisfies some qi ∧ si ∈ R → label = ci
x doesn’t satisfy any rules in R → assigned with default function
x satisfies multiple rules → one rule is picked via a tie-breaker
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201917 / 33
Quantifying Fidelity
Simply how often R agrees with B over Ddisagreement(R) =
∑|{x |x ∈ D, x satisfies qi ∧ si ,B 6= ci}|
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201918 / 33
Quantifying Fidelity
Simply how often R agrees with B over Ddisagreement(R) =
∑|{x |x ∈ D, x satisfies qi ∧ si ,B 6= ci}|
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201918 / 33
Quantifying Umambiguity
Goal 1: prevent rules from overlapping too much
goal 2: maximize coverage of the rules
ruleoverlap(R) = number of times a conjunction was repeated in Rcover(R) = number of x covered by some rule in R
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201919 / 33
Quantifying Umambiguity
Goal 1: prevent rules from overlapping too much
goal 2: maximize coverage of the rules
ruleoverlap(R) = number of times a conjunction was repeated in Rcover(R) = number of x covered by some rule in R
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201919 / 33
Quantifying Umambiguity
Goal 1: prevent rules from overlapping too much
goal 2: maximize coverage of the rules
ruleoverlap(R) = number of times a conjunction was repeated in Rcover(R) = number of x covered by some rule in R
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201919 / 33
Quantifying Umambiguity
Goal 1: prevent rules from overlapping too much
goal 2: maximize coverage of the rules
ruleoverlap(R) = number of times a conjunction was repeated in Rcover(R) = number of x covered by some rule in R
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201919 / 33
Quantifying Interpretability
size(R) = number of rule triples
maxwidth(R) = length of longest rule in number of predicates
numpreds(R) = number of predicates in R (non-unique)
numdsets(R) = number of descriptors
featuroverlap(R) = overlap of features in descriptors and inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201920 / 33
Quantifying Interpretability
size(R) = number of rule triples
maxwidth(R) = length of longest rule in number of predicates
numpreds(R) = number of predicates in R (non-unique)
numdsets(R) = number of descriptors
featuroverlap(R) = overlap of features in descriptors and inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201920 / 33
Quantifying Interpretability
size(R) = number of rule triples
maxwidth(R) = length of longest rule in number of predicates
numpreds(R) = number of predicates in R (non-unique)
numdsets(R) = number of descriptors
featuroverlap(R) = overlap of features in descriptors and inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201920 / 33
Quantifying Interpretability
size(R) = number of rule triples
maxwidth(R) = length of longest rule in number of predicates
numpreds(R) = number of predicates in R (non-unique)
numdsets(R) = number of descriptors
featuroverlap(R) = overlap of features in descriptors and inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201920 / 33
Quantifying Interpretability
size(R) = number of rule triples
maxwidth(R) = length of longest rule in number of predicates
numpreds(R) = number of predicates in R (non-unique)
numdsets(R) = number of descriptors
featuroverlap(R) = overlap of features in descriptors and inner logic
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201920 / 33
Quantified Metrics
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201921 / 33
Setting up the Objective Function
Goal: maximize each fi reward function
Wmax = max width of any rule in either ND or DL
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201922 / 33
Objective Function
Find R ⊆ ND ×DL× C to maximize:
M∑i=1
λi fi (R)
subject to:
size(R) ≤ ε1maxwidth(R) ≤ ε2numdsets(R) ≤ ε3
λi : non-negative weight set by user or found via CV.εi : set by user.
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201923 / 33
Objective Function Optimization
Optimization is NP-Hard; instance of Budgeted Maximum CoverageProblem
Use “approximate local search” algo (Lee at al. 2009) for1/5-approximation
Gist: select a rule that maximizes the objective; repeatedly performdelete or exchange operations to optimize the solution set
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201924 / 33
Objective Function Optimization
Optimization is NP-Hard; instance of Budgeted Maximum CoverageProblem
Use “approximate local search” algo (Lee at al. 2009) for1/5-approximation
Gist: select a rule that maximizes the objective; repeatedly performdelete or exchange operations to optimize the solution set
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201924 / 33
Objective Function Optimization
Optimization is NP-Hard; instance of Budgeted Maximum CoverageProblem
Use “approximate local search” algo (Lee at al. 2009) for1/5-approximation
Gist: select a rule that maximizes the objective; repeatedly performdelete or exchange operations to optimize the solution set
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201924 / 33
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201925 / 33
Outline
1 Background
2 Related Work
3 The MUSE Framework
4 Results
5 Conclusion
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201926 / 33
Datasets
1 Bail outcomes (released on bail or not) for 86K defendants
2 High school performance (graduated on time or not) for 21K students
3 Depression diagnoses (depressed or not) 33K patients
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201927 / 33
Datasets
1 Bail outcomes (released on bail or not) for 86K defendants
2 High school performance (graduated on time or not) for 21K students
3 Depression diagnoses (depressed or not) 33K patients
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201927 / 33
Datasets
1 Bail outcomes (released on bail or not) for 86K defendants
2 High school performance (graduated on time or not) for 21K students
3 Depression diagnoses (depressed or not) 33K patients
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201927 / 33
Baselines
Locally Interpretable Model agnostic Explanations (LIME) (Ribeiro,Singh, and Guestrin 2016)
Interpretable Decision Sets (IDS) (Lakkaraju, Bach, and Leskovec2016)
Bayesian Decision Lists (BDL) (Letham et al. 2015)
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201928 / 33
Baselines
Locally Interpretable Model agnostic Explanations (LIME) (Ribeiro,Singh, and Guestrin 2016)
Interpretable Decision Sets (IDS) (Lakkaraju, Bach, and Leskovec2016)
Bayesian Decision Lists (BDL) (Letham et al. 2015)
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201928 / 33
Baselines
Locally Interpretable Model agnostic Explanations (LIME) (Ribeiro,Singh, and Guestrin 2016)
Interpretable Decision Sets (IDS) (Lakkaraju, Bach, and Leskovec2016)
Bayesian Decision Lists (BDL) (Letham et al. 2015)
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201928 / 33
Results
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201929 / 33
Results
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201930 / 33
Outline
1 Background
2 Related Work
3 The MUSE Framework
4 Results
5 Conclusion
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201931 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Shortcomings
Can’t work on image classifiers; needs to be combined with featureextraction from middle layers of NN
What if we value some features more than others?
Asks end users to do a lot of work
create D,NL, and DL setsSet objective function weightsSet ε constraint values
Is 85% tolerable in high stakes situations?
Possibly encourages bad practice
Might be better as an analysis tool for ML developers
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201932 / 33
Lessons Learned
Potentially good idea to build an interpretable approximation of yourmodel using logic rules
Valuable for sanity checking or helping others use model
More work is needed on interpretable black box algorithms
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201933 / 33
Lessons Learned
Potentially good idea to build an interpretable approximation of yourmodel using logic rules
Valuable for sanity checking or helping others use model
More work is needed on interpretable black box algorithms
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201933 / 33
Lessons Learned
Potentially good idea to build an interpretable approximation of yourmodel using logic rules
Valuable for sanity checking or helping others use model
More work is needed on interpretable black box algorithms
Presenter : Derrick Blakely https://qdata.github.io/deep2Read (University of Virginia)Faithful and Customizable Explanations of Black Box Models Himabindu Lakkaraju1, Ece Kamar2, Rich Caruana2, Jure Leskovec3 1Harvard University 2Microsoft Research 3Stanford University AIES 201933 / 33