i-Algebra: Towards Interactive Interpretability of Deep Neural Networks

Xinyang Zhang† Ren Pang† Shouling Ji‡ Fenglong Ma† Ting Wang†

†Pennsylvania State University, {xqz5366, rbp5354, fenglong, ting}@psu.edu‡Zhejiang University, sji@zju.edu.cn

Abstract

Providing explanations for deep neural networks (DNNs) isessential for their use in domains wherein the interpretabilityof decisions is a critical prerequisite. Despite the plethora ofwork on interpreting DNNs, most existing solutions offer in-terpretability in an ad hoc, one-shot, and static manner, with-out accounting for the perception, understanding, or responseof end-users, resulting in their poor usability in practice.In this paper, we argue that DNN interpretability should beimplemented as the interactions between users and models.We present i-Algebra, a first-of-its-kind interactive frame-work for interpreting DNNs. At its core is a library of atomic,composable operators, which explain model behaviors atvarying input granularity, during different inference stages,and from distinct interpretation perspectives. Leveraging adeclarative query language, users are enabled to build variousanalysis tools (e.g., “drill-down”, “comparative”, “what-if”analysis) via flexibly composing such operators. We proto-type i-Algebra and conduct user studies in a set of represen-tative analysis tasks, including inspecting adversarial inputs,resolving model inconsistency, and cleansing contaminateddata, all demonstrating its promising usability.

IntroductionThe recent advances in deep learning have led to break-

throughs in a number of long-standing artificial intelligencetasks, enabling use cases previously considered strictly ex-perimental. Yet, the state-of-the-art performance of deepneural networks (DNNs) is often achieved at the cost of theirinterpretability: it is challenging to understand how a DNNarrives at a particular decision, due to its high non-linearityand nested structure (Goodfellow, Bengio, and Courville2016). This is a major drawback for domains wherein theinterpretability of decisions is a critical prerequisite.

A flurry of interpretation methods (Sundararajan, Taly,and Yan 2017; Dabkowski and Gal 2017; Fong and Vedaldi2017; Zhang, Nian Wu, and Zhu 2018) have since been pro-posed to help understand the inner workings of DNNs. Forexample, in Figure 1, the attribution map highlights the mostinformative features of input x with respect to model predic-tion f(x). A DNN (classifier), coupled with an interpreta-tion model (interpreter), forms an interpretable deep learn-

Copyright © 2021, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

ClassifierPrediction

f

?

Interpretation InterpreterEnd User

Input

Figure 1: Interactive interpretation of DNNs.

ing system (IDLS), which is believed to improve the modeltrustworthiness (Tao et al. 2018; Guo et al. 2018).

Yet, despite the enhanced interpretability, today’s IDLSesare still far from being practically useful. In particular, mostIDLSes provide interpretability in an ad hoc, single-shot,and static manner, without accounting for the perception,understanding, and response of the users, resulting in theirpoor usability in practice. For instance, most IDLSes gen-erate a static saliency map to highlight the most informativefeatures of a given input; however, in concrete analysis tasks,the users often desire to know more, for instance,

• How does the feature importance change if some other fea-tures are present/absent?

• How does the feature importance evolve over differentstages of the DNN model?

• What are the common features of two inputs that lead totheir similar predictions?

• What are the discriminative features of two inputs that re-sult in their different predictions?

Moreover, the answer to one question may trigger followupquestions from the user, giving rise to an interactive pro-cess. Unfortunately, the existing IDLSes, limited by theirnon-interactive designs, fail to provide interpretability tai-lored to the needs of individual users.

Our Work – To bridge the striking gap, we present i-Algebra, a first-of-its-kind interactive framework for inter-preting DNN models, which allows non-expert users to eas-ily explore a variety of interpretation operations and performinteractive analyses. The overall design goal of i-Algebra isto implement the DNN interpretability as the interactions be-tween the user and the model.

Specifically, to accommodate a range of user preferences

arX

iv:2

101.

0930

1v1

[cs

.LG

] 2

2 Ja

n 20

21

Drill-Down

Analysis

Analysis Tasks

Comparative

Analysis

What-If

Analysis

Interactive Declarative Query

Projection Selection Anti-JoinJoin

Atomic Operators

….

{ Select, From, Where, Join, Left Join, … }

Figure 2: A framework of interactive interpretation of DNNs.

for interactive modes with respect to different DNN mod-els and analysis tasks, we design an expressive algebraicframework, as shown in Figure 2. Its fundamental buildingblocks are a library of atomic operators, which essentiallyproduce DNN interpretability at varying input granularity,during different model stages, and from complementary in-ference perspectives. On top of this library, we define a SQL-like declarative query language, which allows users to flexi-bly compose the atomic operators and construct a variety ofanalysis tasks (e.g., “drill-down,” “what-if,” “comparative”analyses). As a concrete example, given two inputs x andx′, the query below compares their interpretation at the l-thlayer of the DNN f and finds the most discriminative fea-tures of x and x′ with respect to their predictions.

select l from f(x) left join (select l from f(x'))

We prototype i-Algebra and evaluate its usability in threerepresentative tasks: resolving model inconsistency, inspect-ing adversarial inputs, and cleansing contaminated data. Thestudies conducted on Amazon MTurk show that comparedwith the conventional interpretability paradigm, i-Algebrasignificantly improves the analysts’ performance. For exam-ple, in the task of resolving model inconsistency, we ob-serve over 30% increase in the analysts’ accuracy of iden-tifying correct predictions and over 29% decrease in theirtask execution time; in the task of identifying adversarialinputs, i-Algebra improves the analysts’ overall accuracyby 26%; while in the task of cleansing poisoning data, i-Algebra helps the analysts’ detecting over 60% of the datapoints misclassified by the automated tool.

Our Contributions – To our best knowledge, i-Algebrarepresents the first framework for interactive interpretationof DNNs. Our contributions are summarized as follows.

• We promote a new paradigm for interactive interpretationof DNN behaviors, which accounts for the perception, un-derstanding, and responses of end-users.

• We realize this paradigm with an expressive algebraicframework built upon a library of atomic interpretation op-erators, which can be flexibly composed to construct vari-ous analysis tasks.

• We prototype i-Algebra and empirically evaluate it inthree representative analysis tasks, all showing its promis-ing usability in practice.

Background and OverviewDNN Interpretation

We primarily focus on predictive tasks (e.g., image classi-fication): a DNN f represents a function f : X → C, whichassigns a given input x ∈ X to one of a set of predefinedclasses C. We mainly consider post-hoc, instance-level in-terpretation, which explains the causal relationship betweeninput x and model prediction f(x). Such interpretations arecommonly given in the form of attribution maps. As shownin Figure 1, the interpreter g generates an attribution mapm = g(x; f), with its i-th element m[i] quantifying the im-portance of x’s i-th feature x[i] with respect to f(x).Overview of i-Algebra

Despite the rich collection of interpretation models, theyare used in an ad hoc and static manner within most existingIDLSes, resulting in their poor usability in practice (Zhanget al. 2020). To address this, it is essential to account for theperception, understanding, and response of end-users. Weachieve this by developing i-Algebra, an interactive frame-work that allows users to easily analyze DNN’s behaviorthrough the lens of interpretation.

Mechanisms – i-Algebra is built upon a library of com-posable atomic operators, which provides interpretability atdifferent input granularities (e.g., within a user-selected win-dow), during different model stages (e.g., at a specific DNNlayer), and from different inference perspectives (e.g., find-ing discriminative features). Note that we only define thefunctionality of these operators, while their implementationcan be flexibly based on concrete interpretation models.

Interfaces – On top of this library, we define an SQL-likedeclarative query language to allow users to flexibly com-pose the operators to construct various analysis tasks (e.g.,“drill-down,” “what-if,” “comparative” analysis), which ac-commodates the diversity of analytical needs from differentusers and circumvents the “one-size-fits-all” challenge.

An Interpretation AlgebraWe begin by describing the library of atomic operators.

Note that the operators can inherently be extended and all theoperators are defined in a declarative manner, independent oftheir concrete implementation.Atomic Operators

At the core of i-Algebra is a library of atomic operators,including identity, projection, selection, join, and anti-join.We exemplify with the Shapley value framework (Ancona,Oztireli, and Gross 2019; Chen et al. 2019) to illustrate onepossible implementation of i-Algebra, which can also be im-plemented based on other interpretation models.

Identity – The identity operator represents the basic inter-pretation φ(x; x, f), which generates the interpretation of agiven input xwith respect to the DNN f and a baseline inputx (e.g., an all-zero vector).1 Conceptually, the identity oper-ator computes the expected contribution of x’s each featureto the prediction f(x). Within the Shapley framework, withx as a d-dimensional vector (x ∈ Rd) and Ik as a k-sized

1When the context is clear, we omit x and f in the notation.

subset of I = {1, 2, . . . , d}, we define a d-dimensional vec-tor xIk , with its i-th dimension defined as:

[xIk ]i =

{xi (i ∈ Ik)xi (i 6∈ Ik)

(1)

Intuitively, xIk substitutes x with x along the dimensionsof Ik. Then the attribution map is calculated as:

[φ(x)]i =1

d

d−1∑k=0

EIk [f(xIk∪{i})− f(xIk)] (2)

where Ik is randomly sampled from I \ {i}.Input Identity Projection

Figure 3: Sample inputs and their interpretation under the identityand projection operators (ImageNet and ResNet50).

Projection – While the basic interpretation describes theglobal importance of x’s features with respect to its predic-tion f(x), the user may wish to estimate the local impor-tance of a subset of features. Intuitively, the global impor-tance approximates the decision boundary in a high dimen-sional space, while the local importance focuses on a lower-dimensional space, thereby being able to describe the localboundary more precisely.

The projection operator Π allows the user to zoom in agiven input. For an input x and a window w (on x) se-lected by the user, Πw(x) generates the local importanceof x’s features within w. To implement it within the Shap-ley framework, we marginalize x’s part outside the win-dow w with the baseline input x and compute the marginal-ized interpretation. Let w corresponds to the set of indices{w1, w2, . . . , w|w|} in I . To support projection, we definethe coalition Ik as a k-sized subset of {w1, w2, . . . , w|w|},and redefine the attribution map as: [Πw(x)]i ={

1|w|∑|w|−1

k=0 EIk [f(xIk∪{i})− f(xIk)] i ∈ w0 i 6∈ w

(3)

Figure 3 illustrates a set of sample images and their inter-pretation under the identity and projection operators (withinuser-selected windows). Observe that the projection opera-tor highlights how the model’s attention shifts if the featuresout of the window are nullified, which is essential for per-forming the “what-if” analysis.

Selection – While the basic interpretation shows the staticimportance of x’s features, the user may also be interestedin understanding how the feature importance varies dynam-ically throughout different inference stages. Intuitively, thisdynamic importance interpretation captures the shifting ofthe “attention” of DNNs during different stages, which helps

Figure 4: Implementation of the selection operator.

the user conduct an in-depth analysis of the inference pro-cess (Karpathy, Johnson, and Fei-Fei 2016).

The selection operator σ allows the user to navigatethrough different stages of DNNs and investigate the dy-namic feature importance. Given an input x, a DNN f whichconsists of n layers f[1:n], and the layer index i selected bythe user, σi(x) generates the interpretation at the i-th layer.

One possible implementation of σl(x) is as follows. Wetruncate the DNN f at the l-th layer, concatenate it with theoutput layer, and re-train the linear connections between thel-th layer and the output layer, as illustrated in Figure 4. Letfl denote the truncated DNN. Then the selection operatorgenerates a d-dimensional map σl(x) defined as:

[σl(x)]i = [φ(x; x, fl)]i (4)

which substitutes f in Eqn (2) with the truncated DNN fl.Input Selection

Figure 5: Sample inputs and their interpretation under the selectionoperator (ImageNet and ResNet50).

Figure 5 illustrates a set of sample inputs (from ImageNet)and their attribution maps under the selection operator.Specifically, we select i = 2, 3, 4 of the DNN (ResNet50). Itis observed that the model’s attention gradually focuses onthe key objects within each image as i increases.

Join – There are also scenarios in which the user desiresto compare two inputs x and x′ from the same class and findthe most informative features shared by x and x′, from theperspective of the DNN model. The join of two inputs x andx′, denoted by x ./ x′, compares two inputs and generatesthe interpretation highlighting the most informative featuresshared by x and x′. Note that the extension of this definitionto the case of multiple inputs is straightforward.

Within the Shapley framework, x ./ x′ can be imple-mented as the weighted sum of the Shapley values of x andx′ (given the weight of x’s map as ε):

[x ./ x′]i = ε · [φ(x; x, f)]i + (1− ε) · [φ(x′; x, f)]i (5)

Intuitively, a large value of [x ./ x′]i tends to indicate thatthe i-th feature is important for the predictions on both x andx′ (with respect to the baseline x).

Figure 6 illustrates two pairs of sample inputs and theirattribution maps under the join operator as ε = 0.5, whichhighlight the most important features with respect to theirpredictions (“horse” and “plane”) shared by both inputs.

=“Airplane”

=“Horse”

c<latexit sha1_base64="0jIMiY3Xg6FeHydWT6UzrJgEy0o=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48t2FpoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCoreNUMWyxWMSqE1CNgktsGW4EdhKFNAoEPgTj25n/8IRK81jem0mCfkSHkoecUWOlJuuXK27VnYOsEi8nFcjR6Je/eoOYpRFKwwTVuuu5ifEzqgxnAqelXqoxoWxMh9i1VNIItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2ooszYbEo2BG/55VXSrlW9i2qteVmp3+RxFOEETuEcPLiCOtxBA1rAAOEZXuHNeXRenHfnY9FacPKZY/gD5/MHx7OM6w==</latexit>

x ./ x0,<latexit sha1_base64="4v0YCrrDL+l2+pafNohxYfyY76A=">AAAB9HicbVBNTwIxEJ3FL8Qv1KOXRmL0YMgumuiR6MUjJvKRwIZ0S4GGbru2XYRs+B1ePGiMV3+MN/+NBfag4EsmeXlvJjPzgogzbVz328msrK6tb2Q3c1vbO7t7+f2DmpaxIrRKJJeqEWBNORO0apjhtBEpisOA03owuJ369SFVmknxYMYR9UPcE6zLCDZW8keoFcgnwyganZ638wW36M6AlomXkgKkqLTzX62OJHFIhSEca9303Mj4CVaGEU4nuVasaYTJAPdo01KBQ6r9ZHb0BJ1YpYO6UtkSBs3U3xMJDrUeh4HtDLHp60VvKv7nNWPTvfYTJqLYUEHmi7oxR0aiaQKowxQlho8twUQxeysifawwMTannA3BW3x5mdRKRe+iWLq/LJRv0jiycATHcAYeXEEZ7qACVSDwCM/wCm/O0Hlx3p2PeWvGSWcO4Q+czx+5CpFp</latexit>

Join

x ./ x0,<latexit sha1_base64="4v0YCrrDL+l2+pafNohxYfyY76A=">AAAB9HicbVBNTwIxEJ3FL8Qv1KOXRmL0YMgumuiR6MUjJvKRwIZ0S4GGbru2XYRs+B1ePGiMV3+MN/+NBfag4EsmeXlvJjPzgogzbVz328msrK6tb2Q3c1vbO7t7+f2DmpaxIrRKJJeqEWBNORO0apjhtBEpisOA03owuJ369SFVmknxYMYR9UPcE6zLCDZW8keoFcgnwyganZ638wW36M6AlomXkgKkqLTzX62OJHFIhSEca9303Mj4CVaGEU4nuVasaYTJAPdo01KBQ6r9ZHb0BJ1YpYO6UtkSBs3U3xMJDrUeh4HtDLHp60VvKv7nNWPTvfYTJqLYUEHmi7oxR0aiaQKowxQlho8twUQxeysifawwMTannA3BW3x5mdRKRe+iWLq/LJRv0jiycATHcAYeXEEZ7qACVSDwCM/wCm/O0Hlx3p2PeWvGSWcO4Q+czx+5CpFp</latexit>

Join

Figure 6: Sample inputs and their interpretation under the join op-erator (CIFAR10 and VGG19).

Anti-Join – Related to the join operator, the anti-join oftwo inputs x and x′, denoted by x ♦ x′, compares two inputsx and x′ from different classes and highlights their most in-formative and discriminative features. For instance, in imageclassification, the user may be interested in finding the mostcontrastive features of two images that result in their differ-ent classifications.

Within the Shapley value framework, the anti-join oper-ator x ♦ x′ can be implemented as the attribution map of xwith respect to x′ and that of x′ with respect to x:

[x ♦ x′]i = ([φ(x;x′, f)]i, [φ(x′;x, f)]i) (6)

It is worth comparing Eqn (5) and Eqn (6): Eqn (5) com-pares x (and x′) with the baseline x, highlighting the contri-bution of each feature of x (and x′) with respect to the dif-ference f(x)−f(x) (and f(x′)−f(x)); meanwhile, Eqn (6)compares x and x′, highlighting the contribution of each fea-ture of x (and x′) with respect to the difference f(x)−f(x′)(and f(x′)− f(x)).

=“Dog” =“Cat”

=“Frog”=“Ship” c2<latexit sha1_base64="gxvawmigqvEzWrjqgYRso8ZdmLo=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0wPq1frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa1a1buo1u4vK/WbPI4inMApnIMHV1CHO2hAExgM4Rle4c0Rzovz7nwsWgtOPnMMf+B8/gDudY2Q</latexit>

c1<latexit sha1_base64="bxLhQ885cIXIOA7SAduGcGiC4fY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0wPpev1xxq+4cZJV4OalAjka//NUbxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bH7qlJxZZUDCWNmShszV3xMZjbSeRIHtjKgZ6WVvJv7ndVMTXvsZl0lqULLFojAVxMRk9jcZcIXMiIkllClubyVsRBVlxqZTsiF4yy+vklat6l1Ua/eXlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwDs8Y2P</latexit>

c2<latexit sha1_base64="gxvawmigqvEzWrjqgYRso8ZdmLo=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0wPq1frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa1a1buo1u4vK/WbPI4inMApnIMHV1CHO2hAExgM4Rle4c0Rzovz7nwsWgtOPnMMf+B8/gDudY2Q</latexit>

c1<latexit sha1_base64="bxLhQ885cIXIOA7SAduGcGiC4fY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0wPpev1xxq+4cZJV4OalAjka//NUbxCyNUBomqNZdz02Mn1FlOBM4LfVSjQllYzrErqWSRqj9bH7qlJxZZUDCWNmShszV3xMZjbSeRIHtjKgZ6WVvJv7ndVMTXvsZl0lqULLFojAVxMRk9jcZcIXMiIkllClubyVsRBVlxqZTsiF4yy+vklat6l1Ua/eXlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwDs8Y2P</latexit>

Anti-Join

Figure 7: Sample inputs and their interpretation under the anti-joinoperator (CIFAR10 and VGG19).

Figure 7 compares sample inputs and their attributionmaps under the join operator. In each pair, one is a legiti-mate input and classified correctly (e.g., “ship” and “dog”);the other is an adversarial input (crafted by the PGD at-tack (Madry et al. 2018)) and misclassified (e.g., “frog” and“cat”). The anti-join operator highlights the most discrimi-native features that result in their different predictions.

Note that the anti-join operator is extensible to the case ofthe same input x but different models f and f ′. Specifically,to compute x’s features that discriminate f(x) from f ′(x),we update the expectation in Eqn (2) as EIk [f(xIk∪{i}) −f ′(xIk)]. Intuitively, for i-th feature, we compute the differ-ence of its contribution with respect to f(x) and f ′(x).Compositions

The library of atomic operators is naturally composable.For instance, one may combine the selection and projection

operators, Πw(σl(x)), which extracts the interpretation atthe l-th layer of the DNN and magnifies the features withinthe window w; it is possible to compose the join and selec-tion operators, σl(x) ./ σl(x

′), which highlights the mostdiscriminative features of x and x′ resulting in their differ-ent predictions from the view of the l-th layer of the DNNf ; further, it is possible to combine two anti-join operators,x1 ♦ x2 ♦ x3, which generates the most discriminative fea-tures of each input with respect to the rest two.

To ensure their semantic correctness, one may specifythat the compositions of different operators to satisfy cer-tain properties (e.g., commutative). For instance, the com-position of the selection (σ) and projection (Π) operatorsneeds to satisfy the commutative property, that is, the or-der of applying the two operators should not affect the in-terpretation result, Πwσl(x) = σlΠw(x)a. Moreover, cer-tain compositions are allowed only under certain conditions(conditional). For instance, the composition of two selectionoperators, σlσl′(x), is only defined if l ≤ l′, that is, it gener-ates the interpretation of x with respect to the layer with thesmaller index in l and l′ of the DNN f . Further, the compo-sition of the join and anti-join operators are undefined.

Interactive InterpretationFurther, i-Algebra offers a declarative language that al-

lows users to easily “query” the interpretation of DNN be-haviors and build interactive analysis tasks as combinationsof queries (cf. Figure 2).A Declarative Query Language

Specifically, we define an SQL-like declarative query lan-guage for interpreting DNN behaviors. Next we first definethe statements for each atomic operator and then discusstheir compositions.

We use a set of keywords: “select” for the selection oper-ator, “where” for the projection operator, “join” for the joinoperator, and “left join” for the anti-join operator. The atomicoperators can be invoked using the following statements:select * from f(x)

– the identity operator φ(x).select * from f(x) where w

– the projection operator Πw(x).select l from f(x)

– the selection operator σl(x).select * from f(x) join (select * from f(x'))

– the join operator x ./ x′.select * from f(x) left join (select * from f(x'))

– the anti-join operator x ♦ x′.Similar to the concept of “sub-queries” in SQL, more

complicated operators can be built by composing the state-ments of atomic operators. Following are a few examples.select l from f(x) where w

– the composition of selection and projection Πwσl(x).select l from f(x) join (select l from f(x'))

– the composition of join and selection σl(x) ./ σl(x′).

Interactive AnalysisThrough the declarative queries, users are able to conduct

an in-depth analysis of DNN behaviors, including:Drill-Down Analysis – Here the user applies a sequence

of projection and/or selection to investigate how the DNNmodel f classifies a given input x at different granularitiesof x and at different stages of f . This analysis helps answerimportant questions such as: (i) how does the importance ofx’s features evolve through different stages of f? (ii) whichparts of x are likely to be the cause of its misclassification?(iii) which stages of f do not function as expected?

Comparative Analysis – In a comparative analysis, theuser applies a combination of join and/or anti-join operatorson the target input x and a set of reference inputs X to com-pare how the DNN f processes x and x′ ∈ X . This analy-sis helps answer important questions, including: (i) from f ’sview, why are x and x′ ∈ X similar or different? (ii) doesf indeed find the discriminative features of x and x′ ∈ X ?(iii) if x is misclassified into the class of x′, which parts ofx are likely to be the cause?

What-If Analysis – In what-if analysis, the user modi-fies parts of the input x before applying the operators andcompares the interpretation before and after the modifica-tion. The modification may include (i) nullification (e.g., re-placing parts of x with baseline), (ii) substitution (e.g., sub-stituting parts of x with another input), and (iii) transforma-tion (e.g., scaling, rotating, shifting). This analysis allowsthe user to analyze f ’s sensitivity to each part of x and itsrobustness against perturbation.

Note that these tasks are not exclusive; rather, they maycomplement each other by providing different perspectiveson the behaviors of DNN models. For instance, both drill-down and what-if analyses help the user gauge the impact ofx’s part x[w] on f ’s prediction; yet, the former focuses onanalyzing the decision boundary within the space spannedby the features w, while the latter focuses on analyzing theoverall contribution of x[w] to f ’s prediction.

Empirical EvaluationWe prototype i-Algebra and empirically evaluate its us-

ability in a set of case studies. The evaluation is designed toanswer the following key questions.• RQ1: Versatility – Does i-Algebra effectively support a

range of analysis tasks?• RQ2: Effectiveness – Does it significantly improve the

analysis efficacy in such tasks?• RQ3: Usability – Does it provide intuitive, user-friendly

interfaces for analysts?We conduct user studies on the Amazon Mechanical Turk

platform, in which each task involves 1,250 assignmentsconducted by 50 qualified workers. We apply the follow-ing quality control: (i) the users are instructed about the taskgoals and declarative queries, and (ii) the tasks are set assmall batches to reduce bias and exhaustion.Case A: Resolving Model Inconsistency

Two DNNs trained for the same task often differ slightlydue to (i) different training datasets, (ii) different training

regimes, and (iii) randomness inherent in training algorithms(e.g., random shuffle and dropout). It is thus critical to iden-tify the correct one when multiple DNNs disagree on theprediction on a given input. In this case study, the user isrequested to use i-Algebra to resolve cases that are incon-sistent between two DNNs f and f ′.

Setting – On CIFAR10, we train two VGG19 models fand f ′. In the testing set of CIFAR10, 946 samples are pre-dicted differently by f and f ′, in which 261 samples are cor-rectly predicted by f and 565 samples by f ′. Within this set,824 inputs are classified correctly by either f or f ′, whichwe collect as the testing set T for our study.

= “Deer” = “Cat”

= “Bird” = “Deer”

Figure 8: Sample inputs, their classification by f and f ′, and theirinterpretation by i-Algebra in Case A.

We randomly sample 50 (half predicted correctly byf and the other half by f ′) inputs from T to form thetesting set. Specifically, the baseline directly generates theinterpretation f(x) and f ′(x) for each input x; i-Algebraapplies the Anti-Join operator to highlight x’s most discrim-inative features (from the views of f and f ′) that result inits different predictions, with the declarative query givenas: select * from f(x) left join (select * from f'(x)).Figure 8 shows a set of sample inputs, their classificationunder f and f ′, and their interpretation by i-Algebra.Observe that the discriminative features on the correctmodel tend to agree with human perception better.

Evaluation – We evaluate i-Algebra in terms of (i) ef-fectiveness – whether it helps users identify the correct pre-dictions, and (ii) efficiency – whether it helps users conductthe analysis more efficiently. We measure the effectivenessusing the metric of accuracy (the fraction of correctly distin-guished inputs among total inputs) and assess the efficiencyusing the metric of user response time (URT), which is mea-sured by the average time the users spend on each input.

User Response Times (s)-100 0 100 200 300 400

0

2

4

6

8

10

12

14

16 i-Algebra

Baseline

Model 1 Model 2 Total

Baseline

i-Algebra

0.0

0.1

0.2

0.3

0.4

Acc

ura

cy

0.5

0.6

0.699

0.415

0.614

0.342

0.663

0.360

0.7

Figure 9: Users’ accuracy and URT measures under baseline andi-Algebra in Case A.

Figure 9 compares the users’ performance using the base-line and i-Algebra on the task of resolving model incon-sistency. We have the following observations: (i) i-Algebrasignificantly improves the users’ accuracy of identifying thecorrect predictions on both f and f ′, with the overall accu-

racy increasing by around 30%; (ii) Despite its slightly morecomplicated interfaces, the URT on i-Algebra does not ob-serve a significant change from the baseline, highlighting itseasy-to-use mechanisms and interfaces.

Case B: Detecting Adversarial InputsOne intriguing property of DNNs is their vulnerability to

adversarial inputs, which are maliciously crafted samples todeceive target DNNs (Madry et al. 2018; Carlini and Wagner2017). Adversarial inputs are often generated by carefullyperturbing benign inputs, with difference imperceptible tohuman perception. Recent work has proposed to leverage in-terpretation as a defense mechanism to detect adversarial in-puts (Tao et al. 2018). Yet, it is shown that the interpretationmodel is often misaligned with the underlying DNN model,resulting in the possibility for the adversary to deceive bothmodels simultaneously (Zhang et al. 2020). In this use case,the users are requested to leverage i-Algebra to inspect po-tential adversarial inputs from multiple different interpreta-tion perspectives, making it challenging for the adversary toevade the detection across all such views.

Setting – We use ImageNet as the dataset and consider apre-trained ResNet50 (77.15% top-1 accuracy) as the tar-get DNN. We also train a set of truncated DNN models(l = 2, 3, 4) for the selection operator σl(x) in i-Algebra(details in § ). We apply ADV2 (Zhang et al. 2020), an at-tack designed to generate adversarial inputs deceiving boththe DNN and its coupled interpreter. Specifically, ADV2 op-timizes the objective function:

minx `prd (f(x), ct) + λ`int (g(x; f),mt)s.t. ∆ (x, x◦) ≤ ε (7)

where `prd ensures that the adversarial input x is misclassifiedto a target class ct by the DNN f , and `int ensures that xgenerates an attribution map similar to a target map mt (theattribution map of the benign input x◦).

From ImageNet, we randomly sample 50 inputs and gen-erate their adversarial counterparts, which are combinedwith another 50 randomly sampled benign inputs to form thetesting set T . We request the users to identify the adversarialinputs through the lens of baseline and i-Algebra. In partic-ular, by varying l and w, i-Algebra provides interpretationat various inference stages and input granularity, with thedeclarative query template as: select l from f(x) where w.Figure 10 shows sample adversarial inputs and their inter-pretation under i-Algebra. Observe that by from multiplecomplementary perspectives, the adversarial inputs showfairly distinguishable interpretation.

Evaluation – We quantitatively assess the usability of i-Algebra in terms of (i) effectiveness – whether it helps usersidentify the adversarial inputs more accurately, and (ii) effi-ciency – whether it helps users conduct the analysis moreefficiently. Specifically, considering adversarial and benigninputs as positive and negative cases, we measure the effec-tiveness using the metrics of precision and recall (as well asaccuracy and F-1 score):

We assess the efficiency using the metric of user responsetime (URT), which is measured by the average time the usersspend on each task with the given tool.

Figure 10: Sample adversarial inputs and i-Algebra interpretation.

Baseline

0.0

0.1

0.2

0.3

0.4

Pe

rfo

rma

nce

Me

asu

re

0.5

0.6

0.7

Precision Recall Accuracy F1 Score

0.726

0.646 0.644

0.684

0.369 0.384

0.450

0.577

i-Algebra

0

5

10

15

20

25

30

35

i-AlgebraBaseline

User Response Times (s)0 200 400 600 800

Figure 11: Users’ performance and URT measures under baselineand i-Algebra in Case B.

Figure 11 compares the users’ performance and URT us-ing the baseline interpretation and i-Algebra on the task ofidentifying adversarial inputs. We have the following ob-servations: (i) through the lens of interpretation from mul-tiple complementary perspectives, i-Algebra improves theusers’ effectiveness of distinguishing adversarial and benigninputs with about 26% increase in the overall accuracy; (ii)compared with the baseline, the average URT on i-Algebragrows from 60.45s to 303.54s, which can be intuitively ex-plained by its requirement for multiple rounds of interactivequeries. Given the significant performance improvement, thecost of execution time is well justified.

Figure 12: Sample trigger-embedded inputs at inference, which aremisclassified as “deer”.

Case C: Cleansing Poisoning DataOrthogonal to adversarial examples, another concern for

the safety of DNNs is their vulnerability to manipulations oftheir training data. In the backdoor attacks (e.g., (Gu, Dolan-Gavitt, and Garg 2017)), by injecting corrupted inputs intraining a DNN, the adversary forces the resultant modelto (i) misclassify the inputs embedded with particular pat-terns (“triggers”) to a target class and (ii) behave normally

on benign inputs. Figure 12 shows a set of trigger-embeddedinputs which are misclassified from “truck” to “deer”.

Since the backdoored DNNs correctly classify benign in-puts, once trained, they are insidious to be detected. Onemitigation is to detect corrupted instances in the training set,and then to use cleansed data to re-train the DNN (Tran, Li,and Madry 2018). Typically, the analyst applies statisticalanalysis on the deep representations of the training data anddetects poisoning inputs based on their statistical anomaly.

We let the analyst leverage i-Algebra to perform fine-tuning of the results detected by the statistical analysis.Through the lens of the interpretation, the users may in-spect the inputs that fall in the uncertain regions (e.g., 1.25∼ 1.75 standard deviation) and identify false positives andfalse negatives by the automated detection, which may fur-ther improve the mitigation of backdoor attacks.

Prediction Ground-truth+ -

+ 48 654- 399 3574

Table 1. Samples statistics in Case C.

Setting – We use CIFAR10 as the underlying dataset andVGG19 as the target DNN. We consider using the back-door attack in (Gu, Dolan-Gavitt, and Garg 2017) to gen-erate poisoning instances in one particular class “truck”.We apply spectral signature (Tran, Li, and Madry 2018) toidentify potential poisoning instances. For each data pointxi in a particular class, it examines xi’s deep representa-tion R(xi) at the penultimate layer. After obtaining the topright singular vector of the centered representation [R(xi)−1n

∑ni=1R(xi)]

ni=1, the data points beyond 1.5 standard de-

viation from the center are identified as poisonous. How-ever, this automated tool is fairly inaccurate. Table 1 summa-rizes the predicted results and the ground truth. In this case,we request users to identify positive and negative cases thatare misclassified by the automated tool, with the declarativequery template given as: select l from f(x).

Precision Recall Accuracy F1-Score0.609 0.6343 0.586 0.622

Table 2. Users’ performance under i-Algebra in Case C.

Evaluation – We measure the users’ effectiveness of dis-tinguishing positive and negative cases. The results are listedin Table 2. Observe that equipped with i-Algebra, the userssuccessfully classify around 60% of the data point misclassi-fied by the automated tool. Figure 13 shows the distributionof URT for this task. Note that a majority of users take lessthan 50s to complete the tasks.

Related workNext, we survey three categories of prior work: inter-

pretable deep learning, model attacks and defenses, and in-teractive learning.

Interpretable Deep Learning – Typically, DNN inter-pretability can be obtained by either designing interpretablemodels (Zhang, Nian Wu, and Zhu 2018) or extracting post-hoc interpretations. The post-hoc interpretation methods canbe categorized as backprop- (Sundararajan, Taly, and Yan

Distribution of URT on Use Case C

0.00

0.01

0.02

0.03

0.04

0.05

i-Algebra

User Response Times (s)0 50 100 150 200 250

Figure 13: URT distribution in Case C.

2017), representation- (Selvaraju et al. 2017), meta-model-(Dabkowski and Gal 2017), and perturbation-based (Fongand Vedaldi 2017). Instead of developing yet another inter-pretation method or enhancing existing ones, this work pro-poses the paradigm of interactive interpretability, which canbe flexibly implemented upon existing methods.

Model Attacks and Defenses – DNNs are becomingthe new targets of malicious attacks, including adversar-ial attacks (Carlini and Wagner 2017) and poisoning at-tacks (Shafahi et al. 2018). Although a line of work strivesto improve DNN robustness (Tramer et al. 2018; Tran, Li,and Madry 2018), existing defenses are often penetrated byeven stronger attacks (Ling et al. 2019), resulting in a con-stant arms race. Our work involves the users in the processof model robustness improvement, which is conducive to en-hancing model trustworthiness.

Interactive Learning – Interactive learning couples hu-mans and machine learning models tightly within the learn-ing process. The existing work can be roughly categorizedas model understanding (Krause, Perer, and Bertini 2016;Krause, Perer, and Ng 2016), which allows users to inter-pret models’ input-output dependence, and model debug-ging (Wu et al. 2019; Nushi, Kamar, and Horvitz 2018),which allows users to detect and fix models’ mistakes. i-Algebra can be leveraged to not only understand DNNs’behaviors but also facilitate diverse security tasks includingmodel debugging, data cleansing, and attack inspection.

ConclusionThis work promotes a paradigm shift from static interpre-

tation to interactive interpretation of neural networks, whichwe believe will significantly improve the usability of exist-ing interpretation models in practice. We present i-Algebra,a first-of-its-kind interactive framework for DNN interpreta-tion. At its core is a library of atomic operators that producethe interpretation of DNN behaviors at varying input gran-ularity, at different inference stages, and from distinct inter-pretation perspectives. A declarative query language is de-fined for users to flexibly construct a variety of analysis tasksby composing different operators. We prototype i-Algebraand conduct extensive studies in three representative analy-sis tasks, all demonstrating its promising usability.

AcknowledgmentsThis work is supported by the National Science Foun-

dation under Grant No. 1951729, 1953813, and 1953893.Any opinions, findings, and conclusions or recommenda-tions are those of the authors and do not necessarily reflectthe views of the National Science Foundation. Shouling Jiwas partly supported by the National Key Research and De-velopment Program of China under No. 2018YFB0804102,NSFC under No. 61772466, U1936215, and U1836202, theZhejiang Provincial Natural Science Foundation for Distin-guished Young Scholars under No. LR19F020003, the Zhe-jiang Provincial Key R&D Program under No. 2019C01055,the Ant Financial Research Funding, and the FundamentalResearch Funds for the Central Universities (Zhejiang Uni-versity NGICS Platform). We thank Hua Shen for contribut-ing to implementing and evaluating i-Algebra.

ReferencesAncona, M.; Oztireli, C.; and Gross, M. 2019. Explaining DeepNeural Networks with a Polynomial Time Algorithm for ShapleyValues Approximation. In Proceedings of IEEE Conference on Ma-chine Learning (ICML).

Carlini, N.; and Wagner, D. A. 2017. Towards Evaluating the Ro-bustness of Neural Networks. In Proceedings of IEEE Symposiumon Security and Privacy (S&P).

Chen, J.; Song, L.; Wainwright, M. J.; and Jordan, M. I. 2019. L-Shapley and C-Shapley: Efficient Model Interpretation for Struc-tured Data. In Proceedings of International Conference on Learn-ing Representations (ICLR).

Dabkowski, P.; and Gal, Y. 2017. Real Time Image Saliency forBlack Box Classifiers. In Proceedings of Advances in Neural In-formation Processing Systems (NeurIPS).

Fong, R. C.; and Vedaldi, A. 2017. Interpretable Explanations ofBlack Boxes by Meaningful Perturbation. In Proceedings of IEEEInternational Conference on Computer Vision (ICCV).

Goodfellow, I.; Bengio, Y.; and Courville, A. 2016. Deep Learning.MIT Press.

Gu, T.; Dolan-Gavitt, B.; and Garg, S. 2017. BadNets: Identify-ing Vulnerabilities in the Machine Learning Model Supply Chain.ArXiv e-prints .

Guo, W.; Mu, D.; Xu, J.; Su, P.; Wang, G.; and Xing, X. 2018.LEMNA: Explaining Deep Learning Based Security Applications.In Proceedings of ACM SAC Conference on Computer and Com-munications (CCS).

Karpathy, A.; Johnson, J.; and Fei-Fei, L. 2016. Visualizing andUnderstanding Recurrent Networks. In Proceedings of Interna-tional Conference on Learning Representations (ICLR).

Krause, J.; Perer, A.; and Bertini, E. 2016. Using Visual Analyticsto Interpret Predictive Machine Learning Models. In Proceedingsof IEEE Conference on Machine Learning (ICML).

Krause, J.; Perer, A.; and Ng, K. 2016. Interacting with Predictions:Visual Inspection of Black-Box Machine Learning Models. In Pro-ceedings of the CHI Conference on Human Factors in ComputingSystems (CHI).

Ling, X.; Ji, S.; Zou, J.; Wang, J.; Wu, C.; Li, B.; and Wang, T.2019. DEEPSEC: A Uniform Platform for Security Analysis ofDeep Learning Model. In Proceedings of IEEE Symposium on Se-curity and Privacy (S&P).

Madry, A.; Makelov, A.; Schmidt, L.; Tsipras, D.; and Vladu, A.2018. Towards Deep Learning Models Resistant to AdversarialAttacks. In Proceedings of International Conference on LearningRepresentations (ICLR).

Nushi, B.; Kamar, E.; and Horvitz, E. 2018. Towards AccountableAI: Hybrid Human-Machine Analyses for Characterizing SystemFailure. In Proceedings of AAAI Conference on Artificial Intelli-gence (AAAI).

Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh,D.; and Batra, D. 2017. Grad-CAM: Visual Explanations fromDeep Networks via Gradient-Based Localization. In Proceedingsof IEEE International Conference on Computer Vision (ICCV).

Shafahi, A.; Ronny Huang, W.; Najibi, M.; Suciu, O.; Studer,C.; Dumitras, T.; and Goldstein, T. 2018. Poison Frogs! Tar-geted Clean-Label Poisoning Attacks on Neural Networks. In Pro-ceedings of Advances in Neural Information Processing Systems(NeurIPS).

Sundararajan, M.; Taly, A.; and Yan, Q. 2017. Axiomatic Attribu-tion for Deep Networks. In Proceedings of IEEE Conference onMachine Learning (ICML).

Tao, G.; Ma, S.; Liu, Y.; and Zhang, X. 2018. Attacks Meet In-terpretability: Attribute-Steered Detection of Adversarial Samples.In Proceedings of Advances in Neural Information Processing Sys-tems (NeurIPS).

Tramer, F.; Kurakin, A.; Papernot, N.; Goodfellow, I.; Boneh, D.;and McDaniel, P. 2018. Ensemble Adversarial Training: Attacksand Defenses. In Proceedings of International Conference onLearning Representations (ICLR).

Tran, B.; Li, J.; and Madry, A. 2018. Spectral Signatures in Back-door Attacks. In Proceedings of Advances in Neural InformationProcessing Systems (NeurIPS).

Wu, T.; Ribeiro, M. T.; Heer, J.; and Weld, D. S. 2019. Errudite:Scalable, Reproducible, and Testable Error Analysis. In Proceed-ings of Annual Meeting of the Association for Computational Lin-guistics (ACL).

Zhang, Q.; Nian Wu, Y.; and Zhu, S.-C. 2018. Interpretable Con-volutional Neural Networks. In Proceedings of IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR).

Zhang, X.; Wang, N.; Shen, H.; Ji, S.; Luo, X.; and Wang, T. 2020.Interpretable Deep Learning under Fire. In Proceedings of USENIXSecurity Symposium (SEC).