Latent Compositional Representations Improve SystematicGeneralization in Grounded Question Answering

Ben Bogin1 Sanjay Subramanian2 Matt Gardner2 Jonathan Berant1,21Tel-Aviv University 2Allen Institute for AI

{ben.bogin,joberant}@cs.tau.ac.il, {sanjays,mattg}@allenai.org

Abstract

Answering questions that involve multi-stepreasoning requires decomposing them andusing the answers of intermediate steps toreach the final answer. However, state-of-the-art models in grounded question answer-ing often do not explicitly perform decom-position, leading to difficulties in general-ization to out-of-distribution examples. Inthis work, we propose a model that com-putes a representation and denotation forall question spans in a bottom-up, compo-sitional manner using a CKY-style parser.Our model effectively induces latent trees,driven by end-to-end (the answer) super-vision only. We show that this induc-tive bias towards tree structures dramati-cally improves systematic generalization toout-of-distribution examples compared tostrong baselines on an arithmetic expres-sions benchmark as well as on CLOSURE,a dataset that focuses on systematic gener-alization of models for grounded questionanswering. On this challenging dataset, ourmodel reaches an accuracy of 92.8%, sig-nificantly higher than prior models that al-most perfectly solve the task on a random,in-distribution split.

1 Introduction

Humans can effortlessly interpret new natural lan-guage utterances, as long as they are composedof previously-observed primitives and structure(Fodor and Pylyshyn, 1988). Neural networks,on the other hand, do not exhibit this systematic-ity: while they generalize well to examples sam-pled from the same distribution as the training set,they have been shown to struggle in generalizingto out-of-distribution (OOD) examples that con-tain new compositions in both grounded questionanswering (Bahdanau et al., 2019a,b) and seman-tic parsing (Finegan-Dollak et al., 2018; Keysers

et al., 2020). For example, consider the ques-tion in Fig. 1. This question requires querying thesize of objects, comparing colors, identifying spa-tial relations and computing intersections betweensets of objects. Neural networks tend to succeedwhenever these concepts are combined in waysthat were seen during training time. However, theycommonly fail whenever these concepts are com-bined in novel ways at test time.

A possible reason for this phenomenon isthe expressivity of modern architectures such asLSTMs (Hochreiter and Schmidhuber, 1997) andTransformers (Vaswani et al., 2017), where richrepresentations that depend on the entire input arecomputed. The fact that token representations arecontextualized by the entire utterance potentiallylets the model avoid step-by-step reasoning, “col-lapse" multiple reasoning steps, and rely on short-cuts (Jiang and Bansal, 2019; Subramanian et al.,2020). Such failures are revealed when evaluatingmodels for systematic generalization on OOD ex-amples. This stands in contrast to pre-neural log-linear models, where hierarchical representationswere explicitly constructed over the input (Zettle-moyer and Collins, 2005; Liang et al., 2013).

In this work, we propose a model for visualquestion answering (QA) that, analogous to theseclassical pre-neural models, computes for everyspan in the input question a representation and adenotation, that is, the set of objects in the imagethat the span refers to (see Fig. 1). Denotations forlong spans are recursively computed from shorterspans using a bottom-up CKY-style parser withoutaccess to the entire input, leading to an inductivebias that encourages compositional computation.Because training is done from the final answeronly, the model must effectively learn to induce la-tent trees that describe the compositional structureof the problem. We hypothesize that this explicitgrounding of the meaning of sub-spans through hi-erarchical computation should result in better gen-

Figure 1: An example from CLOSURE illustrating how our model learns a latent structure over the input, where arepresentation and denotation is computed for every span (for denotation we show the set of objects with probability> 0.5). For brevity, some phrases were merged to a single node of the tree. For each phrase, we show the splitpoint and module with the highest probability, although all possible split points and module outputs are softlycomputed. SKIP(L) and SKIP(R) refer to taking the denotation of the left or right sub-span, respectively.

eralization to new compositions.We evaluate our approach in two setups: (a) a

synthetic arithmetic expressions dataset, and (b)CLOSURE (Bahdanau et al., 2019b), a visual QAdataset that focuses on systematic generalization.On a random train/test split of the data (i.i.d split),both our model and prior baselines obtain nearperfect performance. However, on splits that re-quire systematic generalization to new compo-sitions (compositional split) our model dramati-cally improves performance: for the arithmeticexpressions problem, a vanilla Transformer failsto generalize and obtains 2.9% accuracy, whileour model, Grounded Latent Trees (GLT), gets98.4%. On CLOSURE, our model’s accuracy isat 92.8%, 20% absolute points higher than strongbaselines and even 15% points higher than modelsthat use gold structures at training time or dependon domain-knowledge.

To conclude, we propose a model with an in-herent inductive bias for copositional computa-tion, which leads to large gains in systematic gen-eralization, and induces latent structures that areuseful for understanding its inner workings. Ourwork suggests that despite the undeniable suc-cess of general-purpose architectures built on top

of contextualized representations, restricting in-formation flow inside the network can greatly ben-efit compositional generalization.1

2 Compositional Generalization

Natural language is mostly compositional; hu-mans can understand and produce a potentially in-finite number of novel combinations from a closedset of known components (Chomsky, 1957; Mon-tague, 1970). For example, a person would knowwhat a "winged giraffe" is even if she’s neverseen one, assuming she knows the meaning of“winged” and “giraffe”. This ability, which weterm compositional generalization, is fundamen-tal for building robust models that effectively learnfrom limited data (Lake et al., 2018).

Neural networks have been shown to generalizewell in many language understanding tasks (De-vlin et al., 2019; Raffel et al., 2019), when usingi.i.d splits. However, when models are evaluatedon splits that require compositional generalization,a significant drop in performance is observed. Forexample, in SCAN (Lake and Baroni, 2018) and

1Our code and data can be foundat https://github.com/benbogin/glt-grounded-latent-trees-qa.

gSCAN (Ruis et al., 2020), synthetically gener-ated commands are mapped into a sequence of ac-tions. When tested on unseen command combi-nations, models perform poorly. A similar casewas shown in text-to-SQL parsing Finegan-Dollaket al. (2018), where splitting the training examplesby the template of the target SQL query resultedin a dramatic drop in performance. SQOOP (Bah-danau et al., 2019a) shows the same phenomena ona synthetic visual QA task, which tests for general-ization over unseen combinations of object prop-erties and relations. This also led to developingmethods that construct compositional splits auto-matically (Keysers et al., 2020).

In this work, we focus on answering complexgrounded questions over images. The CLEVRbenchmark (Johnson et al., 2017a) contains pairsof synthetic images and questions that requiremulti-step reasoning, e.g., “Are there any largecyan spheres made of the same material as thelarge green sphere?”. While this task is mostlysolved, with an accuracy of 97%-99% (Perez et al.,2018; Hudson and Manning, 2018), recent work(Bahdanau et al., 2019b) introduced CLOSURE:a new set of questions with identical vocabularybut different structure than CLEVR, asked on thesame set of images. They evaluated generalizationof different model families and showed that all failon a large fraction of the examples.

The most common approach for grounded QAis based on end-to-end differentiable models suchas FiLM (Perez et al., 2018), MAC (Hudsonand Manning, 2018) LXMERT (Tan and Bansal,2019), and UNITER (Chen et al., 2019). Thesehigh-capacity models do not explicitly decomposethe problem into smaller sub-tasks, and are thusprone to fail on compositional generalization. Adifferent approach (Yi et al., 2018; Mao et al.,2019) is to parse the image into a symbolic or dis-tributed knowledge graph with objects, attributes(color, size, etc.), and relations, and then parse thequestion into an executable logical form, whichis deterministically executed. Last, Neural Mod-ule Networks (NMNs; Andreas et al. 2016) parsethe question into an executable program as well,but execution is learned: each program module isa neural network designed to perform an atomictask, and modules are composed to perform com-plex reasoning. The latter two model familiesconstruct compositional programs and have beenshown to generalize better on compositional splits

(Bahdanau et al., 2019a,b) compared to fully dif-ferentiable models. However, programs are notexplicitly tied to spans in the input question, andsearch over the space of possible programs is notdifferentiable, leading to difficulties in training.

In this work, we learn a latent structure for thequestion and tie each question span to an exe-cutable module in a differentiable manner. Ourmodel balances the distributed and the symbolicapproaches: we learn from downstream supervi-sion only and output an inferred tree of the ques-tion, describing how the answer was computed.We base our model on work on latent tree parsers(Le and Zuidema, 2015; Liu et al., 2018; Mail-lard et al., 2019; Drozdov et al., 2019) that pro-duce representations for all spans, and computea soft weighting over all possible trees. We ex-tend these parsers to answer grounded questions,grounding sub-trees in image objects. Closest toour work is Gupta and Lewis (2018), where deno-tations are computed for each span. However, theydo not compute compositional representations forthe spans, limiting the expressivity of their model.Additionally, they work with a knowledge graphrather than images.

3 Model

In this section, we give a high-level overview ofour proposed Grounded Latent Trees (GLT) model(§3.1), explain our grounded CKY-based parser(§3.2), and describe the architecture details (§3.3,§3.4) and training procedure (§3.5).

3.1 High-level overview

Problem setup Our task is visual QA, wheregiven a question q = (q0, . . . , qn−1), and an im-age I , we aim to output an answer a ∈ A froma fixed set of natural language phrases. We traina model from a training set {(qi, Ii, ai)}Ni=1. Weassume we can extract from the image up to nobjfeatures vectors of objects, and represent them as amatrix V ∈ Rnobj×hdim (details on object detectionand representation are in §3.4).

Our goal is to compute for every question spanqij = (qi, . . . , qj−1) a representation hij ∈ Rhdim

and a denotation dij ∈ [0, 1]nobj , which we in-terpret as the probability that the question spanrefers to each object. We compute hij and dij

in a bottom-up fashion, using CKY (Cocke, 1969;Kasami, 1965; Younger, 1967). Algorithm 1 pro-vides a high-level description of the procedure.

Algorithm 1Require: question q, image I , word embedding matrix E,

visual representations matrix V1: H: tensor holding representations hij , ∀i, j s.t. i < j2: D: tensor holding denotations dij , ∀i, j s.t. i < j3: for i = 1 . . . n do4: hi = Eqi , di = fground(Eqi , V ) (see §3.4)5: for l = 1 . . . n do6: compute hij , dij for all entries s.t j − i = l

7: p(a | q, I) = softmax(W [h0n; d0nV ])8: return argmaxa p(a | q, I)

We compute representations and denotations forlength-1 spans (we use hi = hi(i+1), di = di(i+1)

for brevity) by setting the representation hi = Eqi

to be the corresponding word representation in anembedding matrix E, and grounding each word inthe image objects: di = fground(Eqi , V ) (lines 3-4; fground function described in §3.4). Then, werecursively compute representations and denota-tions of larger spans (lines 5-6). Last, we pass therepresentation of the entire question (h0n) togetherwith the weighted sum of the visual representa-tions (d0nV ) through a softmax layer to produce afinal answer distribution (line 7), using a learnedclassification matrix W ∈ RA×2hdim .

Computing hij ,dij for all spans requires over-coming some challenges. Each span representa-tion hij should be a function of two sub-spanshik,hkj . We use the term sub-spans to refer toall adjacent pairs of spans that cover qij , formally,{(qik, qkj)}j−1k=i+1. However, we have no super-vision for the “correct” split point k. Our model(§3.2) considers all possible split points and learnsto induce a latent tree structure from the final an-swer only. We show that this leads to a com-positional structure and denotations that can beinspected at test time, providing an interpretablelayer.

In §3.3 we describe the form of the composi-tion functions, which compute both span repre-sentations and denotations from two sub-spans.These functions must be expressive enough to ac-commodate a wide range of interactions betweensub-spans, but not create reasoning shortcuts thatmight hinder compositional generalization.

3.2 Grounded chart parsing

We now describe how to recursively computehij ,dij from previously computed representationsand denotations. In standard CKY-parsing, eachconstituent over a span qij is constructed by com-

Figure 2: Illustration of how hij is computed. First, weconsider all possible split points and compose pairs ofsub-spans using fh. Then, a weight is computed for allrepresentations, and the output is their weighted sum.

bining two sub-spans qik, qkj that meet at a splitpoint k. Similarly, we define a representationhkij that is conditioned on the split point and

constructed from previously-computed represen-tations of two sub-spans:

hkij = fh(hik,hkj), (1)

where fh(·) is a composition function (§3.3).Since we want the loss to be differentiable with

respect to its input, we do not pick a particu-lar value k, but instead use a continuous relax-ation. Specifically, we compute the probabilitypH(k | i, j) that k is the split point for the span qij ,given the tensorH of all computed representationsof shorter spans. We then define the representationof the span hij to be the expected representationover all possible split points:

hij =∑k

pH(k | i, j) · hkij = EpH(k|·)[hk

ij ]. (2)

The split point distribution is defined as pH(k |i, j) ∝ exp(sThk

ij), where s ∈ Rhdim is a parametervector that determines what split points are likely.Figure 2 illustrates computing hij .

Next, we turn to computing the denotation dij

of each span. Conceptually, computing dij canbe analogous to hij ; that is, a function fd willcompute dk

ij for every possible split point k, andwe will define dij = EpH(k|·)[dk

ij ]. However, thefunction fd (see §3.3) interacts with the visual rep-resentations of all objects and is thus computation-

Figure 3: Illustration of how dij is computed. We com-pute the denotations of all modules, and a weight foreach one of the modules. The span denotation is thenthe weighted sum of the module outputs.

ally costly. Therefore, we propose a less expres-sive but more efficient approach, where fd(·) isapplied only once for each span qij .

Specifically, we compute the expected denota-tion of the left and right sub-spans of qij :

dijL = EpH(k|·)[dik] ∈ Rnobj (3)

dijR = EpH(k|·)[dkj ] ∈ Rnobj . (4)

If pH(k | ·) puts most probability mass on a singlesplit point k′, then the expected denotations willbe similar to picking that particular split point.

Now we can compute dij given the expectedsub-span denotations and representations with asingle application of fd(·):

dij = fd(dijL , dijR ,hij), (5)

which is substantially more efficient than the al-ternative Ep(k|·)[fd(dik,dkj ,hij)]: in our imple-mentation fd is appliedO(n2) times versusO(n3)with the alternative solution. This is important formaking training tractable in practice.

3.3 Composition functions

We now describe the exact form of the composi-tion functions fh and fd.

Composing representations We first describethe function fh(hik,hkj), used to compose therepresentations of two sub-spans (Eq. 1). The goalof this function is to compose the “meanings” oftwo adjacent spans, without having access to therest of the question or to the denotations of the

Figure 4: The different modules used with their inputsand expected output.

sub-spans. For example, composing the represen-tations of “same” and “size” to a representationfor “same size”. At a high-level, composition isbased on a generic attention mechanism. Specif-ically, we use attention to form a convex sum ofthe representations of the two sub-spans (Eq. 6-7),and apply a non-linear transformation with a resid-ual connection (Eq. 8).

αkij = softmax ([aLhik, aRhkj ]) ∈ R2 (6)

hkij = α(1)WLhik + α(2)WRhkj ∈ Rhdim (7)

fh(hik,hkj) = FFrep

(hkij

)+ h

kij ∈ Rhdim (8)

where aL, aR ∈ Rhdim , WL,WR ∈ Rhdim×hdim , andFFrep(·) is a linear layer of size hdim × hdim fol-lowed by a non-linear activation.2

Composing denotations Next, we describe thefunction fd(dijL , dijR ,hij), used to compute thespan denotation dij (Eq. 5). Importantly, this func-tion has access only to words in the span qij andnot to the entire input utterance. We would likefd(·) to support both simple compositions that de-pend only on the denotations of sub-spans, as wellas more complex functions that take into accountthe visual representations of different objects (spa-tial relations, colors, etc.).

2We also use Dropout and Layer-Norm (Ba et al., 2016)throughout the paper, omitted for simplicity.

We define four modules in fd(·) for computingdenotations and let the model learn when to useeach module (we show in §4 that two modulessuffice, but four improve interpretability). Themodules are: SKIP, INTERSECTION, UNION, anda general-purpose VISUAL function, where onlyVISUAL uses the visual representations V . As il-lustrated in Fig. 3, each module m outputs a de-notation vector dm

ij ∈ [0, 1]nobj , and the denotationdij is a weighted average of the four modules:

p(m|i, j) ∝ exp(Wmodhij) ∈ R4 (9)

dij =∑m

p(m|i, j) · dmij ∈ Rnobj , (10)

where Wmod ∈ Rhdim×4. Next, we define the fourmodules (see Fig. 4).

SKIP In many cases, only one of the left or rightsub-spans have a meaningful denotation: for ex-ample, for the sub-spans “there is a” and “redcube”, we should only keep the denotation of theright sub-span. To that end, the SKIP moduleweighs the two denotations and sums them:

(c(1)ij , c

(2)ij ) = softmax (Wskhij) ∈ R2 (11)

dskij = c

(1)ij · dijL + c

(2)ij · dijR ∈ Rnobj , (12)

where Wsk ∈ Rhdim×2.

INTERSECTION and UNION We define twosimple modules that only use the denotations dijLand dijR . The first module corresponds to inter-section of two sets, and the second to union:

dintij = min

(dijL , dijR

)∈ Rnobj , (13)

duniij = max

(dijL , dijR

)∈ Rnobj , (14)

where min(·) and max(·) are computed element-wise, per object. We show in §4.2 that while thesetwo modules are helpful for interpretability, theireffect on performance is relatively small, and theycan be omitted for simplicity.

VISUAL This module is responsible for com-positions that involve visual computation, suchas computing spatial relations (“left of the redsphere”) and comparing attributes of objects(“has the same size as the red sphere”). Unlikeother modules, in addition to sub-span denotationsit also uses the visual representations of the ob-jects, V ∈ Rnobj×hdim . For example, for the sub-spans “left of” and “the red object”, we expect

the function to ignore dijL (since the denotationof “left to” is irrelevant), and return a denotationwith high probability for objects that are left to ob-jects with high probability in dijR .

To determine whether an object with index oshould have high probability in the output, weneed to consider its relation to all other ob-jects. A simple scoring function might be (hij +vo1)T (hij + vo2), which will capture the relationbetween all pairs of objects conditioned on thespan representation. However, this computationis quadratic in nobj. Instead, we propose a lin-ear alternative that again leverages expected de-notations of sub-spans. Specifically, we computethe expected visual representation of the right sub-span and process this representation with a feed-forward layer:

vR = dRV ∈ Rhdim , (15)

qR = FFR (Whhij + vR) ∈ Rhdim . (16)

We use the right sub-span because the syntax inCLEVR is mostly right-branching, but a sym-metric term can be computed if needed. Then,we generate a representation q(o) for every objectthat is conditioned on the span representation hij ,the object probability under the sub-span denota-tions, and its visual representation. The final ob-ject probability is based on the interaction of q(o)and qR:

q(o) = FFvis(Whhij + vo + dL(o)s1 + dR(o)s2

)dvisij (o) = σ

(q(o)TqR

)where Wh ∈ Rhdim×hdim , s1, s2 ∈ Rhdim are learnedembeddings and FFvis is a feed-forward layer ofsize hdim × hdim with a non-linear activation. Thisis the most expressive module we propose.

Relation to CCG Our approach is related toclassical linguistic formalisms, such as CCG(Steedman, 1996), that tightly couple syntax andsemantics. Under this view, one can considerthe representations h and denotations d as anal-ogous to syntax and semantics, and our composi-tion functions and modules perform syntactic andsemantic neural composition.

3.4 GroundingIn lines 3-4 of Algorithm 1, we initialize the rep-resentations and denotations of length-1 spans.The representation hi is initialized as the corre-sponding word embeddingEqi , and the denotation

is computed with a grounding function. A sim-ple implementation for fground would be σ(h>i V ),based on the dot product between the word repre-sentation and the visual representations of all ob-jects. However, in the case of a co-referring pro-noun (“it”), we want to ground the pronoun to thedenotation of a previous span. We now describehow we address this case.

Coreference Sentences such as “there is a redsphere; what is its material?” are harder to an-swer with a CKY parser, since the denotation of“its” depends on the denotation of a distant span.We propose a simple heuristic for this issue, thataddresses the case where the referenced object isthe denotation of a previous sentence. This solu-tion could be potentially expanded in future work,to a wider array of coreference phenomena.

In every example that comprises two sen-tences:3 (a) We compute the denotationdfirst for the entire first sentence as de-scribed (standard CKY); (b) We ground eachword in the second sentence as proposedabove: d

secondi = σ(hsecond

i>V ); (c) For each

word in the second sentence, we predictwhether it co-refers to dfirst using a learnedgate (r1, r2) = softmax

(FFcoref(hsecond

i )),

where FFcoref ∈ Rhdim×2. (d) We definedsecondi = r1 · dfirst + r2 · d

secondi .

Visual representation Next, we describe howwe compute the visual embedding matrix V . Twocommon approaches to obtain visual features are(1) computing a feature map for the entire imageand letting the model learn to attend to the cor-rect feature position (Hudson and Manning, 2018;Perez et al., 2018); and (2) predicting the locationsof objects in the image, and extracting features justfor these objects (Anderson et al., 2018; Tan andBansal, 2019; Chen et al., 2019). We use the latterapproach, since it simplifies learning over discretesets, and has better memory efficiency – the modelonly attends to a small set of objects rather then theentire image feature map.

Specifically, we run CLEVR images through aRESNET101 model (He et al., 2016), pre-trainedon ImageNet (Russakovsky et al., 2015). Thismodel outputs a feature map Vall of size W ×H ×D, where D = 512 and W = H = 28. We thenuse an object detector, Faster R-CNN (Ren et al.,2015), which predicts the location bbpred ∈ R4 of

3in CLEVR, we split sentences based on semi-colons.

all objects in the image, in the format of bound-ing boxes (horizontal and vertical positions, widthand height). We use these predicted locations tocompute Vpred, containing only the features in Vallthat are predicted to contain an object accordingto bbpred. Since Faster R-CNN was trained on realimages, we adapt it to CLEVR images by train-ing it to predict bounding boxes of 5,000 objectsfrom CLEVR images (and 1,000 images used forvalidation), using gold scene data. The boundingboxes and features are extracted and fixed as a pre-processing step.

Finally, to compute V , in a similar fashion toLXMERT and UNITER we augment the objectrepresentations in Vpred with their position embed-dings, and pass them through a single Transformerself-attention layer to add context about otherobjects: V = TransformerLayer(VpredWfeat +bbpredWpos) , where Wfeat ∈ RD×hdim and Wpos ∈R4×hdim .

Complexity Similar to CKY, we go over allO(n2) spans in a sentence, and for each span com-pute hk

ij for each of the possible O(n) splits (thereis no grammar constant since the grammar has ef-fectively one rule). To compute denotations dij ,for all O(n2) spans, we perform a linear computa-tion over all nobj objects. Thus, the algorithm runsin timeO(n3+n2nobj), with similar memory con-sumption. This is higher than end-to-end modelsthat do not compute explicit span representations.

3.5 Training

The model is fully differentiable, and we train withmaximum likelihood, maximizing the log proba-bility log p(a∗ | q, I) of the correct answer a∗ (seeAlgorithm 1).

4 Experiments

In this section, we evaluate our model on both in-distribution and out-of-distribution splits.

4.1 Arithmetic expressions

It has been shown that neural networks can betrained to perform numerical reasoning (Zarembaand Sutskever, 2014; Kaiser and Sutskever, 2016;Trask et al., 2018; Geva et al., 2020). However,models are often evaluated on expressions that aresimilar to the ones they were trained on, whereonly the numbers change. To test for generaliza-tion, we create a simple dataset and evaluate on

Figure 5: Arithmetic expressions: unlike the easysetup, we evaluate models on expressions with oper-ations ordered in ways unobserved at training time.Flipped operator positions are in red.

two splits that require learning the correct opera-tor precedence and outputs. In the first split, se-quences of operators that appear at test time donot appear at training time. In the second split, thetest set contains longer sequences compared to thetraining set.

We define an arithmetic expression as a se-quence containing n numbers with n−1 arithmeticoperators between each pair. The answer a is theresult evaluating the expression.

Evaluation setups The sampled operators areaddition and multiplication, and we take only ex-pressions such that a ∈ {0, 1, . . . , 100} to train asa multi-class problem. During training, we ran-domly pick the length n to be up to ntrain, and dur-ing test time we choose a fixed length ntest. Weevaluate on three setups. In the easy split, wechoose ntrain = ntest = 8, and the sequence ofoperators is randomly drawn from a uniform dis-tribution for both training and test examples. Inthis setup, we only check that the exact same ex-pression is not shared between the training and testset. In the compositional split, we randomly pick3 positions, and for each one randomly assign ex-actly one operator that will appear at training time.On the test set, the operators in all three positionsare flipped, so that they now contain the unseenoperator (see Fig. 5). The same lengths are usedas in the easy split. Finally, in the length split, wetrain with ntrain = 8 and test with ntest = 10. Ex-amples for all setups are generated on-the-fly for 3million steps with a batch size of 100.

Models We compare GLT to a standard Trans-former, where the input is the expression, and theoutput is predicted using a classification layer overthe [CLS] token. All models are trained with

Easy split Op. split Len. split

Transformer 100.0± 0.0 2.9± 1.1 10.4± 2.4GLT 99.9± 0.2 98.4± 0.7 94.9± 1.1

Table 1: Arithmetic expressions results for easy split,operation-position split, and length split.

cross-entropy loss given the correct answer.For both models, we use an in-distribution val-

idation set for hyper-parameter tuning. For theTransformer, we use 15 layers with a hidden sizeof 200 and feed-forward layer size of 300. Forour model, we use hdim = 400. Since in thissetup we do not have an image or any groundedinput, we only compute hij for all spans, and de-fine p(a | q) = softmax(Wh0n).

GLT layers are almost entirely recurrent, that is,the same parameters are used to compute repre-sentations for spans of all lengths. The only excep-tion are layer-normalization parameters, which arenot shared across layers. Thus, at test time whenprocessing an expression longer than observed attraining time, we use the layer-normalization pa-rameters (total of 2 · hdim parameters per layer)from the longest span seen at training time.4

Results Results are reported in Table 1. Wesee that both models almost completely solvethe in-distribution setup, but on out-of-distributionsplits the Transformer performs poorly, while GLTshows only a small drop in accuracy.

4.2 CLEVR and CLOSURE

We evaluate performance on grounded complexquestions using CLEVR (Johnson et al., 2017a),consisting of 100,000 synthetic images with mul-tiple objects of different shapes, colors, materialsand sizes. 864,968 questions were syntheticallycreated using 80 different templates, includingsimple questions ("what is the size of red cube?")and questions requiring multi-step reasoning (seeFigure 1). The split in this dataset is i.i.d: tem-plates used for training are the same as those inthe validation and test sets.

To test compositional generalization after train-ing on CLEVR, we use the recent CLOSURE

dataset (Bahdanau et al., 2019b), which includesseven new question templates, with a total of

4Removing layer normalization leads to improved accu-racy of 99% on the arithmetic expressions length split, buttraining on CLEVR becomes too slow.

TrainPrograms

TestPrograms

DeterministicExecution

CLEVR CLOSURE

MAC no no no 98.5 72.4FiLM no no no 97.0 60.1GLT (our model) no no no 98.4 92.8 ± 3.0

NS-VQA yes no yes 100 77.2

PG+EE (18K prog.) yes no no 95.4 -PG-Vector-NMN yes no no 98.0 71.3

GT-Vector-NMN yes yes no 98.0 94.4

Table 2: Test results for all models on CLEVR and CLOSURE. “Train Programs” stands for models trained withgold program, “Test Programs” for oracle models evaluated using gold programs, and “Deterministic Execution”for models that depend on domain-knowledge for execution (execution is not learned).

25,200 questions, asked on the CLEVR valida-tion set images. The new templates are created bytaking referring expressions of various types fromCLEVR and combining them in novel ways.

A problem found in CLOSURE is that sentencesfrom the template embed_mat_spa are ambigu-ous. For example, in the question “Is there asphere on the left side of the cyan object that isthe same size as purple cube?”, the phrase “thatis the same size as purple cube” can modify ei-ther “the sphere” or “the cyan object”, but theanswer in CLOSURE is always the latter. There-fore, we deterministically compute both of the twopossible answers and keep two sets of question-answer pairs of this template for the entire dataset.We evaluate models5 on this template by takingthe maximum score over these two sets (such thatmodels must be consistent and choose a single in-terpretation for the template to get a perfect score).

Baselines We evaluate against the baselines pre-sented in Bahdanau et al. (2019b). The most com-parable baselines are MAC (Hudson and Man-ning, 2018) and FiLM (Perez et al., 2018), whichare differentiable and do not use any programannotations. We also compare to NMNs thatrequire at least a few hundred program exam-ples for training. We show results for PG+EE(Johnson et al., 2017b) and an improved ver-sion, PG-Vector-NMN (Bahdanau et al., 2019b).Last, we compare to NS-VQA, which in addi-tion to parsing the question, also parses the sceneinto a knowledge graph. NS-VQA also requiresdomain-knowledge and data, as it parses the imageinto a knowledge graph based on gold data from

5We update the scores on CLOSURE for MAC, FiLM andGLT due to this change in evaluation. The scores for the restof the models were not affected.

CLEVR (objects color, shape, location, etc.).

Setup Baseline results are taken from previouspapers (Bahdanau et al., 2019b; Hudson and Man-ning, 2018; Yi et al., 2018; Johnson et al., 2017b),except for MAC and FiLM on CLOSURE, whichwe re-executed due to the aforementioned evalua-tion change. For GLT, we use CLEVR’s valida-tion set for hyper-parameter tuning, and run 4 ex-periments to compute mean and variance on CLO-SURE test set. We train for 40 epochs and per-form early-stopping on CLEVR’s validation set.We use hdim = 400.

Because of our model’s high run-time and mem-ory demands (see §3.4), we found that runningon CLEVR and CLOSURE, where question lengthgoes up to 42 tokens, is difficult. Thus, we deletefunction words that typically have empty deno-tations and can be safely skipped,6 reducing themaximum length to 25.

CLEVR and CLOSURE In this experiment wecompare results on i.i.d and compositional splits.Results are in Table 2. We see that GLT performswell on CLEVR and gets the highest score onCLOSURE, improving by almost 20 points overcomparable models. GLT is competitive even withthe oracle GT-Vector-NMN which uses gold pro-grams at test time.

Removing intersection and union As de-scribed in §3.3, we defined two modules specifi-cally for CLEVR, (INTERSECTION and UNION).We remove these modules to evaluate performancewithout them, and see that the model suffers onlya small loss in accuracy and generalization: accu-racy on CLEVR (validation set) is 98.0± 0.3, and

6The removed tokens are punctuations, ‘the’, ‘there’, ‘is’,‘a’, ‘as’, ‘it’, ‘its’, ‘of’, ‘are’, ‘other’, ‘on’, ‘that’.

CLOSURE FS C.Humans

MAC 90.2 81.5FiLM - 75.9GLT (our model) 96.1 ± 0.9 72.8

NS-VQA 92.9 67.0

PG-Vector-NMN 88.0 -PG+EE (18K prog.) - 66.6

Table 3: Test results in the few-shot setup and forCLEVR-Humans.

accuracy on CLOSURE (test set) is 90.1± 7.1. Re-moving these modules leads to more cases wherethe VISUAL function is used, effectively perform-ing intersection and union as well. While thedrop in performance and generalization is mild,this model is harder to interpret since the VISUAL

function performs multiple functions.

Few-shot We test GLT in a few-shot (FS) setup,where we add a few out-of-distribution exam-ples. Specifically, we use 36 questions for eachCLOSURE template, with a total of 252 exam-ples. Similar to Bahdanau et al. (2019b), we takea model that was trained on CLEVR and fine-tune it by oversampling CLOSURE examples (300times) and adding them to the original training set.To make results comparable to Bahdanau et al.(2019b), we perform model selection based on theCLOSURE validation set, and evaluate on the testset. As we see in Table 3, GLT gets the best ac-curacy. If we perform model selection based onCLEVR alone (the preferred way to evaluate inthe OOD setup, Teney et al. 2020), accuracy onCLOSURE is 94.2 ± 2.1, which is still highest.

CLEVR-Humans To test the performance ofGLT on real natural language, we test on CLEVR-Humans (Johnson et al., 2017b), which consists of32,164 questions based on images from CLEVR.These questions, asked and answered by humans,contain new words and reasoning steps that werenot seen in CLEVR. We take a model that wastrained on CLEVR and fine-tune it on CLEVR-Humans training set, similar to prior work. Weuse GloVe (Pennington et al., 2014) for the em-beddings of words unseen in CLEVR. We showresults in Table 3. We see that GLT gets betterresults than models that use programs, showing itsflexibility to learn new concepts and phrasings, butlower results compared to more flexible modelslike MAC and FILM (see error analysis below).

4.3 Error analysis

We sampled 25 questions with wrong predictionson CLEVR, CLOSURE, and CLEVR-Humans toanalyze model errors. On CLEVR, most errors(84%) are due to problems in visual processing ofthe images such as grounding the word “rubber”to a metal object, problems in bounding box pre-diction or questions that require subtle spatial re-lation reasoning, such as identifying if an object isleft to another object of different size, when theyare at an almost identical x-position. The remain-ing errors (16%) are due to failed comparisons ofnumbers or attributes (“does the red object havethe same material as the cube”).

On CLOSURE, 60% of the errors were similarto those seen in CLEVR, e.g. problematic visualprocessing or failed comparisons. We’ve foundthat in 4% of cases, the execution of the VISUAL

module was wrong, e.g. it collapsed two reason-ing steps (both intersection and finding objects ofsame shape), but did not output the correct deno-tation. Other errors (36%) are in the predicted la-tent tree, where the model was uncertain about thesplit point and softly predicted more than one tree,resulting in wrong answer predictions. In somecases (16%) this was due to question ambiguity(see §4.2), and in others cases the cause was un-clear (e.g., for the phrase “same color as the cube”the model gave similar probability for the split af-ter “same” and after “color”, leading to a wrongdenotation for that span).

On CLEVR-Humans, we see that the modelsuccessfully learned certain new “concepts” suchas colors (“gold”), superlatives (“smallest”,“largest”), relations (“the reflecting object”), po-sitions (“back left”) and negation (see Fig. 6). Italso answered correctly questions with differentstyle than CLEVR (“Are there more blocks orballs?”, “... cube being covered by ...”). How-ever, the model fails on other new concepts, suchas the “all“ quantifier, arithmetic computations(“how many more... are there than...?”), and oth-ers (“What is the most common shape?”).

4.4 Interpretability

A key advantage of latent trees is interpretabil-ity – one can analyze the computation structureof a given question. Next, we analyze when aremodel outputs interpretable, and discuss how in-terpretability is affected by the limitations of GLTand relates to its generalization abilities. Addi-

Figure 6: An example from CLEVR-Humans. The model learned to negate (“not”) using the VISUAL module(negation is not part of CLEVR).

Figure 7: An example from CLEVR-Humans. This question requires reasoning steps that are not explicitlymentioned in the input. This results in a correct answer but non-interpretable output.

tional output examples can be seen in Appendix A.

The model predicts a denotation for each span,which is a probability for all objects in the im-age. Thus, for every question span that shouldcorrespond to a set of objects, the output is inter-pretable, as can be seen in Fig. 1. Having inter-pretable tree structures helps analyze ambiguousquestions, such as the ones found in CLEVR andCLOSURE.

However, span denotations are not always dis-tributions over objects, but rather a number or anattribute. For example, in comparison questions(“is the number of cubes higher than the numberof spheres?”) a fully interpretable model wouldhave a numerical denotation for each group of ob-jects. GLT solves such questions, by groundingthe objects correctly and leaving the counting andarithmetic comparison to the answer function (line7 in Algorithm 1). However, this comes at a costto interpretability (see Fig. 10). In the numeri-cal comparison example, it is easy to inspect thegrounding of objects, but hard to tell what is thecount for each group, which is likely to affect gen-eralization as well. A future research direction isto learn richer denotation structure.

Another case where interpretability is sub-optimal is counting. Due to the expressivity of theanswer function, the denotation in counting ques-tions does not necessarily contain only the objectsto be counted. For example, for a question suchas “how many cubes are there”, the most inter-pretable model would only have all the cubes inthe denotation of the entire question. However,GLT often outputs non-interpretable probabilitiesfor the objects. In such cases, the outputs are inter-pretable for sub-spans of the question (“cubes arethere”), as seen in Fig. 6. This issue could be ad-dressed by pre-training or injecting different countmodules, as shown by Subramanian et al. (2020).

Finally, the hardest case is when the requiredreasoning steps are not explicitly mentioned in thequestion. For example, the question “what is themost common shape?” requires to count the dif-ferent shapes in the image, then take the shapewith the maximum count. While our model an-swers this question correctly (see Fig. 7), it doesso by “falling back” to the flexible answer func-tion, rather than by explicitly performing the re-quired computation. In future work, we will ex-plore combining the compositional generalizationabilities of our model, which grounds intermediate

answers to spans, with the advantages of NMNs,that support more flexible reasoning.

5 Conclusion

We propose a model for grounded question an-swering that strongly relies on compositional com-putation. We show our model leads to large gainsin a systematic generalization setup and providesan interpretable structure that can be inspected byhumans and sheds light on the model’s inner work-ings. Our work suggests that generalizing to un-seen language structures can benefit from a stronginductive bias in the network architecture. By lim-iting our model to compose non-contextualizedrepresentations in a recursive bottom-up manner,we outperform state-of-the-art models a challeng-ing compositional generalization task. Our modelalso obtains high performance on real natural lan-guage questions in the CLEVR-humans dataset.In future work, we plan to investigate the struc-tures revealed by our model in other groundedquestion answering setups, and to allow the modelmore freedom to incorporate non-compositionalsignals, which go hand in hand with compositionalcomputation in natural language.

Acknowledgements

This research was partially supported by The Yan-dex Initiative for Machine Learning, and the Euro-pean Research Council (ERC) under the EuropeanUnion Horizons 2020 research and innovation pro-gramme (grant ERC DELPHI 802800). We thankJonathan Herzig for his useful comments. Thiswork was completed in partial fulfillment for thePh.D degree of Ben Bogin.

References

Peter Anderson, Xiaodong He, Chris Buehler,Damien Teney, Mark Johnson, Stephen Gould,and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and vi-sual question answering. In Proceedings of theIEEE conference on computer vision and pat-tern recognition, pages 6077–6086.

Jacob Andreas, Marcus Rohrbach, Trevor Darrell,and Dan Klein. 2016. Neural module networks.2016 IEEE Conference on Computer Vision andPattern Recognition (CVPR), pages 39–48.

Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E.Hinton. 2016. Layer normalization. ArXiv,abs/1607.06450.

Dzmitry Bahdanau, Shikhar Murty, MichaelNoukhovitch, Thien Huu Nguyen, Harmde Vries, and Aaron Courville. 2019a. System-atic generalization: What is required and canit be learned? In International Conference onLearning Representations.

Dzmitry Bahdanau, Harm de Vries, Timothy J.O’Donnell, Shikhar Murty, Philippe Beaudoin,Yoshua Bengio, and Aaron C. Courville. 2019b.Closure: Assessing systematic generalization ofclevr models. ArXiv, abs/1912.05783.

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed ElKholy, Faisal Ahmed, Zhe Gan, Yu Cheng,and Jingjing Liu. 2019. Uniter: Learninguniversal image-text representations. ArXiv,abs/1909.11740.

Noam Chomsky. 1957. Syntactic Structures.Mouton.

John Cocke. 1969. Programming Languages andTheir Compilers: Preliminary Notes. New YorkUniversity, USA.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-trainingof deep bidirectional transformers for languageunderstanding. In Proceedings of the 2019Conference of the North American Chapterof the Association for Computational Linguis-tics: Human Language Technologies, Volume1 (Long and Short Papers), pages 4171–4186,Minneapolis, Minnesota. Association for Com-putational Linguistics.

Andrew Drozdov, Patrick Verga, Mohit Yadav,Mohit Iyyer, and Andrew McCallum. 2019.Unsupervised latent tree induction with deepinside-outside recursive auto-encoders. In Pro-ceedings of the 2019 Conference of the NorthAmerican Chapter of the Association for Com-putational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers),pages 1129–1141, Minneapolis, Minnesota. As-sociation for Computational Linguistics.

Catherine Finegan-Dollak, Jonathan K. Kummer-feld, Li Zhang, Karthik Ramanathan, Sesh

Sadasivam, Rui Zhang, and Dragomir Radev.2018. Improving text-to-SQL evaluationmethodology. In Proceedings of the 56th An-nual Meeting of the Association for Compu-tational Linguistics (Volume 1: Long Papers),pages 351–360, Melbourne, Australia. Associa-tion for Computational Linguistics.

Jerry A. Fodor and Zenon W. Pylyshyn. 1988.Connectionism and Cognitive Architecture: ACritical Analysis. MIT Press, Cambridge, MA,USA.

Mor Geva, Ankit Gupta, and Jonathan Berant.2020. Injecting numerical reasoning skills intolanguage models. In Proceedings of the 58thAnnual Meeting of the Association for Com-putational Linguistics, pages 946–958, Online.Association for Computational Linguistics.

Nitish Gupta and Mike Lewis. 2018. Neural com-positional denotational semantics for questionanswering. In Proceedings of the 2018 Con-ference on Empirical Methods in Natural Lan-guage Processing, pages 2152–2161, Brussels,Belgium. Association for Computational Lin-guistics.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, andJian Sun. 2016. Deep residual learning forimage recognition. 2016 IEEE Conferenceon Computer Vision and Pattern Recognition(CVPR), pages 770–778.

Sepp Hochreiter and Jürgen Schmidhuber. 1997.Long short-term memory. Neural Computation,9:1735–1780.

Drew Arad Hudson and Christopher D. Manning.2018. Compositional attention networks formachine reasoning. In International Confer-ence on Learning Representations.

Yichen Jiang and Mohit Bansal. 2019. Avoid-ing reasoning shortcuts: Adversarial evaluation,training, and model development for multi-hopQA. In Proceedings of the 57th Annual Meetingof the Association for Computational Linguis-tics, pages 2726–2736, Florence, Italy. Associ-ation for Computational Linguistics.

Justin Johnson, Bharath Hariharan, Laurensvan der Maaten, Li Fei-Fei, C Lawrence Zit-nick, and Ross Girshick. 2017a. Clevr: A di-

agnostic dataset for compositional language andelementary visual reasoning. In CVPR.

Justin Johnson, Bharath Hariharan, Laurensvan der Maaten, Judy Hoffman, Li Fei-Fei,C Lawrence Zitnick, and Ross Girshick. 2017b.Inferring and executing programs for visual rea-soning. In ICCV.

Lukasz Kaiser and Ilya Sutskever. 2016. Neuralgpus learn algorithms. International Confer-ence on Learning Representations.

T. Kasami. 1965. An efficient recognition andsyntax analysis algorithm for context-free lan-guages. Technical Report AFCRL-65-758, AirForce Cambridge Research Laboratory, Bed-ford, MA†.

Daniel Keysers, Nathanael Schärli, Nathan Scales,Hylke Buisman, Daniel Furrer, Sergii Kashu-bin, Nikola Momchev, Danila Sinopalnikov,Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov,Xiao Wang, Marc van Zee, and Olivier Bous-quet. 2020. Measuring compositional general-ization: A comprehensive method on realisticdata. In International Conference on LearningRepresentations.

Brenden M. Lake and Marco Baroni. 2018. Gen-eralization without systematicity: On the com-positional skills of sequence-to-sequence recur-rent networks. In ICML.

Brenden M. Lake, Tomer D. Ullman, Joshua B.Tenenbaum, and Samuel Gershman. 2018.Building machines that learn and think likepeople. The Behavioral and brain sciences,40:e253.

Phong Le and Willem Zuidema. 2015. The forestconvolutional network: Compositional distribu-tional semantics with a neural chart and with-out binarization. In Proceedings of the 2015Conference on Empirical Methods in NaturalLanguage Processing, pages 1155–1164, Lis-bon, Portugal. Association for ComputationalLinguistics.

Percy Liang, Michael I Jordan, and Dan Klein.2013. Learning dependency-based composi-tional semantics. Computational Linguistics,39(2):389–446.

Yang Liu, Matt Gardner, and Mirella Lapata.2018. Structured alignment networks formatching sentences. In Proceedings of the 2018Conference on Empirical Methods in NaturalLanguage Processing, pages 1554–1564, Brus-sels, Belgium. Association for ComputationalLinguistics.

Jean Maillard, Stephen Clark, and Dani Yo-gatama. 2019. Jointly learning sentence embed-dings and syntax with unsupervised tree-lstms.ArXiv, abs/1705.09189.

Jiayuan Mao, Chuang Gan, Pushmeet Kohli,Joshua B. Tenenbaum, and Jiajun Wu. 2019.The neuro-symbolic concept learner: Interpret-ing scenes, words, and sentences from naturalsupervision. In International Conference onLearning Representations.

Richard Montague. 1970. Universal grammar.Theoria, 36(3):373–398.

Jeffrey Pennington, Richard Socher, and Christo-pher Manning. 2014. GloVe: Global vectorsfor word representation. In Proceedings ofthe 2014 Conference on Empirical Methods inNatural Language Processing (EMNLP), pages1532–1543, Doha, Qatar. Association for Com-putational Linguistics.

Ethan Perez, Florian Strub, Harm de Vries, Vin-cent Dumoulin, and Aaron C. Courville. 2018.Film: Visual reasoning with a general condi-tioning layer. In AAAI.

Colin Raffel, Noam Shazeer, Adam Roberts,Katherine Lee, Sharan Narang, MichaelMatena, Yanqi Zhou, Wei Li, and Peter J. Liu.2019. Exploring the limits of transfer learningwith a unified text-to-text transformer. ArXiv,abs/1910.10683.

Shaoqing Ren, Kaiming He, Ross Girshick, andJian Sun. 2015. Faster r-cnn: Towards real-timeobject detection with region proposal networks.In Proceedings of the 28th International Con-ference on Neural Information Processing Sys-tems - Volume 1, NIPS’15, pages 91–99, Cam-bridge, MA, USA. MIT Press.

Laura Ruis, Jacob Andreas, Marco Baroni, Di-ane Bouchacourt, and Brenden M. Lake. 2020.A benchmark for systematic generalization in

grounded language understanding. ArXiv,abs/2003.05161.

Olga Russakovsky, Jia Deng, Hao Su, JonathanKrause, Sanjeev Satheesh, Sean Ma, Zhi-heng Huang, Andrej Karpathy, Aditya Khosla,Michael Bernstein, and et al. 2015. Im-agenet large scale visual recognition chal-lenge. International Journal of Computer Vi-sion, 115(3):211–252.

Mark Steedman. 1996. Surface structure and in-terpretation.

Sanjay Subramanian, Ben Bogin, Nitish Gupta,Tomer Wolfson, Sameer Singh, Jonathan Be-rant, and Matt Gardner. 2020. Obtaining faith-ful interpretations from compositional neuralnetworks. In Proceedings of the 58th AnnualMeeting of the Association for ComputationalLinguistics, pages 5594–5608, Online. Associ-ation for Computational Linguistics.

Hao Tan and Mohit Bansal. 2019. LXMERT:Learning cross-modality encoder representa-tions from transformers. In Proceedings of the2019 Conference on Empirical Methods in Nat-ural Language Processing and the 9th Interna-tional Joint Conference on Natural LanguageProcessing (EMNLP-IJCNLP), pages 5100–5111, Hong Kong, China. Association for Com-putational Linguistics.

Damien Teney, Kushal Kafle, Robik Shrestha,Ehsan Abbasnejad, Christopher Kanan, and An-ton van den Hengel. 2020. On the value of out-of-distribution testing: An example of good-hart’s law. ArXiv, abs/2005.09241.

Andrew Trask, Felix Hill, Scott E Reed, Jack Rae,Chris Dyer, and Phil Blunsom. 2018. Neuralarithmetic logic units. In Advances in NeuralInformation Processing Systems, pages 8035–8044.

Ashish Vaswani, Noam Shazeer, Niki Parmar,Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Ł ukasz Kaiser, and Illia Polosukhin. 2017. At-tention is all you need. In I. Guyon, U. V.Luxburg, S. Bengio, H. Wallach, R. Fergus,S. Vishwanathan, and R. Garnett, editors, Ad-vances in Neural Information Processing Sys-tems 30, pages 5998–6008. Curran Associates,Inc.

Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Tor-ralba, Pushmeet Kohli, and Joshua B. Tenen-baum. 2018. Neural-symbolic vqa: Disentan-gling reasoning from vision and language un-derstanding. In Advances in Neural Informa-tion Processing Systems, pages 1039–1050.

D.H. Younger. 1967. Recognition and parsing ofcontext-free languages in time n3. Informationand Control, 10(2):189–208.

Wojciech Zaremba and Ilya Sutskever. 2014.Learning to execute. ArXiv, abs/1410.4615.

Luke Zettlemoyer and Michael Collins. 2005.Learning to map sentences to logical form:Structured classification with probabilistic cat-egorial grammars. In UAI.

A Output Examples

We show 3 additional examples of our model out-puts, along with the induced trees and denotationsin the following pages.

Figu

re8:

An

exam

ple

from

CL

OS

UR

E.∪

stan

dsfo

rthe

UN

ION

mod

ule.

Figure 9: An example from CLEVR-Humans.

Figure 10: An example from CLEVR.