benchmarking meaning representations in neural semantic ...1in this paper, we focus on grounded...

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1520–1540,November 16–20, 2020. c©2020 Association for Computational Linguistics

1520

Benchmarking Meaning Representations in Neural Semantic Parsing

Jiaqi Guo1∗, Qian Liu2∗, Jian-Guang Lou3, Zhenwen Li4Xueqing Liu5, Tao Xie4, Ting Liu1

1Xi’an Jiaotong University, Xi’an, China 2Beihang University, Beijing, China3Microsoft Research, Beijing, China 4Peking University, Beijing, China

5Stevens Institute of Technology, Hoboken, New Jersey, [email protected], [email protected]@microsoft.com, {lizhenwen, taoxie}@[email protected], [email protected]

AbstractMeaning representation is an important com-ponent of semantic parsing. Although re-searchers have designed a lot of meaning rep-resentations, recent work focuses on only afew of them. Thus, the impact of meaningrepresentation on semantic parsing is less un-derstood. Furthermore, existing work’s per-formance is often not comprehensively evalu-ated due to the lack of readily-available execu-tion engines. Upon identifying these gaps, wepropose UNIMER, a new unified benchmarkon meaning representations, by integrating ex-isting semantic parsing datasets, completingthe missing logical forms, and implementingthe missing execution engines. The resultingunified benchmark contains the complete enu-meration of logical forms and execution en-gines over three datasets × four meaning rep-resentations. A thorough experimental studyon UNIMER reveals that neural semantic pars-ing approaches exhibit notably different per-formance when they are trained to generatedifferent meaning representations. Also, pro-gram alias and grammar rules heavily impactthe performance of different meaning repre-sentations. Our benchmark, execution enginesand implementation can be found on: https://github.com/JasperGuo/Unimer.

1 Introduction

A remarkable vision of artificial intelligence is toenable human interactions with machines throughnatural language. Semantic parsing has emergedas a key technology for achieving this goal. Ingeneral, semantic parsing aims to transform a nat-ural language utterance into a logic form, i.e.,a formal, machine-interpretable meaning repre-sentation (MR) (Zelle and Mooney, 1996; Dahlet al., 1994).1 Thanks to the recent development

∗Work done during an internship at Microsoft Research.1In this paper, we focus on grounded semantic parsing,

where meaning representations are grounded to specific knowl-

MR Geo ATIS JobProlog - - 91.4Lambda 90.4 91.3 85.0FunQL 92.5 - -SQL 78.0 69.0 -Prolog 89.6 - 92.1Lambda - - -FunQL - - -SQL 82.5 79.2 -

Table 1: State-of-the-art performance for MRs on Geo,ATIS, and Job. The top table shows exact-match accu-racy whereas the bottom table shows execution-matchaccuracy. Most existing work focuses on evaluatingonly a small subset of dataset×MR pairs, leaving mostof the table unexplored. A finer-grained table is avail-able in the supplementary material (Table 8).

of neural networks techniques, significant improve-ments have been made in semantic parsing perfor-mance (Jia and Liang, 2016; Yin and Neubig, 2017;Dong and Lapata, 2018; Shaw et al., 2019).

Despite the advancement in performance, weidentify three important biases in existing work’sevaluation methodology. First, although multipleMRs are proposed, most existing work is evaluatedon only one or two of them, leading to less com-prehensive or even unfair comparisons. Table 1shows the state-of-the-art performance of semanticparsing on different dataset × MR combinations,where the rows are the MRs and the columns arethe datasets. We can observe that while LambdaCalculus is intensively studied, the other MRs havenot been sufficiently studied. This biased evalua-tion is partly caused by the absence of target logicforms in the missing cells. Second, existing workoften compares the performance on different MRsdirectly (Sun et al., 2020; Shaw et al., 2019; Chenet al., 2020) without considering the confounding

edge bases, instead of ungrounded semantic parsing.

https://github.com/JasperGuo/Unimerhttps://github.com/JasperGuo/Unimerhttps://github.com/JasperGuo/Unimerhttps://github.com/JasperGuo/Unimer

1521

role that MR plays in the performance,2 causingunfair comparisons and misleading conclusions.Third, a more comprehensive evaluation methodol-ogy would consider both the exact-match accuracyand the execution-match accuracy, because twologic forms can be semantically equivalent yet donot match precisely in their surface forms. How-ever, as shown in Table 1, most existing work isonly evaluated with the exact-match accuracy. Thisbias is potentially due to the fact that executionengines are not available in six out of the twelvedataset ×MR combinations.

Upon identifying the three biases, in this paper,we propose UNIMER, a new unified benchmark, byunifying four publicly available MRs in three of themost popular semantic parsing datasets: Geo, ATISand Jobs. First, for each natural language utter-ance in the three datasets, UNIMER provides anno-tated logical forms in four different MRs, includingProlog, Lambda Calculus, FunQL, and SQL. Weidentify that annotated logical forms in some MR× dataset combinations are missing. As a result,we complete the benchmark by semi-automaticallytranslating logical forms from one MR to another.Second, we implement six missing execution en-gines for MRs so that the execution-match accuracycan be readily computed for all the dataset ×MRcombinations. Both the logical forms and their ex-ecution results are manually checked to ensure thecorrectness of annotations and execution engines.

After constructing UNIMER, to obtain a pre-liminary understanding on the impact of MRson semantic parsing, we empirically study theperformance of MRs on UNIMER by using twowidely-used neural semantic parsing approaches(a seq2seq model (Dong and Lapata, 2016; Jiaand Liang, 2016) and a grammar-based neuralmodel (Yin and Neubig, 2017)), under the super-vised learning setting.

In addition to the empirical study above, wefurther analyze the impact of two operations, i.e.,program alias and grammar rules, to understandhow they affect different MRs differently. First,Program alias. A semantically equivalent programmay have many syntactically different forms. Asa result, if the training and testing data have adifference in their syntactic distributions of logicforms, a naive maximum likelihood estimationcan suffer from this difference because it fails to

2In (Kate et al., 2005; Liang et al., 2011; Guo et al., 2019),it was revealed that using an appropriate MR can substantiallyimprove the performance of a semantic parser.

capture the semantic equivalence (Bunel et al.,2018). As different MRs have different degreesof syntactic difference, they suffer from thisproblem differently. Second, Grammar rules.Grammar-based neural models can guarantee thatthe generated program is syntactically correct (Yinand Neubig, 2017; Wang et al., 2020; Sun et al.,2020). For a given set of logical forms in an MR,there exist multiple sets of grammar rules to modelthem. We observe that when the grammar-basedneural model is trained with different sets ofgrammar rules, it exhibits a notable performancediscrepancy. This finding alias with the one madein traditional semantic parsers (Kate, 2008) thatproperly transforming grammar rules can lead tobetter performance of a traditional semantic parser.

In summary, this paper makes the followingmain contributions:

• We propose UNIMER, a new unified bench-mark on meaning representations, by integrat-ing and completing semantic parsing datasetsin three datasets × four MRs; we also imple-ment six execution engines so that execution-match accuracy can be evaluated in all cases;

• We provide the baseline results for two widelyused neural semantic parsing approaches onour benchmark, and we conduct an empiricalstudy to understand the impact that programalias and grammar rule plays on the perfor-mance of neural semantic parsing;

2 Preliminaries

In this section, we provide a brief description ofthe MRs and neural semantic parsing approachesthat we study in the paper.

2.1 Meaning Representations

We investigate four MRs in this paper, namely,Prolog, Lambda Calculus, FunQL, and SQL, be-cause they are widely used in semantic parsing andwe can obtain their corresponding labeled data inat least one semantic parsing domain. We regardProlog, Lambda Calculus, and FunQL as domain-specific MRs, since the predicates defined in themare specific for a given domain. Consequently, theexecution engines of domain-specific MRs need tobe significantly customized for different domains,requiring plenty of manual efforts. In contrast, SQLis a domain-general MR for querying relational

1522

MR Logical Form

Prolog answer(A, ( flight(A) , tomorrow(A) , during day(A, B) , const (B, period (morning)),from(A, C) , const(C, city(Pittsburgh)), to(A, D), const (D, city(Atlanta))))LambdaCalculus

( lambda A:e ( (flight A) ∧ (during day A morning:pd) ∧ (from A Pittsburgh:ci)∧ (to A Atlanta:ci) ∧ (tomorrow A) ) )

FunQL answer ( flight ( tomorrow ( intersect ( during day ( period ( morning ) ) ,from ( city ( Pittsburgh ) ) , to ( city ( Atlanta ) ) ) ) ) )

SQL

SELECT flight id FROM . . . WHERE city 1.city name = ‘pittsburgh’AND city 2.city name = ‘atlanta’ AND date day 1.year = 1991AND date day 1.month number = 1 AND date day 1.day number = 20AND departure time BETWEEN 0 AND 1200

Table 2: Examples of meaning representations for utterance “what flights do you have in tomorrow morning frompittsburgh to atlanta?” in the ATIS domain.

databases. Its execution engines (e.g., MySQL)can be used directly in different domains. Table 2shows a logical form for each of the four MRs inthe ATIS domain.Prolog has long been used to represent the meaningof natural language (Zelle and Mooney, 1996; Kateand Mooney, 2006). Prolog includes first-orderlogical forms, augmented with some higher-orderpredicates, e.g., most, to handle issues such asquantification and aggregation. Take the first logi-cal form in Tables 2 as an example. The uppercasecharacters denote variables, and the predicates inthe logical form specify the constraints betweenvariables. In this case, character A denotes a vari-able, and it is required to be a flight, and the flightshould depart tomorrow morning from Pittsburghto Atlanta. The outer predicate answer indicatesthe variable whose binding is of interest. One ma-jor benefit of Prolog-style MRs is that they allowpredicates to be introduced in the order where theyare actually named in the utterance. For instance,the order of predicates in the logical form strictlyfollows their mentions in the natural language ut-terance.Lambda Calculus is a formal system to expresscomputation. It can represent all first-order logicand it naturally supports higher-order functions. Itrepresents the meanings of natural language withlogical expressions that contain constants, quan-tifiers, logical connectors, and lambda abstract.These properties make it prevalent in semantic pars-ing. Consider the second logical form in Table 2. Itdefines an expression that takes an entity A as inputand returns true if the entity satisfies the constraintsdefined in the expressions. Lambda Calculus canbe typed, allowing type checking during generationand execution.FunQL, abbreviated for Functional Query Lan-guage, is a variable-free language (Kate et al.,

2005). It abstracts away variables and encodescompositionality via its nested function-argumentstructure, making it easier to implement an efficientexecution engine for FunQL. Concretely, unlikeProlog and Lambda Calculus, predicates in FunQLtake a set of entities as input and return another setof entities that meet certain requirements. Consider-ing the third logical form in Table 2, the predicateduring day(period(morning)) returns aset of flights that depart in the morning. With thisfunction-argument structure, FunQL can directlyreturn the entities of interest.SQL is a popular relational database query lan-guage. Since it is domain-agnostic and has well-established execution engines, the subtask of se-mantic parsing, Text-to-SQL, has received a lot ofinterests. Compared with domain-specific MRs,SQL cannot encapsulate too much domain priorknowledge in its expressions. As shown in Table 2,to query flights that depart tomorrow, one needs tospecify the concrete values of year, month, and dayin the SQL query. However, these values are notexplicitly mentioned in the utterance and may evenchange over time.

It is important to note that although these MRsare all expressive enough to represent all mean-ings in some domains, they are not equivalent interms of their general expressiveness. For example,FunQL is less expressive than Lambda Calculus ingeneral, partially due to the elimination of variablesand quantifiers.

2.2 Neural Semantic Parsing Approaches

During the last few decades, researchers have pro-posed different approaches for semantic parsing.Most state-of-the-art approaches are based on neu-ral models and formulate the semantic parsing prob-lem as a sequence transduction problem. Due tothe generality of sequence transduction, these ap-

1523

Encoder Encoder

River

Encoder Encoder

in California ?

Attention

Decoder

answer

Decoder

answer

(

Decoder

(

river

… Decoder

)

(a) Seq2Seq Model

Encoder Encoder

River

Encoder Encoder

in California ?

Attention Decoder

Stat := “answer(” Var “,” Goal “)”

Decoder

Var := “A”

Decoder

Goal := “(“ Pred “,” Conj “)”

… Decoder

State := “stated(” StateName “)”

StateName := “California”

Stat := “answer(” Var “,” Goal “)”

Var := “A”

(b) Grammar-based Model

Figure 1: Illustrations of the seq2seq model and thegrammar-based model with utterance “Rivers in Cali-fornia?” and its corresponding logical form in Prolog.

proaches can be trained to generate any MRs. Inthis work, without loss of generality, we bench-mark MRs by evaluating the seq2seq model (Dongand Lapata, 2016; Jia and Liang, 2016) and thegrammar-based model (Yin and Neubig, 2017) un-der the supervised learning setting. We select thetwo models because most neural approaches aredesigned based on them.Seq2Seq Model. Dong and Lapata (2016) andJia and Liang (2016) formulated the semanticparsing problem as a neural machine translationproblem and employed the sequence-to-sequencemodel (Sutskever et al., 2014) to solve it. As illus-trated in Figure 1a, the encoder takes an utteranceas input and outputs a distributed representation foreach word in the utterance. A decoder then sequen-tially predicts words in the logical form. When aug-mented with the attention mechanism (Bahdanauet al., 2014; Luong et al., 2015), the decoder canbetter utilize the encoder’s information to predictlogical forms. Moreover, to address the problemcaused by the long tail distribution of entities inlogical forms, Jia and Liang (2016) proposed anattention-based copying mechanism. That is, ateach time step, the decoder takes one of two typesof actions, one to predict a word from the vocabu-lary of logical forms and the other to copy a wordfrom the input utterance.Grammar-based Model. By treating a logicalform as a sequence of words, the seq2seq modelcannot fully utilize the property that logical forms

are well-formed and must conform to certain gram-mars of an MR. To bridge this gap, Yin and Neubig(2017) proposed a grammar-based decoder that out-puts a sequence of grammar rules instead of words,as presented in Figure 1b. The decoded grammarrules can deterministically generate a valid abstractsyntax tree (AST) of a logical form. In this way,the generated logical form is guaranteed to be syn-tactically correct. This property makes it widelyused in a lot of code generation and semantic pars-ing tasks (Sun et al., 2020; Wang et al., 2020; Bo-gin et al., 2019). The grammar-based decoder canalso be equipped with the attention-based copy-ing mechanism to address the long-tail distributionproblem.

3 Benchmark

To provide an infrastructure for exploring MRs, weconstruct UNIMER, a unified benchmark on MRs,based on existing semantic parsing datasets. Cur-rently, UNIMER covers three domains, namely Geo,ATIS, and Job, each of which has been extensivelystudied in previous work and has annotated logicalforms for at least two MRs. All natural languageutterances in UNIMER are written in English.Geo focuses on querying a database of U.S. geog-raphy with natural language. To solve the prob-lem, Zelle and Mooney (1996) designed a Prolog-style MR and annotated 880 (utterance, logicalform) pairs. Popescu et al. (2003) and Kate et al.(2005) proposed to use SQL and FunQL to repre-sent the meanings, respectively. Almost the sametime, Zettlemoyer and Collins (2005) proposed touse Lambda Calculus and manually converted theProlog logical forms to equivalent expressions inLambda Calculus. Following their work, we adoptthe standard 600/280 training/test split.ATIS is a dataset of flight booking questions. Itconsists of 5,418 questions and their correspond-ing SQL queries. Zettlemoyer and Collins (2007)proposed to use Lambda Calculus to represent themeanings of natural language and automaticallymap these SQL queries to its equivalent logicalforms in Lambda Calculus. Following the workof Kwiatkowski et al. (2011), we use the standard4480/480/450 training/dev/test split.Job is a dataset about job announcements postedin the newsgroup austin.jobs (Califf and Mooney,1999). It consists of 640 utterances and their corre-sponding Prolog logical forms that query computer-related job postings. Similarly, in Geo, Zettlemoyer

1524

and Collins (2005) proposed using Lambda Cal-culus and manually converted them to equivalentexpressions in Lambda Calculus. We use the sametraining-test split with them, containing 500 train-ing and 140 test instances.

Since not all the four MRs that we introduce inSection 2.1 are used in these three domains, wesemi-automatically translate logical forms in oneMR into another. This effort enables researchersto explore MRs in more domains and make a faircomparison among them. Take the translation ofLambda Calculus to FunQL in ATIS as an exam-ple. We first design predicates for FunQL based onthose defined in Lambda Calculus and implementan execution engine for FunQL. Then, we translatelogical forms in Lambda Calculus to FunQL andcompare the execution results to verify the correct-ness of the translation. In this process, we findthat there is no ready-to-use Lambda Calculus ex-ecution engine for the three domains. Hence, weimplement one for each domain. These engines, onthe one hand, enable evaluations of semantic pars-ing approaches with both exact-match accuracy andexecution-match accuracy. On the other hand, theyenable exploration of weakly supervised semanticparsing with Lambda Calculus. In addition, wefind some annotation mistakes in logical forms andseveral bugs in existing execution engines of Pro-log and FunQL. By correcting the mistakes andfixing the bugs in the engines, we create a refinedversion of these datasets. Section A.1 in the supple-mentary material provides more details about theconstruction process.

We plan to cover more domains and more MRsin UNIMER. We have made UNIMER along withthe execution engines publicly available.3 We be-lieve that UNIMER can provide fertile soil for ex-ploring MRs and addressing challenges in semanticparsing.

4 Experimental Setup

Based on UNIMER, we take the first attempt tostudy the characteristics of different MRs and theirimpact on neural semantic parsing.

4.1 Experimental Design

Meaning Representation Comparison. To un-derstand the impact of MRs on neural semantic

3Our benchmark, execution engines and and our im-plementation can be found on: https://github.com/JasperGuo/Unimer

Rule Description

Shuffle Shuffle expressions in Select, From, Where,and Having clauses

Argmax Express Argmax/min with OrderBy andLimit clause instead of subqueryIn2Join Replace In clause with Join clause

Table 3: Three basic transformation rules for SQL.

parsing, we first experiment with the two neuralapproaches described in Section 2.2 on UNIMER,and we compare the resulting performance of dif-ferent MRs with two metrics: exact-match accu-racy (a logical form is regarded as correct if it issyntactically identical to the gold standard),4 andexecution-match accuracy (regarded as correct if alogical form’s execution result is identical to thatof the gold standard).5

Program Alias. To explore the effect of programalias, we replace a different proportion of logicalforms in a training set with their aliases (semanti-cally equivalent but syntactically different logicalforms), and we re-train the neural approaches toquantify its effect. To search for aliases of a logicalform, we first derive multiple transformation rulesfor each MR. Then, we apply these rules to thelogical form to get its aliases and randomly sam-ple one. We compare the execution results of theresulting logical forms to ensure their equivalencein semantics. Table 3 presents three transformationrules for SQL. We provide a detailed explanationof transformation rules and examples for each MRin Section A.3 of the supplementary material.Grammar Rules. To understand the grammarrules’ impact on grammar-based models, we pro-vide two sets of grammar rules for each MR. Eachset of rules can cover all the logical forms in thethree domains. We compare the performance ofmodels trained with different sets of rules. Specif-ically, Wong and Mooney (2006) and Wong andMooney (2007) have induced a set of grammarrules for Prolog and FunQL in Geo. We directlyuse them in Geo and extend them to support logicalforms in ATIS and Job. As for SQL, Bogin et al.(2019) have induced a set of rules for SQL in theSpider benchmark, and we adapt it to support theSQL queries in the three domains that we study.

4Following Dong and Lapata (2016), we sort the sub-expressions in the conjunction predicate of Lambda Calculusbefore comparison.

5It should be acknowledged that using the execution-matchaccuracy to evaluate a parser is not enough, as there existspurious programs that lead to the same execution results withthe gold standard but have different semantics.

https://github.com/JasperGuo/Unimerhttps://github.com/JasperGuo/Unimerhttps://github.com/JasperGuo/Unimerhttps://github.com/JasperGuo/Unimer

1525

When it comes to Lambda Calculus, we use theone induced by Yin and Neubig (2018). For com-parison, we also manually induce another set ofgrammar rules for the four MRs. Section A.4 inthe supplementary material provides definitions ofall the grammar rules.

4.2 ImplementationsWe implement each approach with the Al-lenNLP (Gardner et al., 2018) and PyTorch (Paszkeet al., 2019) frameworks. To make a fair compari-son, we tune the hyper-parameters of approachesfor each MR on the development set or throughcross-validation on the training set, with the NNIplatform.6 Due to the limited number of test datain each domain, we run each approach five timesand take the average number. Section A.2 in thesupplementary material provides the search spaceof hyper-parameters for each approach and the pre-processing procedures of logical forms.

Multiple neural semantic parsing ap-proaches (Dong and Lapata, 2016; Iyer et al.,2017; Rabinovich et al., 2017) adopt the dataanonymization techniques to replace entitiesin utterances with placeholders. However, thetechniques are usually ad-hoc and specific fordomains and MRs, and they sometimes requiremanual efforts to resolve conflicts (Finegan-Dollaket al., 2018). Hence, we do not apply dataanonymization to avoid bias.

5 Experimental Results

5.1 Meaning Representation ComparisonTable 4 presents our experimental resultson UNIMER. Since we do not use data anonymiza-tion techniques, the performance is generally lowerthan that shown in Table 1 and Table 8, but theperformance is on par with the numbers reportedin ablation studies of previous work (Dong andLapata, 2016; Jia and Liang, 2016; Finegan-Dollaket al., 2018). We can make the following threeobservations from the table.

First, neural approaches exhibit notably differ-ent performance when they are trained to generatedifferent MRs. The difference can vary by as muchas 20% in both exact-match and execution-matchmetrics. This finding tells us that an apple-to-applecomparison is extremely important when compar-ing two neural semantic parsing approaches. How-ever, we notice that some papers (Sun et al., 2020;

6https://github.com/microsoft/nni

P L F SGeo

0

20

40

60

80

100

120

140

160

P L F SATIS

0

50

100

150

200

250

300

350

400

P L F SJob

0

20

40

60

80

100

120

Figure 2: Statistics of logical forms in the training setof Prolog (P), Lambda Calculus (L), FunQL (F) andSQL (S). The y-axis indicates the number of productionrules in each logical form.

Shaw et al., 2019; Chen et al., 2020) do not clearlynote the MRs used in baselines and compare withthem using different metrics; the attained result canbe somewhat misleading.

Second, domain-specific MRs (Prolog, LambdaCalculus, and FunQL) tend to outperform SQL(domain-general) by a large margin. For example,in Geo, the execution-match accuracy of FunQLis substantially higher than that of SQL in all ap-proaches. This result is expected because a lot ofdomain knowledge is injected into domain-specificMRs. Consider the logical forms in Table 2. Thereis a predicate tomorrow in all three domain-specific MRs, and this predicate can directly alignto the description in the utterance. However, oneneeds to explicitly express the concrete date valuesin the SQL query; this requirement can be a heavyburden for neural approaches, especially when thevalues will change over time. In addition, a recentstudy (Finegan-Dollak et al., 2018) in Text-to-SQLhas shown that domain-specific MRs are more ro-bust against generating never-seen logical formsthan SQL, because their surface forms are muchcloser to natural language.

Third, among all the domain-specific MRs,FunQL tends to outperform the others in neuralapproaches. In Geo, FunQL outperforms the otherMRs in both metrics by a large margin. In Job,the grammar-based (w/ copy) model trained withFunQL achieves the state-of-the-art performance.One possible reason is that FunQL is more com-pact than the other MRs, due to its elimination ofvariables and quantifiers. Figure 2 shows box plotsabout the number of grammars rules in the AST ofa logical form. We can observe that while FunQLhas almost the same number of grammar rules withthe other MRs (Table 5), it has much fewer gram-mar rules involved in a logical form than the others

1526

Approach Prolog Lambda Calculus FunQL SQLExact Execution Exact Execution Exact Execution Exact ExecutionGeoSeq2Seq 70.0 ±2.1 73.9 ±2.4 64.7 ±2.3 70.1 ±2.1 76.8 ±2.4 79.4 ±2.2 58.0 ±2.4 68.7 ±3.7

w/ Copy 72.9 ±2.1 78.7 ±2.4 75.4 ±0.5 80.1 ±1.3 80.3 ±1.4 87.1 ±0.9 72.3 ±1.1 76.8 ±1.8Grammar 68.6 ±1.2 75.7 ±2.3 70.7 ±1.1 75.1 ±1.4 76.1 ±0.3 78.2 ±0.3 63.3 ±2.0 70.8 ±1.9

w/ Copy 74.3 ±2.1 79.5 ±1.1 75.6 ±1.5 80.7 ±0.7 81.8 ±0.3 86.2 ±0.4 67.9 ±0.7 72.1 ±1.0ATISSeq2Seq 65.9 ±1.5 73.8 ±1.3 68.7 ±1.8 76.3 ±0.9 70.8 ±0.6 76.3 ±1.0 5.6 ±0.3 61.1 ±4.6

w/ Copy 73.7 ±1.9 80.4 ±0.5 75.4 ±2.4 83.1 ±1.3 78.0 ±1.1 82.7 ±1.0 8.0 ±0.6 70.0 ±1.5Grammar 70.9 ±0.9 76.3 ±1.0 71.7 ±1.5 77.4 ±1.3 72.1 ±0.8 77.5 ±0.9 5.5 ±0.2 63.7 ±1.3

w/ Copy 73.4 ±1.2 79.2 ±1.1 75.7 ±0.4 82.1 ±0.8 76.5 ±0.7 82.7 ±1.2 7.2 ±0.6 61.0 ±1.1JobSeq2Seq 68.1 ±2.3 75.7 ±1.7 65.8 ±1.3 78.6 ±1.5 71.4 ±2.1 81.4 ±2.3 68.5 ±3.0 75.6 ±3.0

w/ Copy 71.4 ±3.6 79.1 ±2.2 77.9 ±2.8 88.0 ±1.4 75.6 ±2.4 85.9 ±3.0 78.7 ±2.1 87.3 ±2.3Grammar 70.4 ±2.4 79.4 ±2.4 68.4 ±3.9 82.9 ±4.1 72.9 ±2.7 86.3 ±2.5 74.6 ±1.5 83.3 ±1.8

w/ Copy 75.9 ±1.1 84.4 ±2.1 80.2 ±1.8 91.0 ±1.3 78.4 ±2.1 92.4 ±1.5 78.7 ±1.4 87.6 ±2.1

Table 4: Experimental results of the seq2seq model and the grammar-based model. We highlight the best perfor-mance of each approach and MR in both metrics. The poor SQL exact-match accuracy in ATIS is caused by thedifferent distribution of SQL queries in test set. Iyer et al. (2017) rewrote the SQL queries in test set to improvetheir execution efficiency.

MR Geo ATIS Job# Vo # Ru # Vo # Ru # Vo # RuProlog 146 234 503 732 170 230Lambda 180 245 466 532 200 208FunQL 152 188 494 706 195 208SQL 187 269 530 656 211 227

Table 5: Statistics of logical forms in different MRs.‘# Vo’ and ‘# Ru’ indicate the vocabulary size and thenumber of grammar rules of an MR, respectively.

on average. This statistic is crucial for neural se-mantic parsing approaches as it directly determinesthe number of decoding steps in decoders.

A similar reason can be used to explain that theperformance on SQL is lower than others. As Fig-ure 2 shows, SQL has larger medians of the numberof grammar rules, and it also has much more out-liers than domain-specific MRs. It makes neuralmodels more challenging to learn.

Interestingly, this finding contradicts thefinding in CCG-based semantic parsing ap-proaches (Kwiatkowksi et al., 2010), in which theyshow that Lambda Calculus outperforms FunQL inthe Geo domain. The reason is that compared withLambda Calculus, the deeply nested structure ofFunQL makes it more challenging to learn a high-quality CCG lexicon, which is crucial for CCGparsing. In contrast, neural approaches do not relyon a lexicon and directly learn a mapping betweensource and target languages.

5.2 Program Alias

Figure 3 shows the execution-match accuracy ofthe seq2seq (w/ copy) model with different pro-

0 25 50 75 100Replaced Logical Forms (%)

4045505560657075808590

Exec

utio

n-M

atch

(%)

PrologLambdaFunQLSQL

(a) Geo

0 25 50 75 100Replaced Logical Forms (%)

40455055606570758085

Exec

utio

n-M

atch

(%)

PrologLambdaFunQLSQL

(b) ATIS

Figure 3: Execution-match accuracy of the seq2seq(w/ copy) model when different proportions of logicalforms in training set are replaced with their aliases.

MR Geo ATISExact Execution Exact ExecutionProlog 24.2% 15.1% 7.0% 5.4%Lambda 11.9% 8.6% 6.2% 0.9%FunQL 24.7% 8.0% 7.0% 4.1%SQL 19.4% 4.9% 9.1% 3.6%

Table 6: Relative decline of performance when 25% oflogical forms in training data are replaced with aliases.

portions of logical forms in the training set re-placed with aliases.7 Since not all the logical formshave aliases, the curves in Figure 3 stop at differ-ent points. Among all the domain-specific MRs,FunQL has the fewest logical forms with aliases,while Prolog has the largest. For example, in Geo,only 42% of FunQL logical forms in the trainingset have aliases, while more than 70% of logicalforms in Lambda Calculus and Prolog have aliases.

7We have experimented with the grammar-based modeland observed similar results.

1527

MR Grammar Geo ATIS

Prolog G1 72.6 ±1.2 77.5 ±0.9G2 75.7 ±2.3 76.2 ±1.0

Lambda G1 75.1 ±1.4 74.9 ±1.5G2 75.6 ±1.5 77.4 ±1.3

FunQL G1 74.5 ±1.7 74.0 ±2.2G2 78.2 ±0.4 77.5 ±0.9

SQL G1 68.2 ±2.4 63.7 ±1.3G2 70.2 ±1.0 61.3 ±1.7

Table 7: Execution-match accuracy of the grammar-based model (w/o Copy). ‘G1’ denotes the grammarrules induced by (Wong and Mooney, 2006, 2007; Yinand Neubig, 2018; Bogin et al., 2019); ‘G2’ denotes thegrammar rules induced by ourselves.

From the figure, we have two main observa-tions. First, in both domains, as more logical formsare replaced, the performance of all MRs declinesgradually. Among all the MRs, the performanceof Prolog declines more seriously than the othersin both domains. In other words, it suffers fromthe program alias problem more seriously. Thetrends of Lambda Calculus and FunQL in ATISare impressive, as their performance decreases onlyslowly. Selecting an MR with the less effect ofprogram alias could be a better choice when weneed to develop a semantic parser for a new do-main, because we can save many efforts in defin-ing annotation protocols and checking consistency,which could be extremely tedious. Second, theexact-match accuracy declines more seriously thanexecution-match. Table 6 provides the relative de-clines in both exact-match and execution-matchmetrics when 25% of logical forms are replaced.We find that the exact-match accuracy declinesmore seriously than execution-match, indicatingthat under the effect of program alias, exact-matchmay not be suitable as it may massively under-estimate the performance. At last, given a largenumber of semantically equivalent logical forms,it would be valuable to explore whether they canbe leveraged to improve semantic parsing (Zhonget al., 2018).

5.3 Grammar Rules

Table 7 presents the experimental results of thegrammar-based (w/o copy) model trained with dif-ferent sets of grammar rules. As the table shows,there is a notable performance discrepancy betweendifferent sets of rules. For example, in ATIS, wecan observe 2.5% absolute improvement when themodel is trained with G2 for Lambda Calculus.Moreover, G2 is not always better than G1. Whilethe model trained with G2 for Prolog outperforms

G1 in Geo, it lags behind G1 in ATIS. This obser-vation motivates us to consider what factors con-tribute to the discrepancy. We had tried to explorethe search space of logical forms defined by dif-ferent grammar rules and the distribution drift be-tween the AST of logical forms in the training andtest set. However, the exploration results cannotconsistently explain the performance discrepancy.As our important future work, we would explorewhether or not the discrepancy is caused by betteralignments between utterances and grammar rules.Intuitively, it would be easier for decoders to learnthe set of grammar rules having better alignmentswith utterances.

We can learn from these results that similar totraditional semantic parsers, properly transforminggrammar rules for MRs can also lead to better per-formance in neural approaches. Therefore, gram-mar rules should be considered as a very importanthyper-parameter of grammar-based models, andit is recommended to mention the used grammarrules in research papers clearly.

6 Related Work

Meaning representations for semantic parsing.Recent work has shown that properly designingnew MRs often helps improve the performanceof neural semantic parsing. Except for the fourMRs that we study, Liang et al. (2011); Liang(2013) presented DCS and lambda DCS for query-ing knowledge bases and demonstrated their ad-vantages over Lambda Calculus. Guo et al. (2019)proposed SemQL, an MR for relational databasequeries, and they showed improvements in the Spi-der benchmark. Wolfson et al. (2020) designed anMR, named QDMR, for representing the meaningof questions through question decomposition. In-stead of designing new MRs, Cheng et al. (2019)proposed a transition-based semantic parsing ap-proach that supports generating tree-structured log-ical forms in either top-down or bottom-up man-ners. Their experimental results on various seman-tic parsing datasets using FunQL showed that top-down generation outperforms bottom-up in varioussettings. Our work aims to investigate the char-acteristics of different MRs and their impact onneural semantic parsing.

Ungrounded semantic parsing. Except for thegrounded MRs studied in this work, there are alsoungrounded MRs that are not tied to any particu-lar applications, such as AMR (Banarescu et al.,

1528

2013), MRS (Copestake et al., 2005; Flickingeret al., 2017), and UCCA (Abend and Rappoport,2013). Abend and Rappoport (2017) conducted asurvey on ungrounded MRs to assess their achieve-ments and shortcomings. Hershcovich et al. (2019)evaluated the similarities and divergences in thecontent encoded by ungrounded MRs and syntacticrepresentation. Lin and Xue (2019) carried out acareful analysis on AMR and MRS to understandthe factors contributing to the discrepancy in theirparsing accuracy. Partly inspired by this line ofwork, we conduct a study on grounded MRs andinvestigate their impact on neural semantic pars-ing. Hershcovich et al. (2018) proposed to leverageannotated data in different ungrounded MRs to im-prove parsing performance. With UNIMER, we canexplore whether it is feasible in grounded semanticparsing.

Extrinsic parser evaluation. Another line ofresearch that is closely related to our work is extrin-sic parser evaluation. Miyao et al. (2008) bench-marked different syntactic parsers and their repre-sentations, including dependency parsing, phrasestructure parsing, and deep parsing, and evaluatedtheir impact on an information extraction system.Oepen et al. (2017) provided a flexible infrastruc-ture, including data and software, to estimate therelative utility of different types of dependencyrepresentations for a variety of downstream appli-cations that rely on an analysis of grammaticalstructure of natural language. There has not beenwork on benchmarking MRs for grounded seman-tic parsing in neural approaches, to the best of ourknowledge.

Weakly supervised semantic parsing. In thispaper, we focus on supervised learning for semanticparsing, where each utterance has its correspond-ing logical form annotated. But the similar eval-uation methodology could be applied to weaklysupervised semantic parsing, which receives wideattention because parsers are only supervised withexecution results and annotated logical forms areno longer required (Berant et al., 2013; Pasupat andLiang, 2015; Goldman et al., 2018; Liang et al.,2018; Mueller et al., 2019). We also notice thatvarious MRs have been used in weakly supervisedsemantic parsing, and it would be valuable to ex-plore the impact of MRs in such settings.

7 Conclusion

In this work, we propose UNIMER, a unified bench-mark on meaning representations, based on estab-lished semantic parsing datasets; UNIMER cov-ers three domains and four different meaning rep-resentations along with their execution engines.UNIMER allows researchers to comprehensivelyand fairly evaluate the performance of their ap-proaches. Based on UNIMER, we conduct an em-pirical study to understand the characteristics ofdifferent meaning representations and their impacton neural semantic parsing. By open-sourcing oursource code and benchmark, we believe that ourwork can facilitate the community to inform thedesign and development of next-generation MRs.

Implications. Our findings have clear impli-cations for future work. First, according to ourexperimental results, FunQL tends to outperformLambda Calculus and Prolog in neural semanticparsing. Additionally, FunQL is relatively robustagainst program alias. Hence, when developersneed to design an MR for a new domain, FunQLis recommended to be the first choice. Second, toreduce program alias’ negative effect on neural se-mantic parsing, developers should define a concreteprotocol for annotating logical forms to ensure theirconsistency. Specifically, given an MR, develop-ers should identify as many as possible sourceswhere program alias can occur. Take SQL as anexample. To express the argmax semantics, onecan either use subquery or the OrderBy clause.8

Having identified these sources, developers need todetermine using which expression in what context,e.g., argmax is always expressed with subquery,and the unordered expressions in conjunctions arealways sorted by characters.

Acknowledgments

We would like to thank Bei Chen for helpful discus-sions and thank the anonymous reviewers for theirconstructive comments. Ting Liu was partially sup-ported by NSFC (61632015, 61721002) and StateGrid R&D Program (5226SX1800FC).

ReferencesOmri Abend and Ari Rappoport. 2013. UCCA: A

semantics-based grammatical annotation scheme. InProceedings of the 10th International Conference

8A concrete example is provided in Section A.3 in thesupplementary material.

https://www.aclweb.org/anthology/W13-0101https://www.aclweb.org/anthology/W13-0101

1529

on Computational Semantics, pages 1–12, Potsdam,Germany. Association for Computational Linguis-tics.

Omri Abend and Ari Rappoport. 2017. The state ofthe art in semantic representation. In Proceedingsof the 55th Annual Meeting of the Association forComputational Linguistics, pages 77–89, Vancouver,Canada. Association for Computational Linguistics.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2014. Neural machine translation by jointlylearning to align and translate. arXiv preprintarXiv:1409.0473.

Laura Banarescu, Claire Bonial, Shu Cai, MadalinaGeorgescu, Kira Griffitt, Ulf Hermjakob, KevinKnight, Philipp Koehn, Martha Palmer, and NathanSchneider. 2013. Abstract Meaning Representationfor sembanking. In Proceedings of the 7th Linguis-tic Annotation Workshop and Interoperability withDiscourse, pages 178–186, Sofia, Bulgaria. Associa-tion for Computational Linguistics.

Jonathan Berant, Andrew Chou, Roy Frostig, and PercyLiang. 2013. Semantic parsing on Freebase fromquestion-answer pairs. In Proceedings of the 2013Conference on Empirical Methods in Natural Lan-guage Processing, pages 1533–1544, Seattle, Wash-ington, USA. Association for Computational Lin-guistics.

Ben Bogin, Matt Gardner, and Jonathan Berant. 2019.Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Con-ference on Empirical Methods in Natural LanguageProcessing and the 9th International Joint Confer-ence on Natural Language Processing, pages 3659–3664, Hong Kong, China. Association for Computa-tional Linguistics.

Rudy Bunel, Matthew Hausknecht, Jacob Devlin,Rishabh Singh, and Pushmeet Kohli. 2018. Leverag-ing grammar and reinforcement learning for neuralprogram synthesis. In International Conference onLearning Representations.

Mary Elaine Califf and Raymond J. Mooney. 1999. Re-lational learning of pattern-match rules for informa-tion extraction. In Proceedings of the 16th NationalConference on Artificial Intelligence and the 7th In-novative Applications of Artificial Intelligence Con-ference Innovative Applications of Artificial Intelli-gence, page 328–334, USA. American Associationfor Artificial Intelligence.

Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and KaiYu. 2019. Semantic parsing with dual learning. InProceedings of the 57th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 51–64,Florence, Italy. Association for Computational Lin-guistics.

Bo Chen, Xianpei Han, Ben He, and Le Sun. 2020.Learning to map frequent phrases to sub-structures

of meaning representation for neural semantic pars-ing. In Proceedings of the 34th AAAI Conferenceon Artificial Intelligence, pages 7546–7553. AAAIPress.

Jianpeng Cheng, Siva Reddy, Vijay Saraswat, andMirella Lapata. 2019. Learning an executable neu-ral semantic parser. Computational Linguistics,45(1):59–94.

Ann Copestake, Dan Flickinger, Carl Pollard, andIvan A. Sag. 2005. Minimal recursion semantics:An introduction. Research on Language & Compu-tation, 3(4):281–332.

Deborah A. Dahl, Madeleine Bates, Michael Brown,William Fisher, Kate Hunicke-Smith, David Pallett,Christine Pao, Alexander Rudnicky, and ElizabethShriberg. 1994. Expanding the scope of the atis task:The atis-3 corpus. In Proceedings of the Workshopon Human Language Technology, page 43–48, USA.Association for Computational Linguistics.

Li Dong and Mirella Lapata. 2016. Language to logi-cal form with neural attention. In Proceedings of the54th Annual Meeting of the Association for Compu-tational Linguistics, pages 33–43, Berlin, Germany.Association for Computational Linguistics.

Li Dong and Mirella Lapata. 2018. Coarse-to-fine de-coding for neural semantic parsing. In Proceed-ings of the 56th Annual Meeting of the Associationfor Computational Linguistics, pages 731–742, Mel-bourne, Australia. Association for ComputationalLinguistics.

Catherine Finegan-Dollak, Jonathan K. Kummerfeld,Li Zhang, Karthik Ramanathan, Sesh Sadasivam,Rui Zhang, and Dragomir Radev. 2018. Improv-ing text-to-SQL evaluation methodology. In Pro-ceedings of the 56th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 351–360, Melbourne, Australia. Association for Compu-tational Linguistics.

Dan Flickinger, Stephan Oepen, and Emily M. Bender.2017. Sustainable Development and Refinement ofComplex Linguistic Annotations at Scale, pages 353–377. Springer Netherlands, Dordrecht.

Matt Gardner, Joel Grus, Mark Neumann, OyvindTafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Pe-ters, Michael Schmitz, and Luke Zettlemoyer. 2018.Allennlp: A deep semantic natural language process-ing platform. In Proceedings of Workshop for NLPOpen Source Software, pages 1–6, Melbourne, Aus-tralia. Association for Computational Linguistics.

Omer Goldman, Veronica Latcinnik, Ehud Nave, AmirGloberson, and Jonathan Berant. 2018. Weakly su-pervised semantic parsing with abstract examples.In Proceedings of the 56th Annual Meeting of theAssociation for Computational Linguistics, pages1809–1819, Melbourne, Australia. Association forComputational Linguistics.

https://doi.org/10.18653/v1/P17-1008https://doi.org/10.18653/v1/P17-1008https://arxiv.org/abs/1409.0473https://arxiv.org/abs/1409.0473https://www.aclweb.org/anthology/W13-2322https://www.aclweb.org/anthology/W13-2322https://www.aclweb.org/anthology/D13-1160https://www.aclweb.org/anthology/D13-1160https://doi.org/10.18653/v1/D19-1378https://doi.org/10.18653/v1/D19-1378https://openreview.net/forum?id=H1Xw62kRZhttps://openreview.net/forum?id=H1Xw62kRZhttps://openreview.net/forum?id=H1Xw62kRZhttps://www.aaai.org/Papers/AAAI/1999/AAAI99-048.pdfhttps://www.aaai.org/Papers/AAAI/1999/AAAI99-048.pdfhttps://www.aaai.org/Papers/AAAI/1999/AAAI99-048.pdfhttps://doi.org/10.18653/v1/P19-1007https://aaai.org/ojs/index.php/AAAI/article/view/6253https://aaai.org/ojs/index.php/AAAI/article/view/6253https://aaai.org/ojs/index.php/AAAI/article/view/6253https://doi.org/10.1162/coli_a_00342https://doi.org/10.1162/coli_a_00342https://link.springer.com/article/10.1007/s11168-006-6327-9https://link.springer.com/article/10.1007/s11168-006-6327-9https://doi.org/10.3115/1075812.1075823https://doi.org/10.3115/1075812.1075823https://doi.org/10.18653/v1/P16-1004https://doi.org/10.18653/v1/P16-1004https://doi.org/10.18653/v1/P18-1068https://doi.org/10.18653/v1/P18-1068https://doi.org/10.18653/v1/P18-1033https://doi.org/10.18653/v1/P18-1033https://doi.org/10.1007/978-94-024-0881-2_14https://doi.org/10.1007/978-94-024-0881-2_14https://doi.org/10.18653/v1/W18-2501https://doi.org/10.18653/v1/W18-2501https://doi.org/10.18653/v1/P18-1168https://doi.org/10.18653/v1/P18-1168

1530

Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao,Jian-Guang Lou, Ting Liu, and Dongmei Zhang.2019. Towards complex text-to-SQL in cross-domain database with intermediate representation.In Proceedings of the 57th Annual Meeting of theAssociation for Computational Linguistics, pages4524–4535, Florence, Italy. Association for Compu-tational Linguistics.

Daniel Hershcovich, Omri Abend, and Ari Rappoport.2018. Multitask parsing across semantic represen-tations. In Proceedings of the 56th Annual Meet-ing of the Association for Computational Linguistics,pages 373–385, Melbourne, Australia. Associationfor Computational Linguistics.

Daniel Hershcovich, Omri Abend, and Ari Rappoport.2019. Content differences in syntactic and semanticrepresentation. In Proceedings of the 2019 Confer-ence of the North American Chapter of the Associ-ation for Computational Linguistics: Human Lan-guage Technologies, pages 478–488, Minneapolis,Minnesota. Association for Computational Linguis-tics.

Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, JayantKrishnamurthy, and Luke Zettlemoyer. 2017. Learn-ing a neural semantic parser from user feedback. InProceedings of the 55th Annual Meeting of the As-sociation for Computational Linguistics, pages 963–973, Vancouver, Canada. Association for Computa-tional Linguistics.

Robin Jia and Percy Liang. 2016. Data recombinationfor neural semantic parsing. In Proceedings of the54th Annual Meeting of the Association for Compu-tational Linguistics, pages 12–22, Berlin, Germany.Association for Computational Linguistics.

Rohit J. Kate. 2008. Transforming meaning representa-tion grammars to improve semantic parsing. In Pro-ceedings of the 12th Conference on ComputationalNatural Language Learning, pages 33–40, Manch-ester, England. Coling 2008 Organizing Committee.

Rohit J. Kate and Raymond J. Mooney. 2006. Us-ing string-kernels for learning semantic parsers. InProceedings of the 21st International Conference onComputational Linguistics and 44th Annual Meet-ing of the Association for Computational Linguistics,pages 913–920, Sydney, Australia. Association forComputational Linguistics.

Rohit J. Kate, Yuk Wah Wong, and Raymond J.Mooney. 2005. Learning to transform natural toformal languages. In Proceedings of the 20th Na-tional Conference on Artificial Intelligence, page1062–1068. AAAI Press.

Tom Kwiatkowksi, Luke Zettlemoyer, Sharon Goldwa-ter, and Mark Steedman. 2010. Inducing probabilis-tic CCG grammars from logical form with higher-order unification. In Proceedings of the 2010 Con-ference on Empirical Methods in Natural LanguageProcessing, pages 1223–1233, Cambridge, MA. As-sociation for Computational Linguistics.

Tom Kwiatkowski, Luke Zettlemoyer, Sharon Goldwa-ter, and Mark Steedman. 2011. Lexical generaliza-tion in CCG grammar induction for semantic pars-ing. In Proceedings of the 2011 Conference on Em-pirical Methods in Natural Language Processing,pages 1512–1523, Edinburgh, Scotland, UK. Asso-ciation for Computational Linguistics.

Chen Liang, Mohammad Norouzi, Jonathan Berant,Quoc V Le, and Ni Lao. 2018. Memory augmentedpolicy optimization for program synthesis and se-mantic parsing. In Proceedings of the 31st Interna-tional Conference on Neural Information ProcessingSystems, pages 9994–10006. Curran Associates, Inc.

Percy Liang. 2013. Lambda dependency-based compo-sitional semantics. arXiv preprint arXiv:1309.4408.

Percy Liang, Michael Jordan, and Dan Klein. 2011.Learning dependency-based compositional seman-tics. In Proceedings of the 49th Annual Meeting ofthe Association for Computational Linguistics: Hu-man Language Technologies, pages 590–599, Port-land, Oregon, USA. Association for ComputationalLinguistics.

Zi Lin and Nianwen Xue. 2019. Parsing meaning repre-sentations: Is easier always better? In Proceedingsof the First International Workshop on DesigningMeaning Representations, pages 34–43, Florence,Italy. Association for Computational Linguistics.

Thang Luong, Hieu Pham, and Christopher D. Man-ning. 2015. Effective approaches to attention-basedneural machine translation. In Proceedings of the2015 Conference on Empirical Methods in Natu-ral Language Processing, pages 1412–1421, Lis-bon, Portugal. Association for Computational Lin-guistics.

Yusuke Miyao, Rune Sætre, Kenji Sagae, Takuya Mat-suzaki, and Jun’ichi Tsujii. 2008. Task-orientedevaluation of syntactic parsers and their representa-tions. In Proceedings of the 46th Annual Meetingof the Association for Computation Linguistics: Hu-man Language Technologies, pages 46–54, Colum-bus, Ohio. Association for Computational Linguis-tics.

Thomas Mueller, Francesco Piccinno, Peter Shaw,Massimo Nicosia, and Yasemin Altun. 2019. An-swering conversational questions on structured datawithout logical forms. In Proceedings of the 2019Conference on Empirical Methods in Natural Lan-guage Processing and the 9th International JointConference on Natural Language Processing, pages5902–5910, Hong Kong, China. Association forComputational Linguistics.

Stephan Oepen, Lilja Øvrelid, Jari Björne, Richard Jo-hansson, Emanuele Lapponi, Filip Ginter, and ErikVelldal. 2017. The 2017 shared task on extrinsicparser evaluation. towards a reusable community in-frastructure. In Proceedings of the 2017 Shared Taskon Extrinsic Parser Evaluation at the Fourth Inter-national Conference on Dependency Linguistics and

https://doi.org/10.18653/v1/P19-1444https://doi.org/10.18653/v1/P19-1444https://doi.org/10.18653/v1/P18-1035https://doi.org/10.18653/v1/P18-1035https://doi.org/10.18653/v1/N19-1047https://doi.org/10.18653/v1/N19-1047https://doi.org/10.18653/v1/P17-1089https://doi.org/10.18653/v1/P17-1089https://doi.org/10.18653/v1/P16-1002https://doi.org/10.18653/v1/P16-1002https://www.aclweb.org/anthology/W08-2105https://www.aclweb.org/anthology/W08-2105https://doi.org/10.3115/1220175.1220290https://doi.org/10.3115/1220175.1220290https://www.aaai.org/Papers/AAAI/2005/AAAI05-168.pdfhttps://www.aaai.org/Papers/AAAI/2005/AAAI05-168.pdfhttps://www.aclweb.org/anthology/D10-1119https://www.aclweb.org/anthology/D10-1119https://www.aclweb.org/anthology/D10-1119https://www.aclweb.org/anthology/D11-1140https://www.aclweb.org/anthology/D11-1140https://www.aclweb.org/anthology/D11-1140http://papers.nips.cc/paper/8204-memory-augmented-policy-optimization-for-program-synthesis-and-semantic-parsing.pdfhttp://papers.nips.cc/paper/8204-memory-augmented-policy-optimization-for-program-synthesis-and-semantic-parsing.pdfhttp://papers.nips.cc/paper/8204-memory-augmented-policy-optimization-for-program-synthesis-and-semantic-parsing.pdfhttps://arxiv.org/abs/1309.4408https://arxiv.org/abs/1309.4408https://www.aclweb.org/anthology/P11-1060https://www.aclweb.org/anthology/P11-1060https://doi.org/10.18653/v1/W19-3304https://doi.org/10.18653/v1/W19-3304https://doi.org/10.18653/v1/D15-1166https://doi.org/10.18653/v1/D15-1166https://www.aclweb.org/anthology/P08-1006https://www.aclweb.org/anthology/P08-1006https://www.aclweb.org/anthology/P08-1006https://doi.org/10.18653/v1/D19-1603https://doi.org/10.18653/v1/D19-1603https://doi.org/10.18653/v1/D19-1603http://svn.emmtee.net/epe/2017/public/proceedings.pdf#page=11http://svn.emmtee.net/epe/2017/public/proceedings.pdf#page=11http://svn.emmtee.net/epe/2017/public/proceedings.pdf#page=11

1531

the 15th International Conference on Parsing Tech-nologies. Pisa, Italy, pages 1–16.

Panupong Pasupat and Percy Liang. 2015. Composi-tional semantic parsing on semi-structured tables. InProceedings of the 53rd Annual Meeting of the Asso-ciation for Computational Linguistics and the 7th In-ternational Joint Conference on Natural LanguageProcessing, pages 1470–1480, Beijing, China. Asso-ciation for Computational Linguistics.

Adam Paszke, Sam Gross, Francisco Massa, AdamLerer, James Bradbury, Gregory Chanan, TrevorKilleen, Zeming Lin, Natalia Gimelshein, LucaAntiga, Alban Desmaison, Andreas Kopf, EdwardYang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,Junjie Bai, and Soumith Chintala. 2019. Pytorch:An imperative style, high-performance deep learn-ing library. In Proceedings of the 32nd InternationalConference on Neural Information Processing Sys-tems, pages 8024–8035. Curran Associates, Inc.

Ana-Maria Popescu, Oren Etzioni, and Henry Kautz.2003. Towards a theory of natural language inter-faces to databases. In Proceedings of the 8th Inter-national Conference on Intelligent User Interfaces,page 149–157, New York, NY, USA. Association forComputing Machinery.

Maxim Rabinovich, Mitchell Stern, and Dan Klein.2017. Abstract syntax networks for code generationand semantic parsing. In Proceedings of the 55th An-nual Meeting of the Association for ComputationalLinguistics, pages 1139–1149, Vancouver, Canada.Association for Computational Linguistics.

Peter Shaw, Philip Massey, Angelica Chen, FrancescoPiccinno, and Yasemin Altun. 2019. Generating log-ical forms from graph representations of text andentities. In Proceedings of the 57th Annual Meet-ing of the Association for Computational Linguistics,pages 95–106, Florence, Italy. Association for Com-putational Linguistics.

Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, LiliMou, and Lu Zhang. 2020. Treegen: A tree-basedtransformer architecture for code generation. In Pro-ceedings of the 34th AAAI Conference on ArtificialIntelligence, pages 8984–8991. AAAI Press.

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.Sequence to sequence learning with neural networks.In Proceedings of the 27th International Conferenceon Neural Information Processing Systems, page3104–3112, Cambridge, MA, USA. MIT Press.

Adrienne Wang, Tom Kwiatkowski, and Luke Zettle-moyer. 2014. Morpho-syntactic lexical generaliza-tion for CCG semantic parsing. In Proceedings ofthe 2014 Conference on Empirical Methods in Nat-ural Language Processing, pages 1284–1295, Doha,Qatar. Association for Computational Linguistics.

Bailin Wang, Richard Shin, Xiaodong Liu, OleksandrPolozov, and Matthew Richardson. 2020. RAT-SQL:Relation-aware schema encoding and linking fortext-to-SQL parsers. In Proceedings of the 58th An-nual Meeting of the Association for ComputationalLinguistics, pages 7567–7578, Online. Associationfor Computational Linguistics.

Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner,Y. Goldberg, D. Deutch, and Jonathan Berant. 2020.Break it down: A question understanding bench-mark. Transactions of the Association for Compu-tational Linguistics, 8:183–198.

Yuk Wah Wong and Raymond Mooney. 2006. Learn-ing for semantic parsing with statistical machinetranslation. In Proceedings of the Human LanguageTechnology Conference of the NAACL, Main Confer-ence, pages 439–446, New York City, USA. Associ-ation for Computational Linguistics.

Yuk Wah Wong and Raymond Mooney. 2007. Learn-ing synchronous grammars for semantic parsingwith lambda calculus. In Proceedings of the 45thAnnual Meeting of the Association of ComputationalLinguistics, pages 960–967, Prague, Czech Repub-lic. Association for Computational Linguistics.

Pengcheng Yin and Graham Neubig. 2017. A syntacticneural model for general-purpose code generation.In Proceedings of the 55th Annual Meeting of the As-sociation for Computational Linguistics, pages 440–450, Vancouver, Canada. Association for Computa-tional Linguistics.

Pengcheng Yin and Graham Neubig. 2018. TRANX: Atransition-based neural abstract syntax parser for se-mantic parsing and code generation. In Proceedingsof the 2018 Conference on Empirical Methods inNatural Language Processing: System Demonstra-tions, pages 7–12, Brussels, Belgium. Associationfor Computational Linguistics.

John M. Zelle and Raymond J. Mooney. 1996. Learn-ing to parse database queries using inductive logicprogramming. In Proceedings of the 13th Na-tional Conference on Artificial Intelligence, page1050–1055. AAAI Press.

Luke Zettlemoyer and Michael Collins. 2007. Onlinelearning of relaxed CCG grammars for parsing tological form. In Proceedings of the 2007 Joint Con-ference on Empirical Methods in Natural LanguageProcessing and Computational Natural LanguageLearning, pages 678–687, Prague, Czech Republic.Association for Computational Linguistics.

Luke S. Zettlemoyer and Michael Collins. 2005. Learn-ing to map sentences to logical form: Structuredclassification with probabilistic categorial grammars.In Proceedings of the Twenty-First Conference onUncertainty in Artificial Intelligence, page 658–666,Arlington, Virginia, USA. AUAI Press.

https://doi.org/10.3115/v1/P15-1142https://doi.org/10.3115/v1/P15-1142http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfhttp://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfhttp://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfhttps://doi.org/10.1145/604045.604070https://doi.org/10.1145/604045.604070https://doi.org/10.18653/v1/P17-1105https://doi.org/10.18653/v1/P17-1105https://doi.org/10.18653/v1/P19-1010https://doi.org/10.18653/v1/P19-1010https://doi.org/10.18653/v1/P19-1010https://aaai.org/ojs/index.php/AAAI/article/view/6430https://aaai.org/ojs/index.php/AAAI/article/view/6430http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdfhttps://doi.org/10.3115/v1/D14-1135https://doi.org/10.3115/v1/D14-1135https://doi.org/10.18653/v1/2020.acl-main.677https://doi.org/10.18653/v1/2020.acl-main.677https://doi.org/10.18653/v1/2020.acl-main.677https://www.mitpressjournals.org/doi/abs/10.1162/tacl_a_00309#authorsTabListhttps://www.mitpressjournals.org/doi/abs/10.1162/tacl_a_00309#authorsTabListhttps://www.aclweb.org/anthology/N06-1056https://www.aclweb.org/anthology/N06-1056https://www.aclweb.org/anthology/N06-1056https://www.aclweb.org/anthology/P07-1121https://www.aclweb.org/anthology/P07-1121https://www.aclweb.org/anthology/P07-1121https://doi.org/10.18653/v1/P17-1041https://doi.org/10.18653/v1/P17-1041https://doi.org/10.18653/v1/D18-2002https://doi.org/10.18653/v1/D18-2002https://doi.org/10.18653/v1/D18-2002https://dl.acm.org/doi/10.5555/1864519.1864543https://dl.acm.org/doi/10.5555/1864519.1864543https://dl.acm.org/doi/10.5555/1864519.1864543https://www.aclweb.org/anthology/D07-1071https://www.aclweb.org/anthology/D07-1071https://www.aclweb.org/anthology/D07-1071https://dl.acm.org/doi/10.5555/3020336.3020416https://dl.acm.org/doi/10.5555/3020336.3020416https://dl.acm.org/doi/10.5555/3020336.3020416

1532

Kai Zhao and Liang Huang. 2015. Type-driven in-cremental semantic parsing with polymorphism. InProceedings of the 2015 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 1416–1421, Denver, Colorado. Associationfor Computational Linguistics.

Zexuan Zhong, Jiaqi Guo, Wei Yang, Jian Peng, TaoXie, Jian-Guang Lou, Ting Liu, and Dongmei Zhang.2018. SemRegex: A semantics-based approachfor generating regular expressions from natural lan-guage specifications. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing, pages 1608–1618, Brussels, Bel-gium. Association for Computational Linguistics.

A Supplemental Material

Algorithm 1: Translation of Lambda Cal-culus Logical Forms to FunQLInput: A Lambda Calculus logical form pOutput: A FunQL logical form f

1 types = InferTypes(p);2 tokens = Tokenize(p);3 expressions, stack = [], [];4 i = 0;5 for t ∈ tokens do6 if t == ‘(’ then7 Append(stack,i);8 else if t == ‘)’ then9 j = = Pop(stack);

10 e = Search(expressions, j, i);11 Remove(e);12 ne = Translate(tokens[j : i], e,

types);13 Append(expressions, ne);14 i+ = 1;15 end16 f = expressions[0];

A.1 Details of Benchmark ConstructionGeo. Since the four MRs we study have annotatedlogical forms and have execution engines exceptfor Lambda Calculus in Geo, we directly use them(SQL taken from (Finegan-Dollak et al., 2018), Pro-log taken from (Jia and Liang, 2016), FunQL takenfrom (Wong and Mooney, 2006), and Lambda Cal-culus taken from (Kwiatkowksi et al., 2010)). Weimplement an execution engine with Haskell forLambda Calculus. With these resources, we cross-validate the correctness of annotations and execu-tion engines by comparing the execution resultsof logical forms. As a result, we found nearly 30Prolog logical forms with annotation mistakes andtwo bugs in the execution engines of Prolog andFunQL.ATIS. There are only annotated logical formsfor Lambda Calculus and SQL in ATIS. We di-rectly use Lambda Calculus logical forms providedby (Jia and Liang, 2016) and SQL queries providedby (Iyer et al., 2017). To provide annotations forFunQL and Prolog, we semi-automatically trans-late Lambda Calculus logical forms to equivalentlogical forms in FunQL and Prolog. Algorithm 1presents the pseudo code for translating a LambdaCalculus logical form to FunQL. Basically, we first

https://doi.org/10.3115/v1/N15-1162https://doi.org/10.3115/v1/N15-1162https://doi.org/10.18653/v1/D18-1189https://doi.org/10.18653/v1/D18-1189https://doi.org/10.18653/v1/D18-1189

1533

Approach MR Exact ExecGeoKwiatkowksi et al. (2010) FunQL 84.3 -Shaw et al. (2019)∗ FunQL 89.3 -Jia and Liang (2016)∗ Prolog - 89.3

- data augmentation Prolog - 85.0Chen et al. (2020)∗ Prolog - 89.6Kwiatkowksi et al. (2010) λ 87.9 -Wang et al. (2014) λ 90.4 -Zhao and Huang (2015) λ 88.9 -Dong and Lapata (2016)∗† λ 87.1 -Rabinovich et al. (2017)∗† λ 87.1 -Dong and Lapata (2018)∗† λ 88.2 -Sun et al. (2020)∗† λ 89.1 -Iyer et al. (2017)∗† SQL - 82.5ATISWang et al. (2014) λ 91.3 -Zhao and Huang (2015) λ 84.2 -Jia and Liang (2016)∗ λ 83.3 -

- data augmentation λ 76.3 -Dong and Lapata (2016)∗† λ 84.6 -Rabinovich et al. (2017)∗† λ 85.9 -Yin and Neubig (2018)∗† λ 88.2 -Dong and Lapata (2018)∗† λ 87.7 -Cao et al. (2019)∗† λ 89.1 -Sun et al. (2020)∗† λ 89.6 -Shaw et al. (2019)∗ λ 87.1 -Iyer et al. (2017)∗† SQL - 79.2JobsZettlemoyer and Collins (2005) λ 79.3 -Zhao and Huang (2015) λ 85.0 -Dong and Lapata (2016)∗† Prolog 90.0 -Rabinovich et al. (2017)∗† Prolog 91.4 -Chen et al. (2020)∗ Prolog - 92.1

Table 8: Performance of semantic parsing approaches,where ∗ denotes neural approaches and † denotes ap-proaches using data anonymization techniques. ‘λ’ de-notes Lambda Calculus. ‘Exact’ and ‘Exec’ denoteexact-match and execution-match, respectively.

perform a type inference for variables in the logicalform, since some predicates in Lambda Calculuscan take different types of variables as input, e.g.,for the predicate to(A,B), variable B can be ei-ther an airport or a city. Then, we tokenize theinput logical form and recursively translate eachsub-expression in the logical form into FunQL ex-pressions based on the predicates we design forFunQL. Translation from Lambda Calculus to Pro-log is performed in a similar way. The source codefor translation is also available in our Github repos-itory. We also implement execution engines forLambda Calculus, Prolog, and FunQL.Job. There are only annotated logical forms forLambda Calculus and Prolog in Job. We directlyuse Prolog logical forms provided on the web-site.9 To provide annotations for FunQL, we semi-

9http://www.cs.utexas.edu/ ml/nldata/jobquery.html

automatically translate Prolog logical forms toequivalent logical forms in FunQL with an algo-rithm similar to Algorithm 1. In terms of SQL, thelogical forms in Job are relatively simple and canbe expressed with the SELECT, FROM and WHEREclauses of SQL. We simply translate each sub-expression in Prolog to an expression in WHEREclause and use a conjunction to join the resultingexpressions. Since we cannot find the annotatedLambda Calculus logical forms provided by (Zettle-moyer and Collins, 2005), we also translate theProlog logical forms to Lambda Calculus. We im-plement execution engines for Lambda Calculusand FunQL.

A.2 Model Configuration

Preprocessing of Prolog and Lambda CalculusAs Prolog and Lambda Calculus have variables, weneed to standardize their variable naming beforetraining. Following (Jia and Liang, 2016), we pre-process the Prolog logical forms to the De Brujinindex notation. We standardize the variable nam-ing of Lambda Calculus based on their occurrenceorder in logical forms.Attention Copying Mechanism. In Geo and Job,we use the standard copy mechanism, i.e., directlycopying a source word to a logical form. In ATIS,following (Jia and Liang, 2016), we leverage an ex-ternal lexicon to identify potential copy candidates,e.g., slc:ap can be identified as a potential entityfor description “salt lake city airport” in utterance.When we copy a source word that is part of a phrasein the lexicon, we write the entity associated withthat lexicon entry to a logical form.Hyper-Parameters. For the seq2seq model, theembedding dimension of both source and targetlanguages ranges over {100, 200}. We select aone-layer bi-directional LSTM as an encoder. Thehidden dimension of the encoder ranges over {32,64, 128, 256}. Similarly, a one-layer LSTM isselected as the decoder. Its hidden dimension isas same as the encoder. In terms of attention, weselect bi-linear as the activation function, wherethe hidden dimension is 2 times that of the encoder.We employ dropout at training time with rate rang-ing over {0.1, 0.2, 0.3}. We select batch size from{16, 32, 48, 64}, and select learning rate from{0.001, 0.0025, 0.005, 0.01, 0.025, 0.05}. Follow-ing (Dong and Lapata, 2016), we use the RMSPropalgorithm to update the parameters. The smooth-ing constant of RMSProp is 0.95. We initialize all

1534

parameters uniformly at random within the interval[−0.1, 0.1].

Similarly, for the grammar-based model, a one-layer bi-directional LSTM is used as an encoderand another LSTM is employed as a decoder. Thelayers of the decoder is selected from {1, 2}. Thehidden dimension of the encoder ranges over {64,128, 256}. The hidden dimension of the decoderis 2 times that of the encoder. The hidden dimen-sion of both the grammar rule and non-terminalis selected from {64, 128, 256}. We also employdropout in the encoder and decoder at training timewith rate selected from {0.1, 0.2, 0.3}. We selectbatch size from {16, 32, 48, 64}, and select learn-ing rate from {0.001, 0.0025, 0.005, 0.01, 0.025,0.05}. We use the Adam algorithm to update theparameters.

For both models, gradients are clipped at 5 toalleviate the exploding gradient problem, and earlystopping is used to determine the number of epochs.We provide the detailed configurations of the NNIplatform in our Github repository.

A.3 Search for Aliases for Logical Forms.

Algorithm 2: Search for Program AliasInput: A logical form pOutput: A set of aliases of the input logical

form aliases1 ast = Parse(p);2 aliases = [];3 for r ∈ Rules do4 C = Apply(ast, r);5 for c ∈ C do6 if IsEquivalent(p, c) then7 Append(aliases,c);8 end9 end

Algorithm 2 presents the way we search foraliases for a logical form. Transformation rules canbe categorized into two groups based on whetherthey are domain-specific. Considering the follow-ing two logical forms:

(lambda A:e (exists B (and (flight B) (fare BA))))

(lambda A:e (exists B (and (flight B) (equals(fare B) A))))they are semantically equivalent due to the multi-ple meaning definitions of fare. There are also

Rule DescriptionPrologShuffle Shuffle expressions in conjunctionRemove Remove redundant expressionsMerge Put expressions into higher-order predicatesLambda CalculusShuffle Shuffle expressions in conjunctionReplace Replace semantically equivalent predicatesFunQLShuffle Shuffle expressions in conjunctionRemove Remove redundant expressionsSwap Swap unit relation predicateReplace Replace semantically equivalent predicatesSQL

Shuffle Shuffle expressions in Select, From, Where,and Having clauses

Argmax Express Argmax/min with OrderBy andLimit clause instead of subqueryIn2Join Replace In clause with Join clause

Table 9: Transformation rules for Prolog, Lambda Cal-culus and FunQL.

domain-general transformation rules, e.g., permut-ing the expressions in the conjunction predicate:

(lambda A:e (exists B (and (flight B) (fare BA))))

(lambda A:e (exists B (and (fare B A) (flightB))))

In this work, we primarily consider domain-general transformation rules and only when thereis limited aliases found by domain-general rules,we use domain-specific rules. Table 9 presentsthe transformation rules we used in Geo domains.Rules in ATIS are similar. We provide examplesbelow to illustrate the rules.Prolog Shuffle:

answer(A,(area(B,A),const(B,stateid(Texas))))answer(A,(const(B,stateid(texas)),area(B,A),))

Prolog Remove:answer(A,(loc(B,A),city(B),const(B,cityid(Austin, ))))answer(A,(loc(B,A),const(B,cityid(Austin, ))))

Prolog Merge:answer(A,(state(A),loc(B,A),highest(B,place(B))))answer(A,(highest(B,(place(B),loc(B,A),state(A)))))

FunQL Swap:answer(count(major(city(loc 2(stateid(Texas))))))answer(count(city(major(loc 2(stateid(Texas))))))

FunQL Replace:answer(city(loc 2(largest(state(all))))))answer(city(loc 2(largest one(area 1(state(all))))))

SQL Argmax:SELECT city.city name FROM city WHERE

city.population = (SELECT MAX(c1.population)FROM city as c1 WHERE c1.state name = ‘ari-zona’) and city.state name = ‘arizona’;

1535

MR Geo ATISProlog 98.0% 96.2%Lambda Calculus 69.7% 95.3%FunQL 42.5% 87.9%SQL 38.5% 95.5%

Table 10: Number of logical forms in training set ofGeo and ATIS that have aliases.

SELECT city.city name FROM city WHEREstate name = ‘arizona’ ORDER BY city.populationDESC LIMIT 1;SQL In2Join:

SELECT river.river name FROM river WHEREriver.traverse IN (SELECT state.state name FROMstate WHERE state.area = (SELECT MAX(s1.area)FROM state as s1));

SELECT river.river name FROM river, stateWHERE river.traverse = state.state name ANDstate.area = (SELECT MAX(s1.area) FRORM stateas s1);

Table 10 presents the number of logical forms inthe training set of Geo and ATIS that have aliases,from which we can see that with these simple rules,we are able to find a lot of aliases. Since the logicalforms in Geo are generally simpler than that ofATIS, the number of Geo is lower than ATIS.

A.4 GrammarWe present the grammar rules used in our ex-periments for Geo (Figure 4-11). The grammarrules for different MRs in ATIS follow a sim-ilar definition. It is important to note that al-though all logical forms in an MR can be mod-eled by both G1 and G2, they are not neces-sarily equivalent in expressiveness. For exam-ple, answer(largest(place(all))) is asyntactically correct but semantically incorrectFunQL logical form in Geo, because the predicatelargest is not allowed to take places as input.However, G2 of FunQL still can accept this logicalform, while G1 cannot, because G1 explicitly en-codes type constraints in the its rules. Nevertheless,the performance of G1 is lower than G2 in bothGeo and ATIS domains, which is surprising. Forall the grammar rules we present, G1 and G2 forProlog and Lambda Calculus and equivalent. G1and G2 for FunQL and SQL are not equivalent.

statement := "answer(" Var "," Form ")"Form := "(" Form conjunction ")" | "area(" Var "," Var ")"

| "capital(" Var ")" | "capital(" Var "," Var ")" | "city(" Var ")" | "const(" Var ",countryid(usa))" | "const(" Var "," City ")" | "const(" Var "," Place ")" | "const(" Var "," River ")" | "const(" Var "," State ")" | "count(" Var "," Form "," Var ")" | "country(" Var ")" | "density(" Var "," Var ")" | "elevation(" Var "," Num ")" | "elevation(" Var "," Var ")" | "fewest(" Var "," Var "," Form ")" | "high_point(" Var "," Var ")" | "higher(" Var "," Var ")" | "highest(" Var "," Form ")" | "lake(" Var ")" | "largest(" Var "," Form ")" | "len(" Var "," Var ")" | "loc(" Var "," Var ")" | "longer(" Var "," Var ")" | "longest(" Var "," Form ")" | "low_point(" Var "," Var ")" | "lower(" Var "," Var ")" | "lowest(" Var "," Form ")" | "major(" Var ")" | "most(" Var "," Var "," Form ")" | "mountain(" Var ")" | "next_to(" Var "," Var ")" | "not(" Form ")" | "place(" Var ")" | "population(" Var "," Var ")" | "river(" Var ")" | "shortest(" Var "," Form ")" | "size(" Var "," Var ")" | "smallest(" Var "," Form ")" | "state(" Var ")" | "sum(" Var "," Form "," Var ")" | "traverse(" Var "," Var ")"

conjunction := "" | "," Form conjunctionCity := "cityid(" CityName "," StateAbbrev ")"

| "cityid(" CityName ",_)"Place := "placeid(" PlaceName ")"River := "riverid(" RiverName ")"State := "stateid(" StateName ")"Var := "a" | "b" | "c" | "d" | "e" | "f" | "g" | "nv" | "v0" | "v1" | "v2" | "v3" | "v4" | "v5" | "v6" | "v7"StateName := "Texas" | "illinois" | … | "kentucky"CityName := "albany" | "chicago" | … | "columbus"PlaceName := "mount mckinley" | … | "death valley"RiverName := "ohio" | "colorado” | … | "red"StateAbbrev := "dc" | "sd" | … | "me"Number := "0" | "1.0"

Figure 4: The grammar rules for Prolog in Geo inducedby (Wong and Mooney, 2007) (G1).

benchmarking meaning representations in neural semantic ...1in this paper, we focus on grounded...

Documents