Understanding Context-free Grammarsthrough Data Visualization

Felix Hultin

Institutionen för lingvistik

Examensarbete Kandidat 15hp

Datorlingvistik

Vårterminen 2016

Handledare: Mats Wirén

Examinator: Bernhard Wälchli

Expertgranskare: Gintare Grigonyte

Understanding Context-free Grammars throughData Visualization

AbstractEver since the late 1950’s, context-free grammars have played an important role within the field oflinguistics, been a part of introductory courses and expanded into other fields of study. Meanwhile, datavisualization in modern web development has made it possible to do feature rich visualization in thebrowser. In this thesis, these two developments are united, by developing a browser based app, to writecontext-free grammars, parse sentences and visualize the output. A user experience study with usability-tests and user-interviews is conducted, in order to investigate the possible benefits and disadvantagesof said visualization when writing context-free grammars. The results show that data visualization waslimitedly used by participants, in that it helped them to see if sentences were parsed and, if a sentencewas not parsed, at which position parsing went wrong. Future improvements on the software and studieson them are proposed as well as the expansion of the field of data visualization within linguistics.

SammanfattningÄnda sedan det sena 1950-talet har kontextfria grammatiker spelat en viktig roll hos lingvistiska teorier,används i introduktionskurser och expanderats till andra forskningsfält. Samtidigt har datavisualiseringinom modern webbutveckling gjort det möjligt att skapa innehållsrik visualisering i webbläsaren. I dettaexamensarbete förenas dessa två utvecklingar genom utvecklandet av en webbapplikation, gjord för attskriva kontextfria grammatiker, parsa meningar och visualisera utdatan. En användarbarhetsstudie utförs,bestående av användartest och användaintervjuer, för att undersöka möjliga fördelar och nackdelar avvisualisering i skrivandet av kontextfria grammatiker. Resultaten visar att data visualisering användes påett begränsat sätt av deltagarna, i den meningen att det hjälpte dem att se om satser kan parsas och, om ensats inte blir parsad, se på vilket stället parsning misslyckades. Framtida förbättringar av applikationenoch studier av dem föreslås samt en utbyggnad av data visualisering inom lingvistik.

KeywordsContext-free grammar, data visualization, usability-testing, user-interview, D3.js, Backbone.js, JavaScript.

Contents1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2.1. Context-free Grammars in Linguistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1.1. Formal Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22.1.2. Representing Parsed Sentences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.2. Data Visualization in the Browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2.1. Web Technology Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2.2. D3.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2.3. Backbone.js . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3. Previous Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3.1. M-PATR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3.2. PSG Web Laboratory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.3.3. Conclusion on Previous Implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3. Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94. Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.1. CFG-Visualizer: A Software Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.1.1. Program Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.1.2. Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.2. User Experience Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.2.1. Usability Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2.2. User Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.1. Usage of Data Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165.2. Perception and Understanding of Data Visualization . . . . . . . . . . . . . . . . . . . . . . . . . 17

6. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.1. Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6.1.1. Software Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196.1.2. User Experience Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6.2. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22B. Introduction Slides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24C. Results of Lab Task/App Feature Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3

1. IntroductionEver since context-free grammars were introduced by Noam Chomsky in the late 1950’s [Chomsky,1956], and especially expanded upon in his classic Syntactic Structures [Chomsky, 1957], they havebeen a vital part of introductory courses and theories in linguistics. Although the initial rudimentarymodel is not what is used in generative grammar theories today, educationalists and writers alike havefound it to be a fruitful and simple enough example as point of demonstration for undergraduate students.

Meanwhile, the digital revolution has made it possible to visualize data in graphs in an easier waythan was ever possible before. More recently, with the standardization of JavaScript, HTML, SVG, CSSand other web technologies, modern web development has allowed for programmers to embed featurerich graphs into their web content. It has, therefore, become more and more common to see complexand interactive graphs on the web, such as election data graphs1 on The New York Times websitenytimes.com, elaborate geographical data with web mapping on Google’s Google Maps websitemaps.google.com or violence and guns statistics2 on The Guardian’s website theguardian.com, just to mention a few.

In this thesis, I intend to unite these different developments by developing a browser based softwareCFG-Visualizer3, which can be used as an educational tool to write and parse context-free grammarsas well as visualize parsed sentences. My primary area of research will be to see how this sort ofvisualization can be of benefit for writing and constructing context-free grammars and, to this purpose,I will do a user experience study to qualitatively asses the possible benefits and disadvantages of saidvisualization. More specifically, I will do one-on-one usability tests and user interviews.

With this research, I hope to shed light on how modern data visualization can be useful for notonly students of linguistics but perhaps even professionals within the field. Beyond just context-freegrammars, this sort of research might inspire further development of data visualizing tools for linguisticpurposes.

1An example of this would be the results of the 2012 presidential elections at http://elections.nytimes.com/2012/results/president (accessed 03-June-2016).2See the online article from december 14 2015: Gun deaths in your district: what have yourelected representatives done? theguardian.com/us-news/ng-interactive/2015/dec/14/gun-control-laws-congress-shooting-deaths-nra-lobby-campaign-donations (accessed 03-June-2016)3Currently running at http://mumin.ling.su.se/cfg-visualizer/ and source code is available at https://github.com/felixhultin/cfg-visualizer (accessed 03-june.2016).

1

2. BackgroundThe software developed and used for this thesis, CFG-Visualizer, is based, on the one hand, on context-free grammars and, on the other hand, modern web data visualization. In this section, both these areaswill be covered, while section 4 will go into how they are reconciled to form the desired software.Additionally, previous similar implementations will be compared, in order to put the program and theneed for it into perspective.

2.1. Context-free Grammars in LinguisticsContext-free grammars were introduced by Noam Chomsky in his article Three models for the descrip-tion of language [Chomsky, 1956] and was expanded upon in his classic work Syntactic Structures. Itwas later used for the phrase structure component of his transformation grammar, laid out in Aspectsof the Theory of Syntax [Noam, 1965], and has also worked as a skeleton for other more rich linguisticformalisms within linguistics and computational linguistics [Shieber, 1986], for example Generalizedphrase structure grammar [Gazdar, 1985]. Furthermore, it has expanded into the field of computer sci-ence, where it has been used for syntactic specification of programming languages [Aho et al., 2006, p.197].

The reason for context-free grammar’s popularity in education, besides its importance within manyfields of study, can probably be explained by the fact that it is a simple enough model to understand,while at the same time fulfilling the general requirements of a generative grammar. That is, it consistsof a set of rules which produces a set of valid combinations of words (sentences) for a given language.A given combination of words or characters is then considered grammatical if the sentence can bededuced from the rules in the context-free grammar and ungrammatical if no combination of rules canbe deduced to fit the sentence. The process of searching for such rules is called parsing and if they canbe found the sentence is parsed (i.e. grammatical) [Jurafsky and Martin, 2000, p. 425].

2.1.1. Formal DefinitionContext-free grammars have, as specified by Chomsky [Chomsky, 1956], certain formal properties.More specifically a context-free grammar is a 4-tuple of, as summarized by Daniel Jurafsky and JamesH. Martin [Jurafsky and Martin, 2000, p. 425]:

N a set of non-terminals symbols (or variables)

Σ a set of terminal symbols (disjoint from N)

R a set of rules or productions, each of the form A→ β ,where A is a non-terminal,β is a string of symbols from the infinite set of strings (Σ∪N)∗

S a designated start symbol

For a context-free grammar describing a natural language, words are terminals whereas lexical cate-gories and constituents are non-terminals. A grammatical sequence of words is realized if a set of rulesin R, can be put together to reach S, the start symbol. In this model, the way of describing a grammar isto define the appropriate terminals and non-terminals, and, most importantly, the rules which allow forcreating grammatical sequences. In practice, however, only the production rules have to be written. Thisis because terminals and non-terminals are implicitly defined when writing down productions, whichalso makes for a more compact notation.

2

In figure 1, for example, the set of terminals {I, an, my, elephant, pajamas, shot, in} and the non-terminals {S, PP, NP, VP, PRO, DT, N, V, P} can be identified by just looking at the rules. If a symbolcannot be found on the left-hand side of any arrow, it is a terminal, otherwise a non-terminal. S is simplythe non-terminal which cannot be found on any right-hand side.

Constituents and lexical categories are often abbreviated for convenience sake or according to someannotation standard like Penn Treebank [Marcus et al., 1994]. For example, verb phrase is abbreviatedto VP, nominal phrase to NP, prepositional phrase to PP, PRO to pronoun, N to noun and so on and soforth. Non-terminals are capitalized in order to more easily distinguish terminals from non-terminals.The naming fashion is, however, only a social convention between linguists and not a part of the formaldefinition of context-free grammars [Chomsky, 1956].

S→ NP VPPP→ P NPNP→ DT NNP→ DT N PPNP→ PROVP→ V NPVP→ VP PP

PRO→ IDT→ an | myN→ elephant | pajamasV→ shotP→ in

Figure 1: A list of productions allowing for English sentences like I shot an elephant to be parsed.

2.1.2. Representing Parsed SentencesAn important visual aid for linguists to describe natural languages has been to draw parsed sentences astrees, consisting of a list of labels with lines connecting them. This highlights the relations between theconstituents and the words. An example of such a tree is given in figure 2.

S

VP

NP

N

elephant

DT

an

V

shot

NP

PRO

I

Figure 2: A drawn syntax tree of the sentence I shot an elephant with non-terminal labels capitalized andterminal words lower cased.

In this tree, the terminals I, shot, an and elephant are the leaf nodes, S is the root and S, NP, VP, PRO,V, DT, N are parent as well as children nodes [Beskow et al., 1996, ch. 4]. This sort of illustration canbe seen as a precursor to data visualizations of context-free grammar.

3

2.2. Data Visualization in the BrowserIn order to understand the role of D3 and Backbone.js (the two programming libraries used in CFG-Visualizer) in modern web development, some technological fundamentals have to be understood. I,therefore, aim in this section to familiarize non-experienced readers with some of the technologicalfundamentals that make up modern web usage (section 2.2.1) in order to put D3 (section 2.2.2) andBackbone.js (section 2.2.3) into their proper context. More web technologically savvy readers are rec-ommended to skip some or all of these sections.

2.2.1. Web Technology FundamentalsThe Internet is based on the communication between web servers and web clients. Web servers are(often dedicated) computers with server software installed to serve content on request. Requests are sentby web clients, often by a user with the help of a browser. A communication protocol is used by bothserver and client, like a shared language between two speakers, so that the requests and the responsescan be understood. Most notable Internet protocol is the Hypertext Transfer Protocol (HTTP), sowhen communication is done through HTTP, HTTP-responses and HTTP-requests are being sent re-spectively. For instance, when a user enters the URL www.google.com into the Firefox web browser,a HTTP-request is sent to a google server, which sends back a HTTP-response, containing the datawhich Firefox can use to display Google’s homepage. A HTTP-response often contains a host of data,which the browser uses to display the content on the screen [Murray, 2013, p. 15-17]. This type of datawill be covered below.

HTMLHypertext Markup Language (HTML) [Hunt, 2010] is a mark-up language, which defines the overallstructure and content of what is displayed in the browser. In this mark-up language, specific HTML tagsare hierarchically organized into a tree. Unlike other mark-up languages like XML, HTML tags havedefined semantics and are interpreted and displayed accordingly by the browser. Some of these are <p>(paragraph), h3 (header, of smaller size), <ol> and <li> (ordered list and list item respectively), and<a> (URL link).

In HTML code, a tag is separated into an opening tag enclosed by angle brackets <tag> and aclosing tag, enclosed by angle brackets and a dash </tag>. In the opening tag, defined attributes canbe set <tag attribute1=value1 attribute2=value2>. Between the opening and closingtag, the tags content can be defined, which can be children tags and/or simple text. The text is usuallythat which is displayed on the screen [Murray, 2013, p. 17-23].

<html><h3>Felix’s Favorite Websites</h3><p> Here are my favorite websites, in order.</p><ol>

<li><a href="theguardian.com">The Guardian</a>

</li><li>

<a href="stackoverflow.com">Stackoverflow</a></li><li>

<a href="xkcd.com">xkcd</a></li>

</ol></html>

Figure 3: HTML mark-up (left) and how it is displayed in a browser (right).

Figure 3 shows how a simple web page could be written with HTML. First, the root element <html>is added, primarily to indicate that the document indeed is HTML, as opposed any to other mark-uplanguages like XML. Then, a header is defined with the <h3> tag, where the value of the element is theactual text to be displayed. A paragraph with some text is defined with the <p> as well as an ordered

4

list <ol> containing list items <li>. Each <li> contains the special <a> tag, which is used to createa (clickable) link, where the href attribute defines the URL to be taken to and the text content is whatis to be displayed on the screen.

SVGScalable Vector Graphics (SVG) [Dahlström et al., 2011] is a vector image format used to definethe semantics of geometrical shapes. Like HTML, it is a mark-up language with pre-defined tags andattributes [Murray, 2013, p. 49-58]. Figure 4 shows how a rectangle and a circle could be coded anddisplayed with SVG.

<svgxmlns="http://www.w3.org/2000/svg"><rect x="10" y="10" height="100" width="100" fill="green"/><circle cx="170" cy="60" r="50" fill="red"/>

</svg>

Figure 4: SVG mark-up (left) and how it is displayed in a browser (right).

First, an SVG element has to be defined with an xmlns specifying the version. The <rect> rep-resents a rectangle and has an x and y attribute, defining the coordinates on a Cartesian field, wherethe rectangle is to be positioned. The attributes height and width attributes set, unsurprisingly, thewidth and height of the rectangle, while fill fills the interior with a color, in this case green. The cxand cy attributes in the <circle> tag work the same as x and y, only difference being that the centerof the circle is placed on these coordinates. Because a circle’s size can be determined by its radius, onlythe r attribute has to be set, in this case to half of the rectangle’s height in order to align perfectly. An<svg> tag can be embedded in an HTML document as any other HTML tag, which makes it possibleto blend usual web content with animated features.

CSSCascading Style Sheets (CSS) [Bos et al., 2011] is a style sheet language, which can be used to describeor style the presentation of mark-up documents like HTML and SVG. More specifically, it allows forthe programmer to select tags and set their attributes [Murray, 2013, p. 29-35].

JavaScriptJavaScript [ECMA International, 2015] is the only programming language to be natively supported byall major browsers, such as Mozilla Firefox, Google Chrome, Safari and Internet Explorer. Therefore,all tasks which cannot be done solely with HTML or SVG, are usually done with JavaScript.

With JavaScript, the programmer can create programs that do not require any page re-load. Remem-ber, a web-client receives responses from a web-server, however, with JavaScript only the initial re-sponse has to be received and further programming tasks are done by the received JavaScript code [Mur-ray, 2013, p. 35-49].

JavaScript can be used to manipulate HTML. For example the code in figure 5 programmaticallyselects all links <a> in the HTML code and colors them green.

elements = document.getElementsByTagName("a");for (i=0; i < elements.length; i++){

elements[i].style.color="green"};

Figure 5: JavaScript code (left) and the HTML 3 as it is displayed after the code is run (right).

5

2.2.2. D3.jsD3.js is JavaScript library for creating visualizations (or graphs) in the browser. It does so by allowingfor binding, creating and manipulating HTML and, especially, SVG elements. Additionally, it providesfor algorithms to easily calculate the layouts for specific kinds of graphs, such as pie charts, tree chartsand bar charts. D3 makes it easy to make interactive graphs, i.e. graphs that can change over time or onuser-related events, for examples on mouse click or mouse hovering [Bostock et al., 2011].

2.2.3. Backbone.jsBackbone.js is a JavaScript library/framework used to design single-page applications (i.e. applicationsthat run exclusively in the browser without page reload). It does so by tying markup elements (i.e.HTML or SVG) to so called models and views. Models store defined attributes associated to somepre-conceived entity, while views are interfaces that display the data of any number of models in thebrowser. For example, on a book rating website there could be a model containing attributes associatedto a book, such as title, publisher, author, release-date, isbn and rating, and a viewto display this information in the browser with the help of HTML and CSS. Backbone.js also allowsfor defining ways in which different models and views interact with each other, as well as binding useractions to them. For example, an HTML <button>, labeled five stars could be attached to a view,on which user click actions could be bound, resulting in changing the model’s rating attribute tofive. Models can also be stored in a so called collection. In a collection, models are ordered and canbe accessed conveniently. For example, all book models with the release-date of 2015 could beretrieved from a collection called books.

Backbone.js was made to be lightweight and easily customizable. It, therefore, makes it a good com-bination with the already flexible library of D3.js [Ashkenas, 2010].

2.3. Previous ImplementationsThe reason I started from scratch with this new software, CFG-Visualizer, is because previous attemptsof visualizing context-free grammars for linguistic purposes have been too limited in their functionality.In this section, I intend to cover two programs, both of which have been a source of inspiration, andhighlight their functionality as well as their limitations.

2.3.1. M-PATRM-PATR was written by Mats Wirén for his doctoral thesis Studies in Incremental Natural-LanguageAnalysis [Wirén, 1992, p. 41] and was used to generate the graphs illustrated throughout the thesis andlater for educational purposes. Since it was implemented in Interlisp and Common Lisp on Xerox LispMachine in the early 90’s, it posed maintenance problems and is, as of today, neither maintained normigrated to any more modern operating system (such as Linux or Windows).

The features of the system can be seen in figure 6. On the upper left side, there is a a console window(Exec Chart (XCL)), which allows the user to enter a sentence to be parsed by the system. Right beneathit, on the lower left side in the Chart Window, a special kind of graph is drawn, which elegantly illustratesthe back trace of the parsing. It consist of N+1 nodes, where N is the amount of words to be parsed, anda set of links combining these nodes, each labeled with either a word (the bottom links) or a constituentas well as a number indicating its position in the parsing algorithm. The latter links can also be callededges and represent a production rule in the CFG. There is also a distinction beetween active andinactive edges: Inactive edges have fully stroked lines and show that the production rule can cover thewords between the left node and the right node. Active edges, on the other hand, have stroked lines andindicates that the algorithm has tried to parse all the words between the nodes or is in the process ofdoing so, but is not completed.

Whenever the user clicks on the label of an edge link, its syntax tree is drawn in the Parse TreeWindow on the lower right side. In the case of an active edge, everything but the incomplete constituentsis drawn.

6

Figure 6: A screenshot of M-PATR in use, after having parsed the string en svensk tiger i en park.

Technically the chart graph actually illustrates the explicit states created during a chart parsing al-gorithm. M-PATR is highly configurable in that it allows the user to decide which chart parser to use,as well as to adjust in which way the algorithm is performed (for example depth-first/breadth-first). Thetopic of chart-parsing and the particular Earley chart-parsing algorithm will be covered in more depthin section 4.1.24.

Besides not being supported on any modern systems today, M-PATR has some deficiencies. Firstly,being written in, by today’s standards, the rather uncommon programming language Common-lisp, theuser has to unnecessarily download and install compilers and libraries for it to work. Hence, if it were tobe successfully ported to a working system today, it would still have significant accessibility problems.Secondly, having been developed for the purpose of analyzing parsing algorithms, besides context-freegrammars, the program contains a range of complex features presented in a not so orderly fashion. Thiscould potentially make it harder for an unfamiliarized user to understand and use the program.

The chart graph, however, remains the main source of inspiration for CFG-Visualizer and will be oneof the main object of study in this thesis.

2.3.2. PSG Web LaboratoryThe PSG Web Laboratory is a browser based app made by Torbjörn Lager and was “designed as an aidfor students to explore context-free phrase structure grammars for natural language . . . [and] helps thestudent to understand the relationship between grammars, strings and trees, and the notions of syntacticambiguity and recursion.” [Lager, 2002]. It allows the user to enter a context-free grammar on the leftside of the page and parse (analyze) a sentence entered in an input field below it. The sentence parseforest is returned and on the right side a syntax tree is drawn. The user can scroll through the wholeparse forest. Furthermore, the maximum size of the parse-forest as well as the maximum depth of thetrees can be specified.

Being available on the browser by a simple link obviously makes PSG Web Laboratory much moreaccessible than M-PATR, but it comes with its deficiencies as well. The most obvious one is age. Having

4Daniel Jurafsky and James H. Martin also provide an excellent introduction into the topic in chapter 11 of Speech and LanguageProcessing.

7

Figure 7: A screenshot of PSG Web Laboratory in use, after having parsed the string John saw a manwith a telescope.

more than a decade on its back, seemingly without any maintenance support, its functionality has startedto deprecate. It was written partly in the Oz programming language, which is not supported on anymodern browser, hence making much of the application not working. At the time of writing this thesis,the links connecting the labels in the syntax tree window, are not shown and one cannot scroll throughthe parse-forest.

Furthermore, debugging a context-free grammar in PSG Web Laboratory is not an easy task. If, forexample, a sentence cannot be parsed, due to inefficiencies in the grammar, the program does not help tosee where things went wrong or how much of the sentence was parsed. This might be especially difficultfor users without a background in context-free grammars or programming.

2.3.3. Conclusion on Previous ImplementationsIn summary, one can say that M-PATR and PSG Web Laboratory have both seen the end of their days.They are written for technologies that are either not supported or fully out-dated by modern standards.Furthermore, they lack the functionality, aesthetics and simplicity expected of modern applications. Allthese reasons call for a new software to be made to fit the research made in this study.

8

3. PurposeThe purpose of this thesis will be to investigate how data visualizations of context-free grammars canbe used as an educational tool to help users in writing and constructing context-free grammars. To thispurpose, three related research questions have been formulated:

1) Can data visualization facilitate the learning process in connection with understanding and apply-ing context-free grammar?

2) How is data visualization utilized to construct a context-free grammar?

3) How is data visualization of context-free grammars perceived or understood by users?

In the upcoming section, I intend to cover the methods used to answer these questions.

9

4. MethodThe methodology of this study consist of, on the one hand, creating a program which can visualizecontext-free grammars in a suitable way, and on the other hand, defining the methods for investigat-ing user experience in relation to data visualization. In this section, I intend to cover both of theseapproaches.

4.1. CFG-Visualizer: A Software ImplementationCFG-Visualizer is a browser based, single-page application written in JavaScript, D3.js and Backbone.js.It allows for writing and editing a context-free grammar, entering and parsing sentences conforming tothe rules of the grammar, and visualizing the parsed output.

As seen in figure 8, the superficial layout is similar to that of PSG Web Laboratory. On the left side,there is a text area to write and edit a context-free grammar and an input field to enter and test sentences,while on the right side the parsed output is visualized.

Figure 8: A screenshot of CFG-Visualizer having parsed the sentence the old man plays the piano. Onlyone syntax tree has been found for this sentence.

The idea behind CFG-Visualizer was to have a simple layout, which clearly emphasizes the workflowof the program. Instead of having multiple superfluous buttons as in PSG Web Laboratory, or a lotof windows as in M-PATR, the intended purpose of writing, debugging and constructing context-freegrammar is on the forefront of the application.

The complete list of features are listed below. The name of the feature is in bold followed by itsposition in the user interface in parentheses. Subfeatures are indented below their parent feature.

Grammar editor (upper left side) A text area in which production rules of a context-free grammarcan be written and edited.

Sentence generator (lower right corner) A button (randomize), which, when clicked, gen-erates and parses a random sentence from the context-free grammar (with a limit up to eightwords).

10

Sentence parser (lower left side) An input field, where a sentence can be entered for parsing. Ifthe sentence is not successfully parsed, the field’s background becomes red.

Syntax tree (upper right side) A visualized syntax tree of the currently selected tree in the parseforest.

Forest browsing (upper right corner) A right and left arrow button to browse through theparse forest. Next to these buttons, the number of trees in the parse forest and the currenttree index displayed are shown.

Chart graph (lower right side) A chart graph (see section 2.3.1), illustrating the parsed output ofthe sentence input.

Edge inspection On hovering a label of a non-word edge, the complete or incomplete syntaxtree (see section 4.1.2) of that edge is drawn in the syntax tree window.

Edge filtering (lower right corner) Two checkboxes to remove active or inactive edges fromthe chart graph.

4.1.1. Program ArchitectureThe parsing is done with the Earley chart parse algorithm and is written in regular JavaScript. Thevisualization is put to life with D3.js, while Backbone.js does the inner workings of the user interfaceaccording to the MVC framework5.

Figure 9 shows the different Backbone.js/D3 components and how they interact with each other.The Editor view is responsible for showing the text area and whenever something is entered intoit, it is tokenized, syntax checked and each production rule is added to the Grammar collection. TheSentence Parser view displays the input field in the lower left corner and, whenever something isentered into it, parses the sentence according to grammar and updates the Chart model, which containsthe chart structure (more on parsing and chart structure in the next section). The Graphs view is thenresponsible for how and in which order the graphs are drawn in their respective window and data isaccordingly fed into the Syntaxtree and ChartVisualizer module in order to draw the chartgraph and the syntax tree graph. D3.js thus only handles the visualization.

Figure 9: CFG-Visualizers Backbone architecture. V stands for View, C for collection and M for model.

5The topic of MVC (Model-view-controller) will not be covered in this thesis. For those interested in what MVC is and how itworks in Backbone.js the online documentation website http://backbonejs.org/ [Ashkenas, 2010] is recommended.

11

4.1.2. ParsingFor parsing context-free grammars, the Earley algorithm [Earley, 1970] was used. It is a so called chart-parsing algorithm, because it populates its results in a structure called a chart, whose results are reusedduring parsing. It is, thus, a dynamic programming approach. Figure 10 shows the pseudo code of thealgorithm from [Jurafsky and Martin, 2000, p. 478].

The chart contains N +1 arrays, where N is the amount of words in the input string. During parsing,an array is populated with so called states. A state consists of:

• A production rule (X → α) matched.

• A dot •, representing the position of the parsing in the production rule.

• A start position i and an end position j ranging between the words matched [i, j].

In the algorithm, the chart arrays are looped through left-to-right. Either one of the procedures pre-dictor, scanner or completer are applied depending on the status of the state. They all take a single stateand derive new states from it. The predictor makes top-down predictions on states which have a non-terminal immediately to the right of its dot and is not a part-of-speech category. For all production rulesin the grammar which have that non-terminal on its left-hand side, a state is created with i and positionj placed where the input state ends. The scanner is applied when a state has a part-of-speech category tothe right of its dot. If the word at that position is a word of that part-of-speech category in the grammar,a new state is created of the form (wordclass→ word • [0,1]). Finally, the completer is applied to anystate which has its dot at the end of their rules. It makes copy of all states in the previous array whichhave the same rule over the same span of input and advances the dot over that span [Earley, 1970].

The algorithm described so far is only a recognizer, not a parser. In CFG-Visualizer, the parse forest isretrieved by first finding a complete parse S→α • [0,N] in the last array of the chart and then recursivelyfinding other completed states that fit the symbols on the right-hand side [Thompson and Ritchie, 1983].6

The chart graph (see Figure 8) illustrates all the states generated by the Earley recognizer. For eachstate, an edge is drawn from the start position i to the end position j. If the dot is not positioned at theend of the production rule, the edge is active and dotted, otherwise inactive and filled. If the user hoversover an active edge, the parsed input up until the dot is drawn as a syntax tree, while the rest of thesymbols are drawn with dotted lines as children to the root. On hovering an inactive edge, a fully parsedsyntax tree is naturally drawn.

4.2. User Experience ResearchIn order to find out how data visualization can be of benefit for writing context-free grammars, two wellknown user-experience research methods are used: usability testing and user interviews. Subjects forboth of these methods are the same users, where the usability test is done first and the user interviewdirectly afterwards.

Usability testing is a technique where a user is given a set of tasks to complete, while being observedby a researcher [Nielsen, 1994]. Although not explicitly connected in the technical literature, usabilitytesting is an observation technique [Bell and Waters, 2014, chapter 12] which “provides direct infor-mation about how people use computers and what their exact problems are with the concrete interfacebeing tested” [Nielsen, 1994, p. 165]. Because this study is about identifying user behavior relating todata visualization in a user interface, usability testing is an appropriate method to observe and analyzesuch behavior in real time.

A user interview is a one-on-one discussion between a researcher and a participant to discuss in depthwhat the participant thinks of a topic in question. For this study, a user-interview is an appropriatecomplement to usability testing, because it potentially gives more weight to observations made during

6In future versions of CFG-Visualizer, a more efficient and less error-proned parsing method might be implemented, for examplethe one described in Elizabeth Scott’s article SPPF-style parsing from Earley recognisers [Scott, 2008].

12

function EARLEY-PARSE(words, grammar)

ENQUEUE((γ →•S, [0,0]),chart[0])for i← from 0 to LENGTH(words) do

for each state in chart[i] doif INCOMPLETE?(state) and

NEXT-CAT(state) is not a part of speech thenPREDICTOR(state)

else if INCOMPLETE?(state) andNEXT-CAT(state) is a part of speech then

SCANNER(state)else

COMPLETER(state)end if

end forend forreturn chart

procedure PREDICTOR((A→ α •B, [i, j]))for each (B→ γ) in GRAMMAR-RULES-FOR(B, grammar) do

ENQUEUE((B→•γ, [ j, j]),chart[ j])end for

end procedure

procedure SCANNER((A→ α •B, [i, j]))if B⊂ PARTS-OF-SPEECH(word[j]) then

ENQUEUE((B→ word[ j], [ j, j+1]),chart[ j+1])end if

end procedure

procedure COMPLETER((B→ γ•, [ j,k])) )for each (A→ α •Bβ , [i, j]) in chart[ j] do

ENQUEUE((A→ αB•β , [i,k]),chart[k])end for

end procedure

procedure ENQUEUE(state, chart-entry)if state is not already in chart-entry then

PUSH(state, chart-entry)end if

end procedureend function

Figure 10: Pseudo code of the Earley Recognizer [Jurafsky and Martin, 2000, p. 478].

13

usability-testing and, more importantly, can provide an insight to how users perceive data visualizationin the context of context-free grammars, which relates to the third research question of this thesis.

For this study, five participants are chosen, based on the findings made in Nielsen and Landauer’sproceeding A mathematical model of the finding of usability problems [Nielsen and Landauer, 1993]that exactly five users should be enough to make important observations. They show that more thanfive users provide very little more information and, therefore, argue that it is waste of resources. Thesefindings are closely connected to iterative design methodology, which is concerned with improving asoftware design in cycles of development and testing, in order to not just identify weaknesses, but toalso to improve upon it [Nielsen and Landauer, 1993].

Prior to usability-testing, participants’ background in linguistics are recorded and, because CFG-Visualizer requires some understanding of what context-free grammars are and how their syntax work,all participants receive a ten minute short introduction into what context-free grammars are and relatedconcepts. Related concepts, such as root symbol S, terminals, non-terminals, production rules and lexi-con are covered during this introduction. A syntax tree example is given and explained, however, not thechart graph. The slides for this introduction, which are used by the researcher to explain these concepts,can be found in Appendix B7. Because the app is meant to be used as an educational tool for studentsto learn context-free grammars, participants are needed who have no prior experience of context-freegrammars or programming. The latter because prior experience in programming might provide un-known advantages in handling syntax errors, debugging programs and understanding structured code,which might skew the results.

4.2.1. Usability TestDuring the usability test, the user is given a lab assignment (see Appendix A) which consists of twoparts. In the first part, the user is asked to perform a set of tasks having to do with the functions in theprogram based on the default grammar loaded on page-reload and aims to make the user comfortablewith the features of the program. In the other part, the user is asked to construct a context-free grammarwhich can parse four given grammatical sentences but not three ungrammatical sentences8.

The types of tasks given in the second part can be divided into three categories: word-to-wordclassclassification, grammar construction and syntactic disambiguation. Word-to-wordclass classifica-tion (see Appendix A, part 2, exercise 1) is when the user has to classify a set of words according totheir respective wordclass. Grammar construction is when the user has to write the actual productionrules to make a working context-free grammar (see Appendix A, part 2, exercise 2). Syntactic disam-biguation is when the user has to make sure that the grammar can produce more than one syntax treegiven an ambigious sentence.

During the program interaction, the user is allowed to ask a supervisor for assistance. Letting usersdo the tasks without needed assistance proved after some piloting to be too difficult, due to the relativelydifficult task of writing a context-free grammar. Therefore, besides an observer taking notes, a supervisoris also present during the usability testing (in this case, the author of this study).

Two measurable variables are also recorded by the observer, in order to counter-balance and under-stand the possible bias of the supervisor’s role in the test. These are:

Time progress The time the user took to finish the first part and the second part respectively.

Assistance time Time in which either participant or supervisor communicated with one another.

During program usage, the usage of the application is observed. This partly done by filling out a binarytable for each user, illustrated in table 1. In this table, the rows correspond to the tasks in the second partof the lab assignment and the columns correspond to the application features listed in section 4.1. If aparticipant is observed using a specific feature while solving a specific task, the cell of the feature andthe task is filled out with Yes, otherwise with No.

7Credit to Kristina Nilsson Björkensstam for providing me with these slides.8In Appendix A, the parts are not numbered, but the first part is called Uppvärmning: I shot an elephant in my pajamas and thesecond part Konstruktion av egen grammatik.

14

Furthermore, the ways in which graphs and other features of the programs are used for certain tasks,are also qualitatively observed to further complement and explain the binary observations made. Finally,if necessary, questions are asked by the supervisor, in order to get the user’s real time understanding ofsome certain phenomena, most importantly data visualization.

FeaturesGrammar Sentence Sentence Syntax Forest Chart Edge Edgeeditor generator parser tree browsing graph inspection filtering

Word-to-wordclassclassification . . . . . . . . . . . . . . . . . . . . . . . .

Tasks Grammarconstruction . . . . . . . . . . . . . . . . . . . . . . . .Syntacticdisambiguation . . . . . . . . . . . . . . . . . . . . . . . .

Table 1: A binary table to keep track of app feature usage while solving a task. Rows represent the labassignment tasks and columns the features found in CFG-Visualizer. After usability testing, thecells should contain either a binary value Yes, if the feature has been used for the task, or No, ifthe the feature has not been used for the task.

4.2.2. User InterviewDuring the user interview the researcher tries to find out how the user perceives the data visualization,i.e. the graphs, their attitudes towards it and the part it played in constructing context-free grammars. Inthis thesis, the following questions (and eventual follow-up questions) are asked:

1. How would you explain the program to someone else?

2. What there anything especially difficult you encountered while doing the exercises?

3. Was there anything which made it easier for you to finish the exercises?

4. Did you find any of the graphs useful?

5. In which way did the graphs help you to complete the tasks?

The first question attempts find out how the user perceived the program in general. This an importantquestion to ask in order to indirectly understand how the user interprets the workflow of the program andto see to which extent they understand the parsing process. The second question attempts to cover thedirect inefficiencies of the program/exercises. If the user immediately points out some specific feature,it is most probable that this feature indeed has deficiencies. The same logic is applied with question3, where if the user immediately mentions a specific feature, this feature is probably of great benefit.Question 4 and question 5 try more explicitly to find out how the user perceived the data visualization,i.e. the graphs, and how the user viewed their usefulness.

Order is important. The questions regarding the graphs specifically come at the end, so that the par-ticipant is not biased in the previous question to answer what he/she thinks the researcher wants to hearor try to come up with an answer as he/she goes. This is based on the idea laid out by Nielsen [Nielsen,2010], that if a user finds a specific feature to be particularly useful, in our cases the graphs, this shouldcome up naturally in the interview. However, if the view of the interviewer is previously imposed on theuser, that will make the user more prone to say things that he/she would not have otherwise thought.

15

5. ResultsTable 2 shows the measurable variables recorded for each participant. Most participants finished the labassignment within ten minutes of an hour except for subject 5, who finished the exercises in 39 minutes.Furthermore, all participants finished all exercises successfully, except for subject 4, whose context-freegrammar did not produce two syntax trees for sentence (iv) in the second part. Finally, supervisor inputwas mostly limited to the second part and little help was needed for the first part.

An outstanding fact from these result is that all participants could, within an hour and without priorexperience of context-free grammars, write working context-free grammars.

Subject Part 1 + Part 2 = Overall (minutes) Supervisor Input (minutes) Linguistic Background

Subject 1 10 + 45 = 55 24 Yes

Subject 2 10 + 51 = 61 27 Yes

Subject 3 13 + 43 = 56 17 Yes

Subject 4 21 + 48 = 69 29 No

Subject 5 12 + 26 = 38 9 Yes

Table 2: Participant results of the usability study. The first column is the name given to each subject. Thesecond column refers to the time it took for the participant to finish the first part, the secondpart and the overall time. Supervisor Input refers to the time in which participant or supervisorcommunicated with each other and linguistic background to whether the participant has anyacademic background in linguistics or not.

Relating to the the research questions, how data visualization is utilized to construct a context-freegrammar and how data visualization is perceived or understood, the results of the usage and perceptionof context-free grammars will be covered in the following sections.

5.1. Usage of Data VisualizationTable 3 shows the complete results of lab task/feature usage for all participants (for a binary tableof each specific participants, see Appendix C). The results are very consistent across all participants,where same features were used for the same tasks. The only outlier is the usage of edge inspection forgrammar construction, where usage differed slightly between participants.

In the initial task of word-to-wordclass classification (see Appendix A, part 2, exercise 1), neitherthe syntax tree nor the chart graph was utilized by the participants. Actually, no sentence was enteredby any user into the sentence parser during this exercise. A couple of users misunderstood the syntaxof the production rules and wrote for example a dash - instead of a dash and lesser-than sign -> orconfused the position of lexical categories (non-terminals) and words (terminals) for example placing aword like shoot on the left-hand side and the lexical category V on the right-hand side V <- shoot.When facing problems due to these syntax errors, the graphs were not (and to most part could not be)utilized. In the face of these errors, the appropriate syntax convention had to be pointed out by thesupervisor.

During grammar construction, users had a common workflow, wherein they 1) reasoned about theproduction rules, 2) wrote them down and 3) when feeling comfortable with the grammar, used thesentence parser to test if the program could generate any output for the sentences given in the exercises.If it did not work, they would be repeat the process. They did mostly not do any intermediate testingto see if a part of a sentence could be parsed nor did they ever, although being exposed to it in part 1exercise 4, use the sentence generator.

16

FeaturesGrammar Sentence Sentence Syntax Forest Chart Edge Edgeeditor generator parser tree browsing graph inspection filtering

Word-to-wordclassclassification All yes All no All no All no All no All no All no All no

Tasks Grammar Yes, Noconstruction All yes All no All yes All yes All no All yes No, No, Yes All noSyntacticdisambiguation All yes All no All yes All yes All yes All no All no All no

Table 3: The combined results of all binary tables listed in Appendix C. Rows represent lab assignmenttasks and columns the features in CFG-Visualizer. If the same value was observed across allparticipant, the cell contains either All yes or All no, otherwise the result, yes or no for eachparticipant is listed in order and in bold (subject 1, subject 2 . . . subject 5).

When it came to the chart graph, the advanced features of edge inspection or edge filtering wereused by some users but could not be utilized to solve the task at hand. The only way in which the chartgraph was utilized was when a sentence did not successfully parse. More specifically, the absence ofnon-word labeled links between two nodes, would help the user to identify at which word parsing wentwrong and would target that specific word for debugging together with its given lexical categories orconstituents when trying to fix the context-free grammar. Even if a participant had studied the graphcarefully in the introductory part (see Appendix A) or had it shortly explained, they would not utilizethese advanced features more than those who had not studied them closely.

Finally, when the issue of syntactic disambiguation came up (see Appendix A, part 2, exercise 2)and the participant’s grammar had to produce two syntax trees for a given sentence, but only one wasproduced, only the syntax tree was utilized together with its subfeature forest browsing. With it theuser could reason about which interpretation the already parsed tree gave, in order to write rules for theinterpretation that was not showing up. Here, however, the chart graph was not utilized at all by anyparticipant.

5.2. Perception and Understanding of Data VisualizationAll participants, including the one without a linguistic background, understood the contents of the syn-tax tree and the effect to which entering a sentence into the sentence parser input produced it. Thiswas shown in the usability-test, in which each participant (without aid from the supervisor) could comethe conclusion that a drawn syntax tree was the result of a successful parse (although subject 4 did atthe beginning not understand the purpose of the exercises). Furthermore, while solving the first exer-cise concerning syntactical ambiguity (Appendix A, part 1, exercise 2), all participants could answersatisfactory why a sentence was ambiguous with the help of the graph.

On initial usage, the chart graph was not comprehensible for any participant. They were over-whelmed with the complexities of information displayed and it was not clear for them that the graph wasindeed interactive and hence did not use any of the scrollable features, unless advised to do so. Whenasked, during usage, what the user thought the graph displayed, they could not come up with an answer.

In the interviews, all participants expressed dissatisfaction with the chart graph. They admitted tonot knowing what the links between the nodes meant, why some links were dotted and some not, orwhat exactly the buttons did. The only feature which was naturally mentioned in question 2, was thatof the syntax tree, which they thought helped them solve the exercises. When asked what they thoughtwas most difficult with the exercises, most users mentioned keeping track of the production rules and toutilize the production rules in an effective way. The chart graph was never mentioned on question 3.

17

Finally, when asked how the program was structured, users seemed to understand the intended work-flow. It was often described as “something to write grammars with and produce sentences with”, whichexpressed an indirect understanding of parsing.

18

6. Discussion6.1. Method6.1.1. Software ImplementationOne of the main motivations for writing this thesis, was to develop an application which could beused for educational purposes. In this regard, I would argue, that the thesis has been successful. CFG-Visualizer is free software (GNU GPL Version 39) application, publicly available online10, to be used,studied and further improved upon. This is especially true in relation to web technologies, since CFG-Visualizer can serve as a code example on visualizing linguistic data with D3.js and combining D3.jsand Backbone.js to make a feature rich, functional user interface for linguistic purposes. Although thereare examples of visualizing language data with D3.js11 and of combining D3.js and Backbone.js tocreate user interfaces with visualized data12, there is, to my knowledge, no code example of D3.jsand Backbone.js being used to make feature rich user interfaces for linguistic purposes. Seeing thatboth D3.js and Backbone.js are fully intergrated with solid web standards, like HTML, CSS, SVGand JavaScript, whose presence will not go away any time soon, CFG-Visualizer will most likely bea working code example for a long time to come. This stands in contrast to the other two softwareimplementations mentioned in this thesis, M-PATR and PSG Web Laboratory, who both, at the outsetof their development, were less likely to be supported in the long term because they relied on somerelatively fringe technologies.

Furthermore, being accessible by just a click away in the browser, makes it convenient for educators toinclude it in their educational material. With a single-page application like CFG-Visualizer, they do nothave to be concerned about installation, update or cross-platform issues, which is more prevalent withtraditional desktop applications. Finally, the fact that the thesis describes an educational setting in whichthe program is used and the results of it, can be of value for educators who wish to teach context-freegrammars or for those who are in the field of researching the teaching of context-free grammars.

6.1.2. User Experience ResearchThe biggest elephant in the room of the methods for user experience research described in this thesisis probably that of the role of the supervisor in the usability-testing. Although extra care was taken tocounter-act and understand the bias and influence of the supervisor, there is still the possibility that thesupervisor had unknown influences, which might depreciate the credibility of the results. After all, itcannot be fully determined whether the results reflect the actual usage and perception or if the usage orperception is imposed by the input of the supervisor. The fact remains, however, that if the supervisorwas not present, the tests could not have been performed and there would be no data to be analyzed. Inthe light of this, I would argue that any insight is better than no insight. Finally, if the software couldbe improved to such an extent that the supervisor would not have to be present, which would of coursehave to be determined by piloting, the same study could be performed and these results could be putup to test. A final counter-argument against the critique of a supervisor, is that a supervisor is presentin practical situations anyway, mostly in educational settings, so the results might actually reflect morerealistic situations and make them more transferable to real usage than otherwise.

The introduction presented to users could also have been done differently. In this study, only in-formation regarding the properties of context-free grammars were covered. However, if the topic ofparsing and chart-parsing as well as the chart graph and its functionalities had been presented, differentresults might have emerged, which might corresponded more accurately to some real-life educational

9http://www.gnu.org/licenses/gpl-3.0.en.html (accessed 08-june.2016)10Currently running at http://mumin.ling.su.se/cfg-visualizer/ and source code is available at https://github.com/felixhultin/cfg-visualizer (accessed 03-june.2016).

11An example of this would be Ben Podgursky’s nlpviz, which does sentence parse tree visualization with D3.js http://nlpviz.bpodgursky.com/home (accessed 07-june.2016)

12See for example Miles McCrocklin’s six part series of designing and creating reusable visualizations with D3.js: http://bl.ocks.org/milroc/5522467 (accessed 07-june.2016)

19

settings, in which the topic of context-free grammar and parsing/parsing algorithms are taught together.However, I chose not to include these topics, because, firstly, context-free grammars might be taughtindependently of chart-parsing algorithms, in which the relatively complex topic of chart-parsing algo-rithms would add extra time and resources in learning, and, secondly, it would bias the inherent usageand understanding of data visualization. An important goal of data visualization is, after all, to be easilyunderstood at first sight and the insight to that initial understanding would be lost if such an introductionwas made. With that said, however, it would be interesting, in the light of the results in this study, to seethe results of a similar study like this but with a more comprehensive introduction.

Further critique can be directed at the sampling size used in this study. Even though Nielsen et. alargued that five is enough for usability-testing, other researchers have questioned this on the groundthat five subjects is not enough to be representative of the total population wishing to be studied andthat every usability problem is not equally-easy-to-detect [Schmettow, 2008, p. 89-98]. In my opinion,however, these points fall flat in this study, because, firstly, all usability-problems are not searched forbut enough for further research and improvements to be made and, secondly, the features studied inthis study are explicitly defined, namely data visualization, so potential, not-easily-detected, usabilityproblems in the user interface are not as relevant to the research area at hand.

A final note on replicability. Although this would have to be determined by further research, I wouldargue that the results shown are likely to repeat themselves if similar subjects are chosen with similarlinguistic and technical background, because the variables involved in the tests are not likely to change. Itdoes not seem likely that the users workflow would change because of different environmental settings.The only thing, as mentioned above, which could change users behavior might be if the supervisor isnot involved. In the light of this, it would also be interesting to conduct similar research with users whoalready have experience in either programming or context-free grammars.

6.2. ResultsThe results show the limitations and benefits of the kinds of data visualizations described in section4.1. It seems, after reviewing the results, that both graphs helped users in one crucial aspect, namelyto check if a given sentence can be parsed with a specific CFG. This happens to be, I would argue, themost important feature to have for a program dedicated to editing construct-free grammars. If you cannotdetermine that the grammar parses the sentences you want the grammar to generate, you cannot knowif it meets your expectations. Furthermore, for the user to be able to see at which specific word the errorshows up, helped the productivity of writing a context-free grammar, which seems to indicate that this isa more convenient approach of learning context-free grammars than, say, just writing them with pen andpaper. This differs significantly from PSG Web Laboratory (see section 2.3.2), where such informationcould not be attained. However, to make certain that such information does have an influence, it wouldbe interesting to make a similar study like this one, but with PSG Web Laboratory.

However, an abundance of information does not seem to be helpful to users either. Although the chartgraph contained expressive information about the parsed output and the way it was parsed, beginnerusers did not seem to benefit from this information when writing context-free grammars. So even thoughM-PATR contains a lot more information and has a much more flexible configuration when it comes toparsing and the output of the parsing, which probably helps in research endeavors, this informationwould just be superfluous for inexperienced users.

Actually, as the results show, users’ attention were mostly focused on reasoning about and writing pro-duction rules. Feature rich graphs were neither enough to help users understand syntactic organization ofproduction rules nor to grasp the inherent relations between the rules written and the graphs displayed.To solve these deficits, other features would have to be implemented. For example, syntax highlighting,could help users better understand the syntax of production rules and how the root symbol S, terminals,non-terminals and the rules relate to one another. It would also help users with syntax errors, whichoften came up during usability-testing. Also, a feature could be implemented to let users add productionrules or terminals and non-terminals or part-of-speech categories and their words graphically, rather

20

than writing a CFG by hand, which might help to better understand the structure and the formal prop-erties of context-free grammars. Furthermore, visualizing the relationships between the graphs and theproduction rules, might make debugging of context-free grammars easier. For example, if a user scrollsover a link or label in any of the graphs, the rule from which they were deduced could be highlighted inthe text area or the rules could be colorized in the text area and the links in the graphs corresponding tothese rules could have the same color. Also, a feature to let users draw their own syntax tree, for which acontext-free grammar could be generated, might help them better understand the relationships betweenthe output and the grammar. Finally, letting users step through the parsing, wherein edges would bedrawn on word by word or state by state, could help users understand the parsing process easier.

For further research, all or some of the mentioned possible features could be implemented and thesame study conducted, in order to see whether they would yield more satisfying results. This would helpfurther the understanding user interaction with data visualization of context-free grammars, laid out inthis thesis.

One positive outcome from this research is probably the insight on how efficient an app like CFG-Visualizer can be to learn context-free grammars. Within an hour, all participants, without prior expe-rience of context-free grammars, could satisfactorily write working context-free grammars. In the lightof these results and the ones mentioned above describing the applications usefulness in debugging acontext-free grammar, I would argue that the answer to first research question can data visualization fa-cilitate the learning process in connection with understanding and applying context-free grammar is yesand, furthermore, that CFG-visualizer can be an effective tool for educators wishing to teach studentscontext-free grammars. Furthermore, the answer to the second research question how is data visualiza-tion utilized to construct a context-free grammar is, for all the reasons given above, that users testedsentences to see if they could be parsed by specific grammars and, if parsing did not work, identify atwhich word parsing stopped. Finally, the answer to third question how is data visualization of context-free grammars perceived or understood by users? is that users understood that visualization related toparsed output and that words were organized according to constituent relations, but the full expressive-ness of chart-parsing visualization could not be appreciated.

Now beyond context-free grammars and the purpose of this thesis, data-visualization within the fieldof linguistics and the effect to which it can be used by students and professionals, is something which,in my opinion, should be further researched. Seeing how illustrations of linguistic phenomena, whetherbeing dependency trees within dependency grammar or syntax trees within generative linguistics, hasbeen a prevalent and useful practice for many years, expanding it to 21st century technology, might justbe an important tool for us to better understand and communicate theories of language.

21

7. ConclusionThis thesis has investigated how data visualization can help users in writing and constructing context-free grammars. The three research questions 1) can data visualization facilitate the learning process inconnection with understanding and applying context-free grammar, 2) how is data visualization utilizedto construct context-free grammars and 3) how is data visualization perceived or understood by userswere formulated and answered by developing the browser based app CFG-Visualizer and conductingusability-tests and user interviews on participants without prior knowledge in context-free grammars:

The results showed that 1) data visualization was facilitated in the learning process in that it helpedusers to write and test sentences for specific context-free grammars and study the parsed output, 2) userstested sentences to see if they could be parsed by specific grammars and, if parsing did not work, identifyat which word parsing stopped, and 3) users understood that visualization related to parsed output andthat words were organized according to constituent relations, but the full expressiveness of chart-parsingvisualization could not be appreciated. Hence, data visualization and usage thereof was limited in thesense that it failed to be used in keeping track of and formulating production rules, solve ambiguityproblems and identifying syntax errors (of production rules themselves). New features to implementand research thereof has been proposed as well as an expansion of the field of data visualization withinlinguistics.

22

Laboration: Kontextfria grammatiker

Felix Hultin

Uppvärmning: I shot an elephant in my pajamas1. Öppna en webbläsare (Firefox, Chrome eller Internet Explorer) och gå till länken

http://mumin.ling.su.se/cfg-visualizer/.

2. På vänster sida hittar ni en liten, redan definererad, kontextfri grammatik. Stud-era den närmare och fundera över vilka satser den kan generera.

3. Gå ner till rutan i vänstra hörnet där det står “Enter sentence...” och skriv insatsen I shot an elephant i inmatningsfältet. Studera den övre och undre grafenoch fundera på vad de representerar. Vad händer om man sedan tar bort den sistabokstaven, t:et i elephant?

Skriv nu in I shot an elephant in my pajamas. Gå igenom och studera resultateni den övre grafen genom att klicka på pilarna uppe till höger. Varför generar denhär grafen två träd och den andra bara ett?

4. Testa slutligen att trycka på knappen randomize några gånger och analysera re-sultaten. Är alla satser grammatiskt giltiga (på engelska)? Om inte försök förk-lara varför de ändå lyckas genereras och nämn dessutom några nödvändiga gram-matiska egenskaper som du anser saknas i grammatiken.

Konstruktion av egen grammatikTa nu bort den initiala grammatiken och förbered er för att skriva en egen kontextfrigrammatik för svenska (tips: spara grammatiken lite då och då i en textfil så att gram-matiken inte försvinner ifall fönstret råkar stängas eller dylikt). Ni kommer utgå ifrånde följande orden: en, bror, cykeln, vän, damen, killen, den, handväskan, han, sov, min,köpte, träffade, av, med.

1. Fundera över vilka ordklasser orden tillhör och dela in dem.

2. Skriv en kontextfri grammatik som känner igen nedanstående satser men sominte känner igen de rödmarkerade. Exempel (ii) och (iv) bör producera två syn-taxträd

(i) min vän sov(ii) han träffade damen med handväskan

(iii) den gamla cykeln sov(iv) killen köpte cykeln av den gamla damen(v) *sov köpte min vän

(vi) *killen sov cykeln(vii) *min gamla vän

A.

23

Frasstrukturträd

2016-03-15 KNB, DALI LINGVISTIK

B. Introduction Slides

24

Frasstrukturträd● Lexikon:

– V: bakar

– PN: Bengtsson

– Adv: betydligt

– A: bättre

– N: bullar

● Grammatik

– S NP VP

– NP PN

– VP V NP

– NP AP N

– AP AdvP A

– AdvP Adv

2016-03-15 KNB, DALI LINGVISTIK

Frasstrukturträd● Lexikon:

– V: bakar

– PN: Bengtsson

– Adv: betydligt

– A: bättre

– N: bullar

● Grammatik

– S NP VP

– NP PN

– VP V NP

– NP AP N

– AP AdvP A

– AdvP Adv

2016-03-15 KNB, DALI LINGVISTIK

Formella grammatiker

● Ett språk består av en mängd strängar:

Bengtsson bakar betydligt bättre bullar

● En sträng är en ändlig sekvens av noll eller fler

symboler från alfabet Σ

bakar, Bengtsson, betydligt, bullar, bättre

● Symbolerna kombineras enligt ett regelsystem för

språket

2016-03-15 KNB, DALI LINGVISTIK

Kontextfri grammatik

Chomsky (1956):

1. Terminala symboler: ord

2. Icke-terminala symboler: konstituenter

– N (substantiv), V (verb)– NP (nominalfras), VP (verbfras)– S (startsymbol)

3. Regler: består av en enda icke-terminal till vänster och en eller fler terminaler

och/eller icke-terminaler till höger

– S → NP VP– S S CONJ S– NP → Det (A) N– NP NTillåter: disjunktion, optionalitet, rekursion

● Passar t ex för att beskriva frasstruktur

2016-03-15 KNB, DALI LINGVISTIK

Kontextfri grammatik

● Lexikala kategorier

(ordklasser)

PM: egennamnN: substantivV: verbA: adjektivAdv: adverb

● Icke-lexikala kategorier

(frastyper)

S: startymbol/sats

NP: nominalfras

VP: verbfras

AP: adjektivfras

AdvP: adverbfras

● Lexikon:

PM: Bengtsson

N: bullar

V: bakarA: bättre

Adv: betydligt

● Frasstrukturregler

S NP VPNP PMNP AP N

VP V NP

AP AdvP A

AdvP Adv

2016-03-15 KNB, DALI LINGVISTIK

C. Results of Lab Task/App Feature UsageThis appendix includes the raw results of each participant’s usage of specific app features for solvingthe different types of tasks in the lab assignment (see Appendix A). They are presented in table 4, 5, 6,7 and 8, where rows represent the lab tasks and columns the app features in CFG-Visualizer (see section4.1). The cells contain either a binary value Yes, if the feature has been used for the task, or No, if thethe feature has not been used for the task.

The results are very consistent across all participants. The only outlier is the value, in bold, of edgeinspection for grammar construction, where usage differed between participants.

FeaturesGrammar Sentence Sentence Syntax Forest Chart Edge Edgeeditor generator parser tree browsing graph inspection filtering

Word-to-wordclassclassification Yes No No No No No No No

Tasks Grammarconstruction Yes No Yes Yes No Yes Yes NoSyntacticdisambiguation Yes No Yes Yes Yes No No No

Table 4: Subject 1

FeaturesGrammar Sentence Sentence Syntax Forest Chart Edge Edgeeditor generator parser tree browsing graph inspection filtering

Word-to-wordclassclassification Yes No No No No No No No

Tasks Grammarconstruction Yes No Yes Yes No Yes No NoSyntacticdisambiguation Yes No Yes Yes Yes No No No

Table 5: Subject 2

FeaturesGrammar Sentence Sentence Syntax Forest Chart Edge Edgeeditor generator parser tree browsing graph inspection filtering

Word-to-wordclassclassification Yes No No No No No No No

Tasks Grammarconstruction Yes No Yes Yes No Yes No NoSyntacticdisambiguation Yes No Yes Yes Yes No No No

Table 6: Subject 3

30

FeaturesGrammar Sentence Sentence Syntax Forest Chart Edge Edgeeditor generator parser tree browsing graph inspection filtering

Word-to-wordclassclassification Yes No No No No No No No

Tasks Grammarconstruction Yes No Yes Yes No Yes No NoSyntacticdisambiguation Yes No Yes Yes Yes No No No

Table 7: Subject 4

FeaturesGrammar Sentence Sentence Syntax Forest Chart Edge Edgeeditor generator parser tree browsing graph inspection filtering

Word-to-wordclassclassification Yes No No No No No No No

Tasks Grammarconstruction Yes No Yes Yes No Yes Yes NoSyntacticdisambiguation Yes No Yes Yes Yes No No No

Table 8: Subject 5

31

References[Aho et al., 2006] Aho, A. V., Lam, M. S., Sethi, R., and Ullman, J. D. (2006). Compilers: Principles,

Techniques, And Tools. Addison-Wesley.

[Ashkenas, 2010] Ashkenas, J. (2010). Backbone.js online documentation version 1.3.3. http://backbonejs.org/. Accessed: 2016-05-11.

[Bell and Waters, 2014] Bell, J. and Waters, S. (2014). Doing Your Research Project : A Guide ForFirst-Time Researchers., volume 6th ed. McGraw-Hill Education.

[Beskow et al., 1996] Beskow, B., Lager, T., and Nivre, J. (1996). Elementa i generativ grammatik.Studentlitteratur.

[Bos et al., 2011] Bos, B., Çelik, T., Hickson, I., and Lie, H. W. (2011). Cascading Style SheetsLevel 2 Revision 1 (CSS 2.1) Specification. first Edition of a Recommendation, W3C.https://www.w3.org/TR/2011/REC-CSS2-20110607/.

[Bostock et al., 2011] Bostock, M., Ogievetsky, V., and Heer, J. (2011). D3: Data-driven documents.IEEE Trans. Visualization & Comp. Graphics (Proc. InfoVis).

[Chomsky, 1956] Chomsky, N. (1956). Three models for the description of language. InformationTheory, IRE Transactions on, 2(3):113–124.

[Chomsky, 1957] Chomsky, N. (1957). Syntactic Structures. Mouton, The Hague.

[Dahlström et al., 2011] Dahlström, E., Dengler, P., Grasso, A., Lilley, C., McCormack, C., Schepers,D., Watt, J., Ferraiolo, J., Jun, F., and Jackson, D. (2011). Scalable vector graphics (SVG) 1.1 (secondedition). second edition of a recommendation, W3C. https://www.w3.org/TR/2011/REC-SVG11-20110816/.

[Earley, 1970] Earley, J. (1970). An efficient context-free parsing algorithm. Communications of theACM, 13(2):94–102.

[ECMA International, 2015] ECMA International (2015). Standard ECMA-262. ECMAScript R© 2015Language Specification. ECMA International, 6 edition. https://www.w3.org/TR/2011/REC-CSS2-20110607/.

[Gazdar, 1985] Gazdar, G. (1985). Generalized phrase structure grammar. Harvard University Press.

[Hunt, 2010] Hunt, L. (2010). HTML5 Reference. W3c editor’s draft, W3C.https://dev.w3.org/html5/html-author/.

[Jurafsky and Martin, 2000] Jurafsky, D. and Martin, J. H. (2000). Speech and Language Processing:An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recogni-tion. Prentice Hall PTR, Upper Saddle River, NJ, USA, 1st edition.

[Lager, 2002] Lager, T. (2002). PSG Web Laboratory. http://www.ling.gu.se/~lager/Labs/PSG-Lab/. [Online; accessed 02-May-2016].

[Marcus et al., 1994] Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson,M., Katz, K., and Schasberger, B. (1994). The penn treebank: Annotating predicate argument struc-ture. In Proceedings of the Workshop on Human Language Technology, HLT ’94, pages 114–119,Stroudsburg, PA, USA. Association for Computational Linguistics.

[Murray, 2013] Murray, S. (2013). Interactive Data Visualization for the Web. O’Reilly Media, Inc.

32

[Nielsen, 1994] Nielsen, J. (1994). Usability engineering. Elsevier.

[Nielsen, 2010] Nielsen, J. (2010). Interviewing Users. https://www.nngroup.com/articles/interviewing-users/. [Online; accessed 01-May-2016].

[Nielsen and Landauer, 1993] Nielsen, J. and Landauer, T. K. (1993). A mathematical model of thefinding of usability problems. In Proceedings of the INTERACT’93 and CHI’93 conference on Humanfactors in computing systems, pages 206–213. ACM.

[Noam, 1965] Noam, C. (1965). Aspects of the theory of syntax. Cambridge, Mass.

[Schmettow, 2008] Schmettow, M. (2008). Heterogeneity in the usability evaluation process. In Pro-ceedings of the 22Nd British HCI Group Annual Conference on People and Computers: Culture,Creativity, Interaction - Volume 1, BCS-HCI ’08, pages 89–98, Swinton, UK, UK. British ComputerSociety.

[Scott, 2008] Scott, E. (2008). Sppf-style parsing from earley recognisers. Electronic Notes in Theoret-ical Computer Science, 203(2):53–67.

[Shieber, 1986] Shieber, S. (1986). An Introduction to Unification-based Approaches to Grammar.CSLI lecture notes. Center for the Study of Language and Information, Stanford University.

[Thompson and Ritchie, 1983] Thompson, H. and Ritchie, G. D. (1983). Techniques for parsing naturallanguage: two examples. Department of Artificial Intelligence, University of Edinburgh.

[Wirén, 1992] Wirén, M. (1992). Studies in Incremental Natural-Language Analysis. PhD thesis,Linköping University.

33

Stockholm University

SE-106 91 Stockholm

Telefon 08 - 16 20 00

www.su.se