output-directed svg programming · output-directed svg programming brian hempel justin lubin ravi...

Output-Directed SVG ProgrammingBrian Hempel Justin Lubin Ravi Chugh

Department of Computer Science, University of Chicago, USA

Figure 1: A recursive program built only by output-directed program transformations, without any text-based programming.

ABSTRACTWepropose output-directed programming, a paradigmwherebyusers directly manipulate the output they wish to createwhile the system automatically builds a text-based programthat produces the output.

Specifically, we present output-directed programming tech-niques for programs that generate Scalable Vector Graphic(SVG) designs. We show how our new interactions enablea variety of complex, readable programs to be constructedautomatically, without any text-based program editing. Com-pared to prior efforts in this direction, our techniques areapplicable at more stages of program construction, facilitat-ing a greater diversity of programs which may be createdentirely by direct manipulation.

We evaluate our techniques by implementing over a dozenparametric designs, including several from the test suite pro-posed inWatch What I Do: Programming by Demonstration.

September 2018

While some of our output-directed programming interac-tions are—by necessity—domain-specific, we describe howothers are implemented in a generalized way, to support theirapplication to other output-directed programming domains.

CCS CONCEPTS•Human-centered computing→ Human computer inter-

action (HCI); • Software and its engineering→ Integrated

and visual development environments;

KEYWORDSProgram Synthesis, Direct Manipulation, Sketch-n-Sketch

arX

iv:1

907.

1069

9v1

[cs

.HC

] 2

4 Ju

l 201

9

1 INTRODUCTIONText-based programming presents a steep learning curveto novices as they approach programming, and it hobbles“experts” even as they build complex software systems.

In response, many techniques have been developed tofacilitate programming more “directly” than by writing text—such as programming-by-example (PBE), programming-by-

demonstration (PBD), constraint-oriented programming (COP).These techniques have been successful in lowering the barri-ers of programming for creating complex artifacts in a varietyof domains, but—as will discuss in §2—they generally limitwhich building blocks are available to programmers. Ideally,programming environments could retain the power and fullflexibility of text-based programming while providing newcapabilities for directly turning human intent into code.

Prior Work: Sketch-n-Sketch. Hempel and Chugh [14] pro-pose an approach to this challenge, whereby a programmingenvironment augments ordinary text-based programmingwith the ability to directly create and manipulate outputvalues—these user interactions are co-designed with corre-sponding program transformations.

Specifically, they design and implement such techniques—which we refer to as output-directed programming—in a sys-tem, called Sketch-n-Sketch, for creating Scalable VectorGraphics (SVG) drawings. Sketch-n-Sketch provides di-rect manipulation tools for drawing new shapes (by addingnew definitions to the program), relating the attributes ofshapes (by introducing new shared variables and arithmeticexpressions), and grouping shapes (by transforming individ-ual definitions into reusable functions abstracted over designparameters). These Draw-Relate-and-Group output-directedprogramming tools are used to construct several high-level,readable programs to generate non-trivial designs.

Although a useful step towards output-directed program-ming, the techniques in Sketch-n-Sketch [14] exhibit sev-eral drawbacks. The set of Draw-Relate-and-Group toolsimplemented is small, limiting the kinds of SVG designs thatbenefit from output-directed program construction (Limita-tion A). Additionally, only final output values are visualizedand subject to direct manipulation. As programs grow, how-ever, output values constitute only a small fraction of pro-gram execution that may be of interest to the user (LimitationB). A third issue is that many of the program transformationsoperate only on a restricted syntactic subset of the (albeitgeneral-purpose) language—namely, a series of top-level defi-nitions followed by a list literal “main” expression that refersto the top-level definitions. If the program veers out of thislanguage subset, those tools become unavailable (LimitationC). As a result, even within the domain of SVG drawings,the output-directed capabilities of Sketch-n-Sketch are re-stricted to a small set of programming tasks.

Our Contributions. To further the vision of output-directedprogramming established by Hempel and Chugh [14], wedesign and implement new techniques in Sketch-n-Sketchto address the aforementioned limitations.(1) We co-design new Draw-Relate-and-Group user inter-

face tools and program transformations—also improvingexisting ones—and add a new class of repetition tools.

(2) Our techniques embrace the paradigm that intermediate

execution products, in addition to final output values,ought to be available for direct manipulation.

(3) Our techniques embrace the paradigm that programswith arbitrary structure ought to benefit from output-directed interactions. To facilitate this, we develop tech-niques for tracking value provenance that, in our experi-ence, seem amenable to desirable program constructioncapabilities. We also provide mechanisms for the user tofocus on a specific part of the program to be transformed.

The resulting new Sketch-n-Sketch system can constructreusable programs—entirely through direct manipulation—for a variety of SVG tasks, drawn from Watch What I Do:

Programming by Demonstration [1] and other sources.Whatever utility Sketch-n-Sketchmay have as a param-

eterized drawing tool, however, our primary contributionsare the principles on which our new techniques are based,as well as the building blocks developed for implementingand composing output-directed program transformations.

Paper Outline. Our user interface and program transforma-tion techniques are best described by example. Accordingly,the bulk of this paper is a detailed description of how threeexample programs can be built in Sketch-n-Sketch withoutput-directed program transformations (§3). Our Supple-mentary Materials include an (anonymized) video.

After this detailed overview, we briefly describe additionalSketch-n-Sketch tools and example programs we have con-structed (§4), and techniques in our implementation to fa-cilitate the program transformations (§5). We wrap up witha discussion of limitations and avenues for future work onoutput-directed programming, both for the domain of SVGas well as other more general-purpose settings (§6).

2 RELATEDWORKParametric Computer-Aided Design (CAD). Feature-based para-metric CAD systems record user actions as a series of stepsthat together act as a program encoding the creation of thedesign. Among CAD systems, EBP [27] is notable for offeringa PBD workflow to create loops and conditionals.

Drawing With Constraints. Several systems integrate con-straint specification into the visual design process (e.g. [17,

37]). Apparatus [29], Recursive Drawing [30], and Geome-ter’s SketchPad [16] are notable for supporting recursion.

Constraint-Oriented Programming (COP). Other constraint-oriented systems, following in the footsteps of SketchPad [34],explicitly view building a constrained system as a program-ming task [4, 11, 15]. While offering varying degrees of visi-bility into the code, these systems are distinguished by run-ning constraint solvers alongside the program.

Programming by Demonstration (PBD). Several early PBDapproaches [8] used shape drawing as a domain for exploringnon-textual programming techniques. These systems usuallyrely on a visual representation of the program rather than atextual one (e.g. [18, 20]), or show actions step-by-step [23].We also use shape drawing as a concrete application domain,but we do not hide the program text.

Although not as visual as peer PBD systems, Tinker [21] isnotable for supporting recursion by demonstration—indeed,any Lisp expression may be created. Unlike our work, ma-nipulations are performed on a symbolic representation ofthe example not far removed from the underlying Lisp. But,like our work, the underlying code is set in a traditionalprogramming language and that code is featured in the UI.PBD techniques have recently been developed for data

visualization [35], mobile applications for collaboration [9],web scraping [2, 5], and API discovery [38].

Output-Directed Programming. A number of recent systemsoffer a regular text-based programming experience augmentedwith abilities to directly manipulate output to enact codechanges. The transformations available may be “small”, e.g.changing constants [7, 19, 24], strings [19, 24, 32, 36], or listliterals [24], but several systems enable “larger” programchanges via output manipulation. We discuss three below.Like our work, APX [25, 26] is a two-pane (code box and

output canvas) environment for creating programs that drawpictures, although APX additionally supports creation of re-altime visual simulations, updated live as the programmeredits their code. On the output canvas, APX supports directmanipulation of e.g. shape position and size, thus changingnumbers in the program. Larger changes—grouping and in-sertion of new shapes—are supported as well, although mostof APX’s interactions are focused on refactoring code bydirectly manipulating program terms in the code box.Transmorphic [31] re-implements the Morphic UI frame-

work [22] using static, functional (i.e. stateless) views. Trans-morphic retains Morphic’s directly manipulatable morphs byaffecting changes to the view’s (text-based) code rather thanchanging live object state. Adding and removing morphs, orchanging a morph’s primitive properties, are supported.We base our work on Sketch-n-Sketch [7, 14, 24], an

editor for producing shape-drawing programs written in a

traditional, text-based, functional programming language.Like APX and Transmorphic, Sketch-n-Sketch augmentstext editing with direct manipulation of the program’s graph-ical output. We build on the version of Sketch-n-Sketchdeveloped by Hempel and Chugh [14], in which users draw,relate, group, and abstract shapes on the output canvas, butwhich required the program to follow a strict syntactic form:(1) shapes must adhere to a “left-top-right-bot” parameter-ization, and (2) only top-level definitions could be affectedby most output-directed transforms. In this work, we: (1) im-prove and add new tooling, (2) expose intermediate executionproducts for manipulation, and (3) allow transformations ofprograms of more arbitrary structure, additionally offeringfocused editing (and thereby facilitating recursive drawing).Our improvements enable a wider class of programs to beconstructed entirely by direct manipulation on the canvas,with no text-based editing.

3 OVERVIEWWe introduce our improved Sketch-n-Sketch system viathree examples, explaining the workflow to construct eachexample entirely with direct manipulation edits on the can-vas. The “Koch Snowflake” example highlights point widgets,list widgets, and recursive function construction via focusedediting; “Tree Branch” demonstrates offset widgets and repe-

tition over point lists; and “Target” briefly walks through aPBD workflow for other repetition scenarios. Code shown isas produced by our tool—newlines inserted for formattingpurposes in this paper are “escaped” by a “\” backslash.

Example 1: Koch SnowflakeFigure 1 shows a von Koch fractalsnowflake design, created by recur-sively repeating the adjacent motif.Points labeled 0 are inputs; points la-beled 1 will be placed 1

3 and23 of the way between the inputs.

A final point (labeled 2) will be placed equidistant from thelatter, forming an equilateral triangle. Repeating the motifbetween pairs of points will recursively define the fractal.This motif requires two helper functions not provided in

the standard library, so our work proceeds in four phases.(a) We construct a helper function that, given two points,

computes a point 13 of the way between them (Figure 1a).

This function, reused backwards, will produce the 23 point.

(b) We will create a function that, given two points, com-putes a third point that completes the equilateral trianglewith its two input points (Figure 1b).

(c)We use these two helpers to build a function that createsthe motif and recursively repeats the motif within itself. Thisforms the points of a Koch curve, i.e. one side of the finalsnowflake (Figure 1c).

(d) We complete the snowflake by laying out three in-stances of Koch curve points along the sides of an equilateraltriangle and then attaching a polygon to the points.

1a: One-Third Point Function. Tools used: Draw Point,

Relate, Abstract, Rename in Output, Delete

The initial program template provided by Sketch-n-Sketchdefines an empty list of SVG shapes: svg (concat []). Weselect “Point or Offset” from the toolbox (Figure 1d). Whenwe click on the canvas to Draw Point, Sketch-n-Sketchinserts the following definition at the top of the program:[x, y] as point = [87, 206]

Although the new x, y, and point variables arenot yet used—Sketch-n-Sketch has not changed theempty shape list at the end of our program—a point appearson our canvas at (87, 206) as shown on the right. Wheneverthe evaluator encounters a number-number pair, a point wid-get corresponding to that coordinate is emitted as a sideeffect during execution. This is our first example of display-ing widgets for intermediate execution products. We add twomore points in similar fashion.

We drag the points with the “Cursor” tool (utilizing the pre-existing live synchronization [7] of Sketch-n-Sketch) intoroughly the right places, with one point 1

3 of theway betweenthe other two. We would now like to tell Sketch-n-Sketchto relate the 1

3 point in terms of the other two.Like traditional GUIs, Sketch-n-Sketch supports select-

ing items on the canvas, either by clicking the items or bydragging a selection box around the items to select.We drag-select the three

points, which triggers an “Out-put Tools” panel that offerspossible actions to transformthe program based on our se-lection. We hover the mouseover the Relate tool. Onhover, Sketch-n-Sketch at-tempts to discover an arith-metic expression that, were itsubstituted into the program,would define one of points interms of the others and leaveour points in roughly the same place. After about 20 secondsof guess-and-check work, three possible results are shownin the Relate tool submenu.1

1The freeze annotations on 1.5! and 3! indicate that these numbers shouldnot be changed during live synchronization [7].

The second result is mathematically equivalent to whatwe might deduce by hand (e.g. xout = x1 +

13 (x2 − x1), and

similarly for y), so we first hover the result—which previewsthe new program in the code box and the new output on thecanvas—and then click to choose that result.[x3, y3] as point3 = [264, 131]

[x, y] as point = [87, 206]

[x2, y2] as point2 = [ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!]

The point2 definition has been rewritten using the dis-covered arithmetic terms. The point definition, which pre-viously was the last definition, has been moved upward soits x and y variables are in scope for point2.

To turn this concrete expression into a reusable function,we select all the points again and invoke Abstract fromthe Output Tools panel. Abstract attempts to abstract some

expression within our selection over that expression’s freevariables. In our case, there are three possible results, whichabstract over: only the math for the x coordinate; only themath for the y coordinate; and the entire 1

3 point. We choosethe last result, resulting in the following code:[x3, y3] as point3 = [264, 131]

[x, y] as point = [87, 206]

point2Func [x3, y3] [x, y] =[ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!]

[x2, y2] as point2 = point2Func point3 point

A new function—called point2Func—has been placed inour code and the old point2 definition now calls this func-tion to produce its point. If we hover the point emittedby this function on the canvas,a gray outline appears labeledwith the text point2Func. Thegray box is a call widget, indi-cating that the point was pro-duced by a call to point2Funcin our program. A name label on any widget may be clickedto Rename in Output. We thus click point2Func, typeoneThirdPt into the text box that appears, and hit Return.The function definition and its one use are appropriatelyrefactored to be named oneThirdPt.When we created this function, it also became a new

drawing tool in our toolbox (Figure 1e). Using type infer-ence, Sketch-n-Sketch realized that our function takes twopoints and could therefore be drawn with the mouse.

Our first helper function is now complete, so we no longerneed the example points used to build the function. We selectall three points and press the Delete key toDelete. Perhapssurprisingly, Delete only removes the definition for point2,but it is because both of the other points were also involved

in calculating point2—Delete removed something aboutall three by discarding the point2 binding. Selecting theremaining two points and pressing Delete again removestheir two definitions from the program. We are left with ouroneThirdPt helper function and an empty shape list.

1b: Equi-Tri Point Function. Tools used: Distance Fea-

tures, Make Equal

We would like our second helper function to take in twopoints and return a point equidistant from both, as if form-ing an equilateral triangle. As before, we place three pointson the canvas in roughly the appropriate positions. How-ever, the math we need to enforce the points’ equidistanceis too complicated for the Relate tool to discover by guess-ing. Instead, when we select the three points, as shown tothe right, pink lines appear be-tween the points. These pinklines are distance features that,when clicked, select the dis-tance between points. Distancefeatures are only shown be-tween selected points to avoidcluttering the canvas.

We click the three pink linesto select the distances and invoke the Make Eqal tool,which queries an external solver (REDUCE [13]) to discoverhow to replace constants in our program with mathematicalexpressions that enforce the desired equality. In our case,because there are many options for which items might bedefined in terms of the others, we are shown a large numberof solutions. We choose the second result (only to avoidproducing an extraneous offset widget, about which we deferdiscussion until our second example below).Now one of the points is mathematically constrained to

be equidistant from the other two. As before, we select thepoints, Abstract the concrete math into a resuable function,and Rename in Output on the call widget label to nameour function equiTriPt. Again, based on its inferred typesignature our new function appears in the toolbox. We deleteour example points, leaving the final code for our two helperfunctions as shown in Figure 1a and Figure 1b.

1c: Recursive Koch Curve Points Function. Tools used:Draw Function, Snap-Drawing, Call Focusing, Focused

(Recursive) Drawing, Add to Output, Reorder in List

The main part of our work is to build the fractal motif,and then repeat the motif inside itself.

We choose our oneThirdPt helper function in the toolboxand draw it on the canvas. This Draw Function actioninserts the following definition into our program, calling ourhelper with two new points:oneThirdPt2 = oneThirdPt [65, 199] [235, 141]

This gives us the endpoints of themotif, and one of our 1

3 points. To getthe other, we draw oneThirdPt back-wards, starting from one endpoint (in green at right, depictedwhile still drawing) and then ending at other endpoint toSnap-Draw [12] so the existing endpoints are reused in-stead of adding new points—the endpoints are pulled outinto variables and used for both calls. Then we switch to ourequiTriPt helper function and Snap-Draw it between our13 and

23 point. The code after these two steps is:

point = [65, 199]

point2 = [235, 141]

oneThirdPt2 = oneThirdPt point point2

oneThirdPt3 = oneThirdPt point2 point

equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2

We now have our motif interms of example points. Select-ing our points, we then Ab-stract. Abstract notices thatour 1

3 and23 point definitions are only used inside this new

function, and so they are pulled into the new function body(shown after the next step below). Using Rename in Output,we name the function makeKochPts.

Now, we want to repeat the motif inside itself. If we selectmakeKochPts from the toolbox and draw it on the canvas,Sketch-n-Sketchwill just insert another call at the top levelof the program. Instead, we need the function to call itselfrecursively.Our system provides the ability to focus on a particular

definition. Setting focus on a definition has two effects: (a)the canvas only displays the output of the focused definition,and (b) drawing operations add to the focused definitionrather than to the top level of the program.

We focus makeKochPts by clicking on the gray call widgetborder. The focused function, and the example call whichprovide example arguments for its execution and display, arespecified by special comments in the program.-- *** Focused Definition ***makeKochPts point point2 =

let oneThirdPt2 = oneThirdPt point point2 inlet oneThirdPt3 = oneThirdPt point2 point inequiTriPt oneThirdPt3 oneThirdPt2

equiTriPt2 = -- *** Example Call ***makeKochPts point point2

On the canvas (as shown on the next page), the focusedfunction displays its arguments, with buttons for removingor reordering each. The inputs of our function are coloredorange, while the outputs are colored blue. With the functionfocused, we can draw it inside itself to create a recursive call.

We Snap-Draw the func-tion between the first pairof points. As a result,Sketch-n-Sketch insertsan if-then-else recur-sive skeleton and the re-cursive call.makeKochPts point point2 =

let oneThirdPt2 = oneThirdPt point point2 inlet oneThirdPt3 = oneThirdPt point2 point inlet equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2 inif ??terminationCondition thenequiTriPt2

elselet makeKochPts2 = makeKochPts point oneThirdPt3 in

equiTriPt2

To avoid infinite recursion, the if-then-else skeletonbranches on a specially named hole, ??terminationCondition.During evaluation, ??terminationCondition returns Falsethe first time the function is encountered in the call stack,and True if the function has appears earlier in the call stack,affect termination at a fixed depth of two. This allows us torun the program and manipulate its output even before wereplace the hole expression later.

We Snap-Draw machKochPts between the remaining threepairs of points; the calls are inserted in the recursive branch.

...if ??terminationCondition thenequiTriPt2

elselet makeKochPts2 = makeKochPts point oneThirdPt3 inlet makeKochPts3 = makeKochPts oneThirdPt3 equiTriPt2 inlet makeKochPts4 = makeKochPts equiTriPt2 oneThirdPt2 inlet makeKochPts5 = makeKochPts oneThirdPt2 point2 in

equiTriPt2

Our design lookslike a fractal now, butonly the blue pointis being output fromour function. All thewhite points are onlyintermediates. More-over, most of thoseintermediates are in-side calls to the base case of our function—but we are focusedon the recursive case. We must first modify the output ofthe base case, before finalizing the recursive case. We hoveran output point of a recursive call to expose its call widget,whose border we then click to focus the base case.

We want the leftmost four points to all be in the output ofthe base case, rather than just the protruding point. The fifthpoint will be provided by the neighboring call to the basecase.

We click-select the three ad-ditional points we would liketo be in the output,2 and thenchoose Add to Output fromthe Output Tools menu. In or-der for the function to outputmultiple points, the functionmust now return a list of points.The return expressions of both branches of our function aretherefore wrapped in lists, and the three selected points areadded to the list in the base case.

...if ??terminationCondition then

[equiTriPt2, oneThirdPt3, oneThirdPt2, point]else

...[equiTriPt2]

Selections in Sketch-n-Sketch are an unordered set, so,alas, our points have not been added in the proper order. Wemust fix the ordering so the polygon we will attach to all ourKoch points will be drawn correctly.

We select one of our points—which highlights the variableusage in the code so we know where it is in the list—andinvoke Reorder in List as necessary to move the pointforward, backward, to the beginning, or to the end in the list.When all our points are appropriately reordered in this way,we are done with the base case. We hit Escape to defocus thebase case, and then we re-focus the recursive case.The recursive case currently

only returns [equiTriPt2]; in-stead, we need it to combine allthe point lists returned from therecursive calls. To select the listsreturned from the base cases, wehover one of the points of thoselists, which reveals a list widget with a dashed gray borderthat we may select. We select all four returned lists from thecalls to the base case and invoke Add to Output.

if ??terminationCondition then[point, oneThirdPt3, equiTriPt2, oneThirdPt2]

elselet makeKochPts2 = makeKochPts point oneThirdPt3 inlet makeKochPts3 = makeKochPts oneThirdPt3 equiTriPt2 inlet makeKochPts4 = makeKochPts equiTriPt2 oneThirdPt2 inlet makeKochPts5 = makeKochPts oneThirdPt2 point2 in

concat [[equiTriPt2], makeKochPts5, makeKochPts4 \, makeKochPts3, makeKochPts2]

Sketch-n-Sketch realizes we are trying to combine liststogether and inserts a concat (flatten) call so the functionproduces a list of points, rather than a list of lists of points.

2Click-select instead of drag-select because, often, multiple point widgetsare created exactly on top of each other. Drag-select will silently also selectthese covered points!

As before, the lists are out of order, so we examine each of thelist widgets and invoke Reorder in List as necessary. The[equiTriPt] singleton list from the original return valueof the function is extraneous; we find and Delete its listwidget, leaving a return expression of:concat [makeKochPts2, makeKochPts3, makeKochPts4, makeKochPts5]

All points are now in the output of makeKochPts. One lastitem remains: we must choose a termination condition. Thefocused call widget for makeKochPts displays the conditionalfor the recursive case as not <| ??terminationConditionwhich we can click toChoose TerminationCondition.Currently, Sketch-n-Sketch offers only one kind of au-tomatically generated termination, fixed depth, which wechoose. A depth argument is added to our makeKochPtsfunction which is decremented on the recursive calls. Thesechanges are visible in the final code Figure 1e. Additionally,the original example call to our function is given a depth of2 with a {1-5} annotation so that Sketch-n-Sketch drawsa slider for modifying depth in the output [7].equiTriPt2 = makeKochPts 2{1-5} point point2

1d: Koch Snowflake Polygon. Tools used: Group, AttachPolygon to Points

We have a function that produces points for one side ofthe Koch snowflake. We create an equilateral triangle withequiTriPt (shown below) and then Snap-Draw makeKochPointsalong its sides.

To make a single list of all our points,we select the three list widgets for thethree sides’ points and invoke Group.Group means gather into list; here it of-fers either to make a list of lists or toconcat all our lists together. We choosethis latter option. Again, we Reorderin List to arrange the three sides so thepoints are clockwise. To finish the design, we choose the“Polygon” tool from the toolbox and click the snowflakePtslist widget, which affects Attach Polygon to Points.To polish the code, we select and equalize the three depthsliders for each of our three calls to makeKochPts, obtaininga single depth variable, and then perform a bit of renamingto result in code shown in Figure 1. We grab the single depthslider, set the depth to three, and find the menu option tohide all the widgets. We’re done!

Example 2: Tree BranchTools used: Draw Offset, Repeat Over List

To highlight offset widgets and the repetition tools, weconstruct the tree branch design shown in Figure 2b. Webuild a rhombus abstracted over its center point, and thenrepeat it over a list of points we draw in the program. For

this example and the following, we omit mentioning uses ofRename in Output.

We construct the rhombus around a centralpoint using offsets, which are simple additionsor subtractions from an x or y coordinate. The “Point orOffset” tool (Figure 1d), affectsDrawOffsetwhen draggedon the canvas, inserting an addition or subtraction operatione.g. xOffset = x + 102. If not drawn from an existingpoint, a starting point is inserted as well.

Offsets may snap their amounts to each other while draw-ing. If we draw a second offset of the same length in the op-posite direction, a variable is inserted for the offset amount:...num = 102

xOffset = x + num

xOffset2 = x - num

We leverage this amount snapping toquickly create the skeleton of the leaf rhom-bus, as shown at right. We then draw a poly-gon, snapping to each offset endpoint, andAbstract the resulting shape into a func-tion parameterized over [x, y], halfW, andhalfH. We will attach instances of this func-tion over the branch.

The branch is also constructedwith offsets, so that it formsan axis-aligned isosceles triangle.

We place two additional “deadspace” offsets inward from theends of the branch to form the start and end of the attach-ment points for the leaves.We then create these attach-

ment points by drawing thepointsBetweenSepBy functionon the branch. Several functions in the standard toolbox thatreturns a list of points. Among these, pointsBetweenSepByreturns points separated from their neighbors by a fixed dis-tance. With this function, making our branch longer will addmore leaves rather than spacing them out.

Finally, to repeat our leaf rhombus over the points, we firstselect the one copy of the rhombus on the canvas. The Out-put Tools panel then offers multiple tools for repeating theshape. We may Repeat With Function, repeating theshape over a new call to any one of the point-list-producingfunctions available, or we can Repeat Over List, repeat-ing the shape over an existing point list in our program. WeRepeat Over List over the attachment points we just drew.The tool creates a new function abstracted over just a singlepoint (rhombusFunc2 below) and maps that function overour leafAttachmentPts, completing our leafy branch.

...rhombusFunc2 ([x, y] as point) =

let halfW = 40 inlet halfH = 83 inrhombusFunc point halfW halfH

...repeatedRhombusFunc2 =

map rhombusFunc2 leafAttachmentPts...

Example 3: TargetTools used: Repeat by Indexed Merge, Fill PBE Hole

Repeating over a point list allows copies of shapes to varyin their spacial positions, but not over any other attributessuch as size or color. To support other repetitions scenar-ios where the varying attributes could be calculated froman index (i.e. 0, 1, 2, . . . ), we offer a PBD workflow whichwe now briefly demonstrate by the construction of a target(Figure 2c).

To start, we draw three concentric circles snapped to thesame center point and change the color of the middle cir-cle. We now select the three circles and invoke Repeat by

Indexed Merge, from which we select the second of tworesults (which differs from the first only in that it adds areverse on line 7 below, so that i=0 always for the smallestcircle)....circles =

map (\i ->circle \

??(1 => 0, 2 => 466, 3 => 0) \point \??(1 => 114, 2 => 68, 3 => 25))

(reverse (zeroTo 3{0-15}))...

The program maps an anonymous function that takes anindex (\i -> ...) over the list [2,1,0]; each index is thustransformed into one of our circles. The anonymous functioncontains the result of merging of our original three circledefinitions. The differences between the expressions—radiusand color—have been turned in what we call programming-

by-example (PBE) holes, represented by ??(...). The firstPBE Hole above can be read as “the first time this expressionis executed it should return 0, the second time it is executedit should return 466, and the third time 0”.

When a program contains PBE Holes, Sketch-n-Sketchemploys sketch-based synthesis [33] to suggest possible fill-ings for each hole. Only the variable i ever differs in theexecution environments at these holes, so all possible fillingsare based upon i, as shown below.

Draw

Draw Shape★

Draw Custom Func★

Dupe★

Draw PointDraw OffsetDeleteFocused DrawingRelate

Make Eqal★RelateDistance Features

Group/Abstract/Repeat

Group★Abstract★

Merge★

List WidgetsRepeat over Function CallRepeat over Existing ListRepeat by Indexed MergeFill PBE HoleRefactor

Rename in OutputAdd/Remove/Reorder ArgumentReorder ListAdd to OutputSelect Termination Condition

Table 1: Listing of the new and improved functional-

ity in Sketch-n-Sketch. Toolsmarkedwith★ are im-

proved versions of those in Hempel and Chugh [14];

others are new.

For the first hole, we choose the mod i 2 == 0! condi-tional to obtain alternating colors. For the second we choosethe only option, a base+i∗sep expression, to calculate theradii of our circles. Similar to the Koch snowflake example,the inserted code zeroTo 3{0-15} contains a slider annota-tion allowing us to manipulate the number of circles in theoutput. We display five circles for the final design (Figure 2c).

4 TOOLS AND EVALUATIONTable 1 lists the new and improved tools in our updatedversion of Sketch-n-Sketch. To evaluate our approach, wehave implemented a total of sixteen programs without text-based editing. The output of each are shown in Figure 2,along with their program sizes (measured in source lines ofcode and math operations). These designs are taken fromthree main sources:(1) Six of our examples (underlined in Figure 2) are from the

PBD test suite proposed in Watch What I Do: Program-

ming by Demonstration [1]. The WWID: PBD suite isdiverse: 15 of its 32 tasks may be interpreted as paramet-ric drawings. Of those 15, our work is able to complete 4and partially complete another 2.

(2) We take 5 tasks from other drawing literature [3, 6, 7].(3) We additionally implement 5 novel tasks.Below, we discuss the features required to complete the

remaining WWID: PBD parametric drawing benchmarks.

Sketch-n-Sketch File Code Tools View Options Output Tools

Undo Redo Clean Up

Current �le: Untitled *

Run �

⦀

Context: Program

equiTriPt [x3, y3] [x2, y2] = [ (x2 + x3 + sqrt 3! * (y2 - y3))/ 2!, (y2 + y3 - sqrt 3! * (x2 - x3)) / 2!]

oneThirdPt [x, y] [x3, y3] = [ x / 1.5!+ x3 / 3!, y / 1.5! + y3 / 3!]

point = [39, 314]

point2 = [490, 301]

makeKochPts depth point point2 = let oneThirdPt2 = oneThirdPt point2 point in let oneThirdPt3 = oneThirdPt point point2 in let equiTriPt2 = equiTriPt oneThirdPt3 oneThirdPt2 in if depth < 2 then [point, oneThirdPt3, equiTriPt2, oneThirdPt2] else let makeKochPts2 = makeKochPts (depth - 1) point oneThirdPt3 in let makeKochPts3 = makeKochPts (depth - 1) oneThirdPt3 equiTriPt2 in let makeKochPts4 = makeKochPts (depth - 1) equiTriPt2 oneThirdPt2 in let makeKochPts5 = makeKochPts (depth - 1) oneThirdPt2 point2 in concat [makeKochPts2, makeKochPts3, makeKochPts4, makeKochPts5]

depth = 3{1-5}

topPts = makeKochPts depth point point2

botCorner = equiTriPt point2 point

rightPts = makeKochPts depth point2 botCorner

123456789

10111213141516171819202122232425262728293031

(a) Koch Snowflake 31 LOC, 22 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

rhombusFunc [x, y] halfW halfH = let xOffset = x + halfW in let xOffset2 = x - halfW in let yOffset = y - halfH in let yOffset2 = y + halfH in let pts = [[x, yOffset], [xOffset, y], [x, yOffset2], [xOffset let [color, strokeColor, strokeWidth] = [118, 360, 2] in polygon color strokeColor strokeWidth pts

rhombusFunc2 ([x, y] as point) = let halfW = 40 in let halfH = 83 in rhombusFunc point halfW halfH

branchHalfW = 48

y1Offset = branchY - branchHalfW

y1Offset2 = branchY + branchHalfW

x1Offset = branchLeft + 405

branch = let pts = [[branchLeft, y1Offset], [x1Offset, branchY], [branc let [color, strokeColor, strokeWidth] = [29, 360, 2] in polygon color strokeColor strokeWidth pts

deadspace = 72

leafAttachmentStartX = branchLeft + deadspace

leafAttachmentEndX = x1Offset - deadspace

leafAttachmentPts = pointsBetweenSepBy [leafAttachmentStartX, br

leaves = map rhombusFunc2 leafAttachmentPts

svg (concat [ [branch], leaves])

23456789

1011121314151617181920212223242526272829303132333435363738394041424344

(b) Tree Branch 31 LOC, 9 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

point = [236, 241]

circles = map (\i -> circle (if mod i 2! == 0! then 0 else 466) point (22 + i * (reverse (zeroTo 5{0-15}))

svg (concat [ circles])

123456789

10

(c) Target 8 LOC, 3 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

point = [82, 136]

h = 239

w = 444

floorRect = rect 36 point w h

tableRect = rect 188 point (w / 3!) h

svg (concat [ [floorRect], [tableRect]])

123456789

101112131415

(d) Precision Floor Plan☆ 9 LOC, 3 MO

(☆ = 25 LOC, 27 MO)


Undo Redo Clean Up


Run �

⦀

Context: Program

[ , y ] p [ g ( q (p

hangingTray topPoint hangDistance = let [x, y] as point = topPoint in let yOffset = y + hangDistance in let [x1, y1] as point1 = [x, yOffset] in let trayHalfW = 85 in let left = x1 - trayHalfW in let right = x1 + trayHalfW in let tray = ellipse 46 point1 trayHalfW 21 in let color = 434 in let strokeWidth = 5 in let wire2 = line color strokeWidth point [right, y1] in let wire1 = line color strokeWidth point [left, y1] in [tray, wire2, wire1]

baseCenter = [centerX, 477]

color = 214

pillar = line color 20 point2 baseCenter

base = ellipse color baseCenter 147 20

strokeWidth = 15

leftArm = line color strokeWidth point2 point

rightArm = line color strokeWidth point2 point3

hangDistance = 232

hangingTray1 = hangingTray point3 hangDistance

hangingTray2 = hangingTray point hangDistance

svg (concat [ [pillar], [base], [leftArm], [rightArm], hangingTray1, hangingTray2])

9101112131415161718192021222324252627282930313233343536373839404142434445464748495051

(f) Balance Scales☆ 35 LOC, 21 MO (☆ = 53 LOC, 40 MO)


Undo Redo Clean Up


Run �

⦀

Context: Program

fstOffset = x13 + baseW

[_, y13] = onLine2

sndOffset = y13 - cutW

[x14, y14] as point14 = [fstOffset, y13]

y14Offset = y14 - cutW

boxBack =

let pts = [point10, [x10, y10Offset], [x12, y12Offset], point

let [color, strokeColor, strokeWidth] = [color, 360, 2] in

polygon color strokeColor strokeWidth pts

boxRight =

let pts = [point12, [x12, y12Offset], [x14, y14Offset], point



boxBot =

let pts = [point10, point12, point14, onLine2] in



boxLeft =

let pts = [point10, [x10, y10Offset], [x13, sndOffset], onLin



boxFront =

let pts = [[x13, sndOffset], [x14, y14Offset], point14, onLin



svg (concat [

[topDownTemplate],

[boxBack],

[boxRight],

[boxBot],

[boxLeft],

[boxFront]

])

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

(g) Box Volume 76 LOC, 42 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

squareByCenter2Func3 center2 =

squareByCenter fill2 center2 (squareW / 2!)

squareByCenter2Func4 center2 =

squareByCenter fill center2 (squareW / 2!)

boxyXFunc ([x, y] as point) squareW n =

let xOffset = x + squareW in

let xOffset2 = x - squareW in

let [x1, y1] as point1 = [xOffset, y] in

let y1Offset = y1 - squareW in

let ySep =0! - squareW in

let upRightPts = nPointsSepBy n [x1, y1Offset] squareW ( ySep)

let [x2, y2] as point2 = [xOffset2, y] in

let y2Offset = y2 - squareW in

let yOffset3 = y2 + squareW in

let yOffset4 = y1 + squareW in

let upLeftPts = nPointsSepBy n [x2, y2Offset] ySep ySep in

let downRightPts = nPointsSepBy n [x1, yOffset4] squareW squar

let downLeftPts = nPointsSepBy n [x2, yOffset3] ySep squareW i

let squareByCenter1 = squareByCenter 408 point (squareW / 2!)

let repeatedSquareByCenter2Func =

map squareByCenter2Func upRightPts in

let repeatedSquareByCenter2Func21 =

map squareByCenter2Func2 downRightPts in


map squareByCenter2Func3 upLeftPts in


map squareByCenter2Func4 downLeftPts in

let squareByCenterSingleton = [squareByCenter1] in

concat [squareByCenterSingleton, repeatedSquareByCenter2Func4

boxyX = boxyXFunc point squareW n

boxyXFunc1 = boxyXFunc [341, 621] squareW 3{0-10}

boxyXFunc2 = boxyXFunc [513, 216] squareW 1{0-10}

svg (concat [

boxyX,

boxyXFunc1,

boxyXFunc2

])

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63 (h) Xsλ 48 LOC, 14 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

[x, y] as point = [66, 148]

h4 = 141

w = 274

fill = 362

h = 73

batteryFunc ([x, y] as point) h4 w fill h = let body = rect fill point w h4 in let head = rect fill [ x+ w, (h4 - h + 2! * y) / 2!] 40 h in [body, head]

battery = batteryFunc point h4 w fill h

svg (concat [ battery])

123456789

101112131415161718192021

(i) Batteryλ★ 13 LOC, 5 MO (★ = 21 LOC, 2 MO)


Undo Redo Clean Up


Run �

⦀

Context: Program

[left, top] as topLeft = [84, 147]

height = 345

stoneWidth = 85

width = 331

archFunc ([left, top] as topLeft) width height stoneWidth =

let lintel = rect 124 topLeft width stoneWidth in

let pillarTop = top + stoneWidth in

let pillarHeight = height - stoneWidth in

let leftPillar = rect 16 [left, pillarTop] stoneWidth pillarH

let rightPillar = rect 220 [ width - stoneWidth+ left, pillar

[lintel, leftPillar, rightPillar]

arch = archFunc topLeft width height stoneWidth

svg (concat [

arch

])

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

(e) Mondrian Archλ★ 15 LOC, 4 MO (★ = 23 LOC, 9 MO)


Undo Redo Clean Up


Run �

⦀

Context: Program

w = 126

color = 366

strokeWidth = 8

line1Func ([x, y] as point) =

let xOffset = x + w in

line color strokeWidth point [xOffset, y]

left = 104

top = 119

rungs =

map line1Func (nVerticalPointsSepBy 4{0-10} [left, top] 50)

bot = 346

leftLine = line color strokeWidth [left, top] [left, bot]

rightLine = line color strokeWidth [ left+ w, top] [ left+ w, bot

svg (concat [

rungs,

[leftLine],

[rightLine]

])

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

(j) Ladder 18 LOC, 3 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

[left, top] as topLeft = [84, 117]

w = 320

∂ = 23

lambdaFunc ([left, top] as topLeft) w ∂ leftColor botColor bigCo

let bot = top + w in

let [x1, bot] as botLeft = [left, bot] in

let right2 = x1 + w in

let right3 = left + w in

let yOffset2 = top + ∂ in

let xOffset2 = left + ∂ in

let leftOffset = x1 + ∂ in

let botOffset = bot - ∂ in

let rightOffset = right2 - ∂ in

let [right, bot] as botRight = [right2, bot] in

let yOffset = bot - ∂ in

let midpoint2 = midpoint topLeft botRight in

let [x, _] = midpoint2 in

let fstOffset = x - ∂ in

let [_, y] = midpoint2 in

let sndOffset = y + ∂ in

let leftTri =

let pts = [[left, yOffset2], [fstOffset, y], [x1, botOffset]

let [color, strokeColor, strokeWidth] = [leftColor, 360, 2]

polygon color strokeColor strokeWidth pts in

let botTri =

let pts = [[leftOffset, bot], [x, sndOffset], [rightOffset,

let [color, strokeColor, strokeWidth] = [botColor, 360, 2] i


let bigTri =

let pts = [[xOffset2, top], [right3, top], [right, yOffset]]

let [color, strokeColor, strokeWidth] = [bigColor, 360, 2] i


[leftTri, botTri, bigTri]

lambda = lambdaFunc topLeft w ∂ 26 234 144

svg (concat [

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

(k) Logo (via Three Tris)λ 37 LOC, 11 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

boxes = map (\i -> rect 200 [ 50 + i * 76, 110] 55 195) (zeroTo 7{0-15})

svg (concat [ boxes])

123456789

(l) N Boxes 7 LOC, 2 MO

(m) Ferris Wheel 25 LOC, 0 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

point = [307, 334]

r = 166

attachmentPts = nPointsOnCircle 7{0-10} 0.06280000000000001{-3.14

color = 434

spokeFunc point2 =

line color 5 point point2

spokes =

map spokeFunc attachmentPts

carFunc center2 =

squareByCenter 48 center2 25

cars =

map carFunc attachmentPts

capFunc point2 =

circle 364 point2 9

caps =

map capFunc attachmentPts

ring1 = ring color 7 point r

hub = circle 362 point 44

svg (concat [

[hub],

cars,

spokes,

[ring1],

caps

])

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38


Undo Redo Clean Up


Run �

⦀

Context: Program

[x, y] as point = [118, 193]

woodHalfL = 98

taperStartX = x + woodHalfL

[x1, y1] as point1 = [taperStartX, y]

pencilHalfW = 45

top = y1 - pencilHalfW

bot = y1 + pencilHalfW

tipX = x1 + 183

body = rectByCenter 44 point woodHalfL pencilHalfW

ratio = 0.651569678605651

leadStartBotPt = onLine [x1, bot] [tipX, y1] ratio

leadStartTopPt = onLine [x1, top] [tipX, y1] ratio

shavedWood =

let pts = [[x1, bot], [x1, top], leadStartTopPt, leadStartBotP

let [color, strokeColor, strokeWidth] = [464, 360, 0] in


lead =

let pts = [leadStartBotPt, leadStartTopPt, [tipX, y1]] in

let [color, strokeColor, strokeWidth] = [409, 360, 0] in


svg (concat [

[body],

[shavedWood],

[lead]

])

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

(o) Pencil Tip 25 LOC, 4 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

pt2 = [351, 271]

pt1 = [122, 382]

arrowFunc pt1 pt2 = let onLine2 = onLine pt1 pt2 0.7754620659147587 in let onPerpendicularLine2 = onPerpendicularLine onLine2 pt2 1! i let onPerpendicularLine3 = onPerpendicularLine onLine2 pt2 -1! let line1 = line 0 5 pt1 pt2 in let line2 = line 0 5 onPerpendicularLine2 pt2 in let line3 = line 0 5 onPerpendicularLine3 pt2 in [line1, line2, line3]

arrow = arrowFunc pt1 pt2

arrowFunc1 = arrowFunc [295, 378] [432, 428]

arrowFunc2 = arrowFunc [324, 535] [245, 413]

svg (concat [ arrow, arrowFunc1, arrowFunc2])

123456789

10111213141516171819202122232425

(p) Arrowsλ 18 LOC, 0 MO


Undo Redo Clean Up


Run �

⦀

Context: Program

[x1, y1]= [444, 358]

x= 56

halfGauge = 58

yOffset = y1 - halfGauge

yOffset2 = y1 + halfGauge

railOverExtension = 40

firstTieX = x + railOverExtension

y1Offset = y1 - halfGauge

y1Offset2 = y1 + halfGauge

endTiesX = x1 - railOverExtension

pointsBetweenSepBy2 = pointsBetweenSepBy [firstTieX, y1] [endTie

tieOverExtension = 32

rectByCenter1Func point2 =

rectByCenter 24 point2 17.5 (halfGauge + tieOverExtension)

repeatedRectByCenter1Func =

map rectByCenter1Func pointsBetweenSepBy2

color = 446

strokeWidth = 17

line1 = line color strokeWidth [x, yOffset] [x1, y1Offset]

line2 = line color strokeWidth [x, yOffset2] [x1, y1Offset2]

svg (concat [

repeatedRectByCenter1Func,

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

(q) Rails 25 LOC, 7 MO

Figure 2: Examples programs created using only output-directed manipulations in Sketch-n-Sketch, with source lines of

code (LOC) and number of math operations in the code (MO). λ = Final design is abstracted. underline = Task from WWID:

PDB test suite (dashed = only partially completed) [1]. ★ = Can also be created via output-directed manipulations in Hempel

and Chugh [14] (✩ = only partially). Other sources: Battery is from Bernstein and Li [3], Ladder from Cheema et al. [6], Logo(via Three Tris), N Boxes, and Ferris Wheel from Chugh et al. [7].

Tackling the Remaining WWID: PBD TasksTwo of the WWID: PBD tasks we attempt can only be par-tially completed in this work (Figure 2g and Figure 2h). Tocomplete these tasks, the “Box Volume” example would re-quire an interaction to compute and display the numericvolume of the folded box. The “Xs” task would require moreprecise control over what definitions are pulled into theabstraction. (Not all uses of a squareWidth parameter arepulled in to the abstraction so the design breaks if we try toadd an X with a different square size.)

The remaining nine tasks in the WWID: PBD test suite arediverse. No single feature would helpwith anymore than twoor three of the tasks. Our most prominent missing featureis the ability to draw text boxes, with other elements placedrelative to the text size. Beyond this, several examples requirea wide variety of list operations. Sketch-n-Sketch wouldalso need the ability to reason about intersections of lines

with shape edges, to specify overlapping and containment-based constraints, and to solve for different kinds of suchconstraints simultaneously. Finally, one example would re-quire Sketch-n-Sketch to create if-then-else branchesoutside of a recursive or hole-filling setting.

5 IMPLEMENTATIONSketch-n-Sketch is an in-browser app written in the func-tional language Elm [10] with exceptions andmutation addedvia a custom library.

Below, we discuss three of the key technical mechanismsthat facilitate the operation of the tools: value provenance fordeducing which program expressions to change; value holesfor snap-drawing; and roles for determining how to drawprogrammer-defined functions.

Provenance. To affect program transformations, the UI mapsselections to output values (an SVG-specific process, colored

brown below). The provenance of these values is then inter-preted as particular program expressions (a general-purposeprocess, in blue below) to which a transformation is applied.

Selections,Direct Manipulation Values Expressions

New CodeVisualization

Widgets

Output

General PurposeDomain Specific

Canvas Selection Expression(s)Value(s)UI Interpretation

In a functional setting, such as the Elm-like language ex-posed to the programmer in Sketch-n-Sketch, values areimmutable. “Changing” a value actually means creating anew, slightly different value. For output-directed program-ming in such a setting, our key insight is that provenancecan be loose. Although we generally find that expressionslater in the computation are more relevant, we tag each valuesuch that it could be traced all the way back to the primi-tive constant numbers that effected it. The programmer isalways editing within some context (e.g. the program, notthe standard library) and is therefore always looking forsome expression(s) within the context of their attention. Aloose approach to provenance ensures that if there is someexpression in their attention that can be associated with theselected output value, it will be found.

Value Holes. To affect snap-drawing and grouping, we relyon value holes, a mechanism to simplify inserting new tem-plate code that must honor output-directed constraints. Avalue hole is a temporary expression that contains a value.Values inside holes are placed in program where an expres-sion should equal an existing output value, such as at thex and y coordinates of a line endpoint being snapped to anexisting point. Before showing the code to the programmer,the provenance of the value in each hole is inspected andthe hole is replaced by an expression that enforces the snap.

Roles. When the programmer defines a custom function, wewant to the function to be drawable with the mouse withoutrequiring the programmer to annotate which of the func-tion arguments are e.g. the width and height. During typeinference, we tag the types at code locations with additionalside information explaining the expressions role. The stan-dard library drawing functions are already annotated withroles—using these functions causes role information to flowback into program. A similar scheme called brands is usedin APX [26]. We also apply certain usage rules—e.g. a num-ber added to an X value must be a HorizontalDistance—toinfer roles in more cases.

6 CONCLUSION AND FUTUREWORKWe presented new techniques for the output-directed SVGprogramming environment Sketch-n-Sketch, and we usedthe new system to construct a variety of non-trivial programsentirely through direct manipulation. Our system lacks manyfeatures—such as rotation attributes and curved path tools—thatwould be required tomake Sketch-n-Sketch a practicaltool for creating parameterized drawings. We expect that

many of these features would work in much the same wayas the techniques presented.Our goal has not, however, been to create the ideal sys-

tem for parameterized drawings. Instead, we consider thisparticular application domain as a laboratory for develop-ing output-directed programming capabilities for a varietyof future programming settings. There are several naturalavenues to explore in this longer-term direction.

Customizing Widgets. Even in our setting—with small pro-grams with a limited number of widgets—clutter is a problem.We have attempted to reduce clutter in two ways: (a) by dis-playing widgets in a context-sensitive manner (i.e. on hover),and (b) by repositioning the bounding boxes of list and callwidgets to minimize overlap. Even with these techniques,there is still sometimes quite a bit of visual noise on the can-vas. Reducing that visual noise may prove to be an tradeoffwith allowing functionality to be discoverable, and there maybe no clear optimum.

Therefore, it may be necessary to allow programmers (orlibrary writers) to programmatically control what, where,and when widgets are displayed. Mechanisms for customiza-tion may also help address the open question about howto render graphical representations of domain objects andcomputations that are not inherently visual. One approachwould be to design an API for “toSvg” functions that specifyhow to render graphical representations of intermediate datastructures (analogous to “toString” functions that rendertext).

Synthesizing Program Transformations. The building blocksin our implementation—provenance, value holes, and roles—have allowed us to implement a variety of program trans-formations. Each of these has been hand-coded one-by-one.This is time-consuming, error-prone, and limits the compos-ability of transformations. We imagine that constraint-basedprogram synthesis techniques can be developed to elimi-nate the need to directly transform abstract syntax trees.Synthesis techniques have been successfully incorporatedinto refactoring tasks for class-based languages [28]; perhapssuch techniques can be extended to handle the kinds of fine-grained, functional program transformations—at the levelsof both abstract and concrete syntax—required to meet thegoals of output-directed programming.

REFERENCES[1] 1993. A Test Suite for Programming by Demonstration. In Watch

What I Do: Programming by Demonstration, Richard Potter and DavidMaulsby (Eds.). MIT Press.

[2] Shaon Barman, Sarah Chasins, Rastislav Bodik, and Sumit Gulwani.2016. Ringer: Web Automation by Demonstration. In Proceedings of

the 2016 ACM SIGPLAN International Conference on Object-Oriented

Programming, Systems, Languages, and Applications (OOPSLA 2016).ACM, New York, NY, USA, 748–764. https://doi.org/10.1145/2983990.2984020

[3] Gilbert Louis Bernstein and Wilmot Li. 2015. Lillicon: Using TransientWidgets to Create Scale Variations of Icons. Transactions on Graphics

(TOG) (2015).[4] Alan Borning. 1981. The Programming Language Aspects of ThingLab.

Transactions on Programming Languages and Systems (TOPLAS) (Octo-ber 1981).

[5] Sarah E. Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon:Scraping Distributed Hierarchical Web Data. In Symposium on User

Interface Software and Technology (UIST).[6] SalmanCheema, Sumit Gulwani, and Joseph LaViola. 2012. QuickDraw:

Improving Drawing Experience for Geometric Diagrams. In Conferenceon Human Factors in Computing Systems (CHI).

[7] Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. 2016.Programmatic andDirectManipulation, Together at Last. InConferenceon Programming Language Design and Implementation (PLDI).

[8] Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman,David Maulsby, Brad A. Myers, and Alan Turransky (Eds.). 1993. Watch

What I Do: Programming by Demonstration. MIT Press.[9] Jonathan Edwards, Jodie Chen, and Alessandro Warth. 2016. Live

End-User Programming. In LIVE Workshop.[10] Evan Czaplicki. [n. d.]. Elm. ([n. d.]). http://elm-lang.org.[11] Tim Felgentreff, Alan Borning, Robert Hirschfeld, Jens Lincke, Yoshiki

Ohshima, Bert Freudenberg, and Robert Krahn. 2014. Babelsberg/JS - ABrowser-Based Implementation of An Object Constraint Language. InECOOP 2014 - Object-Oriented Programming - 28th European Conference,

Uppsala, Sweden, July 28 - August 1, 2014. Proceedings.[12] Michael Gleicher and Andrew Witkin. 1994. Drawing with Con-

straints. The Visual Computer: International Journal of Computer Graph-

ics (1994).[13] AnthonyC. Hearn. 1968. REDUCE: AUser-Oriented Interactive System

for Algebraic Simplification. In Interactive Systems for Experimental

Applied Mathematics. Academic Press.[14] Brian Hempel and Ravi Chugh. 2016. Semi-Automated SVG Program-

ming via Direct Manipulation. In Symposium on User Interface Software

and Technology (UIST).[15] Allan Heydon and Greg Nelson. 1994. The Juno-2 Constraint-Based

Drawing Editor. In Technical Report 131a, Digital Systems Research,

Digital Equipment Corporation.[16] R. Nicholas Jackiw andWilliam F. Finzer. 1993. The Geometer’s Sketch-

pad: Programming by Geometry. InWatch What I Do: Programming

by Demonstration. MIT Press.[17] Jennifer Jacobs, Sumit Gogia, Radomír Mech, and Joel R. Brandt. 2017.

Supporting Expressive Procedural Art Creation Through Direct Ma-nipulation. In Proceedings of the 2017 CHI Conference on Human Factors

in Computing Systems, Denver, CO, USA, May 06-11, 2017.

[18] David Kurlander. 1993. Graphical Editing by Example. Ph.D. Disserta-tion. Columbia University.

[19] Kevin Kwok and Guillermo Webster. 2016. Carbide Alpha. (2016).https://alpha.trycarbide.com/.

[20] Henry Lieberman. 1993. Mondrian: A Teachable Graphical Editor. InWatch What I Do: Programming by Demonstration. MIT Press.

[21] Henry Lieberman. 1993. Tinker: A Programming By DemonstrationSystem for Beginning Programmers. InWatchWhat I Do: Programming

by Demonstration. MIT Press.[22] JohnH.Maloney and Randall B. Smith. 1995. Directness and Liveness In

the Morphic User Interface Construction Environment. In Proceedings

of the 8th Annual ACM Symposium on User Interface Software and

Technology, UIST 1995, Pittsburgh, PA, USA, November 14-17, 1995.[23] David L. Maulsby, Ian H. Witten, and Kenneth A. Kittlitz. 1989. Meta-

mouse: Specifying Graphical Procedures by Example. In Conference on

Computer Graphics and Interactive Techniques (SIGGRAPH).[24] Mikaël Mayer, Viktor Kunčak, and Ravi Chugh. 2018. Bidirectional

Evaluation with Direct Manipulation. Proceedings of the ACM on

Programming Languages (PACMPL), Issue OOPSLA (2018).[25] McDirmid, Sean. [n. d.]. The Future of Programming will be Live. ([n.

d.]). Curry On 2016. https://www.youtube.com/watch?v=bnqkglrSqrg.[26] McDirmid, Sean. [n. d.]. A Live Programming Experience.

([n. d.]). Future Programming Workshop, StrangeLoop 2015.https://onedrive.live.com/download?cid=51C4267D41507773&resid=51C4267D41507773%2111492&authkey=AMwcxdryTyPiuW8https://www.youtube.com/watch?v=YLrdhFEAiqo.

[27] Guy Pierra, Jean-Claude Potier, and Patrick Girard. 1996. The EBPSystem: Example Based Programming System for Parametric Design.In Modelling and Graphics in Science and Technology. Springer BerlinHeidelberg.

[28] Veselin Raychev, Max Schäfer, Manu Sridharan, and Martin Vechev.2013. Refactoring with Synthesis. In Object-Oriented Programming,

Systems, Languages, and Applications (OOPSLA).[29] Schachman, Toby. [n. d.]. Apparatus. ([n. d.]). http://aprt.us/.[30] Schachman, Toby. 2012. Recursive Drawing. Master’s thesis. New

York University Interactive Telecommunications Program. http://recursivedrawing.com/.

[31] Robin Schreiber, Robert Krahn, Daniel H. H. Ingalls, and RobertHirschfeld. 2016. Transmorphic: Mapping Direct Manipulation to Source

Code Transformations.[32] Christopher Schuster and Cormac Flanagan. 2016. Live Programming

by Example: Using Direct Manipulation for Live Program Synthesis.In LIVE Workshop.

[33] Armando Solar-Lezama. 2008. Program Synthesis by Sketching. Ph.D.Dissertation. UC Berkeley.

[34] Ivan Sutherland. 1963. Sketchpad, A Man-Machine Graphical Commu-

nication System. Ph.D. Dissertation. MIT.[35] Victor, Bret. [n. d.]. Drawing Dynamic Visualizations. ([n. d.]). http:

//worrydream.com/#!/DrawingDynamicVisualizationsTalk.[36] Xiaoyin Wang, Lu Zhang, Tao Xie, Yingfei Xiong, and Hong Mei. 2012.

Automating Presentation Changes in Dynamic Web Applications viaCollaborative Hybrid Analysis. In International Symposium on the

Foundations of Software Engineering (FSE).[37] Haijun Xia, Bruno Araújo, Tovi Grossman, and Daniel J. Wigdor. 2016.

Object-Oriented Drawing. In Proceedings of the 2016 CHI Conference

on Human Factors in Computing Systems, San Jose, CA, USA, May 7-12,

2016.[38] Kuat Yessenov, Ivan Kuraj, and Armando Solar-Lezama. 2017. Demo-

Match: API Discovery from Demonstrations. In Proceedings of the

38th ACM SIGPLAN Conference on Programming Language Design

and Implementation (PLDI 2017). ACM, New York, NY, USA, 64–78.https://doi.org/10.1145/3062341.3062386

https://doi.org/10.1145/2983990.2984020

https://doi.org/10.1145/2983990.2984020

http://elm-lang.org

https://alpha.trycarbide.com/

https://www.youtube.com/watch?v=bnqkglrSqrg

https://onedrive.live.com/download?cid=51C4267D41507773&resid=51C4267D41507773%2111492&authkey=AMwcxdryTyPiuW8

https://onedrive.live.com/download?cid=51C4267D41507773&resid=51C4267D41507773%2111492&authkey=AMwcxdryTyPiuW8

https://www.youtube.com/watch?v=YLrdhFEAiqo

http://aprt.us/

http://recursivedrawing.com/

http://recursivedrawing.com/

http://worrydream.com/#!/DrawingDynamicVisualizationsTalk

http://worrydream.com/#!/DrawingDynamicVisualizationsTalk

https://doi.org/10.1145/3062341.3062386

output-directed svg programming · output-directed svg programming brian hempel justin lubin ravi...

Documents