sketching and communicating designanzasca.net/wp-content/uploads/2014/12/01_05_112.pdf · refers to...

F. Madeo and M. A. Schnabel (eds.), Across: Architectural Research through to Practice: 48th Interna-tional Conference of the Architectural Science Association 2014, pp. 59–70. © 2014, The Architectural Science Association & Genova University Press.

SKETCHING AND COMMUNICATING DESIGN INTENT

An analysis of technology

ELIEL DE LA CRUZ and MARTIN TOMITSCH The University of Sydney, Australia {eliel.delacruz, martin.tomitsch}@sydney.edu.au

Abstract. The schematic design process and the communication of design intent present a good opportunity to integrate technological in-novation. The changes in new technology and the imminent shift to-wards the ubiquitous computing era seem to be setting the direction for technology integration in the design research field. This paper pre-sents an analysis of major variations, expansions, and tendencies in the field of design computing research over the last 50 years, focusing on the areas of computational support for sketching and design intent communication. A significant paradigm shift over the last 50 years in-troduced a variety of methods for performing sketch recognition and managing ambiguity; this shift moves away from the tendency to think of computers just as an “incompatible pencil”. This analysis subdivides the literature in relation to the aforementioned topic in 5 categories: 1) Digitizing sketching, 2) Augmenting sketching, 3) Rec-ognizing sketches, 4) Supporting collaborative sketching, and 5) Cap-turing design intent. After carefully analysing the aforementioned lit-erature, the paper presents a matrix defining a new taxonomy to classify the variables in some of the systems mentioned in the paper, as well as the trends that will continue to set the direction for technol-ogy integration in schematic design process research.

Keywords. Design Computing, Sketching, Digitizing, Augmented Design, Design Intent.

1. Introduction

Sketching is not an activity exclusive to designers: people often use sketches for many different applications in their daily lives. In the architectural de-sign process sketching is intertwined with different phases, but it is in the

60 E. DE LA CRUZ AND M. TOMITSCH

schematic design phase where sketches are more often used as a means to visually represent ideas, concepts, and diagrams.

Arnheim (1969) refers to drawing as a medium for visual thinking. For the designers sketching is not just a means to explore different ideas, it is al-so a conduit to think about solutions, problems, and many other spatial relat-ed tasks. Sketches provide the designer with a quick way to graphically rep-resent ideas without having to commit to any specific concept as a final solu-solution.

The almost whimsical nature of sketches provides designers with a great level of affordance in how they express their ideas, however it is this same ambiguous nature that poses as one of the biggest hurdles that computational support needs to overcome (Ellis et al., 1969; Johnson et al., 2009). Fur-thermore, if the sketches are being used to communicate design intent, they often need to rely on verbal or gestural explanations to decode the drawn concept (Cross, 2011). The exception is when there is a direct implicit rela-tionship between the drawn object and the sketch, in which case the interpre-tative semiotics may not be required.

As it will be mentioned in the next few chapters, there have been numer-ous cases of excellent research and projects done in relation to computational support for sketching. Among all the worthy material the literature selected for this paper was identified for its affinity to the goal of computationally supporting design intent communication. However, all the existing and on-going research further accentuates the question: Why is this technology not fully utilized in the industry practice today? Current technological shift to-wards ubiquitous computing will hopefully allow for a better integration be-tween computational support for sketching and the designer’s task.

2. Schematic design sketching

Traditionally early in the design process, when the problem is still unclear and there is not a specific solution, sketching is the preferred method to bet-ter define the problem and to quickly explain concepts to others.

Sketches can be of different kinds and be used for varied purposes, such as artistic sketches, diagrams, schematics, and blueprints representations. Often the boundaries between their uses blend, yet each use provides a dif-ferent level of communication and intention. In their abstract form sketches need some kind of code, legend, or semiotic specification, be it written or verbal, for a third party such as a design collaborator, to understand or inter-pret the design intent. In other words, if a sketch is handed out without ex-planations it would be difficult to interpret what the designer intended to ex-press through his abstract doodles and blobs. If this interpretative process is

SKETCHING AND COMMUNICATING DESIGN INTENT 61

challenging for humans, it is much harder for computers. Can the computer really recognize and distinguish between shapes? Can it understand the in-tention behind symbols and lines?

3. Design computing supporting schematic sketches

Similar to many other areas of technology, much of the current research in computer support for sketching started several decades ago. Although pen-based interfaces are often seen as a new trend, some pioneering systems such as Sketchpad (Sutherland, 1964) and GRAIL (Ellis et al., 1969) emerged long before the Macintosh introduced the adoption of the mouse as a point-ing device in the mid 1980’s.

Due to the market traction of the mouse, pen interfaces were overlooked until in the early 1990s when some companies developed pen–based tablet devices. IBM, GO and GRiD tried to introduce tablet devices to the market, but by 1995 most of these projects were already out of business. One of the devices that enjoyed acceptance for a little longer by the general population was the stylus-based PDA, specifically the Palm Pilot.

Computational support for sketching is a profoundly exciting and broad area of research, often described as intersecting design research, human-computer interaction (HCI), and artificial intelligence (AI). In an effort to focus on and narrow down the literature to fit our specific area of interest, this paper will subdivide previous and current work into 5 areas: 1) Digitiz-ing sketches. 2) Augmenting sketches. 3) Recognizing sketches. 4) Support-ing collaborative sketching. 5) Capturing design intent. Each sub point will provide a brief description of previous, and current research efforts in rela-tion to computational support of schematic sketches.

3.1 DIGITIZING SKETCHES

Digitizing sketching can be subdivided into 2 sections: A) Pen interfaces for natural sketching. B) From pen and paper to vectors and pixels.

A. Pen interfaces for natural sketching

With his Sketchpad, Sutherland (1964) was the first to provide an interactive design system that allowed the user to create drawings by using a light pen on a graphical display. His system provided the user with the option of us-ing physical buttons on the device to apply simple constraints while drawing (See Figure 1). This system was a milestone that opened up a new area of man machine communication. His work is considered a predecessor to computer-aided drafting (CAD).


Figure 1. Ivan Sutherland's in the TX-2 operating area – Using Sketchpad (taken from

Sutherland (1964)). Part of the bank of toggle switches can be seen to his left.

Although some of the obstacles from 1964, such as design intent disam-biguation, and simulation of real paper texture, still linger in today’s re-search; the rapid evolution of technology has allowed for a generation of de-vices to become surprisingly close to the natural workflow designers want when sketching.

Figure 2. Professional product designer working with ILoveSketch. Bae et al. (2008)

Work such as ILoveSketch by Bae et al. (2008) start to break free from the windows, icons, menus, and pointer (WIMP) paradigm and venture into the gesture-based commands as an effort to move the users closer to a natu-ral user interface (NUI) See Figure 2.


B. From pen and paper to vectors and pixels

There are a number of benefits to having a digitized sketch in the design process, including the flexibility of editing the curves, the rapid transition to other CAD tools, or many other possible augmenting techniques. Although 97% of interaction designers and Human Computer Interaction (HCI) practi-tioners begin projects by sketching (Myers et al., 2008), the majority of de-signers still prefer the pencil and paper over digital options.

In their study, Oviatt et al. (2006) advocate that a pen-based application on a tablet computer is not as effective as working out problems using tradi-tional pencil and paper. Although this particular study was done using high school students as participants, there is still a relationship to the cognitive, problem-solving nature of design. The participants preferred paper and pen-cil over electronic writing, but pens of either sort (traditional or tablet-based) are preferred over keyboard and mouse input. Ideally, users could benefit from both worlds: the traditional pencil and paper, and all the augmented ca-pabilities of the digital sketch. The next few paragraphs elaborate briefly on some work done in this particular area of research.

When transposing a sketch to the digital world, the first option would usually be to scan it, which is how many architecture firms still process their sketches. PARC’S ScanScribe (Saund et al., 2003) evaluates drawings and marks made with traditional pencil and paper to produce a computationally modified version of the sketch. At the moment of their publication Scan-Scribe worked with pixels, not vectors.

“ScanScribe is a very functional and useful bitmap image editor that facilitates the manipulation of foreground markings in rough and for-mal document images.” (Saund et al., 2003)

There is also the digital sketching hardware, some of which are commer-cially available, such as the Wacom Inkling (no longer in production but still available in the market), Medion NoteTaker, Anoto Pens, and devices that use licensed Anoto Technology (Livescribe, Logitech’s io2, LeapFrog’s Fly pen), among others. Other pen based systems, such as West et al. (2007)’s Anoto-based MEMENTO, ModelCraft, and Crosspad (sold until 2001) ena-bled users to manipulate digital models and information using a hybrid sys-tem of pen, paper, and tracking solutions.

One of the benefits of these pen based interfaces is that they open up a whole new realm of possibilities for research related to sketching. Unlike the typical mouse, pen interfaces are capable of recording strokes paths, sketch timeline, pen pressure, user’s preferred style of stroke, and many oth-er data that can help develop better design software to recognize and support sketches computationally.


3.2 AUGMENTING SKETCHES

One of the research trends that seem to be progressing rapidly is the 2D to 3D prototypes. These systems have the capability to infer 3D models based on freehand 2D sketches (Lipson & Shpitalni, 2007; Masry et al., 2005; Xu et al., 2014). Users sketch shapes as they are conventionally drawn, then the user specify the preferred interpretation out of the options provided by the system. These systems rely on computational interpretation of sketches to create 3D models (See Figure 3), rather than using conventional modelling commands.

Figure 3. From left to right they show ground truth curves, their algorithmic output (green),

artist modelled 3D surfaces and alternate view curves sketched by two artists(Lipson & Shpitalni, 2007; Masry et al., 2005; Xu et al., 2014).

Another way of augmenting sketching is through the use of projections to enhance the user’s experience when sketching in traditional paper. Using a virtual camera and projection lines to guide the user while sketching Tolba et al. (1999) developed an application that facilitates the creation of interesting re-projections generated from the initial perspective sketch. The two-dimensional drawing program utilizes a projective representation of points, with the help of a method for aligning a sketch using its vanishing points to guide in the construction of scenes with the correct proportions, based on underlay sketches, other drawings, or photographic panoramas.

3.3 RECOGNIZING SKETCHES

The systems mentioned in the previous chapter, specifically the 2D to 3D prototypes, rely heavily on the computer’s ability to recognize or understand


hand-drawn sketches. Despite some content overlap, the research about rec-ognizing sketches is broad enough to dedicate a chapter focusing explicitly on the work done in this area.

One of the pioneers in computational recognition of sketches, Negroponte (1973) started working on sketch recognition experiments back in the early 1970s, a few years later Negroponte (1975) studied the context of the schematic sketch in architecture in relation to computer recognition. His research precedes the 1980s market saturation of the mouse as an input device, and although very limited by the technology of his time, he had a very clear idea of the affordance offered by a pen interface. Negroponte specifies that his experiment is to be considered a starting point, a mecha-nism that will lead to conversations between the user and the machine rather than as a means of generating house plans

3.4 SUPPORTING COLLABORATIVE SKETCHING

When thinking about collaboration three different scenarios come to mind: 1) Collaboration between peers working on the same sketch, on the same room, 2) Collaboration between peers in different locations (remotely) 3) Collaboration between peers in the same office but with different roles and different uses for the design sketch. All three scenarios can benefit greatly from computational support. The more technology advances, the better job computers do at understanding human gestures, touch, speech, sketching styles, among other functions that have great potential to allow collaboration at different levels in the design process.

Ju et al. (2004) researched computational whiteboards and built Work-spaceNavigator, their system can be used by individuals in different loca-tions to support knowledge capture and reuse in prolonged collaborative de-sign tasks. Qian and Gross (1999) developed a drawing program called Netdraw, the software is Java-based object oriented and uses a server-client connection to provide a collective drawing environment for collaborative de-sign.

There are also a number of whiteboard, larger format systems that are used for collaboration between teams, such as Tivoli An electronic white-board for informal workgroup meetings (Pedersen et al., 1993); Flatland: New dimensions in office whiteboards (Mynatt et al., 1999); and Colab: Be-yond the Chalkboard (Stefik et al., 1987). All these collaborative systems support problem solving and knowledge capture for teams gathering in the same room to brainstorm and in the designer’s case to find solutions to de-sign problems.


3.5 CAPTURING AND COMMUNICATING DESIGN INTENT.

If the aim is to provide computational support in order to better communi-cate the design intent, then it would be logical to have different tools to sup-port different scenarios, contexts, and meanings of sketches. These tools could specialize, or be taught to recognize symbols, diagrams, shapes, num-bers, and other elements that would then be compiled to ‘understand’ the de-signer’s intention.

For her thesis Do (1998) developed a prototype called (Right-Tool-Right-Time) RTRT; this prototype was a system that inferred design intentions from a designer’s sketches. Her system demonstrated that

“it is possible for a computer program to recognize drawing symbols and, based on those symbols, activate different design tools.” Do (1998) pp 169

The RTRT system was based on various assumptions:

• Knowledge based design tools must be available at the right time. • Different design activities need different kinds of supporting tools. • Drawing conventions such as symbols and diagrams can serve as clues to ac-

tivate the appropriate design tools at the right time.

Do’s approach to activate a design tool based on the recognition of draw-ing has many possible applications when trying to enhance the current sketching practices in the design industry (See Figure 4). Instead of trying to create a single computer program that would ‘do it all’, this approach pro-vides a more feasible path, with almost endless customizations and connec-tions to current industry software. In her thesis Do explains very detailed the configuration of her RTRT system.


Figure 4. The conceptual model of a drawing environment. (Taken from Do (1998))

Top: different drawing symbols and configurations. Middle, the Right-Tool-Right-Time system that 1) identifies drawing contexts, 2) design intention recognizer (rounded box) infers design intentions (from symbols and configurations, intentions are illustrated inside boxes), 3) inter-application communicator (rounded box) activates different tools based on detected intentions through drawing symbols and configura-tions. Bottom: different design tools.(Do, 1998)

4. Challenges and opportunities

One of the first challenges when reviewing previous works in this exciting area of research is the broad and extensive nature of all that has already been written about it. When writing this paper, the hardest part was selecting which of the hundreds of documents included information pertinent to sup-porting design intent communication and sketch disambiguation. Works were also selected when they included variables that would lead to a closer understanding of the user acceptance, or industry adoption of the technology being researched.

This paper presents a new taxonomy which would help sort out the com-mon features, characteristics, requirements, and capabilities of some of the systems created over the last 50 years. This new taxonomy would serve a dual purpose, first, for designers and developers that are starting to work on their system it would help them to program the prerequisites that their sys-


tem should have. Second, it will help researchers to guide their research ef-forts when looking for examples of work similar to what they are currently developing.

It is however, important to clarify that the following chart is by no means a thoroughly exhaustive effort to include all relevant work available. These systems were selected as examples of some of the previous work done in computational support for sketching; the selection was also based on their relation to supporting design intent communication, sketch disambiguation, user acceptance, and industry adoption of technology.

Figure 5. Computational support for sketching - systems variables.

In the chart (See Figure 5) the systems’ variables were derived from the frequency by which these variables were used. Input refers to the system ability to handle one or more entry at the time. Commercially Available re-fers to the availability of the system in the market. Mobility, defines if the system is stationary or portable. Depth specifies if the system handles 2 or 3


dimensions (2D, 3D). Sketch Intent Recognition identifies if the system rec-ognizes the intention of the designer while sketching a symbol. Recognition Mode refers to the method used by the system to capture what is currently being sketched, Offline systems transfers the sketch to the computer after the sketch is complete. Online systems on the other hand allow users to see their input, and the results, in real time, usually without delay. Multitouch refers to the system’s ability to handle multiple touch-input, it usually includes ges-tures such as, pinch to zoom in or out, rotate with fingers, etc. Ambiguity Mediator specifies if a system can identify and provide different options for the user when sketching. Some systems provide a partial recognition, allow-ing the designer to choose which interpretation is closer to his/her intended sketch.

Conclusion

As the speed in which technology is quickly developing, we are starting to see faster implementations of concepts and ideas that have been in the aca-demic world for decades. The hope is for the all these useful new (and old) pen-based systems to start gaining traction in the design industry, while maintaining a closer relationship with HCI trends. The matrix in the previ-ous chapters was developed hoping that a new taxonomy could provide some kind of contribution to the field, and that some researchers would use it to help define their deliverables.

People will always draw; it is up to us to provide the computational sup-port, and the enhancement to designers’ current tasks in such a natural way that they would find these hybrid systems seamless, useful, and easy to use.

References Arnheim, R. (1969). Visual thinking: Univ of California Press. Bae, S.-H., Balakrishnan, R., & Singh, K. (2008). ILoveSketch: as-natural-as-possible

sketching system for creating 3d curve models. Paper presented at the Proceedings of the 21st annual ACM symposium on User interface software and technology.

Cross, N. (2011). Design thinking: Understanding how designers think and work: Berg. Do, E. Y.-L. (1998). The Right Tool at the Right Time--Investigation of Freehand Drawing as

an Interface to Knowledge Based Design Tools. Ellis, T., Heafner, J. F., & Sibley, W. (1969). The GRAIL Project: An experiment in man-

machine communications: DTIC Document. Johnson, G., Gross, M. D., Hong, J., & Yi-Luen Do, E. (2009). Computational support for

sketching in design: a review. Foundations and Trends in Human-Computer Interaction, 2(1), 1-93.


Ju, W., Ionescu, A., Neeley, L., & Winograd, T. (2004). Where the wild things work: capturing shared physical design workspaces. Paper presented at the Proceedings of the 2004 ACM conference on Computer supported cooperative work.

Lipson, H., & Shpitalni, M. (2007). Correlation-based reconstruction of a 3D object from a single freehand sketch. Paper presented at the ACM SIGGRAPH 2007 courses.

Masry, M., Kang, D., & Lipson, H. (2005). A freehand sketching interface for progressive construction of 3D objects. Computers & Graphics, 29(4), 563-575.

Myers, B., Park, S. Y., Nakano, Y., Mueller, G., & Ko, A. (2008). How designers design and program interactive behaviors. Paper presented at the Visual Languages and Human-Centric Computing, 2008. VL/HCC 2008. IEEE Symposium on.

Mynatt, E. D., Igarashi, T., Edwards, W. K., & LaMarca, A. (1999). Flatland: new dimensions in office whiteboards. Paper presented at the Proceedings of the SIGCHI conference on Human Factors in Computing Systems.

Negroponte, N. (1973). Recent advances in sketch recognition. Paper presented at the Proceedings of the June 4-8, 1973, national computer conference and exposition.

Negroponte, N. (1975). Soft architecture machines: MIT press Cambridge, MA. Oviatt, S., Arthur, A., & Cohen, J. (2006). Quiet interfaces that help students think. Paper

presented at the Proceedings of the 19th annual ACM symposium on User interface software and technology.

Pedersen, E. R., McCall, K., Moran, T. P., & Halasz, F. G. (1993). Tivoli: An electronic whiteboard for informal workgroup meetings. Paper presented at the Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems.

Qian, D., & Gross, M. D. (1999). Collaborative design with NetDraw Computers in Building (pp. 213-226): Springer.

Saund, E., Fleet, D., Larner, D., & Mahoney, J. (2003). Perceptually-supported image editing of text and graphics. Paper presented at the Proceedings of the 16th annual ACM symposium on User interface software and technology.

Stefik, M., Foster, G., Bobrow, D. G., Kahn, K., Lanning, S., & Suchman, L. (1987). Beyond the chalkboard: computer support for collaboration and problem solving in meetings. Communications of the ACM, 30(1), 32-47.

Sutherland, I. E. (1964). Sketch pad a man-machine graphical communication system. Paper presented at the Proceedings of the SHARE design automation workshop.

Tolba, O., Dorsey, J., & McMillan, L. (1999). Sketching with projective 2D strokes. Paper presented at the Proceedings of the 12th annual ACM symposium on User interface software and technology, Asheville, North Carolina, USA.

West, D., Quigley, A., & Kay, J. (2007). MEMENTO: a digital-physical scrapbook for memory sharing. Personal and Ubiquitous Computing, 11(4), 313-328.

Xu, B., Chang, W., Sheffer, A., Bousseau, A., McCrae, J., & Singh, K. (2014). True2form: 3d curve networks from 2d sketches via selective regularization. ACM Transactions on Graphics (TOG), 33(4), 131.

sketching and communicating designanzasca.net/wp-content/uploads/2014/12/01_05_112.pdf · refers to...

Documents