situated prototyping of data-driven applications in augmented … · 2019-04-26 · situated...

Situated Prototyping of Data-DrivenApplications in Augmented Reality

MattWhitlockUniversity of Colorado - BoulderBoulder, CO, [email protected]

Danielle Albers SzafirUniversity of Colorado - BoulderBoulder, CO, [email protected]

ABSTRACTAugmented reality (AR) technologies couple digital content with the real world, enabling contextualizedimmersive analytics experiences. Though effective prototyping tools are critical to crafting theseexperiences, most prototyping approaches decouple the physical environment from the virtual, makingit difficult to evaluate and refine the experience as a whole. With this work, we discuss how an insitu approach to AR prototyping would combat the decontextualization issue and enable holisticprototyping of immersive analytics applications. We propose a more robust prototyping suite to enableincreased collaboration and an opportunity to ideate on the full breadth of the immersive analyticsapplication, situating our discussion in the context of data-driven storytelling for museum exhibits.

CCS CONCEPTS•Human-centered computing→Mixed / augmented reality; Visualization design and evaluationmethods; User interface toolkits;

KEYWORDSAugmented Reality; Prototyping; Museum Exhibits; 3D User Interface.

CHI ’19, May 2019, Glasgow, Scotland© 2019 Association for Computing Machinery.This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version ofRecord was published in Proceedings of ACM Conference on Human Factors (CHI ’19), https://doi.org/10.475/123_4.

https://doi.org/10.475/123_4

Situated Prototyping of Data-Driven Applications in Augmented Reality CHI ’19, May 2019, Glasgow, Scotland

ACM Reference Format:Matt Whitlock and Danielle Albers Szafir. 2019. Situated Prototyping of Data-Driven Applications in AugmentedReality. In Proceedings of ACM Conference on Human Factors (CHI ’19). ACM, New York, NY, USA, 6 pages.https://doi.org/10.475/123_4

INTRODUCTIONAugmented reality (AR) is well-suited for immersive analytics, allowing the display of virtual objectsand 3D visualizations alongside physical referents, enabling a seamless blend of physical and virtualworlds. Effectively leveraging the affordances of AR requires applications that sensibly tie thesephysical and virtual components together. Achieving this blend is challenging because existing toolsdo not allow designers to prototype using AR’s full set of capabilities, including virtual object placement,use of physical referents, 3D data visualizations and interactivity. Existing AR content creation toolsallow designers to pick, place and manipulate objects [10], and immersive visualization prototypingtools allow designers to adjust datasets and encodings [4, 12]. However, these tools are limited intheir abilities to capture and share the experience of an immersive, data-driven story, hindering theiterative prototyping process of such applications.With this work, we discuss the need to develop prototyping tools that support the breadth of

capabilities of AR applications and how an in situ approach can allow designers to build rich prototypesof immersive analytics experiences. We argue that in situ prototyping approaches are important forimmersive analytics because they allow for holistic, contextualized prototypes; rapid and accessibledevelopment; and support for collaboration. To illustrate the potential gains of in situ AR prototyping,we consider prototyping of interactive museum exhibits as a motivating use case. Museums oftencouple data with physical artifacts to tell educational stories. However, current approaches offerlimited support for domain users to readily prototype such exhibits effectively. We use this scenarioto identify where and how situated prototyping might support more effective data-driven storytellingby allowing for holistic, contextualized, and interactive prototyping environments.

RELATEDWORKAR Content Creation: Research in AR content creation tools has yielded systems that enableincreased participation with novel AR technology, deployed in domains such as tourism [6], furnitureassembly [14], cultural experiences [1], and 3D E-books [5]. These tools allow designers to pick andplace virtual objects, manipulate transforms and adjust descriptive characteristics (such as color)to create AR scenarios, using a WYSIWYG interface targeting a wide range of technical abilities.They allow users to focus on the creation of the scene as a whole, rather than crafting individual 3Dmodels. While these tools provide features for effective desktop prototyping, they lack support forunderstanding how a prototype will function in a given operational environment.

https://doi.org/10.475/123_4


Prototyping Immersive Visualizations: Research in immersive analytics has looked at systemsfor interactive data exploration in a 3D immersive space. ImAxes allows users to directly pick up andplace axes representing columns of a dataset to manipulate the visualization [4]. DXR offers a scalableframework to build immersive visualizations that provides accessible packages for both WYSIWYGan programming-based prototyping [12]. These tools allow users to quickly prototype individualvisualizations, but crafting immersive data-driven stories requires designers to manipulate how thevisualizations build a broader story and couple with the physical environment. s

Figure 1: We extend DXR [12] to providea feature-rich iterative in situ prototypinginterface. For example, a designer can pro-totype a target display with associated vi-sualizations, using annotations to furtherdescribe and clarify, in this case describ-ing rotating 3D visualizations on an inter-active map. Visualizations depict temper-ature, humidity and nitrogen dioxide con-centration in a particular location.

As with AR prototyping tools, design tools for 2D data-driven storytelling incorporate functionalitybeyond manipulation of datasets and encodings but allow designers to use visualizations to drivehome a larger narrative to readers/viewers. Recent work in visualization design provides designerswith the tools to explore the visualization design space before refining and finalizing the visualization[2]. DataToon allows designers to build data-driven stories using an interface for manipulation ofvisualizations as well as highlighting, labeling, sketching and further visual embellishment, speakingto the breadth of tasks required to craft data-driven stories [9]. AR offers significant promise for data-driven storytelling as it allows designers to incorporate both physical and virtual artifacts alongsidevisualizations. In situ methods for immersive analytics prototyping can allow designers from a broadarray of domains to craft these immersive, data-driven stories that seamlessly blend physical andvirtual content.

SCENARIO: MUSEUM EXHIBIT CURATIONInteractive museum exhibits can engage people with educational content by combining physicalartifacts with public displays. The ability to prototype the digital artifacts and interactive immersivedisplays and visualizations of a museum exhibit could allow designers to effectively explore thedesign space for a mixed reality exhibit. Recent work in museum prototyping has explored curators’interactions with physical artifacts when prototyping a potential exhibit [7]. For example, exhibitdesigners can associate virtual representations of historical artifacts with physical image markers andadd touch-based and proxemic interactions to trigger animations, sounds, videos, etc [11]. Museumexhibit prototyping could be extended to leverage the full design space of different interactionmodalities and supporting visualizations that would enable compelling data-driven stories thatenhance an exhibit’s physical artifacts.In exhibit design, designers wish to navigate the exhibit space, prototyping novel digital content,

configurations of potential new physical artifacts, and paired data visualizations that augment theexhibit’s physical artifacts. Digital annotations can further clarify physical-virtual associations andrelationships between objects. An in situ approach allows for intuitive understanding of the interactionbetween virtual objects and the physical environment to holistically verify the experience, unlike aprototyping approach decoupled from physical context, such as a those provided by traditional design


tools like Unity. Designers can prototype visualizations and interactivity associated with these virtualand physical objects. In situ navigation of the prototype would enable designers to verify intuitiveengagement with the interactive components of the exhibit and to understand, refine, and evaluatethe holistic experience museum-goers will have.

Figure 2: Example prototype of gestural in-teraction to trigger animations (top, mid-dle) and proxemic interaction to triggeraudio (bottom).

Beyond Individual Elements. In situ prototyping allows designers to focus on the holistic experienceof the immersive application. Museum curators can evaluate and refine the full experience of aprototype exhibit by viewing how different components of the exhibit work together in situ to achieveeducational goals in real-time (Fig. 1). Existing tools such as DXR allow users to quickly adjust theview by changing not only the positions and orientations of visualizations, but also the datasets andencodings of visualizations in situ [12]. Part of this holistic prototyping experience is accessibility:domain experts with limited technical expertise can evaluate the exhibit holistically and interactivelyrefine the encodings, artifacts, and configurations to best match their intended goals. Designerscould further describe or clarify relationships between these visualization prototypes and the physicalenvironment via small sketch-based [3] and/or text-based [8] annotation interfaces. Prototyping toolsthat allow designers to enumerate the relationships between objects and physical referents wouldyield greater breadth in AR prototypes. Functionality to prototype at this breadth in situ could come atthe expense of precise modeling control, but would enable ideation and enumeration of the experienceas a whole.

Contextualized Visualizations. A room scale immersive application incorporates the physical space,additional artifacts and spatially relevant visualizations. Additional artifacts imported as 3D virtualmodels can serve as placeholders in the environment as a proposed installation. Immersive technologiesallow museum curators to bind interactive data visualizations with these artifacts to enrich the storythe exhibit is intended to tell. For example, dinosaur bones could be paired with isotypes comparingthe dinosaur’s size to that of other species. These embedded data representation tightly couplevisualizations to physical referents and specific locations in the environment [13]. Instead of sharingsketched prototypes or first person view captures, designers could collaborate on prototypes in thesame space with the same physical context. In situ prototyping enables a seamless blend of physicaland virtual artifacts with embedded visualizations, allowingmanipulation of an artifact or visualizationwhile viewing all components relative to one another.

Interaction Prototyping. With the breadth of possible technologies in an interactive museum exhibit(head-mounted displays, mobile AR, spatial AR tabletop displays, etc), the designer has a number ofoptions on how to make the environment interactive. Gesturing in a particular direction could enableanimation of a virtual object, and proximity to a physical artifact could trigger an audio file (Fig. 2). Insitu solutions allow designers to evaluate and refine interactions within the physical constraints of the


exhibit. For example, designers can test configurations of artifacts such that proxemic interactive zonesdo not overlap with those of other artifacts or within fragile or confined spaces. Multiple designerscould further prototype interactivity by visualizing multi-user engagement with the exhibit. In situprototyping of interactions provides the ability to verify and manipulate the intended experience.

CONCLUSIONThere is great potential for AR in room-scale prototyping of interactive environments. Leveragingbenefits of AR content creation and visualization prototyping could enable effective design of immersiveAR applications. The in situ approach discussed allows for visualization of the environment withproposed changes, offering an intuitive understanding of the proposition in full. There should becontinued work to elicit intuitive in situ content creation from designers and domain experts to worktoward effective AR prototyping. With effective prototyping tools in place, immersive analytics designcould realize a more iterative and collaborative workflow, resulting in compelling applications.

ACKNOWLEDGMENTSWe thank Jake Mitchell, Nick Pfeufer, Brad Arnot, Ryan Craig, Bryce Wilson and Brian Chung and theComputer Science Senior Projects program at CU for their support in the design and implementationof the supporting system in Figures 1 and 2. We also thank the CU Department of Museum Studiesfor assistance. This research was funded by NSF CHS #1764092 and a grant from the University ofColorado IGP Program.

REFERENCES[1] Nagore Barrena, Andrés Navarro, Sara García, and David Oyarzun. 2016. CoolTour: VR and AR Authoring Tool to Create

Cultural Experiences. In Intelligent Interactive Multimedia Systems and Services 2016. Springer, 483–489.[2] Alex Bigelow, Steven Drucker, Danyel Fisher, and Miriah Meyer. 2017. Iterating between tools to create and edit

visualizations. IEEE transactions on visualization and computer graphics 23, 1 (2017), 481–490.[3] Yun Suk Chang, Benjamin Nuernberger, Bo Luan, and Tobias Höllerer. 2017. Evaluating gesture-based augmented reality

annotation. In 3D User Interfaces (3DUI), 2017 IEEE Symposium on. IEEE, 182–185.[4] Maxime Cordeil, Andrew Cunningham, Tim Dwyer, Bruce H Thomas, and Kim Marriott. 2017. Imaxes: Immersive axes as

embodied affordances for interactive multivariate data visualisation. In Proceedings of the 30th Annual ACM Symposiumon User Interface Software and Technology. ACM, 71–83.

[5] Trien V Do and Jong Weon Lee. 2009. Creating 3d e-books with ARBookCreator. In Proceedings of the InternationalConference on Advances in Computer Enterntainment Technology. ACM, 429–430.

[6] Steven Feiner, Blair MacIntyre, Tobias Höllerer, and Anthony Webster. 1997. A touring machine: Prototyping 3D mobileaugmented reality systems for exploring the urban environment. Personal Technologies 1, 4 (1997), 208–217.

[7] Andrew J Hunsucker, Kelly McClinton, Jennifer Wang, and Erik Stolterman. 2017. Augmented Reality Prototyping ForInteraction Design Students. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in ComputingSystems. ACM, 1018–1023.


[8] Javier Irizarry, Masoud Gheisari, Graceline Williams, and Bruce N Walker. 2013. InfoSPOT: A mobile Augmented Realitymethod for accessing building information through a situation awareness approach. Automation in Construction 33 (2013),11–23.

[9] NamWook Kim, Nathalie Henry Riche, Benjamin Bach, Guanpang A. Xu, Matthew Brehmer, Ken Hinckley, Michel Pahud,Haijun Xia, Michael McGuffin, and Hanspeter Pfister. 2019. DataToon: Drawing Dynamic Network Comics With Pen +Touch Interaction. ACM, 1–12.

[10] Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter. 2004. DART: a toolkit for rapid design exploration ofaugmented reality experiences. In Proceedings of the 17th annual ACM symposium on User interface software and technology.ACM, 197–206.

[11] Dariusz Rumiński and Krzysztof Walczak. 2013. Creation of interactive ar content on mobile devices. In InternationalConference on Business Information Systems. Springer, 258–269.

[12] Ronell Sicat, Jiabao Li, JunYoung Choi, Maxime Cordeil, Won-Ki Jeong, Benjamin Bach, and Hanspeter Pfister. 2019. Dxr:A toolkit for building immersive data visualizations. IEEE transactions on visualization and computer graphics 25, 1 (2019),715–725.

[13] Wesley Willett, Yvonne Jansen, and Pierre Dragicevic. 2017. Embedded data representations. IEEE transactions onvisualization and computer graphics 23, 1 (2017), 461–470.

[14] Jürgen Zauner, Michael Haller, Alexander Brandl, and Werner Hartmann. 2003. Authoring of a mixed reality furnitureassembly instructor. In ACM SIGGRAPH 2003 Sketches & Applications. ACM, 1–1.

situated prototyping of data-driven applications in augmented … · 2019-04-26 · situated...

Documents