mixing realities for sketch retrieval in virtual reality · sketching in virtual reality (vr)...

10
Mixing realities for sketch retrieval in Virtual Reality Daniele Giunchi University College London United Kingdom Stuart James Center for Cultural Heritage Technology Istituto Italiano di Tecnologia Italy Donald Degraen German Research Center for Artificial Intelligence (DFKI) Saarland Informatics Campus Germany Anthony Steed University College London United Kingdom ABSTRACT Drawing tools for Virtual Reality (VR) enable users to model 3D designs from within the virtual environment itself. These tools employ sketching and sculpting techniques known from desktop- based interfaces and apply them to hand-based controller interac- tion. While these techniques allow for mid-air sketching of basic shapes, it remains difficult for users to create detailed and compre- hensive 3D models. In our work, we focus on supporting the user in designing the virtual environment around them by enhancing sketch-based interfaces with a supporting system for interactive model retrieval. Through sketching, an immersed user can query a database containing detailed 3D models and replace them into the virtual environment. To understand supportive sketching within a virtual environment, we compare different methods of sketch interaction, i.e., 3D mid-air sketching, 2D sketching on a virtual tablet, 2D sketching on a fixed virtual whiteboard, and 2D sketch- ing on a real tablet. Our results show that 3D mid-air sketching is considered to be a more intuitive method to search a collection of models while the addition of physical devices creates confusion due to the complications of their inclusion within a virtual environment. While we pose our work as a retrieval problem for 3D models of chairs, our results can be extrapolated to other sketching tasks for virtual environments. CCS CONCEPTS Human-centered computing Virtual reality; HCI design and evaluation methods; Computing methodologies Object recognition; Ranking. KEYWORDS Sketch, Virtual Reality, CNN, HCI ACM Reference Format: Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed. 2018. Mixing realities for sketch retrieval in Virtual Reality. In Woodstock ’18: ACM Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Woodstock ’18, June 03–05, 2018, Woodstock, NY © 2018 Association for Computing Machinery. ACM ISBN 978-1-4503-9999-9/18/06. . . $15.00 https://doi.org/10.1145/1122445.1122456 Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY . ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/1122445.1122456 1 INTRODUCTION Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation of complex 3D designs, existing methods are often supported by a retrieval algorithm capable of finding complex designs based on a simple sketch made by the user by searching a model database. Common approaches can be divided into methods focusing on gestural inter- action [14, 42] or techniques allowing to freely draw sketches in either 2D [19] or 3D space [20]. Gestural interaction techniques are widely used to execute an action as a trigger mechanism or depict a simple trajectory in the design space. While gestures are generally easy to use, they are usually not suitable for characterizing detailed features of an object. A dictionary of gestures additionally limit the ability of the user to freely express their desires. This is especially so for the task of retrieval where flexibility is key to finding the relevant content. However, both 2D and 3D sketches allow the user to convey complex structures including their details. These tech- niques extend the scope of potential designs to a large number of objects within a collection with significant variations in terms of both shape, color and texture. While 2D drawing is familiar, the extension to 2D sketching of 3D objects is non-trivial where perspective and in turn the users depictive ability plays an important factor. However, understanding whether mouse or pen interaction, the familiar, translate into an immersive environment is important to evaluate the contribution of 3D approaches for retrieval and VR interaction in general. Despite the growing interest in methods for Sketch-based Re- trieval [9, 10, 21, 29, 30], only few examples apply to VR [20, 25]. The reason for this is twofold: (1) the ability for sketch descriptions of an object to adequately express the model, and (2) ability for users to get confidence with the depiction of structural elements. Prior work by Giunchi et al.[20] utilized a Multi-View CNN to solve for the first, and introduced on-model sketching in VR for the second. In the same study, Giunchi et al.also performed a detailed study of descriptors for use in 3D model retrieval. While Giunchi et al.[20] showed how a combination of sketch and model improves the user’s retrieval query, they only explore a single modality of interaction, i.e., 3D mid-air sketching, and its level of effectiveness. However, it is essential to understand how different interaction methodologies can impact on both user arXiv:1910.11637v2 [cs.HC] 5 Nov 2019

Upload: others

Post on 21-Aug-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Mixing realities for sketch retrieval in Virtual RealityDaniele Giunchi

University College LondonUnited Kingdom

Stuart JamesCenter for Cultural Heritage Technology

Istituto Italiano di TecnologiaItaly

Donald DegraenGerman Research Center for Artificial Intelligence (DFKI)

Saarland Informatics CampusGermany

Anthony SteedUniversity College London

United Kingdom

ABSTRACTDrawing tools for Virtual Reality (VR) enable users to model 3Ddesigns from within the virtual environment itself. These toolsemploy sketching and sculpting techniques known from desktop-based interfaces and apply them to hand-based controller interac-tion. While these techniques allow for mid-air sketching of basicshapes, it remains difficult for users to create detailed and compre-hensive 3D models. In our work, we focus on supporting the userin designing the virtual environment around them by enhancingsketch-based interfaces with a supporting system for interactivemodel retrieval. Through sketching, an immersed user can query adatabase containing detailed 3D models and replace them into thevirtual environment. To understand supportive sketching withina virtual environment, we compare different methods of sketchinteraction, i.e., 3D mid-air sketching, 2D sketching on a virtualtablet, 2D sketching on a fixed virtual whiteboard, and 2D sketch-ing on a real tablet. Our results show that 3D mid-air sketching isconsidered to be a more intuitive method to search a collection ofmodels while the addition of physical devices creates confusion dueto the complications of their inclusion within a virtual environment.While we pose our work as a retrieval problem for 3D models ofchairs, our results can be extrapolated to other sketching tasks forvirtual environments.

CCS CONCEPTS• Human-centered computing → Virtual reality; HCI designand evaluationmethods; •Computingmethodologies→Objectrecognition; Ranking.

KEYWORDSSketch, Virtual Reality, CNN, HCI

ACM Reference Format:Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed. 2018.Mixing realities for sketch retrieval in Virtual Reality. InWoodstock ’18: ACM

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected] ’18, June 03–05, 2018, Woodstock, NY© 2018 Association for Computing Machinery.ACM ISBN 978-1-4503-9999-9/18/06. . . $15.00https://doi.org/10.1145/1122445.1122456

Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY .ACM,New York, NY, USA, 10 pages. https://doi.org/10.1145/1122445.1122456

1 INTRODUCTIONSketching in Virtual Reality (VR) allows users to design and createobjects in 3D virtual space. To aid users in the creation of complex3D designs, existing methods are often supported by a retrievalalgorithm capable of finding complex designs based on a simplesketch made by the user by searching a model database. Commonapproaches can be divided into methods focusing on gestural inter-action [14, 42] or techniques allowing to freely draw sketches ineither 2D [19] or 3D space [20]. Gestural interaction techniques arewidely used to execute an action as a trigger mechanism or depict asimple trajectory in the design space. While gestures are generallyeasy to use, they are usually not suitable for characterizing detailedfeatures of an object. A dictionary of gestures additionally limit theability of the user to freely express their desires. This is especiallyso for the task of retrieval where flexibility is key to finding therelevant content. However, both 2D and 3D sketches allow the userto convey complex structures including their details. These tech-niques extend the scope of potential designs to a large number ofobjects within a collection with significant variations in terms ofboth shape, color and texture.

While 2D drawing is familiar, the extension to 2D sketching of3D objects is non-trivial where perspective and in turn the usersdepictive ability plays an important factor. However, understandingwhether mouse or pen interaction, the familiar, translate into animmersive environment is important to evaluate the contributionof 3D approaches for retrieval and VR interaction in general.

Despite the growing interest in methods for Sketch-based Re-trieval [9, 10, 21, 29, 30], only few examples apply to VR [20, 25].The reason for this is twofold: (1) the ability for sketch descriptionsof an object to adequately express the model, and (2) ability forusers to get confidence with the depiction of structural elements.Prior work by Giunchi et al. [20] utilized a Multi-View CNN tosolve for the first, and introduced on-model sketching in VR for thesecond. In the same study, Giunchi et al.also performed a detailedstudy of descriptors for use in 3D model retrieval.

While Giunchi et al. [20] showed how a combination of sketchand model improves the user’s retrieval query, they only explorea single modality of interaction, i.e., 3D mid-air sketching, andits level of effectiveness. However, it is essential to understandhow different interaction methodologies can impact on both user

arX

iv:1

910.

1163

7v2

[cs

.HC

] 5

Nov

201

9

Page 2: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Woodstock ’18, June 03–05, 2018, Woodstock, NY Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed

A B

C D

Figure 1: To understand supportive sketching within a vir-tual environment, we investigate sketching in virtual envi-ronments and consider 4 different interaction methods, i.e.,(a) 3Dmid-air sketching, (b) 2D sketching on a virtual tablet,(c) 2D sketching on a fixed virtual whiteboard, and (d) 2Dsketching on a real tablet.

performance and user experience. In addition it is crucial to separatethe model component from the sketch component because the firstone tends to have supremacy over the second one when combined.Therefore, we present a study to understand how users interactwith physical and virtual devices framed in a retrieval context.Our work investigates different techniques for users to provideinitial sketch designs as input for sketch-based retrieval algorithmsin virtual environments. Therefore our contribution are as follows:

• Four methods of Sketch-based Retrieval interaction in VR:– 3D Mid-Air Sketching, based on the method of Giunchi etal. [20] using mid-air drawing using a controller;

– 2D Sketching on a VR Tablet, using a 2D tablet within thevirtual environment;

– 2D Sketching on a VR Whiteboard, using a VR plane toannotate the model;

– 2D Sketching on a physical tablet, using a real world tablettracked in VR to annotate the model with strokes.

• An extensive user study over the four methods identifyingthe advantages of methods with regards to the user.

In the remainder of this paper we outline related work (sec. 2), anddetail theon four interaction methods in sec. 3. We provide thedetails of our user study (sec. 4), evaluate our results (sec. 5), andsummarize our contributions (sec 6).

2 RELATEDWORKUsing Sketches for interaction has a long history, e.g., in the contextof the retrieval of images [15, 18, 21] or videos [22, 23], interaction in2D [19], or more recently for interaction in 3D [20]. Given the multi-modal interaction style there is a vast amount of literature that canbe considered relevant. We therefore focus on core techniques in2D sketching (sec. 2.1) and 3D sketching in AR/VR (sec. 2.2).

2.1 2D Sketching for RetrievalSketch-based Retrieval techniques using 2D sketches query rely ona predefined database to obtain either a resulting image (SBIR) orvideo (SBVR). SBIR approaches can be classified into blob-basedtechniques that try to extract features for shape, color or texture forthe blob, or contour-based techniques that characterize the imagewith curves and lines. Blob-based SBIR methods try to describe im-ages using descriptions of the segments within the image, for exam-ple, QBIC [4], which creates separate descriptors for the modalities.Alternatively, topology models [35] can describe the blob charac-teristics. Contour-based methods include elastic matching [5] andgrid and interest points approaches [11]. SBVR approaches mainlyaddress the optimization of objects between video and sketch query.Collomosse et al. [13] applied Linear dynamic system to solve theobject matching. Hu et al. [22] introduced a hybrid technique thattakes advantage of object semantics. While, James et al. [23] pro-posed a descriptor-based technique for fast indexable retrieval. Toimprove on early methods for sketch-based retrieval, more recentliterature started to employ neural networks. An early examplewas SketchANet [45], which performed sketch recognition usingAlexnet [24]. More recently, triplet convolutional neural networkshave gained interest as they have the capacity to deal with deepembedding spaces [9]. Improving the image similarity metric is amain challenge as these triplet architectures are used to measuresimilarities between images and sketches [10]. Examples includethe work ofWang et al. [41], where they developed a learning modelbased on triplets to describe the images considering fine-grainedsimilarities. This approach was extensively used in recent workslike in Wang et al.[40] where the embedding spaces of sketches and3d models were joint, and in Li et al. [26] for images and 3D models.

All the aforementioned approaches focus on ’Drawing on awhite-board’ with generally singular or interactive search. While improve-ments in the field of model retrieval increase accuracy, supportingthe user during sketching is imperative to increase performance.One example is ShadowDraw [34], where the user is provided withfeedback related to texture, color, and shapes to improve imageretrieval task. During a sketching activity, the system provides theuser with a real-time shadow image to help free-form drawing andachieve a better final sketch of the object. iCanDraw [16] helpsto solve drawing issues by exploiting algorithms for sketch recog-nition. Lastly, Sketch-to-Collage [31] uses a query-by-sketch togenerate collages from an image collection and deploys an indexingmechanism based on colors and shapes.

With the rapid growth of 3D model collections the retrievalof a 3D model from a collection has received attention. However,the nature of the sketch is abstract and iconic, and it does noteasily lend itself to describing an object in its details from a 2Dquery. However for image retrieval two classes of descriptors wereintroduced: model-based descriptors and view based descriptors.For model-based there consists three types of descriptors: geometricmoment [8], surface distribution [28] and volumetric descriptors[32]. The common goal for all of them is to extract the featuresthat describe the shape of the objects. View-based extracts featuresfrom a collection of the 2D projections of the 3D models. Ansary etal. [1] implement an indexing method that exploits 2D views andSu [36] implemented a stack of convolutional neural network that

Page 3: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Mixing realities for sketch retrieval in Virtual Reality Woodstock ’18, June 03–05, 2018, Woodstock, NY

takes multiple 2D projections of the 3D model generating a singlecompact descriptor for the object.

2.2 3D Sketching in AR/VRRather than transforming 2D sketches into 3D representations, im-mersive modeling environments allow users to sketch and design3D content using mid-air interaction techniques. Such techniquesutilizing 3D space are intuitive to learn regardless of the user’sexpertise with the VR system [42]. A very early example of suchan approach called HoloSketch [14] combined head-tracked stereo-scopic shutter glasses and a CRT monitor with a six-axis mouse orwand for mid-air freehand drawing.

While 3D freehand drawing can support expert 3D artists [33]and shows improving accuracy and uniformity of sketched objectsover time [43], interaction techniques for sketching in 3D are stilllimited in terms of accuracy and user fatigue. Realizing an intendedstroke during sketching is firstly bound by the user’s perception ofdepth [3]. When users are unable to determine their active drawinglocation and spatial relationships with other contents inside thescene, sketching errors occur. Additionally, the added complexityof 3D compared to 2D sketching causes a higher cognitive loadand increasingly taxes the sensorimotor system [43]. Lastly, as theabsence of a physical resting surface leads to fatigue, accuracy isnegatively influenced accuracy over time [3].

3D freehand sketching approaches can be further classified incategories that highlight the motivation as well as the limitation ofsuch type of interaction: metaphors and both virtual and physicalsupport surfaces.

For Sketch metaphor Wacker et al.[38] designed and implementeda tool called ARPen, whose real-world position is tracked by a smart-phone app, that lets the user interact via mid-air sketch. In addition,they evaluated through a user study different techniques to selectand move virtual objects with such a tool. 3D sketching is used inthe work of Giunchi et al.[20] where the task is retrieving a targetmodel from a large collection of chairs combining sketch input with3D models by using a multi-view CNN to improve accuracy duringthe search. A user study where a searching task is proposed to theparticipants compared the search based on sketch interaction withsimple browsing of the chair database. We develop our study consid-ering the 3D sketch method based on the same freehand interactionbut focusing specifically on the aspect of sketch generation.

In recent years in VR environments, the use of 2D surfaces,Virtualsurfaces, has been explored to replace the inaccuracy brought by3D tracking. Despite the lack of one dimension some tasks aresuited for a 2D device such as terrain editing [6], user interfaceediting [7] or even object selection and manipulation [37]. Arora etal.developed SymbiosisSketch AR [2] that combines 3D drawingin mid-air with freehand drawing on a surface. They equipped theuser with an Hololens, a tracked stylus, and a tablet and createda hybrid sketching system suitable for professional designers andamateurs. Machuca et al.supported 3D immersive sketching usingmultiple 2D planes to create 3D VR drawings [27].

A virtually rendered tablet in VR can not provide the samelatency-free response that a 2D tool or Physical surfaces can have,different attempts of integrating a real physical 2D tablet have beenmade. Arora compared traditional sketching on a physical surface

to sketching in VR, with and without a physical surface to restthe stylus on [3]. Additionally, they investigated visual guidancetechniques to devise a set of design guidelines. Wacker et al.[39]studied how accurately visual lines and concave/convex surfaceslet users draw 3D shapes attached to physical virtual objects in ARand Dorta [17] draws on virtual planes using a tracked tablets.

3 SKETCH MODALITIES FOR VRIn the following section, we describe our implemented model re-trieval system, our four sketch interaction methods and our back-end which acts as retrieval system. For each method, the user isimmersed within a virtual environment and sketches either in 3Dmid-air (3D Sketching), on a virtual tablet, a virtual whiteboard oron a tracked physical tablet.

3.1 Interaction MethodologiesWe propose four distinct methods of interaction, 3 of them use 2Dsketch generated on different canvas and with different actions,and only one of them make use of 3D sketch. We outline them indetail below.

3D Mid-Air Sketching. This method, shown in Figure 2a, issimilar to existing systems for sketching in VR and is directly basedon the method for 3D mid-air sketching described in [20]. The userdirectly sketches in 3D space using a hand-held controller. Whileholding down the trigger button, a virtual stroke is applied in theair at the current position. By dragging the controller through theair, strokes are extended towards a continuous line. Once the triggerbutton is released, the active stroke is considered to be completed,and a new stroke can be initiated. There is no theoretical limit to thevolume the sketch can occupy and the user can create the sketchin any position.

2D Sketching on a VR Tablet. With this method, we mimic anatural method of sketching, but placed within VR.

A 2D panel is attached to the user’s non-dominant hand con-troller, see Figure 2b, referencing the familiar painting palette. Weavoided a user movable panel due to the additional complexityof modelling an unconstrainted position and orientation. As thismethod aims to simulate sketching on a portable tablet, we de-signed the 2D panel with a similar size to a commonly used tabletdevice (Galaxy Tab A 10.1") with the largest dimension as side ofthe squared panel. The actual sketching of lines is done using thecontroller in the user’s dominant hand. This makes the interactiontechnique a bi-handed approach as both hands are involved in theprocess of sketch creation, i.e., one hand performs the sketch whilethe other hand stabilizes the drawing canvas. Here, the 2D sketchis not only limited in the third dimension, but also by the size ofthe panel.

2D Sketching on a VR Whiteboard. Similar to VR Tablet, thewhiteboard method provides a panel onto which the user can sketchin 2D, see Figure 2c. A familiar design paradigm, the whiteboardtechnique extends the size of the tablet to that of a larger whiteboardin order to provide more space for sketching.

The dimension of the whiteboard is linearly five times than thevirtual tablet, with the same pixel resolution. As the whiteboard ispositioned on a fixed location (centre of the room) inside the virtual

Page 4: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Woodstock ’18, June 03–05, 2018, Woodstock, NY Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed

(a) 3D Mid-Air Sketching (b) 2D Sketching on a VR Tablet

(c) 2D Sketching on a VR White-board

(d) 2D Sketching on a PhysicalTablet

Figure 2: Overview of the four implemented interactionmodalities for sketch-based retrieval.

environment, this method does only requires the use of the user’sdominant hand. The user could ask, before starting the session toadjust the position of the whiteboard.

2D Sketching on a Physical Tablet. Using a real-world tablet(Galaxy Tab A 10.1") approach offers the user a physical tablet toperform 2D sketching while immersed in the virtual environment,see Figure 2d. This mimics the most commonly technique used bydigital artists.

The tablet is positioned on a table and requires a short regis-tration procedure before the start of a sketching session. Whilethe tablet is still limited in drawing space, the physical feedbackprovided from the actual device aims to improve the stability dur-ing sketching. The user is able to sketch using her finger, thus thisapproach does not require the use of a controller. The user’s handsare tracked using a LEAP motion device as the virtual environmentneeds to visualize the correct position of the hands of the user. Thisadditional tracking is necessary as we noticed during a first imple-mentation that the absence of the visual feedback for the fingerposition led to an unpleasant experience. This was mainly due tothe user being unable to find the right location of contact betweenher finger and the tablet.

3.2 Sketch and 3D collectionThe 3D sketch is implemented as a colored strip with a wider sectionthat is exposed to the current camera. Consequently, all the virtualcameras involved in the multi-view generation render the sketch-oriented along the wider section independently from the length orthe trajectory of the stroke. The user can change the color of thenext sketch using the palette without modifying the strokes alreadygenerated. The 2D sketches are simply implemented as sequencesof connected points on a canvas.

As chairs database we used a set of 3370 chairs that are part ofShapeNet [44]. ShapeNet is a large collection of 3D models wherean extensive subset is made by chairs with or without colors ortextures.

3.3 Model RetrievalTo perform sketch retrieval, we implemented a back-end whichhosts a pre-loaded CNN model. This back-end answers the visualqueries containing the sketches by producing a list of the 40 modelsthat are considered more similar to the input sketch. We used theVGG-M Matlab implementation of Su [36]. This implementationprovides a single visual descriptor after elaborating the snapshotstaken by VR software. The method works by generating a set ofstructured camera views (snapshots) around the models. Theseimages are then passed through the CNN and a final descriptor isgenerated. During the generation of the snapshots, the camerasin the virtual environment move at different angles looking at thecenter mass of the sketch and fit in the image its maximum extent.The CNN uses the VGG-M model [12] which has 5 convolutionallayers and 3 fully connected ones. The encodings from each view arethen merged through view-pooling using a max function over theviews, using the final 3 fully connected layers of VGG-M to createa descriptor. We removed the final layer of the model describedby Su et al., as it is normally used for classification purposes. Incontrast, for the other methods of interaction we bypass the poolingstep. An overview of the entire system is depicted in Figure 3.

4 EXPERIMENTTo compare our four interaction methods for sketching in VR, wedesigned and conducted a user study performed in our lab.

Participants: We used a within-subjects experimental design tohelp to reduce the number of participants and errors associatedwith individual differences. To counterbalance possible carryovereffects, the methods were randomized between the users. As ourindependent variable, we distinguish the methods used to sketch,3D sketch, 2D sketch on a virtual tablet, 2D sketch on a fixed vir-tual whiteboard and 2D sketch on a real tablet. We distinguish 3dependent variables, namely the success rate, the completion timeof the task and the number of submitted queries during the search.

A total of 5 participants (4male, 1 female, 25− 43 age range withavg. 34) volunteered for our study that was approved by ANONethics board. All participants had previous experiences with VRand already used an Oculus RIFT and Touch.We opted for a within-subjects study with a high number of searching tasks to minimizethe possible fluctuations in drawing skills emerging from a studywith many users on few chairs. A total of 32 searches were per-formed by each user, 8 for each of the 4 methods. In total we have160 searching sessions. The users did not declare to have particularabilities in drawing or sketching, and they were compensated forthe test.

Apparatus: The rendering of the Virtual Environment was per-formed in Unity 2018.2.13 using an Oculus RIFT DK1 headset witha connected laptop computer. The specification of the laptop is:Intel i7 CPU, 64 GB RAM with Nvidia GeForce GTX 980M graphicscard. The interaction with the 3D environment was provided byboth the Oculus Touch, i.e., two controllers paired with the headset,and hand-tracking using a LEAP Motion device. For the real tabletsession, we use a Galaxy A6 tablet, tracked within Unity applicationvia an Oculus Touch controller attached on the top right corner

Page 5: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Mixing realities for sketch retrieval in Virtual Reality Woodstock ’18, June 03–05, 2018, Woodstock, NY

A B...

x12

C DFigure 3: Overview of the system’s model retrieval mechanic. Here, (A) the sketch created by the user results in a total set of 12images (B) which are processed by 12 versions of the same CNN. After a max-pooling procedure, one descriptor is generatedand (C) compared through Euclidean distance with the descriptors previously calculated for all the chairs of the collection.The result of the search is (D) a small subset of the most similar chairs from which the user can select.

with Oculus Rockband VR Touch Guitar Mount Attachment.

Procedure: Before starting the experiment, each participant signeda consent form and was instructed on the searching task. A periodof 15 minutes was dedicated to training the user to develop confi-dence with the controllers, the sketch mechanism and the virtualenvironment. Between each method, users had 3 minutes of restand they can perform the task seated or standing up.

Upon completion of the introduction, the experiment commenced.Inside the virtual environment, users are placed inside a virtually

furnished room with a chair model in the center. The user searchesby sketching and then triggering the search system when he orshe wants. Results are displayed on a floating panel. The panel ispositioned on the left controller, and the selection is made via theright-hand controller. The panel can display 10 chairs at a time,and 40 chairs are the result of each query. Therefore the user cannavigate 4 pages with simple controls. We decided to display onlyten models per page in order reduce occlusion of the scene whileproviding enough variety for the user to choose from.

For each method, participants were asked to perform sketchsearches for a given set of 8 different chairs in shape and textures.For each session, the participant started with a randomly selectedsketch interaction method and performed the search for each tar-get chair of the 8 proposed in a random order. Using the selectedmethod, the participant started sketching to initiate the search forthe presented target chair. Upon confirmation, the system providedthe user with a set of potential chairs considered to be most similarto the created sketch. The participant could refine the search re-sults by editing or detailing the sketch. When the participant wassatisfied with the results, one of the chairs from the proposed setwas selected. This selection concluded the search task and wouldreplace the presented 3D model inside the scene. Each session isgiven a time limit of 3 minutes after which the search was consid-ered terminated without a successful result. Additional functionsavailable to the participants during sketching with all the differentmethods were: undo function to erase the sketch either entirelyor partially, and a color palette to characterize the sketch with achosen color.

The experimenter recorded success rate and completion time foreach task. The user’s experience with the current sketching methodwas measured asking the user to rate the interaction on a scalefrom 1 (very bad) to 5 (very good).

5 EVALUATIONWe investigated the differences between the four different methodsof interaction using sketches within a virtual environment. Weevaluated our study over the 8 distinct chairs presented to eachparticipant and 4 methods to test in terms of the accuracy of thereturned model, time, and the number of queries to complete thetask. To evaluate the accuracy we counted the number of success-ful searches among the total number of searches. The number ofsuccessful task completions for the 3D sketch was 37 out of 40(92.5%), the 2D sketch with the whiteboard was 9 out of 40 (22.5%),the 2D sketch with virtual tablet was 6 out of 40 (15%) and the the2D sketch with real tablet was 5 out of 40 (12.5%). To evaluate theefficiency we measured the time elapsed from the beginning to theend of the search and count for each search the number of attemptsto submit the sketch to the system. The average time among all thechairs for the the 3D sketch was 71 seconds, the 2D sketch with thewhiteboard was 156 seconds, the 2D sketch with virtual tablet was169 seconds and the the 2D sketch with real tablet was 166 seconds.The average number of attempts among all the chairs for the the 3Dsketch was 1.85, the 2D sketch with the whiteboard was 4.88, the2D sketch with virtual tablet was 5.65 and the the 2D sketch withreal tablet was 2.9. We demonstrated how the variation in chairseffects the different methods in the radar plot of Figure 4.

We showed in Figure 7 the cumulative number of attempts overthe different methods of interaction. It can be seen that the 2Dvirtual tablet required a significant numbers of search triggers,while the 3D sketch required the least. In Figure 5, we show howthe difference in 3D model effects the number of required searchtriggers. These results mimic those of fig. 7, but also demonstratehow different chairs provide challenges to the different interactionmethods. Interestingly, for 3D Sketching and Physical tablet thefigure has a similar profile across models as opposed to VR Tabletand Virtual Whiteboard.

Finally, at the conclusion of the experiment the user evaluatedthe different methods of interaction shown in Figure 6 is the cumu-lative score. 3D sketch appears to provide the best user experience.However, the trend is inconsistent with the number of triggers offigs 7 and 5.

5.1 DiscussionThe difficulty in finding some 3D models with 2D techniques in thedatabase becomes evident from Figure 4. Despite all the methods

Page 6: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Woodstock ’18, June 03–05, 2018, Woodstock, NY Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed

Figure 4: The inner circles in each radar represent 45 seconds. The center of each circle corresponds to time 0. Each radarshows the average time to complete the task for each chair considering all the methods. The time is normalised to 3 minutesas the upper limit allowed for a search attempt.

Figure 5: The number of search iterations for the different types of chairs for the different methods of interaction.

being intuitive, the 3D sketch is more accurate and satisfying tothe user experience.

In the case of 3D Sketch the user can depict the target chairusing more naturally the depth information, while in all the othermethods user can draw only a 2D projection of a three dimensionaldata on a texture. Intuitively the search results will be improvedin terms of precision since more sketch features can be extractedand evaluated by the system. The real tablet method introducesphysical feedback for the user as the draws are generated by thefinger when touching the surface, but essentially the output ofthe interaction is the same for all the 2D techniques. As each userperforms 32 searches in total, they develop a plan to optimize thesearch, exploring how the input impacts on the neural network andexploiting eventual winning strategies for each method.

As 3D sketch interaction generates 12 input images from differentpoints of view, the 2D sketch can contribute only with one. With 2Dsketch, each user quickly developed the idea of selecting the mostsignificant view angle and tries to depict that projection. Despitethis, as the success rate for 2D methods is lower than 3D Sketching.Some users tried to draw the different points of view in the sametexture, in some cases achieving successfully the target chair. Also,while the 3D sketch does not require an accurate depiction, wenoticed that for each 2D methods the user tended to detail thedrawing to increase the probability of finding the target.

Figure 6: Each bar is the cumulative score given by each userfor a specific method.

As a consequence, a typical behavior noticed on the 2D canvasis that the user preferred to fill accurately areas between the edges.This disposition is not necessary for 3D where a few quick strokescan be interpreted by the system as a filled surface. The motivationfor this differences is again the lack of information of 2D sketchcompared with the 3D counterpart, so the need to detail the 2Ddrawing as much as possible emerges naturally.

Page 7: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Mixing realities for sketch retrieval in Virtual Reality Woodstock ’18, June 03–05, 2018, Woodstock, NY

Figure 7: Each bar is the cumulative number of search triggerevents given by each user for a specific method.

This lack of efficiency for the 2D sketches harm the user experi-ence. Essentially the 2D drawing is conditioned by a continuoussearch of the feature that can trigger the right system’s response.Therefore, this attitude denatures the process of intuitive and icono-graphic representation of a model.

In terms of user interaction, for the aforementioned reasons 2Dsketch modalities required from the user more query submission tothe system (as shown by Figure 7), more attention, a firmer hand,and in particular with virtual tablet two hands working at the sametime. This could have caused discomfort after a few minutes ofsketching and eventually a loss of accuracy.

Between 2D methods, the whiteboard shows better results thanthe virtual tablet for two main reasons: firstly because of the fixedtexture to draw on, and secondly because of the larger canvas thatcan include more details. Despite the distance between the surfaceand the drawing hand is the largest between the 2D methods, thisresult shows that this aspect does not have a negative impact onthe task.

The real tablet is interesting because of the physical interactionthat is completely absent in all the other methods. While at thebeginning of the session this is a pleasant novelty, contrary to whatwas imagined the sketch done directly with the finger was lessprecise if compared to the sketch generated by a remote controller.Moreover, mixing virtual reality with a tracked canvas was notsufficient to guarantee decent user experience. The absence of thepositional information of the finger lets the user, still immersedin the scene, become disoriented during the drawing process asit is unclear where is the finger respect to the canvas position.Despite that we managed this issue using a Leap Motion to trackcontinuously the finger displaying an avatar, sometimes noise wasintroduced and the user experience deteriorated rapidly. In addition,as it inherited the lack of performance of the 2D techniques, thismethod does not improve the results of the other 2D methods. Thiscould indicate that even synthetic haptic feedback could aid in thedrawing for the other methods even in 3D Sketching.

Both Figure 7 and Figure 5 combined with the success rate of themethods motivate the low value of triggering events for 3D sketchand real tablet techniques. 3D sketch had a high rate of success anda low number of queries that indicates the high efficiency of themethod. On the contrary, even if the real tablet had few triggeringevents it does not mean that is efficient. Its low success rate and

problematic interaction lead the user to give up quickly, avoidingan extensive search as in the other 2D methods.

All the aforementioned considerations impact on the user expe-rience as shown in Figure 6.

5.2 LimitationsDespite 3D sketch showing positive feedback from the user studycompared to the other methods, we investigate some aspects andlimitations of this system to improve the search accuracy or experi-ence. Below we outline the most important ones:Query Descriptors In our study, we used VGG-M deep descriptor,but recently a large number of CNN architectures has been designedguided by the work of Giunchi et al.. We implemented our systemas distinct modules that can be replaced easily, threrefore beingrelatively simple to extend the study to different neural networksand machine learning models in order to make a comprehensivesurvey. These solutions could also be fine-tuned to VR or learntthrough active learning to become bespoke to the users style. Wehowever focus on the user expeirence with the different modalitiesin this study.Expanded comparisons Although our study aimed to compare 4comprehensive methods for retrieval using sketches in the style ofboth 3D or 2D, the study could extend the study to other methods ofinteraction. An additional method could be a tracked pen coupledwith the tablet to increase accuracy. An interesting follow-up wouldbe comparing the user experience and accuracy obtained in a virtualenvironment with a real counterpart. In such case the interactionwith the physical tablet would likely create a more pleasing userexperience.Sketch in Augmented Reality Understanding how users sketchwithin a virtual reality allows us to perform controlled user stud-ies, but many application fields require real-world interaction. Bytrivial adaptations to this method, we could see applications withinFilm & TV for set design or architecture for model creation. Suchapplications require their nuances to be consider and would needto take advantage of the aforementioned extensions.

In addition, we aim to compare the sketch mechanism with moreadvanced user interfaces paired with state-of-the-art mechanismsfor searching objects in immersive environments such as text inputand faceted search. Moreover, we want to explore possible ways tointegrate sketch with additional information coming from a morecomplex interface. Furthermore, several benefits can be achievedproviding functionalities such as brush size, erase and transformoperations that could mimic the basic functionalities of 2D photoediting software.

6 CONCLUSIONIn the task of scene modeling we have investigated different strate-gies of interaction within Virtual Reality. Framed within the contextof database navigation and retrieval for scene modeling, sketch in-teraction has been shown to be a satisfying method when comparedwith text queries or even the simple database scrolling allowingexpressive visual description of the query and accurate results tobe retrieved. In addition, 3D and 2D sketch are extensively studiedin different contexts related to database navigation and we have

Page 8: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Woodstock ’18, June 03–05, 2018, Woodstock, NY Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed

Figure 8: This figure shows in the first four columns some representative images from the 3D sketch. The fifth column is thesketch from the virtual tablet method. The sixth column is the outcome of the whiteboard method and the seventh column,the real tablet sketch. The last column is the image of the target chair.

Page 9: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Mixing realities for sketch retrieval in Virtual Reality Woodstock ’18, June 03–05, 2018, Woodstock, NY

filled a needed missing comparative study performed within VR.Our study fills this gap by comparing different sketch-based mech-anisms including 3D, 2D sketch in virtual reality with a tablet or awhiteboard and a method that considers the use of a physical tabletwhile the participant is immersed in the virtual environment. Inour experiment, we collected the time to perform the task, the rateof success, and the number of queries.

We analyzed the methods and discovered that amongst the 2Dmethods, the provision a physical tablet did not improve the user ex-perience. It is intuitive to conclude 3D sketching as a more suitableuser experience within VR, therefore overcoming the familiarityof 2D based retrieval. We therefore pose 3D mid-air sketching as asuitable direction within VR retrieval and related tasks.

REFERENCES[1] Vandeborre J.P. Ansary T.F., Daoudi M. 2007. A Bayesian 3-D Search Engine

Using Adaptive Views Clustering. IEEE Transactions on Multimedia 9, 1 (2007),78–88. https://doi.org/10.1109/TMM.2006.886359

[2] Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and KaranSingh. 2018. SymbiosisSketch: Combining 2D & 3D Sketching for DesigningDetailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on HumanFactors in Computing Systems (CHI ’18). ACM, New York, NY, USA, Article 185,15 pages. https://doi.org/10.1145/3173574.3173759

[3] Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh,and George Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfacesin VR. In Proceedings of the 2017 CHI Conference on Human Factors in ComputingSystems (CHI ’17). ACM, New York, NY, USA, 5643–5654. https://doi.org/10.1145/3025453.3025474

[4] Jonathan Ashley, Myron Flickner, James L. Hafner, Denis Lee, Wayne Niblack,and Dragutin Petkovic. 1995. The Query By Image Content (QBIC) System. InProceedings of the 1995 ACM SIGMOD International Conference on Management ofData, San Jose, California, May 22-25, 1995. 475. https://doi.org/10.1145/223784.223888

[5] Alberto Del Bimbo, Pietro Pala, and Simone Santini. 1994. Visual Image Retrievalby Elastic Deformation of Object Sketches. In Proceedings IEEE Symposium onVisual Languages, St. Louis, Missouri, USA, October 4-7, 1994. 216–223. https://doi.org/10.1109/VL.1994.363615

[6] Doug A. Bowman, Jean Wineman, Larry F. Hodges, and Don Allison. 1998. De-signing Animal Habitats Within an Immersive VE. IEEE Computer Graphics andApplications 18, 5 (1998), 9–13. https://doi.org/10.1109/38.708555

[7] Doug A. Bowman and Chadwick A. Wingrave. 2001. Design and Evaluationof Menu Systems for Immersive Virtual Environments. In Virtual Reality 2001Conference, VR’01, Yokohama, Japan, March 13-17, 2001, Proceedings. 149–156.https://doi.org/10.1109/VR.2001.913781

[8] Kimmel R. Bronstein M. A., Bronstein M. M. 2009. Topology-Invariant Similarityof Nonrigid Shapes. Int. J. Comput. Vis. 81 (2009), 281 âĂŞ– 301. https://doi.org/10.1007/s11263-008-0172-2

[9] Tu Bui, Leonardo Ribeiro, Moacir Ponti, and John Collomosse. 2017. Compactdescriptors for sketch-based image retrieval using a triplet loss convolutionalneural network. Computer Vision and Image Understanding 164 (2017), 27–37.https://doi.org/10.1016/j.cviu.2017.06.007

[10] Tu Bui, Leonardo Ribeiro, Moacir Ponti, and John P. Collomosse. 2016. General-isation and Sharing in Triplet Convnets for Sketch based Visual Search. CoRRabs/1611.05301 (2016).

[11] Abdolah Chalechale, Golshah Naghdy, and Alfred Mertins. 2005. Sketch-basedimage matching Using Angular partitioning. IEEE Trans. Systems, Man, andCybernetics, Part A 35, 1 (2005), 28–41. https://doi.org/10.1109/TSMCA.2004.838464

[12] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. 2014. Return of theDevil in the Details: Delving Deep into Convolutional Nets. In British MachineVision Conference. arXiv:cs/1405.3531

[13] John P Collomosse, G McNeill, and Leon Watts. 2008. Free-hand sketch groupingfor video retrieval. https://doi.org/10.1109/ICPR.2008.4761466 19th InternationalConference on Pattern Recognition, ICPR 2008 ; Conference date: 08-12-2008Through 11-12-2008.

[14] Michael F. Deering. 1995. HoloSketch: A Virtual Reality Sketching/AnimationTool. ACM Trans. Comput.-Hum. Interact. 2, 3 (Sept. 1995), 220–238. https://doi.org/10.1145/210079.210087

[15] A. del Bimbo and P. Pala. 1997. Visual Image Retrieval by Elastic Matching ofUser Sketches. IEEE Trans. Pat. Ana. & Mach. Int. 19, 2 (February 1997), 121–132.https://doi.org/10.1109/34.574790

[16] Daniel Dixon, Manoj Prasad, and Tracy Hammond. 2010. iCanDraw: Using SketchRecognition and Corrective Feedback to Assist a User in DrawingHuman Faces. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’10). ACM, New York, NY, USA, 897–906. https://doi.org/10.1145/1753326.1753459

[17] Tomás Dorta, Gokce Kinayoglu, and Michael Hoffmann. 2016. Hyve-3D and the3D Cursor: Architectural co-design with freedom in Virtual Reality. InternationalJournal of Architectural Computing 14, 2 (2016), 87–102. https://doi.org/10.1177/1478077116638921

[18] Mathias Eitz, Kristian Hildebrand, Tamy Boubekeur, and Marc Alexa. 2009. ADescriptor for Large Scale Image Retrieval Based on Sketched Feature Lines. InProceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces andModeling (SBIM ’09). ACM, New York, NY, USA, 29–36. https://doi.org/10.1145/1572741.1572747

[19] Manuel J. Fonseca, Alfredo Ferreira, and Joaquim A. Jorge. 2004. Towards3D Modeling Using Sketches and Retrieval. In Proceedings of the First Euro-graphics Conference on Sketch-Based Interfaces and Modeling (SBM’04). Euro-graphics Association, Aire-la-Ville, Switzerland, Switzerland, 127–136. https://doi.org/10.2312/SBM/SBM04/127-136

[20] Daniele Giunchi, Stuart James, and Anthony Steed. 2018. 3D Sketching for Inter-active Model Retrieval in Virtual Reality. In Proceedings of the Joint Symposiumon Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering (Expressive ’18). ACM, New York, NY,USA, Article 1, 12 pages. https://doi.org/10.1145/3229147.3229166

[21] Rui Hu and John Collomosse. 2013. A Performance Evaluation of Gradient FieldHOG Descriptor for Sketch Based Image Retrieval. Comput. Vis. Image Underst.117, 7 (July 2013), 790–806. https://doi.org/10.1016/j.cviu.2013.02.005

[22] Rui Hu, Stuart James, and John P. Collomosse. 2012. Annotated Free-HandSketches for Video Retrieval Using Object Semantics and Motion. In Advancesin Multimedia Modeling - 18th International Conference, MMM 2012, Klagenfurt,Austria, January 4-6, 2012. Proceedings. 473–484. https://doi.org/10.1007/978-3-642-27355-1_44

[23] Stuart James and John Collomosse. 2014. Interactive Video Asset RetrievalUsing Sketched Queries. In Proceedings of the 11th European Conference on VisualMedia Production (CVMP ’14). ACM, New York, NY, USA, Article 11, 8 pages.https://doi.org/10.1145/2668904.2668940

[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNetClassification with Deep Convolutional Neural Networks. (2012), 1106–1114. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks

[25] Bo Li, Yijuan Lu, Fuqing Duan, Shuilong Dong, Yachun Fan, Lu Qian, HamidLaga, Haisheng Li, Yuxiang Li, Peng Liu, Maks Ovsjanikov, Hedi Tabia, YuxiangYe, Huanpu Yin, and Ziyu Xue. 2016. 3D Sketch-Based 3D Shape Retrieval. InEurographics Workshop on 3D Object Retrieval. The Eurographics Association.https://doi.org/10.2312/3dor.20161087

[26] Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, andLeonidas J. Guibas. 2015. Joint embeddings of shapes and images via CNNimage purification. ACM Transactions on Graphics 34, 6 (2015), 234:1–234:12.https://doi.org/10.1145/2816795.2818071

[27] Mayra D. Barrera Machuca, Paul Asente, Wolfgang Stuerzlinger, Jingwan Lu,and Byungmoon Kim. 2018. Multiplanes: Assisted Freehand VR Sketching. InProceedings of the Symposium on Spatial User Interaction (SUI ’18). ACM, NewYork, NY, USA, 36–47. https://doi.org/10.1145/3267782.3267786

[28] Robert Osada, Thomas A. Funkhouser, Bernard Chazelle, and David P. Dobkin.2002. Shape distributions. ACM Trans. Graph. 21, 4 (2002), 807–832. https://doi.org/10.1145/571647.571648

[29] Kaiyue Pang, Ke Li, Yongxin Yang, Honggang Zhang, TimothyM. Hospedales, TaoXiang, and Yi-Zhe Song. 2019. Generalising Fine-Grained Sketch-Based ImageRetrieval. In The IEEE Conference on Computer Vision and Pattern Recognition(CVPR).

[30] Yonggang Qi, Yi-Zhe Song, Honggang Zhang, and Jun Liu. 2016. Sketch-basedimage retrieval via Siamese convolutional neural network. In 2016 IEEE Interna-tional Conference on Image Processing, ICIP 2016, Phoenix, AZ, USA, September25-28, 2016. 2460–2464. https://doi.org/10.1109/ICIP.2016.7532801

[31] David Gavilan Ruiz, Suguru Saito, and Masayuki Nakajima. 2007. Sketch-to-collage. In 34. International Conference on Computer Graphics and InteractiveTechniques, SIGGRAPH 2007, San Diego, California, USA, August 5-9, 2007, Posters.35. https://dl.acm.org/citation.cfm?id=1280759

[32] Raif M. Rustamov. 2010. Robust Volumetric Shape Descriptor. In EurographicsWorkshop on 3D Object Retrieval, Norrköping, Sweden, May 2, 2010, Proceedings.1–5. https://doi.org/10.2312/3DOR/3DOR10/001-005

[33] Ryan Schmidt, Azam Khan, Gord Kurtenbach, and Karan Singh. 2009. On ExpertPerformance in 3D Curve-drawing Tasks. In Proceedings of the 6th EurographicsSymposium on Sketch-Based Interfaces and Modeling (SBIM ’09). ACM, New York,NY, USA, 133–140. https://doi.org/10.1145/1572741.1572765

[34] Eugenio Di Sciascio, G. Mingolla, and Marina Mongiello. 1999. Content-BasedImage Retrieval over the Web Using Query by Sketch and Relevance Feedback.In Visual Information and Information Systems, Third International Conference,VISUAL ’99, Amsterdam, The Netherlands, June 2-4, 1999, Proceedings. 123–130.

Page 10: Mixing realities for sketch retrieval in Virtual Reality · Sketching in Virtual Reality (VR) allows users to design and create objects in 3D virtual space. To aid users in the creation

Woodstock ’18, June 03–05, 2018, Woodstock, NY Daniele Giunchi, Stuart James, Donald Degraen, and Anthony Steed

https://doi.org/10.1007/3-540-48762-X_16[35] Pedro Manuel Antunes Sousa and Manuel J. Fonseca. 2010. Sketch-based retrieval

of drawings using spatial proximity. J. Vis. Lang. Comput. 21, 2 (2010), 69–80.https://doi.org/10.1016/j.jvlc.2009.12.001

[36] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015.Multi-view Convolutional Neural Networks for 3D Shape Recognition. In Pro-ceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)(International Conference on Computer Vision ’15). IEEE Computer Society, Wash-ington, DC, USA, 945–953. https://doi.org/10.1109/ICCV.2015.114

[37] Zsolt Szalavári and Michael Gervautz. 1997. The Personal Interaction Panel - aTwo-Handed Interface for Augmented Reality. Computer Graphics Forum 16, 3(1997), 335–346. https://doi.org/10.1111/1467-8659.00137

[38] Philipp Wacker, Oliver Nowak, Simon Voelker, and Jan Borchers. 2019. ARPen:Mid-Air Object Manipulation Techniques for a Bimanual AR System with Penand Smartphone. In Proceedings of the 2019 CHI Conference on Human Factors inComputing Systems (CHI ’19). ACM, New York, NY, USA, Article 619, 10 pages.https://doi.org/10.1145/3290605.3300849

[39] Philipp Wacker, Adrian Wagner, Simon Voelker, and Jan Borchers. 2018. PhysicalGuides: An Analysis of 3D Sketching Performance on Physical Objects in Aug-mented Reality. In Proceedings of the Symposium on Spatial User Interaction (SUI’18). ACM, New York, NY, USA, 25–35. https://doi.org/10.1145/3267782.3267788

[40] Fang Wang, Le Kang, and Yi Li. 2015. Sketch-based 3D Shape Retrieval usingConvolutional Neural Networks. CoRR abs/1504.03504 (2015). arXiv:1504.03504

http://arxiv.org/abs/1504.03504[41] Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James

Philbin, Bo Chen, and Ying Wu. 2014. Learning Fine-grained Image Similaritywith Deep Ranking. CoRR abs/1404.4661 (2014). arXiv:1404.4661 http://arxiv.org/abs/1404.4661

[42] Gerold Wesche and Hans-Peter Seidel. 2001. FreeDrawer: A Free-form SketchingSystem on the Responsive Workbench. In Proceedings of the ACM Symposium onVirtual Reality Software and Technology (VRST ’01). ACM, New York, NY, USA,167–174. https://doi.org/10.1145/505008.505041

[43] E. Wiese, J. H. Israel, A. Meyer, and S. Bongartz. 2010. Investigating the Learnabil-ity of Immersive Free-hand Sketching. In Proceedings of the Seventh Sketch-BasedInterfaces and Modeling Symposium (SBIM ’10). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 135–142. http://dl.acm.org/citation.cfm?id=1923363.1923387

[44] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xi-aoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representa-tion for volumetric shapes. In IEEE Conference on Computer Vision and Pat-tern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015. 1912–1920.https://doi.org/10.1109/CVPR.2015.7298801

[45] Yongxin Yang and Timothy M. Hospedales. 2015. Deep Neural Networks forSketch Recognition. CoRR abs/1501.07873 (2015). http://arxiv.org/abs/1501.07873