Creating Augmented Reality USDZ Files to Visualize 3D Objects onStudent Phones in the ClassroomBabak Sanii*

Keck Science Department, Claremont McKenna, Pitzer, and Scripps Colleges. 925 N. Mills Avenue, Claremont, California 91711,United States

*S Supporting Information

ABSTRACT: Augmented reality (AR) is a means of superimposing artificial 3D objectsover the real world via a mobile device. An AR standard file format has been recentlyimplemented on mobile devices that current students commonly own. Here we describethree relatively nontechnical methods to produce 3D AR objects for chemistry coursesand demonstrate their use as both quick in-lecture activities and as part of an extendedlaboratory. Preliminary in-course student assessments (32 responses out of 42 students)indicated that this technology was successfully implemented on students’ devices (100%),that it generated interest/excitement (78%), and that it helped students visualize 3Dobjects (81%).

KEYWORDS: First-Year Undergraduate/General, Interdisciplinary/Multidisciplinary, Computer-Based Learning,Hands-On Learning/Manipulatives, Molecular Modeling, Laboratory Computing/Interfacing

■ INTRODUCTION

Modes of 3D Visualization

There is evidence suggesting that the difference betweenmemorization and critical reasoning is understanding aphenomenon’s 3D structure.1 Consider for example how 3Dvisualizing enhances understanding of orbitals, moleculargeometry, chirality, and complex biomolecular docking.Although we currently lack low-cost 3D representations (e.g.,holograms), considerable progress has been made in extracting3D insight from available 2D screens. Three-dimensionalunderstanding is traditionally imparted by actively manipulat-ing the viewpoint of an object (Figure 1A). Activemanipulation allows students to closely examine areas ofconfusion and orient themselves via changes in perspective andshading. This manipulation on traditional computers is usuallyperformed by keyboard/mouse gestures that change the viewof the object on a fixed screen; on mobile devices, it is usuallyperformed by finger gestures.In virtual reality (VR) approaches students wear headsets

that block out the real world and represent the object relativeto their gaze and position, with unique images per eye invokingstereoscopic depth perception (Figure 1B). VR can alsoincorporate haptic feedback, which has been used in chemistryeducation.2 While one could readily imagine VR as a helpfulindividual learning tool, it is rarely used as part of lectures inpart because it separates students from the collective classroomexperience. Additionally, it requires significant hardware andtime to start and stop its use.

In contrast to the isolated, purely digital environment in VR,augmented reality (AR) embeds virtual content into the realworld and can be implemented in a variety of form factors thatspan from smartphones to head-mounted displays. In the caseof smartphone- and tablet-based AR, three-dimensional objectsare overlaid on a live video of the classroom (Figure 1C). LikeVR, the student gains 3D structural understanding as their gazechanges relative to an object, but via an arm’s length viewportsuch as a phone’s screen. In this configuration the instructorand classroom are not blocked out, which is arguably moredistractive than VR, but are visible and interactive in context tothe student both on their screen and in their real field of vision.Considerable progress has been made in technology toincorporate augmented reality into chemical education.Examples range from molecular representations (both static3

and spatio-dynamic4−6) to tactile virtual laboratory techni-ques7−9 and games.10 However, general barriers for ARimplementation as a quick activity in the classroom remain:(1) creating augmented reality objects is complex and (2)initiating the activity is complex (e.g., distributing hardware orphysical markers,11,12 downloading a program/app13). A recentreview by Akcayir et al. found that reported challenges for ARuse in educational settings included that AR is difficult forstudents to use, requires more time, is not suitable for largegroup settings, and is difficult to design.14 These are the

Received: June 26, 2019Revised: November 8, 2019

Technology Report

pubs.acs.org/jchemeducCite This: J. Chem. Educ. XXXX, XXX, XXX−XXX

© XXXX American Chemical Society andDivision of Chemical Education, Inc. A DOI: 10.1021/acs.jchemed.9b00577

J. Chem. Educ. XXXX, XXX, XXX−XXX

Dow

nloa

ded

via

CL

AR

EM

ON

T C

OL

G o

n N

ovem

ber

21, 2

019

at 2

0:14

:16

(UT

C).

See

http

s://p

ubs.

acs.

org/

shar

ingg

uide

lines

for

opt

ions

on

how

to le

gitim

atel

y sh

are

publ

ishe

d ar

ticle

s.

barriers we seek to address via easier object-generationworkflows incorporating Apple’s implementation of Pixar’sUniversal Screen Description (USDZ) file format, andclassroom implementation techniques. This contributionenables students to visualize 3D chemistry objects in anactivity that only takes a few moments of precious lecture time.Recent Technology for Quick 3D Chemistry Activities

Recent advances in mobile device software (e.g., Google’sARCore and Apple’s ARKit) transform devices most studentsin the United States already own15 into AR viewports. Thesemobile devices typically have cameras to capture theenvironment, accelerometers to detect device orientation,sufficient processor power to process 3D objects withenvironment-blending shading, and high-resolution screens.16

The software advances are implemented directly to theoperating system of existing mobile devices through a regular(often automatic) update and support location-based (marker-less) AR.17

As of September 2018, Apple has incorporated technology(AR Quick Look) directly into its Safari mobile browser whichenables viewing both augmented reality and less distractiveblank-background gesture manipulation of 3D “USDZ” fileswithout downloading or starting a new app. The open USDZfile format flexibly and efficiently contains 3D structuralinformation, enabling a mode of incorporating AR into lectureas a quick visualization tool. The implementation does not yethave the capability of structure manipulations or sequentialstaging, which are areas suitable for dedicated apps. Converting3D objects to USDZ is readily performed by a command-linetool which is provided freely by Apple (USDPython). Toproduce the original 3D objects we employed open-sourcesoftware products of several communities of developers (e.g.,Octave,18 PyMol,19 MeshLab,20 and Blender21).

This paper presents practices for incorporating the USDZfile format for 3D visualization into a chemistry lecture byaddressing (1) three methods to build USDZ chemistryobjects using free/open-source methods, (2) effective in-lecture use of AR, and (3) a laboratory where students producetheir own AR objects. Preliminary feedback from implementa-tion in an introductory chemistry course and laboratorysuggest it is an effective tool for 3D visualization and promotessignificant student interest.

■ GENERATING CHEMICAL USDZ OBJECTS

We present three methods to produce the USDZ chemicalobjects that students can visualize via either AR or gesturemanipulation. The methods are suited for (1) structure−function relationships in biomolecules, (2) scene creation withmultiple objects to convey physical scale, and (3) descriptionsof mathematical functions like orbitals. While these methodscreate objects differently, the three paths converge toultimately produce an “.obj” file that contains 3D structuralinformation, which is subsequently passed to Apple’s USDZfile converter through a single command-line command. Thecurrent step-by-step instructions for their production areincluded in Supporting Information.

Importing Molecules via PyMol, e.g., from the ProteinDatabase

PyMol is multiplatform open-source molecular/biomolecularvisualization software that supports multiple 3D file formats. Itcan be used to import and manipulate 3D files, to directlymanufacture 3D objects, or to import structures directly fromthe protein database.23 In order to produce the file format(.obj) that USDPython can convert into USDZ, this methodincludes the use of a separate multiplatform open-sourceprogram (MeshLab).This method is particularly helpful in biochemistry topics, as

it enables structure−function investigations through theprotein database. This method was implemented into anassociated teaching laboratory where first-year students utilizedit to produce AR objects. In our classroom we used it toexamine biomolecules such as double-stranded DNA to discussits structural stability and major/minor grooves (see Figure 2,first column).

Figure 1. Three common modes of inferring 3D structuralinformation from 2D displays. 3D object is a “crystal structure ofC/EBPbeta Bzip homodimer V285A mutant bound to a high affinityDNA fragment” (PDB 2E4222) as converted into a USDZ file by anundergraduate student as part of the course-associated laboratory(USDZ object created by C. Ylagan and used with permission).

Figure 2. Flowchart of three methods to generate USDZ files.

Journal of Chemical Education Technology Report

DOI: 10.1021/acs.jchemed.9b00577J. Chem. Educ. XXXX, XXX, XXX−XXX

B

Composing Scenes via Blender

Blender is a multiplatform open-source software package usedby the animation community. While it does not have themolecular-specific tools of PyMol by default, and is not tiedinto the Protein Database, it is a more full-featured 3Dcomposition and animation environment. The added capa-bilities dramatically steepen its learning curve, though due toits wide use there are many online tutorials available. Ourprimary use of it was to compose scenes with multiple objects,for example, demonstrating the relative size of an atom’snucleus (see Figure 2, second column).Scripting Mathematical Representations via Octave

Octave is a multiplatform open-source software package usefulfor mathematical scripting and data analysis. It has an activecommunity that freely shares scripts that can be readilycombined. For this class, we used a script that generatesmathematical representations of orbitals written by Peter vanAlem24 and paired it to a script that produces .obj files from3D data written by Anders Sandberg.25 The collective resultwas a webpage that lists electron orbitals out to n = 7 as ARobjects.26 This approach is particularly useful if one wants todepict a functional representation or 3D data (see Figure 2,third column). This is the only method we include thatrequires one to be familiar with writing code.Distributing USDZ Files

USDZ files can be distributed to students via direct methodssuch as email and AirDrop. Serving them on a Web site is alsopossible but may require minor configuration changes on theserver side to accommodate the new file format (detailed in theSupporting Information). A benefit of hosting the object on aWeb site is that one can generate 2D barcodes (QR codes)that point to the file.27 By displaying the QR code to the class,students can use the cameras on their phones to quickly accessthe file (e.g., see Figure 3). The USDZ files will be viewable asboth a manipulable 3D object and as an AR object on anydevice running iOS 12 or newer. On Apple desktops andlaptop it will load, but as a traditional manipulatable 3D object.

■ IMPLEMENTATION IN THE CLASSROOM ANDLABORATORY

Three-dimensional object visualizations via the USDZ fileformat were implemented at the Keck Science Department ofClaremont McKenna, Pitzer, and Scripps liberal arts colleges ofClaremont, California, in the Spring semester of 2019, as partof Integrated Biology and Chemistry (Chem/Bio 042L). Thiscourse is a double-credit interdisciplinary science course for 42first-year students that satisfies one semester of GeneralChemistry and one semester of Introductory Biology. The 3Dvisualizations discussed here occurred in the chemistryportions of the course.Strategies to Address Student Device Disparities

A reasonable concern with this (and any technology-driven)pedagogy is the danger of making assumptions about studentdevice access.28 Erroneous assumptions about device accesscould make a student feel that they did not belong in thatlearning environment. This is a reasonable concern given thatmodern devices incur significant financial burdens and thatbrand use could be localized (e.g., phone family plans). Ourapproach began by anonymously surveying students aboutdevice access prior to the first use in class, which revealed that17% (5 out of 29 respondents) did not regularly bring Apple

phones/iPads to class. This was consistent with a US nationalindustry survey,15 but one cannot assume each class will berepresentative, and we recommend assessing each teachingenvironment individually. We sidestepped bringing attentionto these 17% by communicating that many students’ devicesdid not support this demo yet, so we would be doing the AR ingroups of two. We also indicated that we would bringadditional devices with us, and that if anyone would like toborrow a device in advance they should reach out to theinstructors. Ultimately no extra devices were needed, andworking in groups of two more than sufficed for classroomdevice access. We conducted a survey the day after the first in-class AR use, and 100% of the 24 respondents indicated thatthe augmented reality demonstration functioned for their team.In-Classroom Implementation

The first implementation was to visualize the structure ofdouble-stranded DNA (Figure 3A). The instructor wirelesslymirrored his iPhone’s screen on the classroom projector via alaptop and additional software.29 Once students and instructorall had the object loaded, they identified geometries wherehydrogen bonding or π−π stacking interactions may occur. Asurvey sent to students the day after the first demonstrationrevealed that 100% of 23 respondents indicated that there wasvalue in the instructor mirroring their viewpoint on theprojector. Additionally, 96% indicated the AR helped themvisualize the structure of double-stranded DNA, and 65% and

Figure 3. Objects used in the Spring 2019 Integrated Biology andChemistry course as part of augmented reality activities. QRbarcodes27 in the corners lead to each USDZ file and are accessiblefrom the iPhone camera app (proteins are larger files that may takelonger to load and require a second tap). (A) An ideal double-stranded DNA. (B) A scene demonstrating the scale of an atom’snucleus. Looking “up” from the scene reveals the relatively enormoussize of a biomolecule (DNA), not shown in this view. (C) Crystalstructure of human erythrocyte catalase (PDB 1QQW32). (D) Crystalstructure of α-hemolysin from straphylococcus aureus (PDB7AHL33). QR codes resolve to corresponding USDZ objects asfound on http://faculty.kecksci.claremont.edu/bsanii/augmented-reality-for-teach.html; the files can also be found in the SupportingInformation.

Journal of Chemical Education Technology Report

DOI: 10.1021/acs.jchemed.9b00577J. Chem. Educ. XXXX, XXX, XXX−XXX

C

70% indicated it helped them visualize and understand base-pair interactions/DNA conductivity,30 respectively. 13%indicated that the AR was not more useful than a static viewof the molecule on the projector.Beyond the initial AR application discussed above, the class

also used it to explore the size of the atomic nucleus relative toelectron orbitals and biomolecules as a segue into nuclearchemistry (Figure 3B), to hunt for heme groups in the catalaseenzyme that reduces hydrogen peroxide as a segue intoelectrochemistry (Figure 3C), and to examine the porestructure of α-hemolysin when discussing next generationDNA sequencing technology (Figure 3D). In a student surveyat the end of the course, the 32 responses indicated that 81%found the quick in-class ARs helped them visualize 3D objects.78% felt it generated excitement/interest, and 3% (onerespondent) found it was not a good use of class time.When asked which method helped them better visualize 3Dobjects, 48% indicated manipulating the object with gestures,30% indicated AR, and 21% preferred seeing the instructormanipulate the object on the projector. The finding that theimplementation of AR generated excitement/interest isconsistent with increased “enjoyment” that was found in aprevious study where AR was used to teach laboratory safety.31

In-Laboratory Implementation

We dedicated a teaching laboratory session for students tobecome familiar with 3D manipulations (see SI). In this“experiment” students created and manipulated a fictitiousmolecule in PyMol, viewed a molecule of their choice from theprotein database, and finally generated an AR file in the firstmethod outlined in Figure 2. In a student survey at the end ofthe course with 32 responses, 22% indicated that they foundthe software too complicated to be helpful, but that 88%indicated it helped them either visualize or gain confidence inmanipulating 3D objects.

■ CONCLUSION AND OUTLOOK

Here we developed practices to generate and implement in-class 3D chemical visualizations on student mobile phones,both by gesture manipulation and augmented reality. Theimplementation of the USDZ file format natively in Apple’sSafari web browser enabled rapid deployment to the class,without the need to download an app or distribute hardware.This implementation is currently limited to iOS devices whichare currently popular with many student environments, but thefuture landscape of the AR format (including open cross-platform standards such as OpenXR) is difficult to predict. Wepresent three open-source methods to produce the USDZ files:using Pymol for biomolecules, Blender for comparingstructures, and Octave for electron orbitals. We also presentinclusive practices and preliminary assessments of in-class andin-laboratory use of this technology. Notably the implementa-tions we describe support both augmented reality (whichstudents indicated generates significant interest/excitement)and object manipulation by gestures (which students foundmore helpful in 3D visualization).Looking forward, 84% of students indicated that AR would

be more helpful if it were in color. This capability exists in theUSDZ file format, but the pathways to its implementation inour three pipelines involved individual text-editing of files andso are omitted here. Implementing color may become eveneasier in the near future.34 44% of students indicated the ARwould be more helpful if it were animated, which is also an

existing capability of the USDZ file format and is readilyimplementable in the Blender generation method. Aspiration-ally, we are aiming for an environment where students could allview the same augmented reality object in the classroom whilethe instructor manipulates its structure to demonstratedynamic chemical/biochemical phenomena.

■ ASSOCIATED CONTENT*S Supporting Information

The Supporting Information is available at https://pubs.ac-s.org/doi/10.1021/acs.jchemed.9b00577.

Specific protocols for making AR objects using the threemethods outlined above (PyMol, Blender, and Octave)and brief laboratory materials for students to create theirown objects (PDF, DOCX)USDZ AR objects portrayed in Figure 3 and Octavecode to produce AR objects of electron orbitals (ZIP)

■ AUTHOR INFORMATIONCorresponding Author

*E-mail: bsanii@kecksci.claremont.edu.ORCID

Babak Sanii: 0000-0002-8265-5988Notes

The author declares no competing financial interest.

■ ACKNOWLEDGMENTSThe author gratefully acknowledges input from R. Held, K.Black, research students A. DeShazo, J. Juarez, I. Lopez, B.Louie, L. Trihy, coinstructors E. Jones and C. Purser, and thestudents of the 2019 Integrated Biology and Chemistry course,particularly N. Gutierrez-Cazarez, J. Lahr, and A. Xue.

■ REFERENCES(1) Talley, L. H. The use of three-dimensional visualization as amoderator in the higher cognitive learning of concepts in college levelchemistry. J. Res. Sci. Teach. 1973, 10, 263−269.(2) Edwards, B. I.; Bielawski, K. S.; Prada, R.; Cheok, A. D. Hapticvirtual reality and immersive learning for enhanced organic chemistryinstruction. Virtual Reality 2019, 23, 363−373.(3) Berson, M.; Ng, D.; Shin, J.; Jenkinson, J. Assessing augmentedreality in helping undergraduate students to integrate 2D and 3Drepresentations of stereochemistry. J. Biocommun. 2018, 42, 42.(4) Gillet, A.; Sanner, M.; Stoffler, D.; Goodsell, D.; Olson, A.Augmented reality with tangible auto-fabricated models for molecularbiology applications. IEEE Visualization 2004; IEEE, 2004; pp 235−241.(5) Maier, P.; Tonnis, M.; Klinker, G. Augmented Reality forteaching spatial relations. Conference of the International Journal of Arts& Sciences; Toronto, 2009; pp 1−8.(6) Plunkett, K. N. A Simple and Practical Method for IncorporatingAugmented Reality into the Classroom and Laboratory. J. Chem. Educ.2019, 96, 2628.(7) Wildan, A.; Cheong, B. H.; Xiao, K.; Liew, O. W.; Ng, T. W.Growth measurement of surface colonies of bacteria using augmentedreality. J. Biol. Educ. 2019, 1−14.(8) Gan, H. S.; Tee, N. Y. K.; Bin Mamtaz, M. R.; Xiao, K.; Cheong,B. H.; Liew, O. W.; Ng, T. W. Augmented reality experimentation onoxygen gas generation from hydrogen peroxide and bleach reaction.Biochem. Mol. Biol. Educ. 2018, 46, 245−252.(9) Tee, N. Y. K.; Gan, H. S.; Li, J.; Cheong, B. H.; Tan, H. Y.; Liew,O. W.; Ng, T. W. Developing and demonstrating an augmented realitycolorimetric titration tool. J. Chem. Educ. 2018, 95, 393−399.

Journal of Chemical Education Technology Report

DOI: 10.1021/acs.jchemed.9b00577J. Chem. Educ. XXXX, XXX, XXX−XXX

D

(10) Boletsis, C.; McCallum, S. The table mystery: An augmentedreality collaborative game for chemistry education. InternationalConference on Serious Games Development and Applications; Springer,2013; pp 86−95.(11) Yang, S.; Mei, B.; Yue, X. Mobile Augmented Reality AssistedChemical Education: Insights from Elements 4D. J. Chem. Educ. 2018,95, 1060−1062.(12) Cai, S.; Wang, X.; Chiang, F. A case study of AugmentedReality simulation system application in a chemistry course. Comput.Hum. Behav. 2014, 37, 31−40.(13) Alchemie Solutions, Inc. ModelAR iPhone App; https://itunes.apple.com/us/app/modelar-organic-chemistry/id1438760201?mt=8(accessed 11/07/2019).(14) Akcayır, M.; Akcayır, G. Advantages and challenges associatedwith augmented reality for education: A systematic review of theliterature. Educational Research Review 2017, 20, 1−11.(15) Teen Survey: iPhone Ownership Up, With ‘Intent to Buy’ Flat.Piper Jaffray Survey; http://www.piperjaffray.com/private/pdf/TSWT_Survey_Fall_2019.pdf (accessed 11/07/2019).(16) Williams, A. J.; Pence, H. E. Smart phones, a powerful tool inthe chemistry classroom. J. Chem. Educ. 2011, 88, 683−686.(17) Cheng, K.; Tsai, C. Affordances of augmented reality in sciencelearning: Suggestions for future research. J. Sci. Educ. Technol. 2013,22, 449−462.(18) Eaton, J. W.; Bateman, D.; Hauber, S.; Wehbring, R. GNUOctave version 4.2.2 Manual: A High-Level Interactive Language forNumerical Computations; 2018; Vol. 4.2.2.(19) De Lano, W. L. The PyMOL Molecular Graphics System, Version1.2 r3pre; Schrodinger, LLC, 2002.(20) Cignoni, P.; Callieri, M.; Corsini, M.; Dellepiane, M.;Ganovelli, F.; Ranzuglia, G. Meshlab: an open-source mesh processingtool. Eurographics Italian Chapter Conference 2008, 2008, 129−136.(21) Blender Online Community. BlenderA 3D Modelling andRendering Package; https://www.blender.org (accessed 11/07/2019).(22) Tahirov, T. H.; Inoue-Bungo, T.; Sato, K.; Shiina, M.; Hamada,K.; Ogata, K. PDB ID: 2E42. Structural Basis for Flexible BaseRecognition by C/Ebpbeta. https://www.rcsb.org/structure/2E42(accessed 11/07/2019).(23) Berman, H. M.; Westbrook, J.; Feng, Z.; Gilliland, G.; Bhat, T.N.; Weissig, H.; Shindyalov, I. N.; Bourne, P. E. The protein databank. Nucleic Acids Res. 2000, 28, 235−242.(24) van Alem, P. Hydrogen Orbitals. https://www.mathworks.com/matlabcentral/fileexchange/64274-hydrogen-orbitals (accessed11/07/2019).(25) Sandberg, A. vertface2obj.m. http://www.aleph.se/Nada/Ray/matlabobj.html (accessed 11/07/2019).(26) Sanii, B. Augmented Reality for Teaching Chemistry. http://faculty.kecksci.claremont.edu/bsanii/augmented-reality-for-teach.html (accessed 11/07/2019).(27) Wolle, P.; Muller, M. P.; Rauh, D. Augmented reality inscientific publicationstaking the visualization of 3D structures tothe next level. ACS Chem. Biol. 2018, 13, 496−499.(28) Arvanitis, T. N.; Petrou, A.; Knight, J. F.; Savas, S.; Sotiriou, S.;Gargalakos, M.; Gialouri, E. Human factors and qualitativepedagogical evaluation of a mobile augmented reality system forscience education used by learners with physical disabilities. Personaland ubiquitous computing 2009, 13, 243−250.(29) AirServer Software. http://www.airserver.com (accessed 11/07/2019).(30) Slinker, J. D.; Muren, N. B.; Renfrew, S. E.; Barton, J. K. DNAcharge transport over 34 nm. Nat. Chem. 2011, 3, 228.(31) Zhu, B.; Feng, M.; Lowe, H.; Kesselman, J.; Harrison, L.;Dempski, R. E. Increasing Enthusiasm and Enhancing Learning forBiochemistry-Laboratory Safety with an Augmented-Reality Program.J. Chem. Educ. 2018, 95, 1747−1754.(32) Ko, T.; Safo, M. K.; Musayev, F. N.; Di Salvo, M. L.; Wang, C.;Wu, S.; Abraham, D. J. Structure of human erythrocyte catalase. ActaCrystallogr., Sect. D: Biol. Crystallogr. 2000, 56, 241−245.

(33) Song, L.; Hobaugh, M. R.; Shustak, C.; Cheley, S.; Bayley, H.;Gouaux, J. E. Structure of staphylococcal α-hemolysin, a heptamerictransmembrane pore. Science 1996, 274, 1859−1865.(34) Apple Reality Composer. https://developer.apple.com/augmented-reality/reality-composer/(accessed 11/07/2019).

Journal of Chemical Education Technology Report

DOI: 10.1021/acs.jchemed.9b00577J. Chem. Educ. XXXX, XXX, XXX−XXX

E