SEE: Synthetic Environment Enhancement for Unity

Vaughn Whisker, PhD

Synthetic Environment Applications Lab

Penn State Applied Research Lab

vew101@arl.psu.edu

SOFTWERX Future Directions Forum

Unity Game Engine

• Unity is a commercially-available, “flexible and high-performance end-to-end development platform used to create rich interactive 2D, 3D, VR and AR experiences.”

• Personal, plus, and professional versions available.• Supports mobile platforms, PC-platforms,

embedded systems, and home-entertainment systems (XBox, PS4, etc)

• Multi-player framework for collaboration• Asset store with diverse first and 3rd party-

developed capabilities.• “5.5 million registered developers including large

publishers, indie studios, students and hobbyists around the globe”

Source: Unity3d.com

DIRE: Dynamic Immersive Runtime Environment• Penn State ARL-developed framework quick configuration of Unity

applications• Enhances application support for:

• VR HMDs: Rift, Vive, Hololens, Meta2, etc.• Xbox Gamepad, Oculus Touch Controllers, Vive Wands

• Stereo Displays• Motion Tracking Systems (ART, Optitrack, etc via VRPN or trackD)• Multi-screen displays

• Tools/Interactions: Compass, Query, Measuring Tape, Grab• Additional ARL-developed Modules:

• Data Processing – Read and interpret input files, create data maps, create simple sonifications

• Satellite – Generate and animate satellite orbits from TLEs• Characters – Adds support for quickly populating a scene with virtual people• Geodesy - Adds basic geographical support to Unity via Coordinate system

projections (Flat Earth and UTM) and geodetic component that allows objects to be placed with lat/lon

Synthetic Environment Enhancement

• UI Features• Terrain Import• Geospatial Feature Support• Geospatial Database

Integration• CSV/Excel Import Support• Location Data Ingest• Transit Data Ingest

User Interface Features

• Additional Tools• Range Finder, Highlight, Attach, Draw, Marker,

Waypoint Finder• MenuUI

• Mimics hand-held tablet interface• Support for Tools, Bookmarks, Map, Routes

• MegaLabels• Large labels to provide general situational

awareness • Visible only above a user-defined altitude

• Compass Bar• Static and dynamic object locations• Icons w/ Text Labels• Cardinal directions

Terrain Model Ingest• Automated import of photogrammetry mesh

tiles• Assigns collider and changes shader to unlit/texture

• Import commercial city models • Initially FBX with textures, > 10K objects• Worked with vendor to develop unitypackage

delivery• Automated runtime import of pre-built Unity

Asset Bundles• Investigating automating import of:

• Multi-LOD Collada models• FBX tiles

Geospatial Feature Support• Coordinate system/projection

support – UTM and Flat-earth• Place objects using Lat-Lon-alt vs x,y,z

• Shapefile rendering • Shapefile attribute display for

buffered regions• e.g. Display building names, road

names, language spoken, crime rate, etc

• Shapefile-based data management• Load/unload high-res and low-res

versions in a region• Open Layers web-based map

• Online or cached map tiles• Feature layers

Neighborhoods

Named Buildings

Streets

Shapefile Sources

Geospatial Data Integration with Unity• Unity + backend data/geospatial service = “virtual location services”

• Application reports user’s location to the server• Unity web requests enable query of nearby data to be displayed in 3D environment• User position and nearby data shown on virtual handheld map • System designed to support eventual AR application

Content Interaction

Places

Data/Analytics Service

Map Tiles Analytics

Data Integration

Multiple options for integrating external geospatial data into simulation

https://xxx.xxx.xxx

Data/Analytics Service

127.0.0.1

Data/Analytics Service

Online Server Standalone w/ Locally Cached Data

In-applicationFile Access

CSV/Excel Import Support

• CSV-based model placement• Place and ground-clamp models or

labels from file with name/lat/lonformat

• CSV-based foliage management/placement

• Support for multiple tree types –map attribute to model

• Load only trees around user within user-defined distance

Location Data Ingest• Cached online sources• Open Street Map Binary (*.PBF)• User-defined data

• Web pages for data entry• Supported Location types:

• Business, Education, Entertainment, Entity, Financial, Government Building, Healthcare, Miscellaneous, Parks and Recreation, Place of Worship, Residential, Shop, Sustenance, Transportation, Unknown, plus other application specific icons.

Transit data support• GTFS: Google Transit Feed Specification

(https://developers.google.com/transit/gtfs/) • Transit authorities worldwide publish GTFS data• Spec supports stop locations, routes, trip times,

calendars, fare attributes for a transit agency’s bus and rail services

• Published as a zip file

• SEE GTFS support• Schedule display at bus stops• Animation of bus routes

• Bus Model• Icons

SEE: Summary

Transit Info: Local Transit Agency

Place Info:OpenStreetMapOnline DatabasesUser InputOthers

Trees: Locality’s Geospatial DatabaseCyclomediaNDVI Technique

Buildings:Commercial VendorsLocal Geospatial Database

MiniMap:OpenStreetMapESRIMapnik/Leaflet

Text Overlays:Neighborhood NameBuilding NameApproximate Street AddressGPS PositionNearest IntersectionCardinal Directions

Compass Bar:Cardinal DirectionNearby Locations

Challenges and Lessons Learned • Data Wrangling

• Tradeoffs of using Commercial, Open Source, Govt. sources• Missing/incomplete data

• Filtering/Curating Data• Crowdsourced sources must be filtered significantly to remove irrelevant/extraneous data

• Volume of Data vs Visual Fidelity• Limit range of data being queried• Manage high-resolution data visibility to only show nearby data

• Insufficient Spatial Resolution• Challenge: Some placemarks appear inside of buildings, Trees in/on buildings• Solution: Use GIS tools/methods to adjust source data to make them visible

• Transportation Data Update Interval• GTFS schedule has widespread support from numerous transit agencies worldwide• Challenge: “Real-time” telemetry data not updated frequently enough for real-time simulation• Solution: Use supplied schedule and route data to interpolate tracks.

• How do we make the information more consumable?• Allow user to select relevant sources, user defines relevancy• Use filters or layer managers• Use familiar modalities for delivering information – phone/tablet, virtual GPS, etc.

Questions?