from 2d map to mobile 3d mirror world

From 2D Map to Mobile 3D Mirror World

Yu You, David MurphyMixed Reality SolutionsNRC, Tampere, Finland

A Live Virtual Advertising Use Case

Nokia Research Center

Mobile 3D Mirror World•Mirror world (MR) is “informationally-enhanced virtual models or ‘reflections’ of the physical world” [2].

• Provides increased precision of an added dimension with more lively and immersive UX

• A nice complement to Augmented Reality (AR), which is commonly used in real-time and on-line situation

• Mobile MR can take advantage of mobile AR where mobile sensors and other positioning and tracking technologies are used


Motivation

•study the mobile advertising scenarios from a narrow subject (street-level mirror world) and try to answer an immediate question: how and where to publish the advertisement content?

•This paper focuses on:• Accurate but simple content automatic placement• Mobile content creation and visualization


•Data set• 360 degree panoramic street-level imagery• 3D building models with global geo. coordinates• Ground-terrain 3D mesh data

•Technology• OpenGL-based client renders the building models and ground

around the chosen panoramic image onto which the panoramic image is projectedA mobile client

•Video 1: Nokia CityScene

Experiment setup


The system in a nutshell

•Web-centric end-to-end setup with RESTful Web APIs for ad. content management and mobile 3D mirror world client

• WGS84 geo. boundary or proximity queries

• KML/KMZ is the main data representation format

• 3D objects are modeled separated in COLLADA files


Scalability consideration

• Unlike ad. content data, which is rather dynamic, any other static data are stored in different servers, i.e. provided by Content Delivery Network (CDN)

• Typically CDNs does not offer geo. query functionality • Each 3D data has a geo. coordinate and then its geo-hash is

calculated (e.g. we use quad-key method) at a fixed resolution (zooming) level (i.e. 17, appr. 200m2)

• Therefore an index is created for all data in that geo. boundary and stored in the file system using a directory naming convention, which is correspondent to its geo-hash value. So does the real data.

• Client needs to fetch the metadata (index) files at one given geo. coordinate and then all 3D data.


Mobile 2D map-based content planning• Two types of ad. content representation

1. Static object like billboards standing on one location

2. Mobile 3D object along the vehicle driving road• 2D coordinates are needed only; the altitude is calculated

from the ground terrain mesh• Problems

• Hard to position content accurately to the buildings, not to mention the façade

• That reflects again the common problem in other systems where geo-tagged data (e.g. point of interest) are misplaced in the middle of streets; or even worse - on the wrong side of the street

1

2


3D content manual placement

• High-value campaigns requires pixel-level accuracy

• Developed a on-device authoring tool for non-expert users with following few steps

• Choose the appropriate building façade• Drawing the size of the bounding box as the

content placement or drawing canvas • Saving the drawing or inserted content with

metadata (e.g. the normal to the façade and bounding box data in the 3D)

• The content placement layer can be used to render any media data properly in 3D scene

• However, manual placement is time-consuming and not scalable


3D content heuristic/auto placement

• Auto alignment from only 2D coordinates• Select the closest building model (the center of mass of the

surrounding models)• Determine the intersected façade by casting a light-ray to the model

from a predefined view point• External media like banner images are converted to SVG format to

scaling of the scene (e.g. zooming in/out)


Video 2

Moving box for parking info


Findings and open issues• Alternative ad. placement (e.g. façade vs. ground)

• Placing banners altered the appearance of the building facade, and thus deteriorated the representational value of the mirror world

• Lighting, visual improvement• Illumination enhancement e.g. artificial lighting and content blending with

proper shedder, but lighting blobs on facades and tree shadows are difficult to be segmented

• Proper ad. content selection for moving artifacts• Fixed geo. location as the context wont always work for mobile artifacts

• Study more metrics concerning mobile/3D-specific user actions for analytical processing

• Navigation, messaging etc.• Zooming and panning

• Various pricing strategies vs. moving speed and frequencies • Mobile Web app. vs. native application

• One solution for all OS


Summary

•Bringing ad. content to 3D Mirror World allows innovative types of advertising, coming along with new challenges

•We’ve proposed two basic types of ad. approaches but expect more realistic representations to appear