AUGMENTED REALITY BASED

PRODUCT IDENTIFICATION

AND ADVERTISING SYSTEM

- UDARA ABHILASH ALWIS (CB003982)

INTRODUCTION

PROBLEM BACKGROUND

There are huge product varieties in the market and advertising has

a great impact in today’s life. Customers are always looking for

extended information when they buy products or even regarding

an advertisement, whereas it is very crucial to provide users with

updated and accurate information regarding them. QR Codes are

being deprecated in modern end consumerism due to its

disadvantages and real time visual recognitions technologies are

coming up with powerful mobile devices. Augmented Reality is

becoming the next mass media advertising in today’s world.

Problem Statement

PROBLEM BACKGROUND

• Product information could change anytime

• Advertisements can not display all the information at once

• Customers are looking for more extended information

• No proper method to look for information, time waste

• Augmented Reality as the next mass advertising media

• Extinction of QR Codes from end user consumerism

PROBLEM OBJECTIVES

• Identify Product or Advertisement

• View the extended information, and related advertisements or

products

• Implement Augmented Reality based concepts in the

application

• Insert Products and Advertisements information as

Administrators

• Simulate Product or Advertisement Identification process in

admin system.

SOLUTION OUTLINE

• Capture image when the user points at a product or an

advertisement.

• Identify the product or the advertisement from the server side.

• Provide extended information about the product or the

advertisement to the user.

• Present the information in according to Augmented Reality

principles.

• Allow admins to add new information to the System.

MAIN CHALLENGES

• Implementing Augmented Reality into the Mobile Application.

• Getting the identification results from the server from the

fastest way possible.

• Implementing an Image Matching Algorithms.

• Identifying a product or an advertisement by its Brand Logo.

• Adding and maintaining Product and Advertisement

information

RESEARCH

RESEARCH

• User Requirements Investigation

• Customers

• Difficulty of Finding information

• Having to wait for a Salesperson to show up

• QR Codes, useless

• A New Experience

• Advertisers

• The search for new creative advertising strategies

• Modern Adverts focus simplicity, therefore less information

• Visual Recognition over QR Code Recognition

RESEARCH

• Similar Systems Study

• Aurasma

• Vuforia Qualcomm

• Layar

• The IBM Augmented Reality Personal

Shopping Assist

• Conclusions

• Used for Advertising and real time

Image target identification

• Only available for US and Europe

based countries

• Cloud technologies

• No Local implementation

RESEARCH

• Augmented Reality

• Mobile Augmented Reality

• Mobile Application Development Platform

• Apple

• Android

• Windows Phone

• Mobile Video Stream Frame Monitoring

• Image Processing

RESEARCH

• Image Enhancements

• Spatial Domain Enchantement

• Image Negatives

• Contrast Stretching

• Histogram Processing

• Frequency Domain Enchantement

• Smoothing Frequency-Domain Filters

• Low pass Filtering

• High pass Filtering

• Computer Vision

RESEARCH

• Feature Extraction

• SIFT - Scale Invariant Feature Transforms

• SURF – Speeded Up Robust Features

• Matching

• 3D Object Detection in SURF

• FAST - Corner Detection

• FREAK - Fast Retina Key point Detection

• BRISK - Binary Robust Invariant Scalable Key-points

• Character Recognition and Reading

RESEARCH

• Selection of APIs and Tools

• Image Processing and Image Matching APIs

• OpenCV

• MatLab

• Simple CV

• Character Recognition APIs

• Tesseract

• ABBY OCR

RESEARCH

• Augmented Realty APIs and SDKs

• Vuforia

• FastCV

• Aurasma

• Layar

DEVELOPMENT METHODOLOGY

Feature Driven Development

• Optimized for Scalable Systems

• Executes on iterative development

cycles

• Suited for time intensive application

• Features list that is being built at the

beginning of the development method

cycle

• Feature by Feature

• Each feature gets much better focus

ANALYSIS

• Augmented Realty APIs and SDKs

• Mobile Augmented Reality

• Mobile Application Development Platform

• Apple

• Android

• Windows Phone

• Mobile Video Stream Frame Monitoring

• Image Processing

RESEARCH

RESEARCH

• Similar Systems Study Analysis

• Augmented Reality

• Image Processing and Recognition

• Cloud Computing

• User Experience

• Availability for 3rd Parties

• Not available for 3rd World Countries

• No actual implementation in Sri Lanka (Experimental)

RESEARCH

• Mobile Application Development Analysis – Android

• Mobile Platform Usage

• Developer Support and Features

• Multitasking and Heavy Weight Processing Support

• Image Processing

• User Friendliness and UI Features

RESEARCH

• Computer Vision Analysis

• Robust

• Faster

• High Accuracy

• Feature Extraction Analysis - SIFT, SURF, FREAK and BRISK

• SURF

• Speed and Accuracy

• Implementation Support

REQUIREMENT SPECIFICATION

REQUIREMENT SPECIFICATION

• Functional Requirements

• Ability to identify Products or Advertisements

• Augmented Reality should be implemented

• Provide Related Products and Advertisements

• Automatically Sort and categorize related Products and

Advertisements

REQUIREMENT SPECIFICATION

• Non-functional requirements

• User friendly and easy to use Interface.

• An Interface suitable for Augmented Reality based UI designs.

• Mobile application should be smooth and quickly responsive.

• Recognize Products and advertisements from the fastest way

possible.

DESIGN

DESIGN

• Use case – Mobile Application

DESIGN

• Use case – Admin Server System

DESIGN • Class Diagram – Admin Server System

DESIGN • Activity Diagram – SURF Image Matching

DESIGN • Sequence Diagram – Brand Logo Identification

DESIGN • Sequence Diagram – Product Identification

DESIGN • Sequence Diagram – Mobile Application

IMPLEMENTATION

• Image matching using SURF

• Character Recognition and key words identification

• Mobile Application UI development

• Mobile Application Backend

• Hosting a Web Service

IMPLEMENTATION

• SURF Feature Extraction and Matching

• Extract Feature Points and Descriptors

• Brute Force Match and Vote for Feature Point Matches

IMPLEMENTATION

• Further Voting and Filtering

• Draw Matching Feature Points and Return Results

IMPLEMENTATION

• SURF Feature Extraction and Matching Sample

IMPLEMENTATION

• Experimentations

• In need of Experimentation and Modifications

• Could use “mask” Matrix

• Started to give even greater matched points count of

incorrect images

IMPLEMENTATION

• Experimentation Conclusions

• Get the number of matched feature points count

• Calculate the Area of the formed Polygon

• Identify whether the formed Polygon is a proper Polygon

• Calculate the percentage Ratio of the Detected Key points of

both images

• Adjust the Uniqueness threshold

IMPLEMENTATION

• Determining Best Match

IMPLEMENTATION

• Threshold of an Image Matching Process

• Image Pre-Processing Usage

• Character Recognition and key words identification

IMPLEMENTATION

• Mobile Image Processing

• Image Resizing and Scaling

IMPLEMENTATION

• Image Encoding and Decoding – Base64

IMPLEMENTATION

• Web Service Connectivity

IMPLEMENTATION

• Hosting a Web Service

• Runtime Exception – Heavy EmguCV Libraries

• Cv.Invoke Exceptions

• Server Out Of Memory Exceptions

IMPLEMENTATION

• Dummy Web Service

IMPLEMENTATION

• Overall Admin System Implementation

• Object Oriented Methodologies

• Surf Execution Class

IMPLEMENTATION

• Product Identification and Advertisement Identification Process

• Find the Brand Logo Information

• Load the Product Images under that Brand Logo

• Take the given image through that Product Images list to find

the best match and retrieve information.

TESTING

• Unit Testing

• Functional Testing

• Performance Testing

TESTING

• Unit Testing - Server System

Test Case 1 Test Name : Recognize Given Product Image’s Brand Logo at Server Simulation

Test Criteria Check if the Given Product’s Brand type is identified.

Input / Action Performed Drag and Drop a Product Image and Click on “Search and Detect”

Expected Results System should identify the brand logo and Display the Brand Information

Actual Results The System successfully identified the Brand Type and showed the information of the

Brand.

Test Case 2 Test Name : Recognize Given Product Image at Server Simulation

Test Criteria Check if the given Product’s front view Image is Recognized and Show the information

of the Product

Input / Action Performed Drag and drop Product Image and click on “Search and Detect” button.

Expected Results System should recognize the Product and show information about it as well as Related

Products.

Actual Results Product was Successfully Recognized and information were showed.

TESTING

• Unit Testing - Server System

Test Case 3 Test Name : Determine when a given Image of a Product or an Advertisement cannot be

Identified

Test Criteria Check if the given Threshold condition is able to determine whether a given image can

be Recognized or not.

Input / Action Performed Insert a random unrelated Product/advertisement Image to the Matching Algorithm and

execute matching algorithm

Expected Results The algorithm should determine that the given Image cannot be Recognized based on

the Threshold condition.

Actual Results The Matching process successfully determines the given Image cannot be identified.

TESTING

• Unit Testing – Server System

Test Case 4 Test Name : Automatically Recognize the Brand Type of the Product/Advert image when

adding a new Product/Advert info

Test Criteria Check if the System is able to Automatically Recognize the Brand Type of a given

Product or Advertisement Image when Inserting their information.

Input / Action Performed Drag and drop the Product/Advert Image and click on “Auto Detect”

Expected Results The matching algorithm will detect the Brand logo of the Product/Advert Image and

identify the Brand.

Actual Results The system successfully identifies the Brand Type by detecting the brand logo.

TESTING

• Unit Testing – Mobile Application

Test Case 5 Test Name : Retrieve Recognition results from the Web Service

Test Criteria Check if the Application is able to retrieve the identification result of a capture image

Input / Action Performed Click on “Scan and Detect” button

Expected Results After sending the image to server application should wait for the response and

retrieve the results.

Actual Results The Application send the image to the server and waits till the response is being

given, while the other UI Controls are being disabled. When the results are retrieved,

they are being displayed successfully.

TESTING

• Performance Testing

Case Expected Accuracy Actual Result

Capturing Image and Sending to Web

service

100% 100%

Retrieving Product/Advert Recognition

Results

100% 100%

Capturing product/advert image from

a different angle and recognizing it

100% 50%

Overall Recognition Results 90% 70%

𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑅𝑒𝑠𝑢𝑙𝑡𝑠

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑇𝑒𝑠𝑡𝑠× 100%

CRITICAL EVALUATION

• Efficiency

• Execute all the heavy processing in Server

• Holding the Pre-Processed – SURF Features Extracted Images in the Memory

• Mobile Application to Server Communication – Data compression and Base64 Encoding

• Utilizing the SURF based Image Matching process to achieve more accuracy and Speed

• Utilizing the Recognition Process

• Automated Brand Logo detection Process

CRITICAL EVALUATION

• Ease of Use

• Accuracy

• Increased User Satisfaction

• Maintainability and Extendibility

• The Centralized Database

• Web Service Implementation

CRITICAL EVALUATION

• System Utility and Usability

• Consumers, Advertisers and Marketers

• Providing Access for Extended information

• Information are updated and centralized

• implement creative strategies into Augmented Reality

• intuitive way to recognize products for end users rather than QR Codes

• easily upgraded and extended due to the architecture

• The interface has been designed in a very minimalistic, simple, and user friendly

CRITICAL EVALUATION

• Further Enhancements

• Further Developing the Image Matching Process from ground

up to use even more efficient Algorithms such as FREAK or

BRISK Feature Extraction algorithms

• Adopt an Augmented Reality API such as Vuforia

• High end Cloud Service which will radically increase the

processing power.

• Adopt a new lightweight API Library for Image Processing

DEMONSTRATION

Q & A

THANK YOU !