markus paasovaara: face recognition

Face

recognition_

The state of the art

and how we applied it

at Finnair

• Face Detection & Tracking

• Head pose and eye-gaze estimation

• Emotion recognition

• 2D & 3D Face recognition

• Face Morphing

All with real-time performance!

Background & State of the Art_

3D Face Recognition_

Source: Ayonix.com

From face landmarks

into a 3D model and

feature vector

Terminology:

• Match Rate

• False Negative

• False Positive

• Security & Surveillance

• AI & Robotics

• Consumer behavior

• Smart Shopping

• Convenience

Business Cases_

Domain specific

problem: Human-made

Identification_+ Interaction with another human is

natural

+ Reacting to extraordinary situations is

much easier.

- We are vulnerable to tiredness and

biases.

Fixed camera

Identification_

+ Precise, no biases or overload

+ Works 24/7

- Lacks human interaction, no room for

improvising

What has been figured

TECH

SOFTWARE

HARDWARE

Current State

Analysis_

Walk in experience

What did we do differently?

strength in pattern-based classification with

strength in making choices

O U R A P P R O A C H

0 1

An invisiblewalk-paced identification

focusing on human interaction & convenience

S E R V I C E V I S I O N

Registration

App_

Premium Passengers of Finnair will

register themselves at home using the

Android App deployed to their phones.

First digital touchpoint

Arrive to the airport_

Embedded into the normal UX of the

passenger.

Check in desk 206

interface_

The “Hands in the pocket” experience

for the platinum and gold passengers

of Finnair at the priority check in desk

Second digital touchpoint

Technical Architecture

OVERVIEW

• PoC system architecture is separated into two clients and one backend server

• Android application is used to register new people to the system

• Airport Client Application is used for detecting faces from camera stream and sending those for recognition.

Recognize

Backend

Processes the recognitions and

stores the data

Android

Airport Client Application

• Detects faces in camera stream

• Sends facial images for recognition

• Visualizes results

• Gathers feedback from the agent

Register

How did we do?

R E S U L T S

0 2

Passenger Feedback

“It's amazing that new

technologies are being

sought and tested actively”

4.6/5Overall experience rating

4.75/5Ease of use of the App

4.1/5Not worried about privacy

01_Check in

attendants

"The app is easy to use,

fast and reliable. Would like

to use it in the future,

especially if it is integrated

to our systems.”

- Tiia, Check in agent

Publicity Overview

Our proof of concept has been featured in

various travel and aviation magazines

worldwide.

Big national and international media outlets

have featured the Proof of Concept such as

AP, HS and SVD.

Ten interview requests including big

technology and aviation magazines, including

PTW, ATW and ZD NET.

One more little thing..

03

Privacy_

• GRPD: Penalties up to 4% Revenue / 20MEur

• Scope: “Personal data is any information relating to an

individual, whether it relates to his or her private, professional

or public life. It can be anything from a name, a home address,

a photo, an email address, bank details, posts on social

networking websites, medical information, or a computer’s IP

address.“ Source: European Commission press release

• Privacy by design, Consent, Encryption,

Pseudonymisation, Right to be forgotten, …

What we learnt

04

Control the illumination

(as much as you can)

Study your

environment and users

Triggers can help to

achieve “real-time”

Control the number &

quality of the images

in the database

• Registration

• Self-adjusting

Test the algorithm

Test the code

Test the UX!!

Key Insights_

•We can do this in real time, in real world environment

•You can use an app for sourcing the images, but free form facial image sourcing does not work.

•Equal illumination on both side of the face is crucial

•Minimum 3 photos with the illumination variation in the database is a requirement

•Privacy is your major concern, not your customers

•We need to compromise from zero UI approach

• If you are wearing glasses, we have a problem

Privacy_

Who cares?

• Data controller, Data processor

• System developers?

• End users, You and me?

How to “protect” yourself and your client?

• Privacy by Design

• Know the rules

• Know what you’re doing

• Don’t store ANY data you don’t need

Thank You!