From Human Machine Interactionto Human Machine Integration and Intelligence

Le Thanh Ha, Ph.DAssoc. Prof. at University of Engineering and Technology,

Vietnam National Universityltha@vnu.edu.vn; lthavnu@gmail.com; +84983 692 592

About myself

• Full name: Le Thanh Ha

• 2005-2010: Ph.D at Korea University, Korea

• 2010-now:

– Assoc. Prof. at University of Engineering and Technology (UET), VNUH

– Head of HMI Laboratory

• Expertise: Computer vision, Image/video processing and analysis, Machine learning

5/14/2018 2Le Thanh Ha, HMI Laboratory

HMI Laboratory

IntelligenceIn

terf

ace

Standardization

Standardization

Standardization

ISO/IEC JTC 1 Information Technology

Interaction

Integration

Intelligence

Inte

rfac

e https://hmiuet.wordpress.com

Computer Vision

Artificial Intelligence

Computer Graphics

Video Compression

Single intelligent humanoid

They look like machines. But there is human soul inside!!!

5/14/2018 Le Thanh Ha, HMI Laboratory 4

NS-5 Eva R2-D2 Terminator

Swamp and crowd inspired from nature

5

Collective transportation Flock of birds School of fishes

Seasonal migrationCollective hunting Celebrating crowd

5/14/2018 6

Human and Machine senses

Le Thanh Ha, HMI Laboratory

Human and Machine collaborations

Interface Interaction

Integration Intelligence

5/14/2018 Le Thanh Ha, HMI Laboratory 7

Human Machine Interface

• How do humans manipulate with machine and how information exchanged between human and machines?

• Answer: Standardization

5/14/2018 Le Thanh Ha, HMI Laboratory 8

Keyboard layout

ISO/IEC 9995-3 Information technology: the Complementary Latin group layout according to ISO/IEC 9995-3:2010

5/14/2018 Le Thanh Ha, HMI Laboratory 9

Video standards

ISO standards

243 Committee in ISOIT Committee publish 3004/21347 (14%) ISO standards

5/14/2018 Le Thanh Ha, Lab of HMI 11

April 2017

April 2018: 3149

Information technology

Committee TitlePublished standards

Standards under development

ISO/IEC JTC 1 INFORMATION TECHNOLOGY 3004 546

ISO/IEC JTC 1/SC 17 Cards and personal identification 114 31

ISO/IEC JTC 1/SC 2 Coded character sets 54 3

ISO/IEC JTC 1/SC 22 Programming languages, their environments and system software interfaces 108 20

ISO/IEC JTC 1/SC 23 Digitally Recorded Media for Information Interchange and Storage 138 2

ISO/IEC JTC 1/SC 24 Computer graphics, image processing and environmental data representation 79 8

ISO/IEC JTC 1/SC 25 Interconnection of information technology equipment 188 27

ISO/IEC JTC 1/SC 27 IT Security techniques 165 63

ISO/IEC JTC 1/SC 28 Office equipment 48 2

ISO/IEC JTC 1/SC 29 Coding of audio, picture, multimedia and hypermedia information 574 (20%) 103

ISO/IEC JTC 1/SC 31 Automatic identification and data capture techniques 117 30

ISO/IEC JTC 1/SC 32 Data management and interchange 77 17

ISO/IEC JTC 1/SC 34 Document description and processing languages 80 11

ISO/IEC JTC 1/SC 35 User interfaces 67 16

ISO/IEC JTC 1/SC 36 Information technology for learning, education and training 38 10

ISO/IEC JTC 1/SC 37 Biometrics 118 28

ISO/IEC JTC 1/SC 38 Cloud Computing and Distributed Platforms 10 5

ISO/IEC JTC 1/SC 39 Sustainability for and by Information Technology 6 12

ISO/IEC JTC 1/SC 40 IT Service Management and IT Governance 19 9

ISO/IEC JTC 1/SC 6 Telecommunications and information exchange between systems 352 38

ISO/IEC JTC 1/SC 7 Software and systems engineering 171 425/14/2018 Le Thanh Ha, Lab of HMI 12

April 2018: 3149

ISO/IEC JTC 1 - Information technology

5/14/2018 Le Thanh Ha, Lab of HMI 13

From Interaction to Integration and Intelligence

5/14/2018 Le Thanh Ha, HMI Laboratory 14

IntelligenceIntegrationInteraction

implies stimulus-response

implies partnerships WORKING together

implies partnerships THINKING together

From Interaction to Integration and Intelligence

Interaction

• Stimulus-response

• Machine simply take task orders from human

• Like client-server architecture

Integration

• Machine and human work as partners

• Work done by exchange sequence of information between Machine and Human

• Engage all kinds of physical sensors: Geometric-Visual, Sound-Auditory, Haptic-Tactile

Intelligence

• Information exchanged just like human and human

• Engage higher abstraction of information exchanged between Machine and Human

• Machine can be able to draw both physical and emotional meaning from human

5/14/2018 Le Thanh Ha, HMI Laboratory 15

From Interaction to Integration

Interaction can be described as stimulus-response

Integration implies partnership WORK together

5/14/2018 Le Thanh Ha, HMI Laboratory 16

Desktop applications: MS Offices.doc, .rtf, .html

Online applications: Google drivers.html version 5.0

From Interaction to Integration

5/14/2018 Le Thanh Ha, HMI Laboratory 17

U. Farooq, et al., Human-Computer Integration, Interaction vol. 23, iss. 6, pp. 26-32, 2016

From Integration to Intelligence

• There is a continuum from Integration to Intelligence. Intelligence extends but does not replace interaction an integration.

• Engage higher abstraction of information exchanged between Machine and Human

• Machine can be able to draw both physical and emotional meaning from human

5/14/2018 Le Thanh Ha, HMI Laboratory 18

Integration implies partnerships WORKING

together

Intelligence implies partnerships THINKING

together

From Integration to Intelligence

5/14/2018 Le Thanh Ha, HMI Laboratory 19

Integration implies partnerships WORK

together

Intelligence implies partnerships THINK

together

Open issue in Computer science:+ How to present level of abstract of information?+ How to autonomous create new thing? (Unsupervised learning)+ Knowledge presentation and discovery?+ …

Abstraction level of visual information

5/14/2018 Le Thanh Ha, HMI Laboratory 20

Physical Abstract

Low level Middle level High level

Light frequenciesBrightness

emotion

Edges

Lines

ContoursColors

Regions

Depth

Pixel Intensity

anomaly events

a facesa pen

a person

attention

quality assessmentan animal

Intelligence degree

a banknote

social awareness

Standardization for AI

5/14/2018 Le Thanh Ha, HMI Laboratory 21

Intelligence

Inte

rfac

e

Standardization

Standardization

Standardization

ISO/IEC JTC 1 Information Technology

ISO/IEC JTC 1/SC 42 - ArtificialIntelligence

Recently created in 2017

Interaction

Integration

Intelligence

SOME WORKS FROM OUR GROUP

Vietnamese banknote speaker

Smart camera for healthcare

Human visual system based Image quality assessment

5/14/2018 Le Thanh Ha, HMI Laboratory 22

Vietnamese banknote speaker

Money speaker helps visual impaired people to identify Vietnamese banknotes (iOS & Android)

Make use of Compact descriptors for Visual Search (CDVS):• A standard proposed by the Moving Picture Experts Group [3] (MPEG) organization• Implemented with SIFT at its core• CDVS operators:

– Extract – extract and encode compact descriptors from a set of images– Make index – generate a tree-structured codebook from descriptors extracted from a set of images– Pairwise matching – compare the descriptors extracted from query and reference images– Retrieval – search and discovery of images contained within a large collection, some of which depict

the same objects or scenes as those depicted by a query image.

CDVS

5/14/2018 Le Thanh Ha, HMI Laboratory 24

Extraction pipeline

Pairwise Matching pipeline

Retrieval pipeline

a SIFT vector represented by 128 bytes

Image database

5/14/2018 Le Thanh Ha, HMI Laboratory 25

Smart camera for healthcare

• Fall detection

• Video source from IP cameras

• Detection accuracy from testing dataset: 90%.

• Speed of detection on Rasberrypi 2: 20fps

• Automatically send sms or notifyVN Patent pending Number: 1-2017-02136

5/14/2018 Le Thanh Ha, HMI Laboratory 26

Detection pipeline

5/14/2018 Le Thanh Ha, HMI Laboratory 27

5/14/2018 Le Thanh Ha, HMI Laboratory 28

5/14/2018 Le Thanh Ha, HMI Laboratory 29

HVS-based image quality assessment• Objective measures do

not meet subjective• All current IAQs grades

images as a whole– Can’t be aware of

subject context– Can’t be applied to

current block based video coding

➢ Needs method assess quality of images locally

5/14/2018 Le Thanh Ha, HMI Laboratory 30

The same amount of noise has been added to these two images, such that their PSNR isidentical. High-frequency noise was inserted into the bottom region of the left image, whereas band-pass filtered noise was inserted into the top region of the right image. The noise is hardly visible in the left image due to our low sensitivity to high-frequency stimuli and the strong masking by highly textured content in the bottom region.

Subjective IQA database

• Since there is now benchmark available

• We need to create that database for our own purposes.

• Adapt ITU-R BT.500-11 – “Methodology for the Subjective Assessment of the Quality of Television Pictures,” 2002.

5/14/2018 Le Thanh Ha, HMI Laboratory 31

Create new tool for subjective testing

5/14/2018 Le Thanh Ha, HMI Laboratory 32

5/14/2018 Le Thanh Ha, HMI Laboratory 33

IQA databases

Simple IQA database• 20 subjects• 20 images• Distortion levels: 5 QPs• Image patch size: 64 × 64.• 2 resolution levels• 3 types of image patches

– Edge– Smooth– Texture (complex)

• Total pairs of image patchs: 600

Huge IQA database• 1480 subjects• 308 images (HD and FHD)• Distortion levels: random• Image patch size: 64 × 64• Types of image patches: Spatially

random• Total pairs of image patches: 61 600

5/14/2018 Le Thanh Ha, HMI Laboratory 34

Experiment results on 1st database

5/14/2018 Le Thanh Ha, HMI Laboratory 35

• Relationship between MOSs and QP values are approximated by a logistic function

where 𝑎1 and 𝑎2 are estimated from data.

Experiment results on 1st database

• Applied on H.264/AVC video coding standard and obtain good results.

• A patent has been filed for this HVS-based video coding method.

5/14/2018 Le Thanh Ha, HMI Laboratory 36

Experiment results on 1st database

5/14/2018 Le Thanh Ha, HMI Laboratory 37

5/14/2018 Le Thanh Ha, HMI Laboratory 38

Works on 2nd database

• Data cleaning to remove noise and outliers

• Deep neuron network architectures for learning how HVS assess image quality (locally)

• Applied to the HVS-based IQA DNN to video coding scheme H.265

5/14/2018 Le Thanh Ha, HMI Laboratory 39

Reference DNNs

5/14/2018 Le Thanh Ha, HMI Laboratory 40

Jongyoo Kim, Sanghoon Lee, "Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework," CVPR 2017

THANK YOU VERY MUCH!

5/14/2018 Le Thanh Ha, HMI Laboratory 41