multidimentional classification automation with human interface based on metromaps

.

MDC Background

M.Zhanikeev -- [email protected] -- Multidimentional Classification Automation with HI based on Metromaps -- http://bit.do/150714 -- 2/23...

2/23

.

MDC Basics: Binary Relevance (BR)

• binary: YES or NO for each Y 11

• problem: no relation between classes Y

11 J.Read+3 "Classifier chains for multi-label classification" Machine Learning, SpringerS (2011)


3/23

.

MDC Basics: PairWise Sets (PW)

• relations can be found by creating new classes for all unique pairs in Y 11

• problem: many classes = fuzzy results = low reliability

Training Tuples x1 x2 Y1 Y2 Y3

1 0.7 0.4 1 1 0 2 0.6 0.2 1 1 0 3 0.1 0.9 0 0 1 4 0.3 0.1 0 0 0

h1: X → Z1 h2: X → Z2

Z1 Z2 1 0 0 1 0 0 0 0



4/23

.

MDC Basics: Label Combination (LC)

• basically, the extreme case of PW 11

• the same problem only worse -- there are too many classes!


1 0.7 0.4 1 1 0 2 0.6 0.2 1 1 0 3 0.1 0.9 0 0 1 4 0.3 0.1 0 0 0

h: X → Z

Z 1 0 0 0



5/23

.

MDC Basics: Classifier Chains (CC)• classes are used in sequence 11

• merit: small number of classes -- only the necessary ones are used

• demerit: what is the correct order?


1 0.7 0.4 1 1 0 2 0.6 0.2 1 1 0 3 0.1 0.9 0 0 1

0.3 0.1 0 0 0

h1: X → Y1 h2: Y1 → Y2 h3: Y2 → Y3

h2 h1 h3

4



6/23

.

Metromaps Classifier (proposal)


7/23

.

Metromap Classifier : Core Situation• problem: human load is too high!, ex: disaster scenarios 07b

Human judgment

Auto judgement

Folksonomy

07bmyself+0 "Black Swan Disaster Scenarios" PRMU研 (2014)


8/23

.

Metromap: The Basic Concept• merit: e2e paths in the ontology graph

07 myself+0 "On Context Management Using Metro Maps" SOCA, Matsue, Japan (2014)

14 K.Nesbitt+0 "Getting to more abstract places using the metro map metaphor" 8th IV (2004)


9/23

.

Definitions, Objectives, Terminology.Different Viewpoint..

.

classifier is not for finding hidden relations, but for clear separationbetween known and new.Learning Classifier...... a classifier that improves its inference over time based on human feedback

.Metromaps..

.... are used as the graphical interface between humans and robots

• MDC: Multi-Dimensional Classification• MC: Metromap Classifier

• folksonomy: BigData with very frivolous management of metadata


10/23

.

Metromap Classifier : Design

Human judgment

Auto judgement

Folksonomy


11/23

.

Metromap Classifier : Features

• human role1. build the metromap = relations between classes2. when robot fails, do the work manually3. do the human part (by design) of the work

• robot role1. classify incoming data into YES or NO for question: should human seethis?


12/23

.

Experiment


13/23

.

Experiment : Setup

• IEICE/ken is the source of data -- over 3000 presentations over 2-3 lastyears

• various combinations of title, keywords, abstract• usecase: which presentations should I look at closely?

◦ ... meaning the metromap reflects my personal research interests• Dumb Classifier (DC): one-dimensional yes or no


14/23

.

Metromap Design: The Human


15/23

.

Metromap Classifier: Logic• logic followed by the MC Robot

Metromap Classifier

Human

Check Metromap

Fuzzy?

Cold? Hot?

Robot (Automatic Classification)

Bad

Input

No Yes

No

No


16/23

.

Results: Title only

0 20 40 60 80 100 120Time sequence

0102030405060708090

Goo

d c

ount

Dumb ClassifierMetromap Classifier(smart) Hits on a timelinetitle


17/23

.

Results: Title + Keywords

0 20 40 60 80 100Time sequence

0

10

20

30

40

50

60

70

80G

ood

cou

ntDumb ClassifierMetromap Classifier(smart) Hits on a timeline

title:keywords


18/23

.

Results: Title + Keywords + Abstract

0 20 40 60 80 100 120Time sequence

0102030405060708090

Goo

d c

ount

Dumb ClassifierMetromap Classifier(smart) Hits on a timelinetitle:keywords:abstract


19/23

.

Wrapup: Not Good Enough

• not perfect: about 30% of wrong decisions◦ FP: robot makes human look at bad stuff (false positive)◦ FN: robot passes on good stuff (false negative)

• future improvements: need a solid logic which avoids FP and FN cases

• note: current naive and MDCs are at most 40-60% reliable -- no help here!


20/23

.

That’s all, thank you ...


21/23

.

Learning = Social (software) Robotics


22/23

.

Social Robotics in Knowledge

Rebot

(careless) Input

Human Human

{structure}

(pinpoint) Select

Browse (or use otherwise)

Some Knowledge

(folksonomies, knowledge bases, databases, indexes, ontologies, etc.)

(metromaps )

07 myself+0 "On Context Management Using Metro Maps" SOCA, Matsue, Japan (2014)


23/23

multidimentional classification automation with human interface based on metromaps

Technology

learning classifier

metromaps classifier

metromap classifier

classifier chains cc

classes y

mdc basics

y1 y2 h3

new classes