android and openni - nui application treasure hunter robot

©SIProp Project, 2006-2012 11

Android and OpenNI - NUI Application Treasure Hunter Robot

Hirotaka Niisato@hirotakaster

©SIProp Project, 2006-2012 2

About Myself

SpecialServer/Network Technology

Community (OSS)SIProp.org

CompanyGMO Internet, Inc

©SIProp Project, 2006-2012 3

NUI - Natural User Interface

©SIProp Project, 2006-2012 4

NUI Device

©SIProp Project, 2006-2012 5

Device Specifications

Kinect

(Windows)

Xtion Pro Live

RGB Camera ○ ○

Depth Sensor ○ ○

Audio ○ ○

Commercial Use ○ ○

Library Kinect for

Windows SDK

OpenNI

(GPL/LPGL)

©SIProp Project, 2006-2012 6

Shop

©SIProp Project, 2006-2012 7

Robotics

©SIProp Project, 2006-2012 8

others

©SIProp Project, 2006-2012 9

OpenNI

Open Source

Environment

● OS:Windows/Linux(+ARM)/MacOS/Android

● Java/C++/C#

Modules

● OpenNI

● Sensor

● Middleware(NITE)

©SIProp Project, 2006-2012 10

OpenNI with Android Application Pattern.1

USB

©SIProp Project, 2006-2012 11

OpenNI with Android Application Pattern.2

USB NW

Android : Toshiba T-01C(Regza Phone)

©SIProp Project, 2006-2012 12

Treasure Hunting Robot

©SIProp Project, 2006-2012 13

Summary

This is an "AR(augmented reality) Treasure Hunting Game“

You get virtual treasures by controlling real robot!

©SIProp Project, 2006-2012 14

Manual

Look at radar window like dragon radar.Show some treasure on radar.

Center is a place in which a robot is present.

N(North) is a robot's front.

If you move the robot to a treasure place, you get some point.

Controlling the robot is by brain wave.

©SIProp Project, 2006-2012 15

How to control

Keep 3 plots/5 plots in same area.

Upper(exciting) areaTurn left

Middle(active) areaGo toward

Lower(relax) areaTurn right

©SIProp Project, 2006-2012 16

How to make

©SIProp Project, 2006-2012 17

NUI Devices

©SIProp Project, 2006-2008 18

Hardwares & Softwares

HardwaresBase computer

Pandaboard

Brain Wave SensorMindWave

NeuroSky

Depth SensorXtion pro live

ASUS

DisplayAiRscoter

Brother Industries

Walking RobotKHR-3WL

Kondo science

SoftwaresUbuntu & Android

Linaro11.11

Depth SensorOpenNI

Bone skeleton trackerNITE for ARM

Only binary…

UI FrameworkopenFrameworks

Android

©SIProp Project, 2006-2012 19

Hardwares’ photo

©SIProp Project, 2006-2012 20

Softwares’ photo

©SIProp Project, 2006-2012 21

Calculate robot position

Recognizing a robot by bone skeleton.Using OpenNI

Calculate direction & position of robot.Calculate by moving robot.

Using depth camera.

©SIProp Project, 2006-2012 22

Recognizing bone skeleton

NITE is NOT OSS!Only binary for ARM/Ubuntu.

Want to use Android&openFrameworks UI.

Made dual boot OS for Ubuntu&Android.

Linaro Kernel for Ubuntu&Android on Pandaboard

UserLand: Linaro/Ubuntu (OpenNI runs on this userland)

UserLand: Linaro/Android (Run on Ubuntu by chroot)

©SIProp Project, 2006-2012 23

Recognizing bone skeleton

NITE can NOT recognize our robot…Head is too SMALL as human.

Legs & arms are too THIN as human.

Upright pose is not like human.

©SIProp Project, 2006-2012 24

Calculate robot position

Calculate a current center position.Moving distance is calculated by using depth camera.

Calculate a robot's front.This robot has just 3 ways action.

Go toward, Turn left, Turn right

Direction of “Go toward” is the robot's front.

©SIProp Project, 2006-2012 25

How to control

Keep 3 plots in same area.

Upper(exciting) areaTurn left

Middle(active) areaGo toward

Lower(relax) areaTurn right

©SIProp Project, 2006-2012 26

Brain wave 1/2

What wave is What feeling?γ(gamma) wave is exciting.

β(beta) wave is active.

α(alpha) wave is relax.

θ(theta ) wave is sleep.

δ(delta) wave is deep sleep.

ExcitingRelax

β γαθδ

©SIProp Project, 2006-2012 27

Brain wave 2/2

Upper area γ(gamma) wave is exciting.

Middle areaβ(beta) wave is active.

Lower areaα(alpha) wave is relax.

ExcitingRelax

β γαθδ

©SIProp Project, 2006-2012 28

Download source code

We release all source code on our site.http://www.siprop.org/ja/2.0/index.php?product%2FTreasureHuntingRobot

　 Do It Yourself ⇒Do It With Others!

©SIProp Project, 2006-2012 29

Thank you!