Interactive Computer Graphics

Interactive Computer Graphics Interactive Computer Graphics

Allow users to control how graphical Allow users to control how graphical elements are displayed. elements are displayed.

Human-Computer Interaction (HCI) is critical to Interactive Computer Graphics and Visualization.

HCIHCI

HCIHCI

Three types of HCI:

CLI: command line interface (with keyboard)

GUI: graphical user interface (mouse)

NUI: natural user interface with A/V (Kinect)

A GOOD user interface (UI) allows users to perform interactive tasks with ease and joy.

WYSIWYG (What you see is what you get).

Four basic interaction TASKS:

position, select, quantify, text

Basic design PRINCIPLES:

Look (appearance) and Feel (action).

HCIHCIHardwareHardware

QWERTY(slow down typing)

Dvorak, order by frequency of use

Alphabetic order

HCI Hardware: Keyboards

HCI Hardware: Locators

relative devices:

mice, trackballs, joysticks

absolute devices:

data tablets, touch screen, Kinect, LeapMotion

direct devices:

light pens, touch screens, Kinect, LeapMotion

indirect devices:

mice, trackballs, joysticks

continues devices:

mice, trackballs, joysticks, Kinect, LeapMotion

discrete devices:

control keys

HCI Hardware: Locators

Bounded:

volume control on radio

Unbounded:

clock, dial

Choice Devices:

function keys

foot switches

HCI Hardware: Valuators

Haptic Devices:

pressure-sensitive stylus

force-feedback controls (haptic)

Phantom from Sendable

HCI Hardware: Valuators

3D Interaction Devices

Joysticks with a shaft that twists for a third dimension

Kinect 3D camera, LeapMotion

VR: virtual reality, immersive, head-mounted sensors/markers for tracking

Polhemus 3D sensors

Data glove

Interactive Interactive TasksTasks

Position:

by pointing (GRAPHICS)

Selection:

by name (DB), by pointing

GUI:

hierarchical pull-down menu, radio-buttons

e.g. Format->Paragraph…

Interactive TasksInteractive Tasks

Text Interaction:

keyboard-->text-string

Quantify Interaction:

dials, sliders

3D Interaction Tasks:

Z value from 3D input devices,

multiple views, shift + button-down

Interactive TasksInteractive Tasks

Interaction Toolkits

get user input (control) ==>events

graphical output (feedback) ==>display

graphic device interfaces: GDI

X Windows Toolkit (UNIX)

Windows API (PC)

JAVA Swing

CUDA/CL (GPU)

WebGL (web)

Standard APIs for input hardware: not there, need standardization.

NUI: OpenNI, NIUI, MicroSoft Kinect SDK

Haptic: OpenHaptics Toolkit

NUINUI

Natural User Interfaces: • Voice controls• Kinect 3D sensor• LeapMotion

NUI-based Interactive Computer GraphicsNUI-based Interactive Computer Graphics

UserUser

ControllerController

DisplayDisplay

GraphicsApplicationGraphics

Application

NUI:NUI:

Three parts of NUI:

Hardware: e.g., Kinect

Software: drivers (OpenNI), middleware

Application: integration of HW enabling software with applications.

OpenNIOpenNI

OpenNIOpenNIProduction Nodes:

• a set of components that have a productive role in the data creation process required for Natural Interaction based applications.

• the API of the production nodes only defines the language.

• The logic of data generation must be implemented by the modules that plug into OpenNI.

• E.g. for a production node that represents the functionality of generating hand-point data, the logic of hand-point data generation must come from an external middleware component that is both plugged into OpenNI, and also has the knowledge of how to produce such data.

Production Nodes:

• a set of components that have a productive role in the data creation process required for Natural Interaction based applications.

• the API of the production nodes only defines the language.

• The logic of data generation must be implemented by the modules that plug into OpenNI.

• E.g. for a production node that represents the functionality of generating hand-point data, the logic of hand-point data generation must come from an external middleware component that is both plugged into OpenNI, and also has the knowledge of how to produce such data.

OpenNI OpenNI

(1) body imaging (2) joint recognition (3) hand waving

OpenNI: Sensor-Related Production NodesOpenNI: Sensor-Related Production Nodes

Device: represents a physical device (a depth sensor, or an RGB camera). Its main role is to enable device configuration.

Depth Generator: generates a depth-map. Must be implemented by any 3D sensor that wishes to be certified as OpenNI compliant.

Image Generator: generates colored image-maps. Must be implemented by any color sensor that wishes to be certified as OpenNI compliant

IR Generator: generates IR image-maps. Must be implemented by any IR sensor that wishes to be certified as OpenNI compliant.

Audio Generator: generates an audio stream. Must be implemented by any audio device that wishes to be certified as OpenNI compliant.

OpenNI: Middleware-Related Production NodesOpenNI: Middleware-Related Production Nodes

Gestures Alert Generator: Generates callbacks to the application when specific gestures are identified.

Scene Analyzer: Analyzes a scene, including the separation of the foreground from the background, identification of figures in the scene, and detection of the floor plane. The Scene Analyzer’s main output is a labeled depth map, in which each pixel holds a label that states whether it represents a figure, or it is part of the background.

Hand Point Generator: Supports hand detection and tracking. This node generates callbacks that provide alerts when a hand point (meaning, a palm) is detected, and when a hand point currently being tracked, changes its location.

User Generator: Generates a representation of a (full or partial) body in the 3D scene.

OpenNI: Recording Production NotesOpenNI: Recording Production Notes

Recorder: Implements data recordings

Player: Reads data from a recording and plays it

Codec: Used to compress and decompress data in recordings

OpenNI: CapabilitiesOpenNI: Capabilities

Supports the registration of multiple middleware components and devices. OpenNI is released with a specific set of capabilities, with the option of adding further capabilities in the future. Each module can declare the capabilities it supports.

Currently supported capabilities:

Alternative View: Enables any type of map generator to transform its data to appear as if the sensor is placed in another location.

Cropping: Enables a map generator to output a selected area of the frame.

Frame Sync: Enables two sensors producing frame data (for example, depth and image) to synchronize their frames so that they arrive at the same time.

OpenNI: CapabilitiesOpenNI: Capabilities

Currently supported capabilities:

Mirror: Enables mirroring of the data produced by a generator.

Pose Detection: Enables a user generator to recognize when the user is posed in a specific position.

Skeleton: Enables a user generator to output the skeletal data of the user. This data includes the location of the skeletal joints, the ability to track skeleton positions and the user calibration capabilities.

User Position: Enables a Depth Generator to optimize the output depth map that is generated for a specific area of the scene.

OpenNI: CapabilitiesOpenNI: Capabilities

Currently supported capabilities:

Error State: Enables a node to report that it is in "Error" status, meaning that on a practical level, the node may not function properly.

Lock Aware: Enables a node to be locked outside the context boundary.

Hand Touching FOV Edge: Alert when the hand point reaches the boundaries of the field of view.

OpenNI: Generating and Reading DataOpenNI: Generating and Reading Data

• Production nodes that also produce data are called Generator.

• Once these are created, they do not immediately start generating data, to enable the application to set the required configuration.

• The xn::Generator::StartGenerating() function is used to begin generating data.

• The xn::Generator::StopGenerating stops it.

• Data Generators "hide" new data internally, until explicitly requested to expose the most updated data to the application, using the UpdateData request function.

• OpenNI enables the application to wait for new data to be available, and then update it using the xn::Generator::WaitAndUpdateData() function.

Interactive Game Interactive Game with with

KinectKinect

Video GameVideo Game

Interactive animation: Interactive animation:

user->user->interface -> interface ->

game object action -> game object action -> feedback (A/V, haptic)feedback (A/V, haptic)

Game objects can represent data.Game objects can represent data.

Video GameVideo Game

UserUser

ControllerController

DisplayDisplay

Game (Software)

Game (Software)

Video GameVideo Game

Input Device Driver

Input Device Driver

DisplayDeviceDriver (GDI)

DisplayDeviceDriver (GDI)

Game (Software)

Game (Software)

Software for Kinect-based game developmentSoftware for Kinect-based game development

OpenNI: a general-purpose framework for obtaining data from 3D sensors

SensorKinect: the driver for interfacing with the Microsoft Kinect

NITE: a skeleton-tracking and gesture-recognition library

Unity3D: a game engine

ZigFu: Unity Package for Kinect (Assets and Scripts)

OpenNI: a general-purpose framework for obtaining data from 3D sensors

SensorKinect: the driver for interfacing with the Microsoft Kinect

NITE: a skeleton-tracking and gesture-recognition library

Unity3D: a game engine

ZigFu: Unity Package for Kinect (Assets and Scripts)

EDP(Event-driven Programming)

Concepts: EDP

Event-Driven Programming (EDP): Application waits (idles)

after initialization until the user generates an event trough an input device (keyboard, mouse, …). The OS dispatches the event to the application who owns the active window. The corresponding event handler(s) of the application is invoked to process the event.

A menu in C++:char c;bool done = false;while(!done) {

cout << “Please make your selection, q to end:”cin >> c;switch(c) {case “+”:

add( );break;

case “-”:sub( ); break;

case “q”:done = true; break;

}}

Event Loop

Event Mapping&

Even Dispatching

Event

Event Handler

(1) Event Generators: keyboard, GUI items (buttons, menus, …), NUI devices.

(2) Events / Messages:MouseClick, KeyDown, …

(3) Event Loop:an infinite loop constantly waits for events.

(4) Event Mapping / Event Registration: inform event dispatcher which event an event hander is

for.(5) Event Dispatcher:

dispatch events to the corresponding event handlers.(6) Event Handlers:

methods for processing events.OnMouseClick(), …

Key Components of EDP

(1) Event Generators- usually predefined by the system. - used by programmers / GUI designers.

(2) Events / Messages - predefined by the system.

(3) Event Loop: - usually taken care of by the system. (4) Event Mapping / Registration:

- implemented by the system. - to be used by programmers

with appropriate matching: event => even handler

(5) Event Dispatching: - usually taken care of by the system. (6) Event Handlers - to be implemented by programmers.

Key Components of EDP for Programmers

Most common EDP steps:

(1) Select input devices.(2) Identify event generators and events to use.(3) Map events to event handlers.(4) Implement event handlers.

EDP Programming

SummarySummary

HCI: CLI, GUI, NUI

HCI Hardware

Interactive Tasks

NUI

Kinect

Kinect-based Games

EDP

Interactive Computer Graphics

for

Visualization