EyesWeb XMIMultimodal data recording,

playing and analysisM. Mancini, Università di Genova (Italy)

maurizio.mancini@unige.it

Goal

1. record multimodal data:• video (rgb or silhouette)• audio• sensor

2. play multimodal data3. analyze multimodal data in real-time• silhouette (if available)

• Contraction Index• Quantity of Motion

• sensor• energy• smoothness

1. Recording

media file writerN-th video frame

clock

csv file writer

tsv file writer

N

M-th sensor frame

K-th audio buffer

current time

current time

2. Player

media file reader

N-th video frame

csv file reader

tsv file reader

N

current time

M-th sensor frame

Global measures depending on full body movement (e.g., body orientation, overall motion direction).

Measures from psychological research, e.g., Boone & Cunningham’s amount of upward movement.

Cues from Rudolf Laban’s Theory of Effort, e.g., directness, impulsiveness.

Cues derived from analogies with audio analysis, e.g., Inter Onset Intervals, frequency analysis.

Kinematic measures such as velocity, acceleration, average and peak velocity and acceleration.

3. Analysis: Expressive Features

• SMIs (Silhouette Motion Images) carry information on variations of a blob (usually the silhouette of a user) in the last few frames.

Silhouette Motion Images

• SMIs are different with respect to MHIs, since they do not include the last silhouette, i.e., the current posture.

• Thus, SMIs carry information about the movement detected by the video-camera in the last n frames.

• The SMI area can be therefore considered as a measure of the detected amount of motion.

• The SMI area, normalized by the silhouette area, is called Motion Index (or Quantity of Motion).

Motion_Index[t] = Area(SMI[t, n]) / Area(Silhouette[t])

• Note that this is an approximated measure: e.g., movement against the video-camera is not detected.

SMIs and Motion Index

• Motion Index can also be computed by differently weighting pixels in the input blob.

• So, it is possible to compute a Motion Index where pixels near to the centre of the blob weight more than pixels near to the contour (i.e., something more similar to the physical concept of momentum).

• Or it is possible to compute a Motion Index where pixels near to the contour weight more than pixels near to the centre of the blob (i.e., a more perceptual measure where limbs have a stronger impact on perception of movement).

Weighted Motion Index

• Contraction Index is a measure of how the user’s body uses the space surrounding it.

• The simplest way to compute it is the ratio between the area of the blob and the area of the bounding rectangle.CI = Area(Blob) / Area(Bounding rectangle)

Contraction Index

Silhouette Features Extraction

media file reader

N-th silhouette frame blob extractor

QoMextractor

CIextractor

csv file writer

current time

Sensor Features Extraction

tsv file reader

accelerationdifferentiate

jerk (1/fluidity)

integrateenergy

csv file writer

current time