1

Virtualizing Real-life Lectures with vAcademia

and Kinect

Workshop on Off-The-Shelf Virtual Reality IEEE Virtual Reality Conference

March, 16 2013 | Orlando, FL, USA

Mikhail Fominykh and Ekaterina Prasolova-Førland Norwegian University of Science and Technology, Norway

Andrey SmorkalovVolga State University of Technology, Russia

2

Goal

o A low-cost technological setup for translating real-life presentations and lectures into a 3D virtual environment– Streaming real-life lectures into 3D virtual environment– Automatically creating immersive 3D recordings

3

Motivation: learning with VR

o Virtual worlds have recognized affordances for learning, but also many challenges

o Cost is a limiting factor for learning with virtual worlds and other VR

o Industry, military, and healthcare are the major areas where VR is currently used for educational purposes

o Exploring new ways of using 3D virtual worlds for learning: capturing lectures and creating asynchronous content out of synchronous learning activities

4

Motivation: capturing lectures

o ‘Traditional’ video recording of lectures and web conferences change the context of learning and do not provide immersion or sense of presence as in 3D virtual worlds

o ‘2D’ recordings, including Machinima, do not provide a possibility for collaborative work or a method for further developing the content

o Kinect was previously used to improve video recording of presentations by designing an automatic camera control system

o => Combining 3D recording in vAcademia with Kinect for advanced, immersive capturing of lectures

5

Virtualizing real-life lectures mode interface of vAcademia

Kinect plugin Animation library Cal3D

vAcademia

Script Executing Library

Scripts

vAcademiagraphic engine

First prototype: system implementation

6

First prototype: system implementation

o Five body parts: left arm, right arm, left leg, right leg, and head

o Standing mode (all body parts) and sitting mode (only arms and head)

o “Adequately recognized” status for each parto If a body part is not recognized adequately, last

adequate state is used for 0.2–0.5 sec, and then the default state

7

First prototype: system performance

o Requirements of the components– vAcademia requires and actively uses one CPU core. – Kinect requires a dual-core CPU, but uses only one core, as the

second is reserved for the application that uses Kinect data.o The process of animating the lecturer’s

avatar based on the data from Kinect is not computationally complex.

o System’s performance is satisfactory if component requirements are satisfied, which has been confirmed during the evaluation.

8

First prototype: system evaluation

o Non-systematic evaluation during iterative development process– Several evaluation sessions two-three different courses– Auditoriums of different configurations and lightning– Involving different teachers

o Data– Short interviews with the lecturer while watching the 3D recording

created vAcademia

o Most common feedback– Too many restrictions on the lecturer’s movements– Suggestions on how to increase the educational value

9

Applying Kinect Motion Capture in vAcademia: Challenges

1. Low accuracy in capturing gestures– We could not build a reliable avatar model that can move without

unnatural poses

2. Kinect does not recognize the turn of the lecturer– Left and right arms are mixed up, unnatural pose is returned

3. Kinect cannot capture parts of the body that are covered by other body parts or foreign objects– Additional requirements to the setup– Lower recognition accuracy

10

Applying Kinect Motion Capture in vAcademia: Solutions for 1

o Positioning Kinect device and the lecturer – < 1.8 m. for standing mode– < 1.3 m. for sitting mode– Kinect device at 0.5 m. from the floor – Software-based turn into a zero-degree position by the vertical axis

o Additional filtration mechanism for sorting out unnatural positions of the body parts– Limited the acceptable values of Euler angles between the bones– Separated hands as distinct body parts

11

Applying Kinect Motion Capture in vAcademia: Solutions for 2

o The turn is recognized relatively as a function of the position of the pelvis end points– The resultant value is valid within the range from -110 to 110

degrees against the “facing Kinect device” direction.o Colored markers

– Two markers are placed on the body of a lecturer on the left and on the right side, facing the Kinect device.

– The colors should be different from the lecturer’s clothing and the material should not be shimmering.

– If they are recognized, the system considers that the lecturer is in the acceptable turn range. If not -> last correctly recognized state -> default state.

12

Applying Kinect Motion Capture in vAcademia: Solutions for 2

o Testing colored markers

13

Applying Kinect Motion Capture in vAcademia: Proposal for 3

o Multiple Kinect devices– Three Kinect devices: to the left, to the right, and in front of the

lecturero New challenges:

– Increased price of the system– Data from the multiple Kinect devices should be adjusted to a single

coordinate system => increased requirements for the accuracy of locating Kinect devices

– Additional requirements to the auditorium (>7 m. across)– Merging the data from multiple Kinect devices

14

Supporting Slide Presentations: Challenges

1. Matching relative positions in real and virtual worlds– The position of the lecturer against the whiteboard should match

the position of the avatar against the virtual whiteboard.

2. Capturing a physical pointer – It is an important part of lecture experience, but Kinect cannot

capture it.

3. The gestures switching slides in real world do not have the same meaning in 3D virtual world

15

Supporting Slide Presentations: Solutions for 1

o Precise match between the physical whiteboard and the virtual one.– Performed once after installing the physical whiteboard and the

Kinect device in the classroom.– Capturing left and right edges of the physical whiteboard in Kinect

coordinate system.– Installing the Kinect device and the physical whiteboard on a

specified distance from the floor.o Further improvement

– Recognizing the borders of the physical whiteboard and creating the replica in the 3D virtual world keeping the proportion automatically.

16

Supporting Slide Presentations: Solutions for 2

o Directing the virtual pointer based on the position of the lecturer’s hand.– If the half line that extends from the lecturer’s hand towards the

physical whiteboard crosses it, the avatar in the 3D VW directs a virtual pointer to the same point.

– In order to keep the lecturer aware of his or her hand being captured, we display a semi-transparent yellow area on the physical whiteboard on top of the slides.

17

Supporting Slide Presentations: system in work

o Capturing lecture o Streaming lecture

18

Supporting Slide Presentations: Solutions for 3

o Switching slides functionality in PowerPoint by recognizing standard gestures Swipe Left and Swipe Right

19

Learning Scenarios

o Scenario 1: Lecturing as a synchronous mixed reality activity– Interactions between students in the physical and virtual

classrooms– Recording student and lecturer activities in the same context

o Scenario 2: Round-table discussion as a synchronous mixed reality activity– Participants joining through the 3D virtual world or captured from

the real world – Multiple Kinect-based systems can be installed in remote locations,

each of them can capturing two participants– The designed system provides a significant advantage over pure 3D

virtual worlds in the non-verbal communication support

20

o Scenario 3: Motion capture for synchronous mixed reality educational role plays– Taking turns in the physical classroom or letting the users

captured by Kinect play the roles of facilitatorso Scenario 4: Creating immersive 3D

recordings out of live lectures– Easy and low-cost creation of educational content for later

(asynchronous) use, such as lectures and simulations– Any activity, including streaming Kinect-captured lectures, in

the 3D virtual world can be easily saved and revisited later– The resultant 3D recordings combine the convenience of video

and immersive qualities of 3D virtual worlds

Learning Scenarios (2)

21

Questions?Feedbacks?

Acknowledgments

Andrey SmorkalovsmorkalovAY@volgatech.net

Mikhail MorozovmorozovMN@volgatech.net

Multimedia Systems LaboratoryVolga State University of Technology

http://mmlab.ru

Virtual Spaces LLCvAcademia

http://vacademia.com

Mikhail Fominykhmikhail.fominykh@ntnu.no

Ekaterina Prasolova-Førlandekaterip@ntnu.no