kinectballs: an interactive tool for ball throwing games

17
Introduction Architecture of KinectBalls Lessons learned and future work Conclusion KinectBalls An Interactive Tool for Ball Throwing Games Jonathan Schoreels, Romuald Deshayes, and Tom Mens UMons 4 July 2013 J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 1 / 17

Upload: tom-mens

Post on 15-May-2015

523 views

Category:

Technology


2 download

DESCRIPTION

Presentation by Jonathan Schoreels of KinectBalls during the INTETAIN 2013 conference in Mons, 4 July 2013. KinectBalls is an interactive computer game in which a player throws real balls toward a virtual scene, using the Kinect 3D sensor. This work is the result of a master student project at the Département d'Informatique, Faculté des Sciences, UMONS, Belgium

TRANSCRIPT

Page 1: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

KinectBallsAn Interactive Tool for Ball Throwing Games

Jonathan Schoreels, Romuald Deshayes, and Tom Mens

UMons

4 July 2013

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 1 / 17

Page 2: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Table of contents

1 Introduction

2 Architecture of KinectBalls

3 Lessons learned and future work

4 Conclusion

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 2 / 17

Page 3: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Introduction

Context

Recent interest for new games and entertainment applications usingnew HCI techniques like natural interaction :

Nintendo’s Wii console

Microsoft’s Kinect sensor

Leap motion

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 3 / 17

Page 4: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

KinectBalls

Goal

Bridge the gap between the real and virtual world. The aim of thegame is to bring down a pile of virtual boxes by throwing a real balltowards them.

For this, we exploited the raw information of the Kinect’s 3D sensorto track the moving ball.

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 4 / 17

Page 5: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Realisation

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 5 / 17

Page 6: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Limitations

The Kinect has a low frame-rate (30 Hz). This could beaddressed by using an other sensor with better frame-rate.

Ball moving too fast can be unseen by the device.

Nothing can be seen at a distance below 50 centimetres.

The shape of the ball can be distorted.

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 6 / 17

Page 7: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Table of contents

1 Introduction

2 Architecture of KinectBalls

3 Lessons learned and future work

4 Conclusion

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 7 / 17

Page 8: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Implementation overview

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 8 / 17

Page 9: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Object detection and tracking

For each frame Fn, we have for each pixel the depth. We cancompare a frame Fn with the previous frame Fn−1.

If that difference exceeds a threshold T (to filter out noise) :Something has moved on that particular pixel. We create the matrixMoveMap.

Creation of MoveMap

MoveMap(i , j) =

{1 if |Fn(i , j)− Fn−1(i , j)| > T0 otherwise

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 9 / 17

Page 10: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Object detection and tracking

We search the biggest square of 1’s in MoveMap.

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 10 / 17

Page 11: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Trajectory prediction

At least 3 positions of the ball : Approximation with 3second-degree polynomials.

At each new frame where the ball is detected, a newapproximation is computed.

At the predicted time of impact, a virtual ball is created atthe predicted position.

Knowing the exact 3D position of the wall, we can approximate thecollision between the trajectory approximated and the wall.

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 11 / 17

Page 12: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Trajectory prediction

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 12 / 17

Page 13: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Table of contents

1 Introduction

2 Architecture of KinectBalls

3 Lessons learned and future work

4 Conclusion

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 13 / 17

Page 14: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Lessons Learned

Kids of 5 years and older tested the game for the ”Printemps dessciences 2013” :

They were enthusiasts.

The approximated impact point was good (between 1 to 5centimetres of precision)

Some adults had difficulties to interpret the 3D virtual world.

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 14 / 17

Page 15: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Future work

Use a sensor with a frame-rate of 60 Hz

Exploit the Kinect’s RGB camera to create a virtual ball of thecolor of the thrown ball.

Automate the calibration between the Kinect and the screen.

Use an extra Kinect to track the head of the player and create avirtual window effect.

Multi-player game.

Performance improvements of algorithms.

Use a 3D beamer.

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 15 / 17

Page 16: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Table of contents

1 Introduction

2 Architecture of KinectBalls

3 Lessons learned and future work

4 Conclusion

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 16 / 17

Page 17: KinectBalls: An Interactive Tool for Ball Throwing Games

Introduction Architecture of KinectBalls Lessons learned and future work Conclusion

Conclusion

KinectBalls requires only one very affordable 3D sensor.

Developed algorithms are fast enough to run in real time on astandard computer.

The solution worked fine in all tested indoor situations

J. Schoreels, R. Deshayes, and T. Mens (UMons) KinectBalls 4 July 2013 17 / 17