Probabilistic modelling for sensor fusion(using inertial sensors and magnetometers)

Manon Kok

Delft University of Technology, The Netherlands

September 5th, 2018

Brief background

• MSc in Philosophy of Science, Technology and Society at the Universityof Twente, the Netherlands.

• MSc in Applied Physics at the University of Twente, the Netherlands.

• Research Engineer at Xsens Technologies (February 2009 – July 2011).

• PhD in Automatic Control from Linkoping University (August 2011 –February 2017).

• Postdoc in the Machine Learning Group at the University of Cambridge(March 2017 – February 2018).

• Assistant Professor in the Delft Center for Systems and Control at theDelft University of Technology (April 2018 – Now).

2 / 29

Probabilistic modelling for sensor fusion(using inertial sensors and magnetometers)

3 / 29

About my researchSensor fusion: Using disparate sensors in order to infer more informationabout the quantity of interest than would be possible using each sensorindividually.

Quantity of interest: e.g. the position and orientation of the sensor.

• Inertial sensors• Accelerometers• Gyroscopes

• Magnetometers

4 / 29

Accelerometer measurements

5 10 15 20 25 30−20

0

20

Time [s]

y a,t

[m/s

2 ]

Stationary data ⇒ Earth gravityRotating sensor ⇒ Earth gravity

Quickly moving sensor ⇒ Earth gravity + sensor’s accelerationThe combination of earth gravity and sensor acceleration is called the

external specific force.

5 / 29

Accelerometer measurements

5 10 15 20 25 30−20

0

20

Time [s]

y a,t

[m/s

2 ]

Stationary data ⇒ Earth gravity

Rotating sensor ⇒ Earth gravityQuickly moving sensor ⇒ Earth gravity + sensor’s acceleration

The combination of earth gravity and sensor acceleration is called theexternal specific force.

5 / 29

Accelerometer measurements

5 10 15 20 25 30−20

0

20

Time [s]

y a,t

[m/s

2 ]

Stationary data ⇒ Earth gravityRotating sensor ⇒ Earth gravity

Quickly moving sensor ⇒ Earth gravity + sensor’s accelerationThe combination of earth gravity and sensor acceleration is called the

external specific force.

5 / 29

Accelerometer measurements

5 10 15 20 25 30−20

0

20

Time [s]

y a,t

[m/s

2 ]

Stationary data ⇒ Earth gravityRotating sensor ⇒ Earth gravity

Quickly moving sensor ⇒ Earth gravity + sensor’s accelerationThe combination of earth gravity and sensor acceleration is called the

external specific force.

5 / 29

Gyroscope measurements

2 4 6 8 10 12 14 16 18 20 22

−5

0

Time [s]

y ω,t

[rad

/s]

Stationary data ⇒ Zero angular velocityRotating the sensor around the y -axisRotating the sensor around the x-axisRotating the sensor around the z-axis

6 / 29

Gyroscope measurements

2 4 6 8 10 12 14 16 18 20 22

−5

0

Time [s]

y ω,t

[rad

/s]

Stationary data ⇒ Zero angular velocity

Rotating the sensor around the y -axisRotating the sensor around the x-axisRotating the sensor around the z-axis

6 / 29

Gyroscope measurements

2 4 6 8 10 12 14 16 18 20 22

−5

0

Time [s]

y ω,t

[rad

/s]

Stationary data ⇒ Zero angular velocityRotating the sensor around the y -axis

Rotating the sensor around the x-axisRotating the sensor around the z-axis

6 / 29

Gyroscope measurements

2 4 6 8 10 12 14 16 18 20 22

−5

0

Time [s]

y ω,t

[rad

/s]

Stationary data ⇒ Zero angular velocityRotating the sensor around the y -axisRotating the sensor around the x-axis

Rotating the sensor around the z-axis

6 / 29

Gyroscope measurements

2 4 6 8 10 12 14 16 18 20 22

−5

0

Time [s]

y ω,t

[rad

/s]

Stationary data ⇒ Zero angular velocityRotating the sensor around the y -axisRotating the sensor around the x-axisRotating the sensor around the z-axis

6 / 29

Inertial sensors for pose estimation

∫

rotateremove

gravity

x

angular

velocity orientation

external spe-

cific force acceleration position

Accelerometers and gyroscopes (inertial sensors) can be used forposition and orientation (pose) estimation ⋯

7 / 29

Integration drift

however, the position and orientation estimates obtained using inertialsensors suffer from integration drift.

0 2 4 6 8 100

1

2

3

Time [s]

Orie

ntat

ion

[◦ ]

0 2 4 6 8 10

0

5

Time [s]

Posit

ion

[m]

⇒ Combine inertial sensors with other sensors and / or additionalmodels to obtain accurate estimates.

8 / 29

Probabilistic modelling for sensor fusion(using inertial sensors and magnetometers)

9 / 29

Probabilistic modelling

Probabilistic modelling to

• take into account that all models are approximations,

• take into account that all sensor measurements are not perfect,

• represent the best estimate and its uncertainty.

10 / 29

Probabilistic modelling for sensor fusion(using inertial sensors and magnetometers)

11 / 29

Magnetometers

Magnetometers measure the local(earth) magnetic field

12 / 29

Magnetometer measurements

2 4 6 8 10 12 14 16

−200

0

200

400

Time [s]

y m,t

[µT]

Stationary magnetometer data ⇒ Local (earth) magnetic fieldMagnetic disturbance close to sensor ⇒ Local (earth) magnetic field +

magnetic disturbance

Inertial sensors and magnetometers can be combined to estimateorientation.

But how to model the magnetometer measurements?

13 / 29

Magnetometer measurements

2 4 6 8 10 12 14 16

−200

0

200

400

Time [s]

y m,t

[µT]

Stationary magnetometer data ⇒ Local (earth) magnetic field

Magnetic disturbance close to sensor ⇒ Local (earth) magnetic field +magnetic disturbance

Inertial sensors and magnetometers can be combined to estimateorientation.

But how to model the magnetometer measurements?

13 / 29

Magnetometer measurements

2 4 6 8 10 12 14 16

−200

0

200

400

Time [s]

y m,t

[µT]

Stationary magnetometer data ⇒ Local (earth) magnetic fieldMagnetic disturbance close to sensor ⇒ Local (earth) magnetic field +

magnetic disturbance

Inertial sensors and magnetometers can be combined to estimateorientation.

But how to model the magnetometer measurements?

13 / 29

Magnetometer measurements

2 4 6 8 10 12 14 16

−200

0

200

400

Time [s]

y m,t

[µT]

Stationary magnetometer data ⇒ Local (earth) magnetic fieldMagnetic disturbance close to sensor ⇒ Local (earth) magnetic field +

magnetic disturbance

Inertial sensors and magnetometers can be combined to estimateorientation.

But how to model the magnetometer measurements?

13 / 29

My research

Goal:

Push the boundaries of accuracy that can be obtained from sensormeasurements by using

• advanced models,

• advanced algorithms.

Motivation:

• Sensors are becoming smaller, cheaper and more widely available.

• More and more computational power is available.

• More and more application areas are opening up.

14 / 29

Applications

Analysis of motion of:

• Cars

• Airplanes

• Quadcopters

• Trains⋮

Figures courtesy of Xsens Technologies

15 / 29

Applications

Figures courtesy of Xsens Technologies

Analysis of human motion for:

• Sports

• Rehabilitation

• Movies / games

• Robotics⋮

15 / 29

Two examples

• Human motion capture• Take as much information about the situation at hand into account.

• Indoor localisation using measurements of the magnetic field.• Extract as much information from sensor measurements as possible.

16 / 29

Example: Human motion capture

Estimate the relative position andorientation of body segments.

17 sensors placed on the body

Figures courtesy of Xsens Technologies

17 / 29

Example: Human motion capture

Estimate the relative position andorientation of body segments.

17 sensors placed on the bodyFigures courtesy of Xsens Technologies

17 / 29

Example: Human motion capture

Estimate the relative position andorientation of body segments.

17 sensors placed on the bodyFigures courtesy of Xsens Technologies

17 / 29

Model

Inertial sensors provide information aboutthe change in position and orientation.

The position and orientation of the sensorson the body is approximately constant.

The body segments are connected at thejoints. ⇒ constraint

Additional information...

18 / 29

Model

Inertial sensors provide information aboutthe change in position and orientation.

The position and orientation of the sensorson the body is approximately constant.

The body segments are connected at thejoints. ⇒ constraint

Additional information...

18 / 29

Model

Inertial sensors provide information aboutthe change in position and orientation.

The position and orientation of the sensorson the body is approximately constant.

The body segments are connected at thejoints.

⇒ constraint

Additional information...

18 / 29

Model

Inertial sensors provide information aboutthe change in position and orientation.

The position and orientation of the sensorson the body is approximately constant.

The body segments are connected at thejoints. ⇒ constraint

Additional information...

18 / 29

Model

Inertial sensors provide information aboutthe change in position and orientation.

The position and orientation of the sensorson the body is approximately constant.

The body segments are connected at thejoints. ⇒ constraint

Additional information...

18 / 29

Example: human body motion capture

arg minx1∶NT

− NT∑t=2

NS∑i=1

log p(xSit ∣ xSit−1, ySit )´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶

dynamics of the state xSit

− NS∑i=1

log p(xSi1 ∣ ySi1 )´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶prior

− NT∑t=1

NS∑i=1

log p(xBj

t ∣ xSit )´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶

placement of sensor Si on body segment Bj

subject to c(x) = 0

19 / 29

Human motion capture

Use only sensor data

Courtesy of Xsens Technologies

20 / 29

Human motion capture

Use sensor data + a biomechanical model

Courtesy of Xsens Technologies

21 / 29

Human motion capture

Use sensor data + a biomechanical model + a world model

Courtesy of Xsens Technologies

22 / 29

Two examples

• Human motion capture• Take as much information about the situation at hand into account.

• Indoor localisation using measurements of the magnetic field.• Extract as much information from sensor measurements as possible.

23 / 29

Example: Indoor localisation

Indoor localisation

• to find your way in largeshopping malls, train stations,etc,

• for first responders toemergencies,

• ⋮

0 1 2 3 4 5 10 m

30

40

50

60

Mag

nitu

de(µ

T)

Which sensors to use?

• GPS does not work (well) indoors.

• Use sensors that are present in any modern smartphone.

⇒ Use measurements of the magnetic field as a source of positioninformation!

24 / 29

Example: Indoor localisation

Indoor localisation

• to find your way in largeshopping malls, train stations,etc,

• for first responders toemergencies,

• ⋮0 1 2 3 4 5 10 m

30

40

50

60

Mag

nitu

de(µ

T)

Which sensors to use?

• GPS does not work (well) indoors.

• Use sensors that are present in any modern smartphone.

⇒ Use measurements of the magnetic field as a source of positioninformation!

24 / 29

Modelling the magnetic field

• Use machine learning to learn the magnetic field from data.

• Gaussian process models: Learn a map of the magnetic field. Alsolearn where you are certain or uncertain about this map.

25 / 29

Building magnetic field maps

Build a map of the indoor magnetic field to use it for localisation.

https://www.youtube.com/watch?v=enlMiUqPVJo

26 / 29

Magnetic field SLAM

Simultaneous localization and mapping

https:

//www.youtube.com/watch?v=enlMiUqPVJo

27 / 29

Two examples

• Human motion capture• Take as much information about the situation at hand into account.

• Indoor localisation using measurements of the magnetic field.• Extract as much information from sensor measurements as possible.

28 / 29

Thank you for your attention!

Questions?

https://www.tudelft.nl/staff/m.kok-1/

29 / 29