Swimming Style Recognition and Lap CountingUsing a Smartwatch and Deep Learning

Gino Brunner, Darya Melnyk, Birkir Sigfusson, Roger WattenhoferETH Zurich – Distributed Computing Group – www.disco.ethz.ch

ETH Zurich – Distributed Computing Group – www.disco.ethz.ch

Swimming plan: 400m Freestyle warm-up4 × 50m Backstroke

100m Individual Medley...

...

Swimming plan: 400m Freestyle warm-up4 × 50m Backstroke

100m Individual Medley...

...

Swimming Plan

Style Recognition

Butterfly Backstroke

Breaststroke Freestyle

Lap Counting

Smartwatch: Nixon The Mission

aged from 25 to 75

wide range of skills

40 users

Data

Class Duration [hours] Nr. of users

Butterfly: 1.0 23Backstroke: 2.3 31Breaststroke: 1.4 24

Freestyle: 6.8 37Transition: 5.5 40

Data - Example Recording

0 10 20 30 40 50 60 70 80 90 100 110

0

50

Trans. Trans.Freestyle FreestyleBreaststrokeBackst.Butterfly Trans.

Time [s]

Acc

eler

atio

n[m

/s2]

Processing

signal 1

signal 2signal 3

6s

1s

CNN

1 64 64 64 64

11 11 11 11 11

180 5919

51

128

5

Input layer Conv. layer 1 Conv. layer 2 Conv. layer 3 Conv. layer 4 F.C. layer Classification

3x1 conv. 3x1 conv. 3x1 conv. 3x1 conv. Fully Conn. Fully Conn.

ELU ELU ELU ELU ELU Softmax

3x1 max-pool 3x1 max-pool 3x1 max-pool 3x1 max-pool

Style Recognition - Confusion Matrix

[%] Predicted Class

Butterfly Backstroke Breaststroke Freestyle TransitionButterfly 96.5 0.0 0.0 3.4 0.1

True Backstroke 0.0 98.2 0.2 0.4 1.2Class Breaststroke 0.0 1.7 94.9 1.8 1.7

Freestyle 0.5 0.1 0.0 98.5 0.8Transition 0.0 0.3 0.2 0.7 98.7

Recall 96.5 98.2 94.9 98.5 98.7Precision 99.4 97.9 99.4 94.0 96.4

F1 97.9 98.1 97.1 96.2 97.5

Accuracy 97.4Average F1 97.4

Lap Recognition

Deviation1s 2s 4s 6s 8s

True Pos. 421 676 732 747 749False Pos. 331 76 20 5 3False Neg. 336 81 25 10 8

Precision [%] 56.0 89.9 97.3 99.3 99.6Recall [%] 55.6 89.3 96.7 98.7 98.9

Lap Recognition

20 40 60 80 100 120 140

−500

50A

cc.

[m/s

2]

Wrong Transitions

20 40 60 80 100 120 140Transition

FreestyleBreaststroke

BackstrokeButterfly

Time [s]

Style Recognition for a Lap

Predicted ClassButterfly Backstroke Breaststroke Freestyle Mixed

Butterfly 77 0 0 1 6True Backtstroke 0 156 0 0 2Class Breaststroke 0 0 85 0 4

Freestyle 0 0 0 546 5Mixed 0 4 0 0 54

Code and Data: http://bit.ly/2VGEeWN