Real-Time Computing LabUniversity of Michigan

Invisible Sensingof Vehicle Steering with Smartphones

Dongyao Chen Kyong-Tak Cho, Sihui Han, Zhizhuo Jin, Kang G. Shin

May 19, 2015

/23

Camera

Camera helps detect lane markers for• Detecting steering

maneuvers

Autonomous vehicles use cameras to detect• Lane markers• Objects on

road• …

Affordable, easy to

use+

VariousCV

algorithms…

1

https://www.youtube.com/watch?v=NJ8iXYZfUmA

http://www.phonearena.com/news/Nexus-5-likely-coming-with-an-OIS-camera_id47409

http://www.theguardian.com/technology/2013/jul/09/google-android-reader-why

/23

How’s Camera Doing?

CarSafe[MobiSys’ 13] app

2

iOnRoad app

https://play.google.com/store/apps/details?id=com.picitup.iOnRoad

Citroën DS5

http://www.mycitroen.be/nl/citroen-ds5/innovatie/

http://wallpaperswide.com/road_to_death_valley-wallpapers.html

• Performance of camera depends on the environment

/23

How’s Camera Doing?

2

Lighting Placement

http://smartphones.wonderhowto.com/how-to/keep-your-smartphone-charged-during-power-outage-0140088/

Pavement

http://www.annarbor.com/news/residents-frustrated-with-lack-of-solutions-for-pothole-ridden-north-territorial-road/

Weather

CarSafe[MobiSys’ 13] appiOnRoad app

https://play.google.com/store/apps/details?id=com.picitup.iOnRoad

Citroën DS5

http://www.mycitroen.be/nl/citroen-ds5/innovatie/

• Performance of camera depends on the environment

/23

• Real road conditionCarSafe[MobiSys’ 13] app

2

iOnRoad app

https://play.google.com/store/apps/details?id=com.picitup.iOnRoad

Citroën DS5 LDW System

http://www.mycitroen.be/nl/citroen-ds5/innovatie/

Lighting Placement

http://smartphones.wonderhowto.com/how-to/keep-your-smartphone-charged-during-power-outage-0140088/

Pavement

http://www.annarbor.com/news/residents-frustrated-with-lack-of-solutions-for-pothole-ridden-north-territorial-road/

Weather

How well a camera can see

Bottleneck of application’s performance

How’s Camera Doing?

/23

Key Idea: Camera-free Sensing of Vehicle Steering

3

Reliability-Resistant to changing environments

Accuracy-Visual similarity will not effect the detection results

Lightweight-Lower computational consumption

The challenge is to find the right signature

/23

V-Sense

4

Utilizes gyroscope

Works as a middleware

Differentiates maneuvers

/23

Sensing Steering with Gyroscope

• In left turns

• In lane changes

5

/23

• In left & right turns

• Lane changes• Lane changes• How to detect bumps? • How to relate bump detection to

different steering maneuvers?

Detect bumps in gyroscope reading

5

Understanding Vehicle Steering

/23

Bump Detection

• Three-state algorithm:– No bump → One bump→ Waiting for

bump

6

/23

Increase Accuracy: Steering & Curvy Road

• Horizontal displacement:– WCURVY >WLANE , WCURVY >WTURN

– Integrate horizontal displacement at each time step

7

Lane change S-shape road Turn L-shape road

/238

Calculate Horizontal Displacement

0Ts2Ts3Ts4Ts

Velocity v1v2v3v4

Time 0Ts2Ts3Ts4Ts

Velocity v1v2v3v4

θ1

θ2

θ3

θ4

Time

W4

W3

W2

W1

0Ts2Ts3Ts4Ts

Velocity v1v2v3v4

θ1

θ2

θ3

θ4

Time

Figure: Horizontal displacement of lane change

/23

Detect Horizontal Displacement on Curvy Road

102 m

59 m

9Route image adapted from Google map: https://www.google.com/maps/place/2345+Lancashire+Dr,+Ann+Arbor,+MI+48105/@42.3099558,-83.7171271,17z/data=!3m1!4b1!4m2!3m1!1s0x883cac2d8254dfbb:0x2df93ff4aa13179b

/23

Horizontal Displacement onCurvy Road

54 m

101.5m

10

/23

Horizontal Displacement onCurvy Road

54 m

101.5m

10

102 m

59 m

/23

Evaluation

• Setups– Tested V-Sense on Samsung

Galaxy S4 and S3– 5 volunteers, 2 cars– 40+h test

11

/23

Reliability and Accuracy

12

Right Turn Left Turn Lane Change Curvy Road76%

80%

84%

88%

92%

96%

100%MountedIn Pocket

Succ

ess

Rate

• Resilient to different placements:

/23

Computational Costs

13

/23

Comparison with Other Applications

14

V-Sense iOnRoad Black Sensor Drivea AD5%

20%

35%

50%

65%

80%

95%

~1,000,000

installs

~50,000installs

~50,000installs

110 ratings

Figure: Success rate of detection of lane change in urban area

Succ

ess

Rate

/23

When Steering your Car...

16Image: http://www.wakeforesttoday.com/voices-letters/note-to-self/4569-turn-signals-people.html#.VUPwONpViko

/23

Application I: Detection of Careless Steering

17Car icon: http://publicdomainvectors.org/en/free-clipart/Top-view-car-vector/2260.html

/23

Turn Signal Detection

18

/23

Application I: Detection of Careless Steering

！

19

/23

Application I: Detection of Careless Steering

19

/23

Application II: Fine-grained Lane Guidance

http://365stagandhenweekends.co.uk/blog/

https://idroid.us/google-maps-app-for-ios-updated-with-optimized-for-ios-8.html

20

GPS is unstable and its accuracy varies with environment

GPS Accuracy

/23

Application II: Fine-grained Lane Guidance

21

• Lane tracking at intersection:

Right to Left

Left to

Right

/23

Conclusion

• V–A reliable, accurate and light

weighted middleware for detecting steering maneuvers

• Two proof-of-concept apps–Careless steering warning–Fine-grained lane guidance

22

/23

Thank you!

Q&A

23