satellite navigation head to head test

Blauw Research Limited · 7-10 Adam Street · London WC2N 6AA · +44 (0)207 520 [email protected] · www.blauw.co.uk

Satellite NavigationHead to Head Test

London, July/August 2013


Summary

• In a series of Head-to-Head journeys undertaken with live traffic devices from TomTom, Google, Garmin and CoPilot, the routes generated by Tom Tom delivered drivers to their destinations 5% quicker than the next quickest device i.e. 2 minutes quicker on a typical commuting journey of around 40 minutes


Introduction

• The new generation of satellite navigation devices and smartphone apps have the ability to link navigation to live traffic information, with the goal of avoiding delays and optimising journey times by routing around traffic problems

• TomTom wished to compare the performance of its own traffic information supported satellite navigation device – the TomTom Go 500 with that of it’s competitors

• A Head-to-Head Test was devised which tested the performance of 5 satellite navigation devices on the same journeys at the same times on the same days


Introduction

• The objective was to compare the performance of the satellite navigation devices in terms of: – Estimated Time of Arrival (ETA) vs Actual Time of Arrival (ATA)– Estimate Travel Distance (ETD) vs Actual Travel Distance (ATD)– Trip Duration– Trip Distance– Performance in traffic


Introduction

• 5 identical vehicles were be deployed (2013 Model Ford Fiesta), each manned by a driver and co-driver

• Each vehicle had one satellite navigation device and one GPS tracking device installed

• A series of 85 trips of differing lengths/durations was devised making a total of 425 individual journeys (5 devices x 85 trips)

• All 5 vehicles took the same trip (same start and end point) at the same time (separated by 2 minutes) on the same day

• The 2 minute separation was used to ensure that the drivers were following their navigation devices, not each other in convoy, as they made their journeys


Introduction

• Journeys were taken between 6.00 am and 9.30 am in the morning and 4.00 pm and 7.30 pm in the evening, these times were chosen as the busiest times for traffic in London. Although there is no ‘official’ rush hour, Transport For London’s peak fares are between 4.30 am and 9.30 am and there is other anecdotal evidence to suggest that the evening rush hour starts at 4.00pm

• The GPS tracking devices tracked the exact location of each vehicle as they undertook each journey

• Drivers followed the instructions given by the satellite navigation device, co-drivers kept a record of key data for each journey and traffic conditions


Introduction

Each satellite navigation device was set up in the same way:• Factory defaults• Quickest Route• Traffic alerts on (if available)• Speed camera alerts on (if available)• Speed limit alert on (if available)• Audio navigation on• Toll roads = Yes


IntroductionOur target average journey time was based on the ‘typical commute’ of 40 minutes, we achieved an average of 40m 24s across all journeys

Trip TimeNumber of

journeys % of trips

Under 30 mins 145 34%

30-59 mins 216 51%

Over 60 mins 64 15%

Total 425 100%


Device Type Internet Link

Garmin 3597 PND Tethered to Galaxy S2

Garmin 3598 PND Integrated in PND

TomTom Go 500 PND Tethered to iPhone 4

CoPilot Android App on Galaxy S3 Via Galaxy S3

Google Android App on Galaxy S3 Via Galaxy S3

DevicesAll devices needed an internet link to collect live traffic information


Teams and Devices6 driving teams were used, the devices were rotated between the teams each day


DateMorning Journeys

Evening Journeys Total

15/07/13 4 1 516/07/13 3 3 617/07/13 3 3 618/07/13 3 3 619/07/13 2 3 522/07/13 4 3 723/07/13 4 3 724/07/13 3 3 625/07/13 3 3 626/07/13 3 2 529/07/13 3 3 630/07/13 3 3 631/07/13 3 3 601/08/13 2 3 502/08/13 3 0 3Totals 46 39 85

Journey ScheduleThe first morning trip started soon after 6.00am and the first evening trip soon after 4.00pm


Typical Trip Schedule


Device/Team 1 2 3 4 5 6 Totals

Garmin 3597 18 17 15 11 11 13 85

Garmin 3598 13 18 17 15 11 11 85

TomTom Go 500 8 13 18 17 15 14 85

CoPilot 11 11 13 18 17 15 85

Google 18 11 8 13 18 17 85

Total 68 70 71 74 72 70 425

Trips by Team/Device


Journey Start/End Points


Overall Performance


TomTom Go 500

Google Average Garmin 3597

Garmin 3598

CoPilot0:00

4:48

9:36

14:24

19:12

24:00

28:48

33:36

38:24

43:12

48:00

38:1240:11 40:24 40:49 41:14 41:34

Mins:secs

Average Trip TimeOn average, the TomTom GO 500 was faster than any other device

Base: 85 trips


6 3 1 Average 2 5 40:00

4:48

9:36

14:24

19:12

24:00

28:48

33:36

38:24

43:12

48:00

39:02 39:08 39:49 40:24 40:30 40:4543:00

Team

Mins:secs

Average Trip Time

Base: 85 trips

Team 4 were the slowest, over 2½ minutes slower than average


Tom-Tom Go

500

Google Average Garmin 3597

Garmin 3598

CoPilot0:00

4:48

9:36

14:24

19:12

24:00

28:48

33:36

38:24

43:12

48:00

38:1240:11 40:24 40:49 41:14 41:3438:05

40:14 40:24 40:55 41:16 41:30

ActualReweightedMins:secs

Average Trip Time - weighted

Base: 85 trips

Even after reweighting the data to remove driver variation, the TomTom GO 500 was over 2 minutes quicker than Google and over 3 minutes quicker than Garmin 3598 and CoPilot


TomTom Go 500

Google

Garmin 3597

Garmin 3598

CoPilot

20:34

20:26

21:11

20:58

21:00

40:44

42:46

43:19

42:49

43:31

73:15

74:56

84:00

79:32

76:00

Over 60 minutes30-59 minutesUnder 30 minutes

Mins:secs

Average Trip Time by Duration

Base: 85 trips

Number of journeys:

Under 30 minutes: 14530-59 minutes: 216

Over 60 minutes: 64Total: 425

The TomTom GO 500 performed comparatively well on shorter trips and best overall on trips over 30 min


CoPilot

Google

Garmin 3597

Garmin 3598

TomTom Go 500

14:26

19:53

17:45

19:53

14:40

30:56

29:39

28:47

28:33

27:50

50:23

49:58

45:15

49:33

44:57

02:32

56:28

59:43

00:14

59:12

Over 20 miles10-20 miles5-10 milesUnder 5 miles

Mins:secs

Average Trip Time by Distance

Base: 85 trips

No. of journeys:

Under 5 miles: 415-10 miles: 158

10-20 miles: 138Over 20 miles: 88

Total: 425

The TomTom GO 500 performed comparatively well on all lengths of trips, and best overall on those between 5 and 20 miles


CoPilot

Google

Garmin 3597

Garmin 3598

TomTom Go 500

31:32

30:32

33:00

31:52

31:02

57:32

54:28

57:35

58:00

55:41

43:11

43:00

41:27

43:30

42:07

35:04

33:40

33:23

33:23

33:57

Outer to Outer LondonOuter to Inner LondonInner to Outer LondonInner to Inner London

Mins:secs

Average Trip Time by Location

Base: 85 trips

No. of journeys:

Inner to Inner: 15Inner to Outer: 19Outer to Inner: 22Outer to Outer: 29


Under 30 Mins 30-59

Mins 60-90 Mins Over 90

Mins

0102030405060708090

100

Under 5 Miles5-10 Miles

10-20 MilesOver 20 Miles

36

50

0

93

62

30

16

96

18

8

0

53

29

6

Trip Count

Trip Duration vs Trip Distance

Base: 425 journeys


Garmin 3598

TomTom Go 500

CoPilot Average Google Garmin 3597

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

13.53 13.75 13.84 14.01 14.0714.83

Miles

Average Trip DistanceOn average, the Garmin 3597 travelled

an extra mile compared to TomTom

Base: All trips


Garmin 3598

CoPilot Average TomTom Go 500

Google Garmin 3597

0.00

5.00

10.00

15.00

20.00

25.00

20.04 20.22 20.85 20.97 21.1221.90

19.68

21.01 20.8021.60

19.98

21.80

Average of TripsOverall AverageMPH

Average Trip SpeedEven though the Garmin 3597 achieved the best average speed, it

only had the 3rd best trip time as it routed over longer distances

Base: All trips


CoPilot Google Garmin 3597 TomTom Go 500

Garmin 3598

-10.00

-8.00

-6.00

-4.00

-2.00

0.00

2.00

4.00

-7.65

-5.26

-4.06-3.42

1.60

Minutes

Overestimated

Underestimated

ETA Accuracy: ETA-Arrival Time4 of the devices underestimated the trip time,

the CoPilot was underestimating by almost 20%

Base: All trips


Garmin 3597 Google CoPilot TomTom Go 500

Garmin 3598

-0.80

-0.70

-0.60

-0.50

-0.40

-0.30

-0.20

-0.10

0.00

0.10

0.20

-0.70

-0.61

-0.43

-0.28 -0.27Miles

Underestimated

Distance Accuracy: Estimated-Actual Distance

All devices underestimated the trip distance, the Garmin 3597 underestimated by almost 5%

Base: All trips


Rijn Vogelaar

Average difference between original ETA and ETA during trip

0 10 20 30 40 50 60 70 80 90 100-5

0

5

10

15

20

25

30

35

40

45

0 10 20 30 40 50 60 70 80 90 100-5

0

5

10

15

20

25

30

35

40

45

0 10 20 30 40 50 60 70 80 90 100-5

0

5

10

15

20

25

30

35

40

45

0 10 20 30 40 50 60 70 80 90 100-5

0

5

10

15

20

25

30

35

40

45

0 10 20 30 40 50 60 70 80 90 100-5

0

5

10

15

20

25

30

35

40

45

TomTom

Garmin 3597

CoPilotGarmin 3598

Google

As trips got longer, the difference between the original ETA and the in-trip ETA increased;

Garmin 3598 was the most accurate and Copilot the most inaccurate

Journey Time

Average ETA difference

Base: All trips


Traffic Conditions


Weight of traffic

Light; 12%

Normal; 76%

Heavy;

12%

Light; 15%

Normal; 81%

Heavy; 3%

Light; 9%

Normal; 69%

Heavy;

22%

All journeys Morning

Evening

Traffic was at its heaviest during the evenings

Base: All trips


Driving Conditions

Normal; 99%

Abnormal; 1%

All journeys

Driving conditions were almost perfect

Base: All trips


Satnav Performance


Satnav Performance

Normal; 71% Ab-

nor-mal; 29%

Garmin 3597

Normal; 78%

Ab-normal; 22%

Garmin 3598

Normal; 65%

Abnor-mal; 35%

CoPilot

Normal; 67%

Ab-nor-mal; 33%

Google

Normal; 74%

Ab-nor-mal; 26%

TomTom Go 500

Base: All tripsDedicated PNDs performed

more consistently


Satnav PerformanceGarmin 3597

Garmin 3598

CoPilot

Google

TomTom Go 500

Slow to recalculate routesScreen road layout not matching

actual road prioritiesShowing congestion when none

observedNot showing congestion when

congestion observed

Not alerting to traffic congestionVoice instruction lost or late

Difficult to read roundabout turnsLost connection/screen frozen

Lost connection, no traffic information

Routed down one way/closed road

Voice navigation problems, either absent or conflicting with screen

Not redirecting around traffic congestion

Made some routing choices which extended journey time

Confusing directions, turns and lanes not clear

Not showing congestion when congestion observed

Made some routing choices which extended journey time


Garmin 3597

Garmin 3598

TomTom Go 500 CoPilot Google

Satnav shows no traffic problems/no traffic problems observed 0 7 2 3 1Satnav shows no traffic problems/traffic problems observed 25 15 23 28 23Satnav shows traffic problems/no traffic problems observed 14 6 1 10 1Satnav shows traffic problems/traffic problems observed 21 23 10 22 30Satnav shows traffic problems/different traffic problems observed 0 0 0 1 1Red Count 39 21 24 38 24Green Count 21 30 12 25 31

Exception Reporting Mostly queuing traffic observed but not shown on screen

Numbers are the count of each traffic situation type when driving teams felt it necessary to report an exception

Minimal reporting of ‘false positives’

Base: All exception reports for all trips


Garmin 3597

Garmin 3598

TomTom Go 500 CoPilot Google

Satnav shows no traffic problems/no traffic problems observed 834 871 851 850 819Satnav shows no traffic problems/traffic problems observed 93 44 41 57 78Satnav shows traffic problems/no traffic problems observed 21 11 4 11 0Satnav shows traffic problems/traffic problems observed 120 158 88 171 162Satnav shows traffic problems/different traffic problems observed 2 1 3 2 1Red Count 114 55 45 68 78Green Count 954 1029 939 1021 981Red/Green Percentage 11.9% 5.3% 4.8% 6.7% 8.0%

ETA Reporting

Lowest inaccuracy percentage

Base: All ETA reports for all trips

Numbers are the count of each traffic situation type, taken with the ETA every 3 minutes throughout the journey


What did the team think?


Device/Rank 1st 2nd 3rd 4th 5th

TomTom Go 500 6 4 1 0 0

Garmin 3598 4 6 1 0 0

Garmin 3597 1 1 6 0 3

CoPilot 0 0 2 5 4

Google 0 0 1 6 4

Which device did you like the best?

Base: 11 team members Taking everything into account, which of the devices did you like the best? and the next best and so on?


Which device did you like the best?TomTom Go 500

Base: 11 team members Just thinking about the TOMTOM, what did you like about it? Please give as many details as you can.

Very intuitive to use. Screen easy to see. Split screen feature very good. Information given in good time.

Seamless transition from heavy traffic route to alternative route.

It was really simple to use, easy to enter details and it would automatically look up the address as soon as you'd put in some simple details. Liked the traffic

information icons, this was depicted by what looked like Red lava on the screen, it also have a bar down the right side that gave you indication of how many

cars ahead you were behind in built up traffic areas.

Big screen, sat in cradle nicely, tap in postcode numbers on same screen as letters.

Good choice of voices and speaker sound.


Which device did you like the best?Garmin 3598

Base: 11 team members Just thinking about the Garmin 3598, what did you like about it? Please give as many details as you can.

Clear audible navigation and screen directions. Very easy-touch screen. Like the wide screen. Very simple

menu navigation. Logical layout.

This satnav was easy to use, generally gave lots of warning when coming up to turns. The split screen image was, once again, very useful and gave good

alternatives when in heavy traffic. It was very intuitive to use. Taking a wrong turn the satnav was good to offer an alternative route, sometimes did take a

little longer than one would hope.

1. Map easy to follow 2. Traffic information was on point 3. Camera alerts were prompt

4. Spoken directions were clear


Performance AttributeCount of TomTom

ranked 1st

Ease of installation into the car 5Ease of entering target destination 7Ease of reading screen display – while stationary 8Ease of reading screen display – while moving 8Audibility of voice commands 7Ease of navigation using voice commands 7Ease of navigation using screen instructions 6Usefulness of live traffic information 7Consistency and reliability of navigation service 6Accuracy of navigation – en route and to the end destination 7Ability to divert around traffic 6Taking the shortest route to the destination 6

Which device performed best for…?

Base: 11 team membersTomTom ranked highest

on all attributes

satellite navigation head to head test

Documents