user experience of mobile physical activity applications ... · an alternative approach, in situ...

- 1 -

User experience of mobile physical activity applications: contextual, motivational and

experiential factors

Elizaveta Temir

Project report submitted in part fulfilment of the requirements for the degree of Master of Science (Human-Computer Interaction with Ergonomics) in the Faculty of Life Sciences, University College London, [2013].

NOTE BY THE UNIVERSITY

This project report is submitted as an examination paper. No responsibility can be held by London University for the accuracy or completeness of the material therein.

- 2 -

ACKNOWLEDGMENTS

I would like to thank my supervisor Aisling O’Kane for her help and patience, my research participants for being so open and honest, my parents for incredible support and faith in me, my grandmother Tatyana Bezborodova for proof-reading such an enormous amount of text and my partner Konstantin Bychkhoz for making this research possible.

- 3 -

ABSTRACT

Rising popularity of mobile physical activity applications and wider acceptance of user-centered design emphasises the need for extensive user experience testing. Although to this day the most used user experience testing method is laboratory experiments, its limitations and inability to capture real-life contextual factors, which ultimately define mobile technology use, are being increasingly recognised. This study commenced in the wild exploration of user experience of mobile physical activity applications. A comparative four-week field evaluation of two exemplar mobile running applications was conducted using a combination of field testing techniques, interviews and diary studies. Several methodological issues were overcome, such as usability data gathering in a dynamic environment, and design implications related to users’ previous experiences and context of use were discovered. Moreover, two types of runners and subsequently two types of user needs were identified based on users’ fitness motivations. This showed a need to include motivational features into physical activity applications to accommodate health-directed users, and a higher level of customisation to accommodate those who treat exercise as a hobby.

- 4 -

CONTENTS

Chapter 1. Introduction ...................................................................................... 6

Chapter 2. Literature review ............................................................................ 10

1.1. Autoethnography .................................................................................. 10

1.2. Diary studies ......................................................................................... 10

1.3. Field testing techniques ........................................................................ 11

Chapter 3. Main content chapters .................................................................... 14

1.4. Method .................................................................................................. 14

Autoethnography ........................................................................................ 14

Pilot study ................................................................................................... 16

Main user study ........................................................................................... 17

Data Analysis .............................................................................................. 20

1.5. Results ................................................................................................... 22

Results 1: evaluation of Runtastic .............................................................. 22

Results 2: evaluation of Endomondo .......................................................... 47

Results 3: cross-application patterns .......................................................... 57

Chapter 4. General Discussion ........................................................................ 85

1.6. Design implications .............................................................................. 85

Previous experience with running applications affects usage expectations and patterns ....................................................................................................... 86

Need for correlation between elevation and pace ....................................... 86

Lack of integrated music player ................................................................. 86

Preference of big buttons with distance between them .............................. 86

- 5 -

Need for audio feedback for starting, pausing, un-pausing and stopping session ............................................................................................................... 87

Positioning of the iPhone affects usability and visibility during running .. 88

Need for including motivation into features for Health Runners ............... 88

1.7. Discussion of methods .......................................................................... 90

Autoethnography ........................................................................................ 90

Diary study .................................................................................................. 91

Content of diaries ........................................................................................ 94

Using photos for deferred contextual interviews ........................................ 94

Field testing ................................................................................................ 96

1.8. Limitations ............................................................................................ 97

Chapter 5. Conclusion ..................................................................................... 99

- 6 -

CHAPTER 1. INTRODUCTION

Mobile computing has become ubiquitous in our everyday lives (Merchant, 2012), with around 1.7 billion smartphones sold only in 2012 (mobithinking.com). Mobile sports applications market is also increasing: Runtastic has reached 25 million users (Runtastic, May 2013), Endomondo tracker has reached 12 million users (Endomondo October 2012), RunKeeper has more than 14 million users (Wauters, 2012). There are a high number of physical activity applications that users can choose from (e.g. lifehacker.com, telegraph.co.uk) and with such competition importance of great user experiences (UX) and user centered design (UCD) is gaining wider popularity and acceptance (Mao, Vredenburg, Smith, & Carey, 2005; Au et al., 2008; Hussain, Slany, & Holzinger, 2009). UCD emphasises the goals and needs of users and their involvement in all stages of design (Saffer, 2010, p. 33). An integral part of UCD is usability testing (Abras, Maloney-Krichmar, & Preece, 2004), which is conducted by using one or more usability evaluation methods (UEMs) to assess technology.

Thus far laboratory testing has been used in remarkable 31% of HCI research (Kjeldskov & Graham, 2003) and has been viewed as a benchmark UEM (Hartson, Andre, & Williges, 2001). It allows precise capturing of user-system interaction in controlled environment, and helps to overcome some of the practical issues related to mobile evaluations. Such issues include dynamic context, small display size, effect of invisible sensing on user behaviour and interaction with the product, distributed interaction, connectivity problems, and data entry on the go (Crabtree et al., 2006; Zhang & Adipat, 2005). However, in controlling for these problems lab testing creates such artificial conditions that some researchers suggest that it should not be regarded as the only reliable UEM (Hartson et al., 2001).

Context of use of mobile devices should not be disregarded because it significantly affects UX (Riegelsberger & Nakhimovsky, 2008). As McCarthy and Wright (2004) propose, experience of technology is compositional, and formed by an interconnection of several factors, such as sensual, emotional, and spatio-temporal, which are influenced by the context of use. Context as such is not a single static notion but is a dynamic product of interaction, it emerges from an activity; and the two cannot be separated (Dourish, 2004). Context is a product of multiple factors at a given moment, individuals, environment, prior experience, activity being performed and technology used. A context of running then is a combination of a physical environment, individual’s physical and emotional feelings, social interaction, running background, goals and technology used to support running, all specific to a particular situation.

Thus studying how a user interacts with an application in a lab setting means studying artificial interactions and gathering artificial data, because user actions (and therefore interactions with technology) are essentially situated (Suchman, 1987). They highly depend on and come as responses to circumstances and

- 7 -

environment, in which they occur. Conducting user experience testing in a lab is not studying interactions that later can happen parallel to the environment once technology is in the real world, but rather studying situation that will never occur in the real world. Simply put, behaviour that you observe in the lab does not apply to the real world (Rogers, 2011).

A few studies have illustrated discrepancies between the lab and the real world (e.g. Marshall, Morris, Rogers, Kreitmayer, & Davies, 2011; Bonsignore, Quinn, Druin, & Bederson, 2013). As traditional lab usability testing does not allow capturing information on the actual usage of mobile devices in real life (Mark, Christensen, & Shafae, 2001), there have been some attempts to increase ecological validity of lab testing conditions (e.g. Beck, Christiansen, Kjeldskov, Kolbe, & Stage, 2003; Kangas & Kinnunen, 2005; Kjeldskov, Skov, Als, & Høegh, 2004).

An alternative approach, in situ (also called in the wild or field) testing, i.e. conducting usability testing within the target context of use has gained much attention from both researchers (e.g. Kjeldskov et al., 2004; Rogers et al., 2007; Jambon & Meillon, 2009) and practitioners (e.g. Vredenburg, Mao, Smith, & Carey, 2002). Since then it has been debated whether conducting in situ testing added any significant value compared to traditional testing in the lab. Some researchers say that field testing is too difficult and time consuming, and show that it produces the same list of usability problems as testing in labs (e.g. Kjeldskov et al., 2004; Kallio, Kaikkonen, Kekäläinen, Kankainen, & Cankar, 2005). On the other hand, there are experiments that show that not only the overall number of usability problems obtained in a field test was almost twice as large as that of lab testing, but also numbers of problems of each severity level (i.e. critical, serious and cosmetic) were also higher in field testing (Duh, Tan, & Chen, 2006). Moreover, it is believed that in the wild evaluations give insight into the appropriateness of design for users’ everyday life and their true user experience that cannot be captured outside of target usage context (e.g. Rogers et al., 2007). And as Isomursu, Kuutti, & Väinämö (2004) point out, “mobile applications are designed to be used in a mobile context, so they should also be evaluated in a mobile context” (p. 84).

There is currently lack of literature on conducting in situ usability and user experience testing studies of mobile physical activity application that are currently available in the market. A number of methods have been used for situated mobile evaluations in other domains, such as experience sampling method (ESM; e.g. Mancini et al., 2009), diaries (e.g. Barkhuus & Dey, 2003; Palen & Salzman, 2002; Sohn, Li, Griswold, & Hollan, 2008), user-led participatory evaluations (Isomursu, Kuutti, & Väinämö, 2004), technology probes (e.g. Ahtinen, Ramiah, Blom, & Isomursu, 2008), telephone interviews (Riegelsberger & Nakhimovsky, 2008) and combination of screen capture and video recordings from wearable cameras (Brown, McGregor, & Laurier, 2013).

Typically research in the field of human-computer interaction (HCI) is concerned with creation and evaluation of novel technologies to support behavioral

- 8 -

change in physical activity domain rather than studying how current technologies are being used. Two examples are Houston and UbiFit Garden. Houston is a prototype mobile application designed by Consolvo, Everitt, Smith, and Landay (2006) to increase physical activity levels in its users by tracking and sharing their step count. It was evaluated in a three-week in situ study that was run on dedicated phones (i.e. not participants’ own) and used questionnaires, interviews and daily logs of step counts. UbiFit Garden is a system consisting of a mobile application with glanceable display and a fitness device, which uses a representation of a garden to visualize and thus increase physical activity (Consolvo et al., 2008). It was evaluated in a three-week field study with interviews and was also run on phones provided by experimenters.

When choosing among various UEMs it hard to compare them as there is lack of consensus in UEM comparison techniques and measures, and no standardized criteria available (Hartson et al., 2001). Usually comparing UEMs means in some way comparing lists of usability problems they help find (Hartson et al., 2001), however given lack of universal measures the approach to UEMs taken in this study was exploratory rather than comparative.

This research aimed to uncover methodological difficulties and opportunities in field usability testing of mobile physical activity applications and to produce preliminary guidance on how those can be overcome. This was accomplished by taking two widely used commercial running applications as examples of physical activity applications, Runtastic and Endomondo, and conducting comparative usability testing. The aim was two-fold: to explore methodological side of in situ testing of mobile sports applications, and to investigate what constitutes desirable level of usability and user experience in those applications.

I used two criteria for choosing set of UEMs for this study: (1) their relative success in capturing detailed usability and rich contextual and motivational data about in situ usage of mobile technologies in different research domains, and (2) ease with which they could be appropriated for evaluation of mobile physical activity applications.

Outdoors running is a non-traditional environment (Bennett, Lindgaard, Tsuji, Connelly, & Siek, 2006), meaning that it is subject to rapid unpredictable and uncontrollable changes and poses many environmental and other constraints on the use of technology, especially in a busy city like London where this study took place. There is evidence that combining video capturing of user-system interaction with think-aloud protocols can be an effective way of gathering detailed usability problems in a similar environment of city streets (e.g. Brown et al., 2013; van Elzakker, Delikostidis, & van Oosterom, 2008).

As for collecting data about rich longer-term natural usage of mobile technologies within the target contexts of running, the diary study method seems to be more effective than other methods, such as ESM when studying unpredictable

- 9 -

activities (Sohn, Li, Griswold, & Hollan, 2008; Carter & Mankoff, 2005). Using diary studies also leaves participants in control of data collection and would not impact their normal running behaviours. While such participant-directed approach has its disadvantages (e.g. Riegelsberger & Nakhimovsky, 2008), it was reported that it enabled obtaining much richer data because investigation was based on participants’ natural actions and real events in their lives, which otherwise would have been missed if the study was researcher-directed (Palen & Salzman, 2002).

The study consisted of three phases: a two-week autoethnographical study to gain first-hand experience on study methodologies, a weeklong pilot study and a four-week field study with six participants. This research aims to:

1. Start the exploration of studying mobile applications in the context of physical activity, and

2. To uncover design implications for physical activity applications

This study may be potentially valuable for general HCI practitioners and researchers. The rest of this paper presents a brief literature review on methodologies used, describes study set up and procedures, presents the findings and discusses the methodologies and design implications uncovered.

- 10 -

CHAPTER 2. LITERATURE REVIEW

1.1. Autoethnography

Autoethnography is a type of qualitative research that is carried out through the researcher’s self, a method of studying aspects of the world through one’s own personal experience (Ellis, Adams, & Bochner, 2011). It is a combination of autobiography and ethnography and an alternative to the fact that social sciences are dominated by objective, third person literature separated from researcher’s own views, opinions and experiences. What is important, however, is that the research as objective as it might be is carried out and shaped by the researcher and this subjective lens is inherent to any qualitative research and removing it from the equation removes a very valuable part of it (Ellis & Bochner, 2000). Similarly to Dourish’s (2004) view of context, key assumption in autoethnography is that the context depends, changes and is influenced by the viewer and is not static and objective (Duncan, 2004).

Importantly the biggest advantage of autoethnography also presents its biggest limitations. Using autoethnographical data is typically criticised as lacking reliability, validity and use of analytics, and for being very subjective and emotional (Holt, 2003; Ellis et al., 2011). Researchers tend to handle these issues with a strong use of data analysis triangulation (Duncan, 2004; Ellis et al., 2011). As Duncan (2004) has put it, those doing autoethnography should think about how their study reaches a wider audience, clarifies confusing matters, can be applied further, supports discovery, and can be applied in a future setting. They should triangulate the evidence from multiple sources, peer review, create explicit research protocols, and integrate personal findings with universal theories and accounts. She concluded that As long as autoethnography is enlightening, it is acceptable that it does so through a degree of subjectivity as there is fully objective research.

1.2. Diary studies

There are two types of diary studies: (1) feedback diary studies where participants make open-ended entries or answer a series of questions about a particular event, and (2) elicitation studies, where participants are required to capture media of some sort depicting some aspects of an event to use as memory prompts later in research process during deferred contextual interviews (Carter & Mankoff, 2005). Entries in elicitation diary studies tend to be easier to make and are very effective for capturing transient and unpredictable events, but recall of events can lack details and specificity (Carter & Mankoff, 2005) compared to feedback diary studies. Traditionally, in feedback diary studies participants were asked to make paper-based entries with various pieces of information, such as context and

- 11 -

other details of events of interest, intentions and feelings information (Rieman, 1993; Abowd et al., 2005).

Paper diaries method has also been used in research of mobile technologies (e.g. Church & Smyth, 2009; Abowd et al., 2005) but more often it is too cumbersome and inconvenient for making quick data entries in mobile contexts (Palen & Salzman, 2002; another reference). More often electronic diaries are used (e.g. Amin, Townsend, Van Ossenbruggen, & Hardman, 2009; Kellar, Watters, & Shepherd, 2007), and many more lightweight diary study variations are now available, that can be tailored accordingly to various research objectives.

For example, Brandt, Weiss, and Klemmer (2007) propose a “txt 4 l8r” technique, where users are only required to capture little snippets of data in situ, and then complete a structured diary entry in their own time afterwards. A similar approach was used by Sohn, Li, Griswold, and Hollan (2008), where participants were required to send short text messages throughout the day to a server, which would compile them into full size diary entries. Another interesting method was developed by Palen, Salzman and colleagues (Palen, Salzman, & Youngs, 2000; Palen & Salzman, 2002) during their research of emerging mobile telephony use. In their studies, participants were required to dial a designated phone line, and either leave an open-ended diary entry, or be guided by choosing numbered options to report on particular activities.

Given that both feedback and elicitation diary studies have their shortcomings, Carter & Mankoff (2005) proposed using a combination of the two by asking participants to gather various kinds of media into a diary, such as photos, audio and text narratives. Media can then be used as memory prompts in deferred contextual interviews (Mancini et al., 2009). To ensure even deeper data collection, regular contact with participants has been reported as a way of building relationships with participants, increasing loyalty and increasing flow of ideas in diary studies (Palen & Salzman, 2002) and other filed study techniques, such as telephone interviews (Riegelsberger & Nakhimovsky, 2008).

1.3. Field testing techniques

Vredenburg, Mao, Smith, and Carey (2002) showed that field testing is believed to be an important method amongst UCD practitioners but is applied less frequently than other lower cost methods, such as informal expert review. Field testing can undoubtedly be more difficult (Brewster, 2002), but in recent years it has become a popular method for studying mobile technologies in various domains, such as geographical applications (e.g. Goodman, Brewster, & Gray, 2004; van Elzakker et al., 2008) and mobile search (e.g. Brown et al., 2013) amongst others. An array of methods were used and proposed for capturing usability, environmental and user data, such as portable cameras (Kangas and Kinnunen, 2005), wearable

- 12 -

microcameras (Mark et al., 2001), a combination of wearable cameras and screen capturing software (Brown et al., 2013), shadowing and think-aloud protocols (Goodman, Brewster, & Gray, 2004; van Elzakker et al., 2008).

Some of the methods and their combinations were found to be more successful than others, as reported by van Elzakker et al., (2008), who compared three combinations of UEM techniques in a field usability testing of mobile geo-applications. They found that using shadowing observation with think-aloud and audio/video recording was more powerful than using any combination of only two methods. However, as informative these results are, the study had several methodological limitations. Audio and video capturing was accomplished via cameras installed on a participant’s hat and a headset, both of which were provided by experimenters. Even though in their experiment wearing such equipment did not disturb participants, in a study of running it would be too cumbersome because of its dynamic nature.

More lightweight ways of media recording in field studies with proposed success in research are small wearable cameras, such as placed into a plastic pocket on users’ chest (Brown et al., 2013). Mark et al., (2001) raised privacy concerns when using such method, but putting one or more visible signs informing passers-by about video footage being taken and giving them an opportunity to object and discuss it with researchers is usually considered as an appropriate solution (vom Lehn, Heath, & Hindmarsh, 2002; Gutwill, 2003).

Disturbance of natural user behavior in field studies has been a major concern, irrespectively of methodology used. Isomursu et al., (2004) expressed concerns that presence of evaluator filming users interactions with a mobile device is too obtrusive and disturbing for users and makes user experience artificial. Just mere fact that users are being observed in a study can impact their behaviours, a phenomenon referred to as The Hawthorne Effect (Draper, 2013) is important to consider. A way around this effect can be using the participant observation method, in which instead of just observing users performing studied activities, the researchers takes active part in the process, and thus becomes the participant (e.g. Lethbridge, Sim, & Singer, 2005). Combined with participant-directed approach (Palen & Salzman, 2002), i.e. allowing users to carry out their normal activities rather than providing them with scenarios that they have to complete; will enable gathering truly naturalistic data about usage of mobile applications and any usability problems found would be the ones users would encounter in their natural interactions, i.e. real usability problems. The extent to which a UEM can help find real usability problems is one of the proposed measures of methodological success (Hartson et al., 2001).

Using think-aloud protocols is a useful addition to media capturing (e.g. Goodman, Brewster, & Gray, 2004; van Elzakker et al., 2008; Lethbridge, Sim, & Singer, 2005) and generally is not reported to impact participant behaviour, even

- 13 -

though potential difficulty is that due to high cognitive load participants may be less prone to think aloud (Beck et al., 2003).

- 14 -

CHAPTER 3. MAIN CONTENT CHAPTERS

1.4. Method

Autoethnography

The first phase of this research was a two-week autoethnographic study. It was conducted to carefully explore limitations of chosen methodologies and to control for them as much as possible before proceeding with pilot and main user studies. Six self-study runs were completed overall, three using Runtastic and three using Endomondo on an iPhone 4.

Diary studies, think aloud, and video capture similar to Brown et al. (2013) where a video camera was placed into a transparent plastic bag and placed on my chest (Figure 1), were tested. To avoid privacy issues associated with video recording in public places, I checked if running routes were indeed public property, and attached a paper sign to my back, informing passers-by that video footage was being taken, and inviting them to discuss any objections (Figure 2).

Figure 1. Video camera was placed into a transparent plastic bag on my chest and the information paper sign was placed on my back

!"#$"%!&$!'("$)"*+'"%,*"+'!-.*/!0)"/0+1)"2"!%"),+"#*."3,'3.*'.1"#4,+0"4.!'("#33!1.'0#&&)"%!&$.1"5"0#&6"0,"$.7"

Figure 2. Information sign that was placed on my back

- 15 -

Two variations of diary study were tested: open-ended word processor diary study and voice note diary study following research of Palen and Salzman, 2002. Voice note variation was used as a more lightweight method after I found word-processor diary to be cumbersome and time consuming, resulting in poor report and punctuality. In fact voice note diary study fitted into my post-running routine perfectly as I usually do a five-minute cool down walk. As this may not be the case for user study participants, an online questionnaire was launched to discover people’s typical post-running behaviours.

The questionnaire collected 23 responses (see Appendix 1); of them 17 were regular runners. Most frequent behaviours immediately (5-10 minutes) after running reported in open-ended question were: stretching (9 responses), having a drink (8), and walking (5), similarly to my own routine. Some respondents also mentioned that they would like to enter diaries via text.

A decision was made to give participants both options of entering their data, text and voice, and in further study phases Catch (catch.com) was used (Figure 3) because it allows for both voice notes, text memos, and pictures, and this can then be used as a memory trigger for deferred contextual interviews (Carter & Mankoff, 2005; Mancini et al, 2009).

- 16 -

1 2 3

4 5 6

Figure 3. Catch user interface. 1. Users can log in via different options. 2. Log in via Gmail. 3. Instructions screen. 4. Add a note: time, photo, text note, voice note or task. 5. Home screen. 6. Autoethnography folder where notes were saved to

Unfortunately, during main user study Catch announced they were going out of business, and it is no longer be accessible for future use.

Pilot study

To test equipment and the chosen set of methodologies with someone other than a researcher, a one-week long pilot test was conducted prior to the main user study. The Pilot participant was an occasional runner and owned an iPhone 4, and had briefly tried both Runtastic and Endomondo in the past. There were three points

- 17 -

of contact with the participant: at the start of the study, in the middle and at the end. The first and second points of contact involved a semi-structured interview before and after running sessions together, and the last was a debriefing session and semi-structured interview. Pilot participant did two runs by himself throughout the study using Runtastic. Data from pilot study would not be included into the final analysis because based on the findings of the pilot study, the main user study was modified, and participant had a free choice of which application he wanted to use. Data gathered was used to improve methodology, including improving the questions and not attaching the video camera to clothing, which could make marks on participants' clothing.

Main user study

Participants

Six participants took part in the main user study; three males and three females aged 23 to 33 years old (mean age 27 years). Participants were recruited through friends and colleagues rather than general advertising due to potential dangers of running alone with unfamiliar people. To control for motivational factors only those who already ran on a regular basis and already use any mobile devices to track their performance were recruited.

Two participants had used other running applications, not used in this study, on a regular basis to track their performance, three participants had tried applications in the past but started using GPS watches, and one participant had used many applications but did not find the one she was satisfied with entirely.

Three male participants have integrated running into their everyday lives, and it has become a hobby, more or less enjoyable for all three. They will be referred to as Hobby Runners 1,2 and 3. All of them used GPS watches for tracking their runs prior to the study, and Hobby Runner 2 also sometimes used Nike+ for competing with a friend. Hobby Runners had a positive outlook on challenge and struggle in running, and felt generally good when contemplating their runs. For them running was an enjoyable activity and they frequently participated in various races, and Hobby Runner 3 was even a member of a running club. They were fast-paced long-distance runners and challenged themselves to become better than they were before.

Female participants tended to run for health-related reasons, such as staying fit or losing weight, and for them running was more of an activity they felt they had to do, rather they wanted to. These participants will be referred to as Health Runners 1,2 and 3. For Health Runners struggle (or what they called “not being good enough”) in running was a barrier to enjoyment. They displayed a “tick-off” approach to their runs, i.e. to get it done and get out of way so that they can stay healthy or become thinner. They often unfavourably compared themselves to other more skilled runners, resulting in being upset or low self-esteem. Health Runner 1

- 18 -

was a medium-paced medium distance runner, Health Runner 2 was a fast-paced short-distance runner, and Health Runner 3 was a slow-paced long-distance runner.

Apart from Health Runner 1 who had been running for about six months, all had several years of running experience but participants differed in their attitudes to running.

There were several overlaps in some of the variables, such as Health Runner 2 closely monitoring her own performance over time and Hobby Runners sometimes feeling lazy and unwilling to run at times, however the patterns were quite distinct.

Because of a small participant sample and a different direction of this research the finding of two types of runners is not conclusive of a link between gender and motivations for physical activity. Previous research found that body dissatisfaction (and subsequently displaying health and body image motivations for performing physical activity as seen in current research) is correlated with self-esteem, rather than gender as both men and women display body dissatisfaction (Furnham, Badmin, & Sneade (2002).

Materials

The study was run on participants’ own iPhones and audio and video capture was done using Cisco Flip MinoHD pocket camera.

Individual Gmail accounts were supplied to participants specifically for use in this study. I knew their credentials and could log in freely to access diary and running data.

Design

The user study lasted four weeks, and was split into two two-week phases. For every participant each phase started with a semi-structured interview and a field testing session using one of the applications, and they had then to use that application as a part of their normal running routine for the next two weeks, keeping a digital diary. Participant used one application per study phase. The order at which participants were asked to use applications was determined using a Latin square method for counterbalancing (Appendix 2). At the end of each field testing session participants were interviewed about their experience. At the start of second field testing session, and at the end of the study participants went through their diary entries in a form of deferred contextual interviews. Participants were debriefed at the end of the study.

- 19 -

Participants were compensated for their participation using a variable compensation scheme based on recommendations proposed by Palen and Salzman (2002), and consisted of a baseline reimbursement of £15 for general participation and added compensation for every diary entry they made after running with a double rate (£0.70) if a diary entry was made within 30 minutes after the end of the run. This scheme was designed to increase motivation to make timely diary entries and ensured that the overall compensation would still be considerable irrespectively of how many times participants typically ran.

Procedure

User study consisted of four points of contact with each participant: 1. Entry briefing and interview 2. Field testing session with Application 1 and post-run interview 3. Pre-run interview, field testing session with Application 2 and post-run

interview 4. Exit interview

After initial briefing, each participant signed a consent form and was interviewed about their running background and previous experience with tracking devices. All interviews in the study were semi-structured and quite informal to build trust and rapport with participants. Then on the same day the first field testing session took place, in a form of a friendly run together. It was scheduled to take place at times and locations according to their normal running routines. Participants were not given any tasks or scenarios about intended application, as the study investigated their natural usage. Participants were asked to think-aloud whenever they interacted with their phones, and both audio and video was recorded using handheld camera. Whenever a problem of glare occurred, it was partially solved by narrating everything that went on the screen. After running session participants were interviewed about their experience with applications.

Participants were asked to keep a digital diary in Catch every time they go running on their own. Diary pointers were uploaded into Catch to guide participants on matters that they had to report about (Table 2). They were given a choice of data entry method (text or voice), and were asked to take a photo of something memorable within their running environment, or a screenshot of anything interesting within the applications they felt could illustrate their diary well.

- 20 -

Table 1. Diary pointers used in the main user study

Hi! Can you please tell me as much as possible about you run and interaction with the application today? You can also take screenshots of interesting things. Make sure that you cover the following points:

• Goal and reason for running

• You feelings before/during/after the run

• Place where you ran

• Weather conditions

• How many times you interacted with the app and why

• Describe each interaction in detail

• What you liked and disliked in the app

• Anything interfered with experience?

• Describe your perfect interaction for today’s usage – different to reality?

• Any other interesting information

Two deferred contextual interviews took place, one before second field testing session, and another at the end of the study. During these interviews participants were asked to go through photos and screenshots in their diaries and to report as much as possible about their runs. They were asked to step back from what they had already reported in their diaries, and tell all the memories that photographs triggered.

At the end of the study, participants were debriefed, and asked for their general feedback on the study and the methods used.

Data Analysis

All audio and video data was transcribed verbatim. Overall, 13 hours of audio and video recordings and several text-based diaries were collected.

Latent thematic analysis was chosen for systematic data analysis because of exploratory nature of this study as it provides a lot of flexibility in organising

- 21 -

qualitative data and allows describing it in rich detail (Braun & Clarke, 2006), which was especially needed in absence of an a priory hypotheses. I needed a method that would not enforce any data analysis constraints because I did not know beforehand what type of data, and what detail I would discover, and because any methodological constraints could have limited the discovery part of this research.

Due to novelty of this research and a limited participants sample I was not looking to use the data for creation of any theoretical frameworks, which some of methods similar to thematic analysis such as grounded theory allow doing (Charmaz, 2000).

Data analysis was conducted in four steps:

1. Distinct statements from each participants were transferred onto sticky notes and participants were colour-coded; all data were

2. First round of thematic analysis, where several general themes emerged:

a. General running behaviours and attitudes

b. General attitudes towards using technology in running

c. Data about Endomondo

d. Data about Runtastic

e. Data about previous running applications participants had used

3. Second round of thematic analysis was performed on data relating to Endomondo and Runtastic, and lower-level themes emerged in each data set, such as statements about music player, starting the run, usage behaviours, etc.

4. Final round of thematic analysis was performed on themes from step 3 to discover specific patterns

Data was triangulated across three data sources (interviews, diaries and think-aloud protocols) and with literature. Term triangulation refers to “using more than one research approach to address the same question” (Mackay & Fayard, 1997, p.9). It helps in increasing reliability of and confidence in results (Wong & Blandford, 2003).

- 22 -

1.5. Results

Results section consists of three sub-sections. First I report findings specific to Runtastic, secondly I report findings specific to Endomondo, and then general patterns that I observed across both applications.

Results 1: evaluation of Runtastic

General interaction patterns while running: some runners did not interact with Runtastic, whereas others only sought specific information

Overall I gathered information about 18 distinct use cases with Runtastic, including running sessions with participants and runs mentioned in their diaries.

Only five use cases did not involve any direct interaction with Runtastic apart from listening to voice feedback (Table 3). Health Runner 1 and Hobby Runner 3 did not provide reasons for not interacting with Runtastic, but others did.

Health Runner 3 was interested in distance information while running, but she kept estimating it in her head and did not look at Runtastic because “it did not occur to her” to do that [1]. Health Runner 2 in turn did not interact with Runtastic during her run because she generally dislikes using applications while running and wants to focus on the run itself [2]. Similarly, Hobby Runner 2 elaborated on not using Runtastic while running to check statistics because interaction interrupted his pace, which was undesirable for him [3].

Table 2. Reasons for not interacting with Runtastic during running

Participant Reason Quote Data source

Health Runner 3 It did not occur to her to look up at Runtastic when she needed distance information

[1] “I didn’t look at it because I kind of… Actually, that’s interesting because […] when I was asking about how far we’ve gone and I was trying to guess, I could just look at the app. […] it just didn’t occur to me to do it.”

Field testing

- 23 -

Health Runner 2 Generally dislikes interacting with running applications

[2] “I hardly ever used the app because I've mentioned before I don’t really like to use the app while I'm running. I like to focus on the run, so I obviously can't really describe any interactions there.”

Diary

Hobby Runner 2 It disrupts his pace [3] “The biggest problem with using your mobile for your stats is it does not work well during running. Interaction with the app requires an interruption of pace.”

Diary

Other 13 Runtastic use cases involved direct interactions during running, and three most common types of information that participants tended to look at while running were duration, distance and pace; sometimes they looked for more than one (Table 4).

Table 3. Types of information participants sought when interacting with Runtastic while running

Type of information Number of use cases (N users)

Duration 5 (5)

Distance 7 (6)

Pace 4 (4)

Map 1 (1)

Not specified 1 (1)

- 24 -

Interactive dashboard on main run screen is too cluttered and confusing

Main run screen in Runtastic consists of six main user interface (UI) elements (Figure 4):

1. A status bar consisting of Runtastic logo, type of activity being performed, hear rate information, cardinal direction and GPS status

2. An interactive dashboard displaying the main running statistics 3. A tool bar with several controls: music player, camera, screen orientation

switch, and voice feedback settings 4. A live map 5. Advertising 6. Slider to pause/stop session

Here I report findings on interactive dashboard, and findings on the general mode of interaction for stopping and pausing sessions are described in the “Cross-application findings” section; other elements did not yield any feedback.

The dashboard displays the main information about the session, such as duration, distance, speed and average speed. There are five views that can be switched by tapping anywhere on the region of this element (Figure 4), and circular label in the middle of the dashboard indicates, which view is currently chosen.

- 25 -

Generally it was reported as too cluttered and with text that was too small and hard to see during running (Table 5). Some participants also commented on changing dashboard views and general screen visibility.

For instance, Hobby Runner 1 found information on the dashboard hard to read because the font size was small and the four pieces of information were too close to each other in that square alignment [4]. Similarly, Hobby Runner 2 found text too small and hard to read [5], and described the dashboard as “too cluttered” because there was too much information displayed at the same time and too much going on [6]. Health Runner 3 complained that if she was to look for her distance on the dashboard while running she would probably miss it because the whole of dashboard was quite tiny [7]. She did like that it allowed switching between different views, but she found the numbered label in the middle misleading, because she was trying to tap on it exclusively, but then realised that she could tap the whole dashboard [8].

Figure 4. Five different views of interactive dashboard in Runtastic

- 26 -

Table 4. Participant feedback on visibility of interactive dashboard on the main run screen in Runtastic

Participant Complaint Effect on user experience

Quotes Data source

Hobby Runner 1

Hard to read because of small font and square layout

None [4] “On font, stuff could be a bit bigger, I think, and maybe not in a square because that’s just quite close to each other, so reading it is not as easy.”

Post-run interview

Hobby Runner 2

Text too small; cluttered layout and too much information

None [5] “[…] in terms of the information, it's a bit harder to read as well. […] This text is really tiny.”

[6] “It's very cluttered. There's a lot of stuff going on here. You have the username, GPS signal. They could probably do away with the banner, distance. Okay, time. It's a little bit too cluttered here. I don't like it.”

Field testing

Health Runner 3

Text is too small; did not immediately understand that she could switch dashboard view by tapping the whole of dashboard

Confusion [7] “And, to be fair, I don’t think I would have been able to see it on that app because it was tiny. It’s really small, that one. […] When it shows you the distance on screen while you’re running. […] Yes, the button-y thing, the blue-ribbon thing, yes. It’s quite tiny. I probably would have missed that.”

Post-run interview

- 27 -

[8] “I like the way it does that, but it's tiny little buttons. I mean, it's got a large range, but still. Oh, you tap the whole thing. That's a bit misleading. I was trying to tap that one, but actually you can tap the whole thing.”

Hobby Runner 2 also explored the view switching, but did not like the fact that there was no shortcut to get to the view he wanted apart from tapping through, if for example he was on the third view but wanted to get to the first one quickly (Table 6; [9]).

Table 5. Participant feedback on switching between dashboard views in Runtastic


Quotes Data source

Hobby Runner 2

Could not quickly switch to the desired dashboard view and had to tap through all views

None [9] “I guess you have to tap through these to get through them as well. You can't just quickly jump from one to another. […] I think that's bad, because obviously there's only four, but if you're on the third one and you want to get to the first one, you have to tap around.”

Field testing

- 28 -

Pausing and saving the run action sheet showed the need for preventing accidental data deletion

To save running session in Runtastic users first need to pause it, and then they are directed to the “Pause” action sheet where they can choose between three options: stop and save session, stop and discard session or resume session (Figure 5). To avoid users accidentally discarding their session when they want to save it, the “discard” button is coloured in red.

Figure 5. Pause action sheet in Runtastic

All participants who commented in this functionality complained about it. Lack of space between “stop and save” and “stop and discard” buttons was the main problem (Table 7).

According to Hobby Runner 2 it can be very easy to tap the wrong button because buttons are too close to each other and his hands are usually shaky after running [10]; he also wished for the “discard” button to be smaller because it is not an option that he would normally use anyway [11]. Health Runner 1 was literally “a bit terrified” that she would hit the wrong button and discard her session next time, because three “little” buttons require attention, and is prone to not being too focused on the screen while running [12]. Health Runner 3 also had a problem with saving the run as she reported in her diary. She nearly pressed the “discard” button when she actually wanted to save the run because the buttons are too close together [13].

- 29 -

Table 6. Participant feedback on pause action sheet in Runtastic


Quotes Data source

Hobby Runner 2

Save button is too close discard button

Worry [10] “Obviously stop and save – this is a bit stupid, because stop and save is really close to stop and discard. If you're still a bit high after your run, and a bit shaky, you might hit the wrong button and accidentally delete the whole run.”

[11] “[…] you'd not like to want to discard, so that could probably be a smaller button, that's more hidden away.”

Field testing

Health Runner 1

Buttons are too small and require attention

Feeling of terror; worry

[12] “I'm a little bit terrified the next time I go for a run I'll hit "stop and discard" instead of "stop and save" […] Yes, you slide first, but then something grabs your attention, its three little things, so it's like accidentally click the wrong button. I can see myself doing that, not paying attention, being on the run […]”.

Field testing

Health Runner 3

Action sheet buttons are too close together

Nearly deleted her data

[13] “I nearly pressed the discard session option, when I went to save. They're quite close together.”

Diary

- 30 -

Adding post-run reflective information was useful for some but annoying to others

At the end of each running session in Runtastic users are prompted to enter reflective information, such as their general feeling after the run, surface on which they ran, weather conditions, temperature, heart rate or add a text note (Figure 6).

Figure 6. Adding post-run reflective information in Runtastic

Every participant provided rich feedback on this screen. Their opinions ranged from being very excited to being annoyed (Tables 8 and 9).

According to Hobby Runner 3 entering post-run information in Runtastic was very easy and quick, just a few taps on friendly icons and he seemed very positive about it [14]. For Hobby Runner 2 adding post-run information can be useful as a way to find out patterns in his behaviours or preferences, such as to find out that he likes running in the rain; especially if Runtastic could pull that data together [15]. Hobby Runner 1 exhibited a neutral relationship towards this feature, as he indicated that he typically enters some post-run notes just in case, but never re-visits them [16].

- 31 -

Table 7. General participant feedback on adding post-run reflections in Runtastic

Participant Opinion Effect on user experience

Quotes Data source

Hobby Runner 3

Very easy and quick to do

Positive feeling

[14] “It gave me a few things, which were quite easy just to go, “Actually, yes, I was feeling good,” the weather was like this, the trail, just a few presses and it was done […], friendly, […] easy. […] was like, yes, bang, bang, bang, could be done in a few seconds.”

Post-run interview

Hobby Runner 2

Helpful to find out behaviour patterns

None [15] “I probably would [enter reflective information], because I think it's quite helpful to be able to find out – maybe you find out that you really like runs in the rain, for example, or if they have a way to pull that data together.”

Field testing

Hobby Runner 1

Does it but never looks back at that information

None [16] “[Do you usually keep notes?] Sometimes, maybe one or two words. […] In the previews for runs, they usually have a short description of the notes, so I just do that, and that's it. I never really look at them, but it's just there, just in case.”

Field testing

- 32 -

While these two participants felt all right entering reflections, others did not share their opinion. Health Runner 2 stated that it was not important for her because if there were something notable about the run, she would just write it down [17]. Health Runner 1 got slightly annoyed by having an extra step between stopping and saving her session because she did not like interacting with her iPhone in a busy park in London, where it can get stolen easily. With other applications she typically stops her run, switches off her phone and walks home to look at her statistics inside [18].

Table 8. Negative participant feedback on adding post-run reflections in Runtastic


Quotes Data source

Health Runner 2

Not useful None [17] “I don’t really think it’s that important. I think if there is anything I really want to take note of during the run, I’ll write it down.”

Post-run interview

Health Runner 1

Too much interaction at the end of running

Annoyance; worry

[18] “I had a lot more screens than I was used to. I just thought it was going to be "stop." I would turn off the screen and to actually walk home like I normally do and then look at it when I get home. Instead, it was like, "Do this. Do that. Tell us what your run was before we save the run," and then it saves the run. That was a little bit annoying because I don't really like interacting with my phone that obviously in public normally. […] I'm in a park, and it's London. I've got a pretty nice phone.”

Post-run interview

- 33 -

Health Runner 3 was also not very happy about having to enter reflective information before saving the run because there was too much to see on the screen in the iPhone pouch on her arm and she had to take it off (Table 10). She mentioned that in Nike+, which she typically used, she could just skip through this step, but she was not sure that she could do the same in Runtastic. She indicated that a lot of her runs were the same and nothing interesting usually happened [20]. For her adding reflective post-run information can be useful only for exceptional runs, such as when breaking a personal record, or if a run was just particularly good or bad [19]. However, she never looks back at that information because it is not detailed enough to be a memory trigger. In fact she would not want to enter more detailed information or trigger her memory at all [19].

Table 9. Participant feedback on usefulness of adding post-run reflections in Runtastic


Quotes Data source

Health Runner 3

Only useful for exceptional runs

None [19] “If I had a really bad run, then I’ll be more likely to mark, “That was really bad.” For a normal run, I probably wouldn’t do that. Or if it’s a really good run, I would also maybe mark it. Not that I ever look back on that information. […] Because […] it’s a memory trigger for a run, for me. When I look at that information, it’s not really detailed enough for it to be memory trigger, so it doesn’t work. […] I don’t really want it to trigger my memory. […] I feel like if a run’s special, then… Yes.” [20] “ […] more interesting for me to see

Post-run interview

- 34 -

when I’d run my fastest 10K or something and then maybe have some details about it, but not every run because a lot of my runs are the same. There’s nothing really very interesting that happens.”

To conclude, while Hobby Runners 2 and 3 liked adding post-run reflections, Health Runners 1 and 3 felt that it far too much interaction on saving the run, and that they may do it but either later or not all the time. For Health Runner 2 keeping these records was not important, and Hobby Runner 1 indicated that he usually enters some notes but never looks back at them.

Adding surface information showed to be an unnecessary feature that caused confusion

Users can also choose the surface they ran on in the post-run reflective summary. There are five surfaces: road, trail, off-road, mixed and beach. Icons are grey scale until selected and unlabelled (Figure 7).

Figure 7. Adding post-run reflections in Runtastic: surface icons

Adding surface information to the post-run reflections produced quite interesting feedback. Even though I did not discover participants’ opinions in terms of general usefulness of this feature as such, there are a few comments worth mentioning (Table 11).

- 35 -

Health Runner 1 felt that it was unnecessary because she would know what the surface was during her run by looking at the map in the session overview [21].

Hobby Runner 2 stated that she should not have to enter surface information manually because Runtastic should be able to get that information automatically from tracking where he ran [22].

Table 10. General participant feedback on adding surface information to post-run reflections in Runtastic


Quotes Data source

Health Runner 1

Unnecessary feature

None [21] “I would know what the surface was like by looking at where I ran, so it's kind of unnecessary.”

Post-run interview

Hobby Runner 2

It should be entered automatically

None [22] “Some of this I shouldn't have to enter, really, because surely they can tell where I'm running and stuff like that. They should be able to just get it automatically.”

Field testing

Most participants had troubles understanding meaning of surface icons. Generally, the first and the last ones, road and beach everyone named easily, but all others were not as obvious (Table 12).

Hobby Runner 2 had problems when choosing an icon for the surface we ran in, which was a mixed terrain. He got the meaning of most of them except for trail and off-road, which he did not manage to differentiate [24].

Similarly Hobby Runner 3 [25], Health Runners 1 [26] and 3 [27] could not grasp the difference between trail and off-road surface icons, while guessing road, mixed and beach surfaces.

- 36 -

Health Runner 2 had some general troubles choosing a surface icon, it took her several minutes until she picked mixed terrain correctly; the following quote illustrates well how confused she and other participants were while doing it:

[23] “Surface all right? It’s kind of a mixture of both. So I’m guessing this one maybe. No, maybe not. What’s this one? No, not that one. Maybe that one. It might be better.” (Field testing).

Table 11. Participant troubles differentiating surface icons when adding post-run reflections in Runtastic


Quotes Data source

Hobby Runner 2

Could not differentiate trail and off-road icons

Confusion [24] “I guess this is the road. I guess this is running on terrain, but smaller roads still. This one is terrain, the third one. This one is half and half. This one looks like maybe a beach or something. […]”.

Field testing

Hobby Runner 3


Confusion [25] “I think the first one was road, the second one woodland or off-road, the third one, I haven't got a clue. Maybe if I touched it? That doesn't really – maybe that's just off-road, but not in woods. Oh, that one's road and off-road, and then that's beach.”

Field testing

Health Runner 1


Confusion [26] “Well, that would be a road. That would be more like a path with trees and areas. This looks like a beach, maybe. That, I think, is

Field testing

- 37 -

like a dirt and a road. This one, I'm not sure what this is, but I guessed that one because that was trees around these paths. We were running on a path through the trees. Well, not quite forest-y, but it was park-y enough.”

Health Runner 3


Confusion [27] “That's road, […]. This one is trail. I can't actually – what's that? What's the difference between that? It's got two extra trees. That's just trees. [For the second and the third one?] Yes. That's just trail, trail. [The fourth one?] Trail and road, so a mixed run, which is what we've had, actually, and beach.”

Field testing

Adding temperature information was not useful for everyone

Users can also add temperature information to their post-run reflections, and it can be added manually or automatically. Out of three participants who gave feedback on this feature only Health Runner 3 liked it (Table 13; [28]).

For Health Runner 2 it could only be useful if Runtastic could make a correlation between temperature and her happiness while running [29]. Temperature information was not at all important for Hobby Runner 2 and he was not sure if he wanted to input that into his reflections [30].

- 38 -

Table 12. Participant feedback on adding temperature information to post-run reflections in Runtastic


Quotes Data source

Health Runner 3

Liked this feature

None [28] “[What do you think about the temperature?] I actually quite like that, to be honest.”

Field testing

Health Runner 2

Only useful if it makes a correlation between temperature and happiness

None [29] “I don’t think it needs to record temperature, unless it wants to make some sort of like correlation between temperature and my happiness when running.”

Post-run interview

Hobby Runner 2

Not useful None [30] “I'm not sure I would go with temperature, because that's not that important.”

Field testing

Post-run information screen features produced mixed feedback

Straight after session has been saved Runtastic presents its summary statistics. There are four statistics screen that users can flick through: the main run summary, heat map of speed, speed and elevation graphs and splits information and elevation chart (Figure 8).

- 39 -

However as I observed during field testing most participants tend to only seek information that is most important to them ignoring other statistics. Four participants elaborated on what they looked for on the main run summary: everyone needed duration, distance and average pace information, and only two participants were interested in elevation statistics. No one needed average and maximum speed, calorie and pause information (Table 14). Three participants explored heat maps, and their feedback is presented in a separate maps section below.

Figure 5. Four types of post-run information in Runtastic (from top left clockwise): general run summary, heat maps, elevation and splits screen, elevation chart and speed

- 40 -

Table 13. Type of information that participants sought in the post-run statistics.

Information type Number of use cases (N users)

Duration 4 (4)

Average pace 4 (4)

Distance 4 (4)

Elevation 2 (2)

Hobby Runner 3 and Health Runner 2 commented on having multiple statistics for their runs (Table 15). In Hobby Runner’s 3 opinion having so many different graphs and summaries was not very useful but quite interesting because he liked statistics [31].

Health Runner 2 however was generally not interested in any sorts of post-run graphs, and in her opinion graphs in Runtastic were too detailed and hard to make sense of; she felt that they required too much thinking and that she needed to look at them for 10 minutes before she could figure out what they meant [32].

Table 14. General participant feedback on having multiple graphs in Runtastic


Quotes Data source

Hobby Runner 3

Not useful but interesting

None [31] “It's not particularly useful. It's interesting,[…] I quite like statistics.”

Field testing

Health Runner 2

Too detailed and hard to make sense of

Overwhel- med

[32] “A little bit too detailed. I felt like I had to sit down and look at them for 10 minutes to figure out what they meant. It was too much information for me to really make sense of. […] It required too much thinking. […] not interested in graphs.”

Post-run interview

- 41 -

Participants expected live map to be interactive and unlocking post-run map produced frustration

Runtastic offers users two maps (Figure 9). The first is a non-interactive live map on the main run screen, showing where user is currently running. The second is an interactive heat map of pace/speed available in the post-run statistics, which needs to be unlocked before use.

While usefulness of these maps differed across participants (Table 16), most of them experienced problems with maps’ interactivity (Table 17).

Health Runner 1 indicated that she did not need a map while she ran because she could see the area in front of her, especially in the park [33]. However she was interested in having a map feature in her post-run statistics to be able to know where she ran in case she really liked it or ran in a new place [34]. Hobby Runner 3 responded to post-run map with a very enthusiastic feedback. For him having a map was a reminder of what he had done and where he had been, and would be useful if he wanted to retrace his steps [35].

Figure 6. The two types of maps available in Runtastic. Left: non-interactive live map. Right: interactive heat map of pace/speed. The heat map is locked by default so that users can flick through different screens with post-run information, and to interact with it one needs to tap on a padlock icon in the top right corned of the map. To signify that there are more screens available, there are four dots just below the advertising on each screen – page control to allow users to see where they are in the series of multiple information screens.

- 42 -

Table 15. Participant feedback on usefulness of maps in Runtastic


Quotes Data source

Health Runner 1

Live map is not useful

None [33] “I know where I'm running in front of me. Especially if I'm running at the park, I don't need a map at all on the screen.”

Post-run interview

Health Runner 1

Post-run map is useful if running in new place or in case of a great run

None [34] “It's a pretty nice feature to know where I ran, in case I run somewhere and I'm like, "Yes. Where's that run? It was really great." My run's pretty much always the same place, but if I moved or something like that and try and do a different run.”

Post-run interview

Hobby Runner 3

Post-run map is useful for retracing steps and visualisation

None [35] “That's the map […] That's really good as well, because obviously it shows you where you've been, what you've done, and if you're wanting to retrace your steps, do that exact same one again, that can help remind you.”

Field testing

Health Runner 2 actually tried using the live map to navigate her home when she got lost during one of her runs. She wanted to move the map to see the whole route, but since the map is not interactive she got frustrated and gave up and just made a guess on how to get home. She presumed that there was a way to work the map, but it was difficult to solve this while running and she did not want to stop [36].

In fact she was not the only one who expected the live map on main screen of Runtastic to be interactive. Hobby Runner 1 disliked that the map was static, as he suggested that it would be useful for him if he had a route planned and needed to

- 43 -

check his way. However, if a map only displayed only a small part of the route it meant that he would have to either remember the whole route and then the map loses its utility, or check it every couple of blocks, which may be too distracting [37].

Table 16. Participant feedback on interactivity of live map in Runtastic


Quotes Data source

Health Runner 2

Did not figure out how to unlock live map while running

Frustration [36] “Unfortunately, the app did frustrate me because it showed me where I was, but it didn’t let me move the map around to figure out how to get home. […] I couldn’t seem to the route or to move the map around to see where I was. Maybe there was a way to do it, but I couldn’t figure it out while it was running, and I didn’t really want to stop, so I just kind of made a guess as to how to get home.”

Diary

Hobby Runner 1

Wanted live map to be interactive

None [37] “One thing that I didn't like was when I was looking at the map in here, I couldn't actually touch down and move the map. […] it would be nice to have a map that you can zoom and pan out. […] Say, for example, I tend to have a route planned, and I want to go back—backwards, right? I don't really get a good view, so

Field testing

- 44 -

I basically have to remember where I am going or check the map every two blocks because then it'll update and show me what I didn't see before.”

I also discovered an issue with heat maps: three participants who had used them had troubles interacting with them (Table 18). Two out of three participants did not manage to unlock the heat maps; none of them noticed the padlock icon that they had to tap. Health Runner 1 got extremely frustrated when she repeatedly tried to zoom in to the heat map during our post-running interview. She tried pinching-to-zoom a few times and then gave up and left the screen because she could not stand it anymore [38].

Hobby Runner 2 did not understand the meaning of breadcrumb-dots and did not see the padlock icon. He tried a few times but could not interact with the heat map and found it annoying. He mentioned that it was not consistent with other maps on the iPhone [39].

Only Hobby Runner 3 managed to interact with the heat map. He correctly understood what the four dots underneath the map meant, and then after ineffectually trying to zoom into the map he realised that he had to press on the padlock icon to unlock it. It was not immediately obvious but he was quite happy about it [40].

Table 17. Participant feedback on interactivity of post-run map in Runtastic


Quotes Data source

Health Runner 1

Could not unlock and interact with post-run map

Extreme frustration

[38] “I don't know. It's not letting me move, okay? […] then when I zoom in, it doesn't even let me redistribute the screens, which is weird. I can't move the screens. I don't like that. […] Man, that is absolutely terrible. I can't stand the screen anymore.”

Post-run interview

- 45 -

Hobby Runner 2

Could not unlock and interact with post-run map

Annoy-ance

[39] “It's like a heat map kind of thing, but it's really weird that they have the dots on top of an ad. […] This is annoying. I can't actually move around, to adjust this, whereas everywhere else in the iPhone, you would just drag it around. I'm not sure how I would actually do that.”

Field testing

Hobby Runner 3

Had troubles unlocking post-run map but managed to do it

None [40] “I think you have to lock […] to scroll – click and see there's four little dots, which suggest to me there's more information to have a look at. […] I pressed on lock to zoom – I pressed the button to zoom, but then when I tried to swipe, I realized I wasn't getting anywhere. I realized I had to press that padlock again, to allow me to swipe.”

Field testing

Summary of usability problems found in Runtastic

I discovered a variety of usability problems in Runtastic (Table 19). Features and functions with most problems were post-run heat maps where users did not understand the unlocking mechanism, adding post-run reflective surface icons, which produced a lot of confusion because participants were unable to differentiate between them, stopping the run that was confusing, and saving session action sheet, which produced a lot of anxiety and even terror.

- 46 -

Table 18. Full list of usability problems found in Runtastic

Area/feature Problem N reports (N users)

Effect on user experience

Quotes

Stop run Did not immediately understand how to finish the run

3 (3) Frustration (1), extreme confusion and frustration (1), none (1)

77, 78, 84

Pause run Absence of audio feedback on resume

1 (1) Negative (1) 88

Main run screen, dashboard

Text is too small 3 (3) None (3) 4, 5, 7

Layout is cluttered 2 (2) None (2) 4, 6

Not clear that views can be changed by tapping on the whole of dashboard

1 (1) Confusion (1) 7

Could not switch to desired view fast enough

1 (1) None (1) 9

Main run screen

Hard to see in sunlight 1 (1) Annoyance (1), distracted from running (1)

94

Screen hard to see in a pouch; need for more orientations

1 (1) Decreased usability (1)

91

Hard to reach buttons in corners

1 (1) Decreased usability (1)

93

Save/discard action sheet

Buttons are too small and close together

3 (3) Worry (2), feeling of terror (1), near data deletion

10, 12, 13

Post-run reflections/general

Excessive interaction at the end of running

1 (1) Annoyance (1), worry (1)

18

Should be entered automatically

1 (1) None (1) 22

- 47 -

Troubles differentiating icons

5 (5) Confusion (4), extreme confusion (1),

23, 24, 25, 26, 27

Post-run graphs

Too detailed and hard to make sense of

1 (1) Overwhelming (1)

32

Post-run map

Trouble unlocking 4 (4) Frustration (1), extreme frustration (1), annoyance (1), none (1)

36, 38, 39, 40

Voice feedback

User expected it to be free

1 (1) Annoyance (1) 49

Wanted to hear how ahead or behind her target pace she was while running

1 (1) None (1) 50

Lack of integrated music player

n/a 2 (2) Extreme frustration (1), annoyance (1),

71, 72

Results 2: evaluation of Endomondo

Three general usage patterns: no interaction, using voice feedback only and looking at Endomondo throughout runs

I collected 28 use cases of Endomondo. There were three interaction patterns: (i) only to start and stop tracking, (ii) starting and stopping tracking and using audio feedback throughout the run, and (iii) starting and stopping tracking and directly interacting with the app throughout the run.

Three most mentioned reasons for interacting with the app during running were looking up pace information, looking at distance/time half way through the run to know when to turn back and checking time/distance to know how much is left until the target (Table 20).

- 48 -

Table 19. List of reasons for interacting with Endomondo while running

Reason for interaction Number of use cases (N users)

Looking at distance/time half way through the run to know when to turn back

3 (2)

See if the app was working properly 1 (1)

Checking time to decide when it is appropriate to start walking

1 (1)

Check time/distance to know how much is left until the target

2 (2)

Checking time (n reason reported) 1 (1)

Looking up pace information 6 (3)

There were also a few generic comments, such as Hobby Runner 1 who used a GPS watch in the past reported that interacting with the phone was “an annoyance” because he needed to unlock it first, Hobby Runner 3 interacted with Endomondo more when he was comfortable with the terrain he was running on; and Health Runner 3 who went running with a friend used the information on the screen to cheer her friend on into carrying on running.

Voice feedback was too quiet and some required higher frequency

Feature that received the most comments was voice feedback. Four participants reported using voice feedback as guidance to their performance at least once. Generally, it was said to be useful and enjoyable (Table 21).

- 49 -

Table 20. List of positive feedback about voice feedback feature

Comment Number of reports (N users)

Voice feedback useful for checking pace

1 (1)

Likes voice feedback, satisfied 4 (4)

Voice feedback makes user feel accomplished

1 (1)

Enjoys feedback on a good running day 1 (1)

Voice feedback is motivating and encouraging

1 (1)

However participants were not entirely satisfied with the amount of voice feedback they received. Voice feedback features that participants wanted (and not available in Endomondo) were feedback halfway into the target time/distance, feedback based on time intervals in addition to distance intervals (e.g. every five minutes AND every kilometre), information on being ahead or behind target pace, and voice feedback on completing a run. Health Runner 1 drew my attention to the difference between park and street running, saying that street running is more monotonous, and to feel that she was getting somewhere she needed more frequent updates than just every kilometre.

Health Runner 1 became repetitively annoyed with the voice itself, and called it “mechanical” and “robot”, whereas Health Runner 3 reported getting so distracted by the voice quality that she missed the information it was conveying:

[41] “Again, yes, I just didn't get any feedback of the run. I think I heard the robot voice, because I was so distracted by the robot voice, I didn't actually pick up what it said. I realized afterwards that I actually didn't know how fast I'd gone, or what pace, or how to even modify it.” (Interview).

Every participant who used voice feedback said it was too quiet: those who did not wear headphones failed to hear it completely, and those who were listening to their music through headphones said that it was quiet compared to their music, and it did not fade out music enough. In her diary Health Runner 1 reported that she kept mishearing “lap time” as “last time” and got very frustrated as she though that she was doing very poorly. Hobby Runner 1 said he would prefer glancing at his watch than trying to focus on hearing voice feedback.

- 50 -

Post-run information screen can be overwhelming

At the end of running Endomondo users are presented with variety of information, such as distance, duration, average speed, maximum speed, calories, hydration, heart rate, altitude, descent and weather (Figure 10).

However, not all the information presented was useful for participants, and as put by Health Runner 3,

[42] “it's too cluttered. There's too much information. It doesn't think about what you really want to know, […], just puts everything on the screen, […] It should be a bit more intelligent.” (Field testing).

Five participants were interested in total distance ran, four also needed time and pace information, two participants looked at splits, ascend and descend and maximum speed, and one was interested in calorie expenditure (Table 22). No one looked heart rate, altitude and weather information.

Figure 7. Post-run information screen in Endomondo

- 51 -

Table 21. List of information types participants were looking for in the post-running summary

Type of information Number of reports (N users)

Heat maps 6 (3)

Maximum speed 3 (2)

Time 7 (4)

How long max speed was maintained for

1 (1)

Distance 7 (5)

Ascend + descend 2 (2)

Pace 7 (4)

Calories 1 (1)

Splits 2 (2)

Two participants reported not trusting calorie and hydration information (Table 23). Health Runner 3 explained that she liked having calories information and that was a nice idea, but she did not trust the calculations [43]. Hobby Runner 1 indicated that he was not interested in anything other than distance, duration and average pace. According to him, calorie and hydration calculations depend on body weight and type, and since he had not entered that information, he did not trust the calculations [44].

- 52 -

Table 22. Participant feedback on calorie and hydration information in Endomondo


Quotes Data source

Health Runner 3

Likes calorie information but does not trust calculations

None [43] “I think calories as well, even though I don't actually trust the calculations, but it's a nice rough idea. Yes, as I said, I don't really trust it.”

Post-run interview

Hobby Runner 1

Calories and hydration are not useful; calculations are inaccurate

None [44] “Okay, distance and duration, average pace. Yes, I think we'll stop about here. I don't really care about everything else. I don't think it's accurate, anyways. Calories is based on your weight. I didn't even enter my weight. Hydration depends on your body type.”

Field testing

- 53 -

Those who looked at the splits screen, noticed turtle and hare icons representing slowest and fastest kilometres (Figure 11).

Figure 11. Splits screen. Hare icon signifies the fastest split, and turtle icon signifies the slowest split.

Participants’ opinions on those icons varied (Table 24). Hobby Runner 1 indicated that he understood what they meant, but did not really care [45]. Hobby Runner 2 liked the icons and thought that this representation was “pretty clever” [46]. Similarly, Health Runner 2 though that the icons were cute, and the she was immediately able to see, which kilometre was the fastest and which the slowest, and that was “neat” in her opinion [47].

The most interesting feedback came from Health Runner 3, who felt insulted by the icons, because she felt as if Endomondo was calling her a turtle, even though she had no problem understanding icons meaning [48].

- 54 -

Table 23. Participant feedback on turtle and hare icons on the splits screen in Endomondo


Quotes Data source

Hobby Runner 1

Does not care about splits icons; understands what they represent

None [45] “I don't really care about the icons. It's subjective anyways, I think. Fastest and slowest pace? Yes, I guess so.”

Field testing

Hobby Runner 2

Splits icons are clever; easily saw fastest and slowest split

Positive feelings

[46] “I like the icons that are showing – I guess this was the fastest kilometre and this was the slowest one. That's pretty clever.”

Field testing

Health Runner 2

Icons are cute, and able to see fastest and slowest splits instantly

Positive feelings

[47] “I thought it was kind of cute. […] I do like how it immediately tells you which runs or which kilometres you ran the fastest, or the slowest, which is neat.”

Post-run interview

Health Runner 3

Understood meaning of icons, but though she was being called a turtle

Felling insulted

[48] “Is that my fastest and slowest time? No. That's in mile, and it's saying I'm a turtle. That's a bit insulting, right? Thanks.”

Field testing

Summary of usability problems found in Endomondo

I discovered an array of usability problems in Endomondo (Table 25). Most reported problem was quietness of voice feedback, which was reported by every participant. Lack of audio feedback on stopping the run was also reported frequently, as well as lack of heat maps in the post-run statistics.

- 55 -

Table 24. Full list of usability problems found in Endomondo

Area Problem N reports (N participants)

Effect on user experience (N)

Quotes

Pause run Swiping is a hassle; too much interaction

2 (1) Prevented from pausing the run (2)

n/a

Delay of interface feedback

2 (1) Prevented from pausing the run (1); completed with annoyance (1)

n/a

The need to unlock app screen by sliding is unclear

1 (1) Confusion (1) 81

Lack of audio feedback

1 (1) None (1) n/a

Stop run Delay of interface feedback

1 (1) Stopping and starting run several times (1); annoyance (1)

82


3 (3) None (3) n/a

Stop/pause buttons ate too close together

1 (1) Worry (1) 83

Start run Misinterpretation of “start” button

1 (1) User though “start” button would play her music (1)

n/a


8 (6) Uncertainty (2); annoyance (2); confusion (1), none (4)

89, 90

Voice feedback

Too quiet 6 (6) Missed it (2); none (4)

n/a

- 56 -

Unclear speech 1 (1) Anger (1) n/a

Lack of feedback half way into the target

2 (2) None n/a

Lack of feedback on being ahead/behind target pace

1 (1) None 50

Lack of time based feedback

1 (1) None n/a

Feedback not frequent enough

1 (1) Frustration n/a

Dislike of mechanical voice

1 (1) Negative (1) 62, 63

Post-run summary

Too cluttered 1 (1) Annoyance (1) 42

Splits screen User though that by having a turtle icon she was being called a turtle

1 (1) Feeling insulted (1)

48

Post-run graphs

Lack of heat maps 3 (3) None (3) 51, 52, 53


n/a 3 (2) Nervousness (1), annoyance (1), prevented from completing goal (1)

70

Main run screen

Screen hard to see in a pouch; need for more orientations

1 (1) Decreased usability

92

Screen hard to see in sunlight

2 (2) Annoyance (1), decreased usability (1)

95, 96

- 57 -

Results 3: cross-application patterns

Runners with previous application experience have clear expectations of features they need

Participants who had and liked certain features in their previous applications expected both Endomondo and Runtastic to have them, and disliked it if they did not (Tables 26 and 27).

For example, Health Runner 2 used RunKeeper before the study, and one of the features that it offers is voice feedback updates on pace and how much ahead or behind her target pace she was. As neither Endomondo nor Runtastic provided this feature, she repeatedly mentioned their lack of it, and she got annoyed with Runtastic because she expected voice feedback to be a free feature, but it was not [49]. As with Endomondo, she only mentioned that she wanted to hear her pace, and exactly how far away she was from it [50].

Table 25. Participant feedback on voice feedback in both applications

Application and feature

Participant Comment Effect on user experi-ence

Quotes Data source

Runtastic, voice feedback

Health Runner 2

Expected voice feedback to be a free feature

Annoyance [49] “[…] and on the second kilometre it told me I ran two kilometres, and after that it said “to get more voice feedback go pro” and then it didn’t tell me that I was on my third kilometre. […] that kind of annoyed me with the app I expected that it should be a free function of the app. Especially since the other

Post-run interview

- 58 -

one [RunKeeper] does it for free. And it didn’t tell me my pace, and how behind my target my pace I am, so that’s a feature I expected the app to have as well.”

Endomondo, voice feedback

Health Runner 2

Wanted to hear how ahead or behind her target pace she was while running

None [50] “Well, I like what RunKeeper does, in that it says, “You’ve run for X kilometres. Your average pace is,” let’s say it was, “5 minutes 50 seconds. That’s 20 seconds behind your average pace,” or something. That’s what I want to hear. […] so it’s probably better to have it than not.”

Post-run interview

Exactly the same pattern emerged for heat maps that Runtastic provides and two of participants who had used Runtastic first (Table 27). Health Runner 2 said that “it would be nice to see a heat map like Runtastic has” in Endomondo [51], and Hobby Runner 3 disliked that Endomondo only provided heat maps after paid upgrade, when Runtastic had offered him that for free [52]. Similarly, even though Health Runner 3 used Endomondo first she had met heat maps feature in one of the previous applications she had used before the study, and that was the type of post-run information that she was used to having [53].

- 59 -

Table 26. Participant feedback on heat maps in both applications


Participant Comment Effect on user experience

Quotes Data source

Endomondo, heat maps

Health Runner 2

Wanted to have heat maps

None [51] “It would be nice to see a heat map like Runtastic has.”

Post-run interview


Hobby Runner 3

Missed heat maps

None [52] “Looks like you have to upgrade to get the same functions that are available on the free version of Runtastic. Miss the coloured route dictating the pace along the run […]”.

Exit interview


Health Runner 3

Used to heat maps in post-run statistics

None [53] “What I'm normally used to […] kind of have heat spots”.

Field testing

Elevation information in post-run statistics is useful for runners who struggle with running uphill

Two participants, Health Runner 2 and Hobby Runner 3 liked having elevation information on the post-run screen in both of the applications, and were interested in correlation between their pace and elevation (Table 28). Health Runner 2 was interested in that information because she struggled with elevation and her pace always dropped, and she would like to see how her dropped pace correlated with elevation levels [54, 55]. In turn Hobby Runner 3 reported that he usually did a lot of hill running and also struggled with it. He liked having elevation information in both applications, and also indicated a need for being able to see his speed going uphill [56, 57].

Other participants did not comment on elevation information.

- 60 -

Table 27. participant feedback on elevation information in both applications


Partici-pant

Comment Effect on user experien-ce

Quotes Data source

Runtastic, elevation information

Health Runner 2

Likes elevation information and wants correlation between elevation and pace

Positive feelings

[54] “I like how it records elevation because it’s something I struggle with, when there is a lot of elevation climb my speed always drops. And if it could compare the pace as a function of the elevation climb and say, yes you ran slower but you have more elevation so if they could find some way to sort of equate that stuff, that would be useful for me.”

Post-run interview

Endomondo, elevation information

Health Runner 2

Likes elevation information

Positive feelings

[55] “I like total ascent and descent. That’s good.”

Field testing

Endomondo, elevation information

Hobby Runner 3

Wants to see more elevation information

None [56] “Would like to see more information on pace and elevation.”

Diary

Runtastic, elevation information

Hobby Runner 3

Likes elevation information and

Positive feelings

[57] “I quite like the elevation thing, because I do a lot of hill ones as well,

Exit interview

- 61 -

wants correlation between elevation and speed

so that's really useful […]. What I would find it useful for, actually, is to see my speed going uphill, because it's obviously hard, running uphill.”

Motivational Features in running applications can help Health Runners

I was able to observe an interesting pattern: for Health Runners voice feedback was linked to emotions, motivation and feeling of accomplishment.

Health Runner 1 was discouraged from trying harder in her run by hearing voice feedback on what she considered bad performance (Table 29; [58]) but hearing voice feedback on what she considered a good performance motivated her to improve her results [59].

Table 28. Feedback of Health Runner 1 on how an application can motivate or discourage her from trying harder

Application Partici-pant

Opinion Effect on user experience

Quotes Data source

Endomondo Health Runner 1

Was discoura-ged to try harder after getting feedback on reduced performan-ce

Negative, discourage-ment

[58] “It didn’t help that the app actually discouraged me a little bit because I had done a pretty normal run the first time I used the app, so my numbers are pretty normal, but since I got a headache, and there were so many people, my numbers were down. Every time, it was telling me,

Diary

- 62 -

“One kilometre, this many minutes.” “Last time, this many minutes,” and it was like two, three minutes difference. So that has made me realized, “Why am I even trying to run right now because it’s just not going to really happen?” And so then I was walking more on top of the walking that I was doing because I had a headache. It was not a very good experience with the app today.”


Motiva- ted by getting feedback on improved performance

Positive, motivation- nal

[59] “We didn't really use the app except for when it gave us the kilometres and how fast we were. It was very motivating to hear that. At the beginning of the run, like always, I was a little bit like, "I don't know if I want to run, but I need to run." Then I kind of just went through. After the first one—the time was really good. It was really motivating. It just kept my spirits up and kept us going.

Diary

- 63 -

Then when our second number was really good, too, it was like, "Wow, I've really improved. It’s really… I'm going to keep going strong. I'm going to keep doing as good as I can." It was really motivating for that, so I think the app was really good today. I really actually enjoyed using it. It's good at the end to see the average runtime and the speeds and everything. It was really impressive and really exciting to be doing that well.”

She also indicated that she would like more motivation from her running application because running was not very pleasurable for her and she did it more for the health benefits than just for the sake of it (Table 30). She suggested that Endomondo should have cheering-on encouragement giving her a little praise for achieving certain milestones while running, a feature that she met in some of the applications she had used in the past [60].

For her the amount of motivation that application gave her strongly affected the extent to which she liked using it: as such she preferred Endomondo because it gave her voice feedback the entire time, which she felt supported and motivated her; Runtastic on the other hand did not do that [61].

- 64 -

Table 29. Feedback of Health Runner 1 on importance of having a motivating application



Quotes Data source


Needs more motivation and encouragement

None [60] “Maybe since running isn’t so much pleasure for me—I mean I want to run, and I like running, but I do it more for the health benefits than for just, “I’m going to go out for a run.” I'd like to be like that one day, but I'm not good enough for that yet. So right now I feel like I need a little bit more motivation from the app, a little bit more encouragement. Some of the apps that I'd use before would be like, “Okay, yes! You've done this many minutes,” or, “You've gone this far. Good.” I’d like something like that.”

Diary

Runtastic Health Runner 1

Was not motivated by applica-tion

Disliked applica-tion

[61] “I had an overall impression of not liking it very much. I don't feel like it motivated me. It didn't help that it didn't give voice feedback the entire time. […] Overall, I think Endomondo

Diary

- 65 -

was so much more – I don't know, motivating to run with. It helped you and supported you on the run, whereas this app did not do that.”

Health Runner 3 indicated that she viewed her running application as a sort of companion, almost having a personal relationship with it, and mechanical voice in Endomondo on the contrary was very impersonal and uncaring (Table 31; [62]) and she felt that it made her pace less of an achievement [63].

One of running applications Health Runner 3 had used in the past provided extra feedback on breaking personal records, and it was an extra confirmation of her achievement that made her feel great and surprised, and she suggested implementing a similar feature in Runtastic [64].

Table 30. Feedback of Health Runner 3 on motivation and running applications



Quotes Data source


Disliked robotic voice

Negative [62] “[…] I just hate the robot voice. […] because it was just so impersonal. Your app is like your running companion, because you're on your own, so it plays that role, in a sense, even though I wouldn't want it to play that role too much. […] You know it's not really your companion, but yes, the voice was just so

Post-run interview

- 66 -

– it's so impersonal, because it's not human. You kind of feel like it doesn't really care, which of course, they don't care anyway, but yes, it felt a lot less like it was really helping you. It was just telling you information and it didn't really care.”


Robotic voice made pace less of an achievement

Negative [63] “[…]it [mechanical voice] made it [pace] less kind of an achievement [...].”

Diary

Another application used in the past

Health Runner 3

Good when it informs about broken records

Positive, pleasant

[64] “It tells you when you’ve broken your record. […] I like that a lot. I think that’s good. [How does it make you feel?] Great. Yes, I know it’s a surprise. You’re surprised. You’re pleasantly surprised. […] It makes a confirmation at the end that you’ve achieved something. Yes, it would be nice if they [Runtastic] did that, as well.”

Post-run interview

Even though Health Runner 2 did not mention the need for general motivation from her running application, she did wish for more encouragement with elevation,

- 67 -

which she struggles a lot with, such as increasing her music volume, or telling her “to pick it up” if she starts slowing down while running uphill (Table 32; [65]).

Table 31. Feedback of Health Runner 2 on including motivation in her running application



Quotes Data source


Wants motivation for going up the hill

None [65] “Maybe if it can tell if I’m gone on elevation it could increase the music volume, […], something to like pump me up more. Like keep me going up the hill. Or see if I’m slacking in my speed a lot and say like “pick it up a bit” or something. […] it could tell me like “you’re slacking, we’re going uphill, but you need to keep going”. Give me an extra push up the hill.”

Post-run interview

Hobby Runners, on the other hand, preferred to set applications aside and concentrate on the running itself, and some of them felt that interacting with the application too much can actually interfere with the running process (Table 33). Hobby Runner 3 reported that running applications are best for keeping track of race training but otherwise he wouldn’t always use one because he prefers to just get on with the running [66, 67].

Hobby Runners 1 and 2 reported on the fact that even mere physical interaction with the phone such as unlocking it during running can cause interruptions to the process [68, 69].

- 68 -

Table 32. Feedback of Hobby Runners on preferred amount of interaction with running applications


Opinion Effect on user experien-ce

Quotes Data source

Runtastic Hobby Runner 3

Interaction with applicati-on interferes with running

None [66] “I quite enjoy running without an app because you just go and get it done. […] lots of times it can interfere, and that's why I’ve not used it religiously, but I've used them as I required. I'd say it's probably best for when training for an event or really keen to see what distances I'm doing or something like that, to keep track. Otherwise, I wouldn't always use one.”

Post-run interview


Does not want to look at application and focus on running

None [67] “Just want to get your head down and not particularly look at the app. You just want to go straight on and get into that position.”

Field testing

Endomondo, interaction while running

Hobby Runner 2

Unloc-king the screen to look at applica-tion is annoying

Annoy-ance

[68] “As I've mentioned before, the fact that I need to unlock the screen to do this was a pretty major annoyance.” Endomondo, diary).

Diary

- 69 -

Runtastic, interaction while running

Hobby Runner 1

Interac-tion with applica-tion disrupts pace

Interrup-tion of running

[69] “The biggest problem with using your mobile for your stats is it does not work well during running. Interaction with the app requires an interruption of pace.”

Diary

To conclude, Health Runners showed an emotional link between applications and motivation. Hearing voice feedback on favourable performance could motivate them, and getting feedback on decreased performance could discourage them.

On the other hand, Hobby Runners preferred no interaction with applications because it interfered with running. Hobby Runners preferred to fully focus on running and only use applications as background performance tracking devices.

Lack of music integration deters health runners

Health Runners 1 and 3 listened to music while running and were annoyed with lack of integrated music player in both applications (Tables 34 and 35). In the end, they found a way around it by going out the applications and setting their music up separately, but they were not happy about it.

Health Runner 1 was nervous every time she went out of Endomondo to start her music that it would stop tracking her [70], and extremely annoyed that she had to upgrade to listen to music within Runtastic [71].

Table 33. Feedback of Health Runner 1 on lack of integrated music player



Quotes Data source


Unsure if it would track her run if she went out

Nervous-ness

[70] “I was a little bit nervous it wasn't going to keep track of my distance, because I

Post-run interview

- 70 -

of it to set up music

went in and out of the app to start my music.”


Wanted music player to be free

Extreme frustration

[71] “Look, I can't even play music without upgrading, and that is bullshit.”

Post-run interview

Health Runner 3 in turn did not like lack of integrated music player because it disrupted her routing of setting her phone up before running (Table 35). She usually sets running application and her playlist up first, then puts her iPhone into a pouch on her arm, and only then plugs in her headphones. During our run with Runtastic, she had to play her music out loud while starting Endomondo to check that it did not stop the music before she put the phone into the pouch, and that was a “real pain” because she felt it disturbed people around [72]. She felt equally annoyed by the lack of integrated music player in Endomondo because she tended to skip through songs [73], but having to exit Endomondo, go to music player and then return back to Endomondo was too complicated, and she just gave that up whatsoever [74].

- 71 -

Table 34. Feedback of Health Runner 3 on lack of integrated music player



Quotes Data source


Disrupted her routine of setting up music

Annoy-ance

[72] “I had to go out of the app to start my music, which is a real pain because I don’t like to play the music out loud because I think it’s disturbing people. […] I would normally set up the app, get it ready to go, so I can just press “play” or let go, and then I’d know my music was set up because you get to choose what playlist you can use. I’d put the iPhone in the pouch, pop the pouch in my arm, and make sure that that was fine, and then I get my headphones and put them around my neck, and make sure, wherever it is, it’s out of the way, and then poke them in. That would be the last thing I’d do. With this app, yes, it’s awkward because you have to start playing music. […]Yes, it was really

Post-run interview

- 72 -

awkward. […] I have to unplug my headphones, plug them in again, unplug them.”


Wanted to skip through songs via integrated player

Annoy-ance

[73] “I can’t skip through song or I should do and so yeah I just had to press sort of a bit kind of stand-by button on my iPhone and then press the home button. It was just that so complicated.”

Post-run interview


Did not bother with changing music because it was a hassle

Prevented from comple-ting her goal

[74] “I think also I wanted to change the music at some point, and I couldn't, or I just didn't bother, because I knew it'd be a hassle with the app.”

Diary

Stop/run buttons can cause frustration/worry when tired

Endomondo and Runtastic offer different modes of interaction for stopping activities. In Endomondo users need to click a big green “Pause” button first, and once they do that a smaller red “Stop” button appears (Figure 12).

- 73 -

In Runtastic, users need to “slide to stop” a slider at the bottom of the screen, which then leads to a “Pause” screen where users can either stop or resume their session (Figure 13).

I observed a general trend towards participants having less troubles and providing more positive feedback about stopping a run in Endomondo than in Runtastic.

Figure 8. Stopping a run in Endomondo. From left to right: locked main run screen, slide to unlock the screen, press pause button, press stop button once it has appeared

Figure 9. : Stopping a run in Runtastic. From left to right: main run screen unlocked by default, slide to stop, pause screen where user can stop or resume the run

- 74 -

Four participants commented on stop-run interaction in Runtastic, and all comments were negative (Table 36). Hobby Runner 2 simply disliked the slider [75], and both Health Runner 1 and 3 reported its inconsistency with the general usage of slider in iOS to unlock the phone [77, 78]. Health Runner 3 called it “completely crazy” [77], and Health Runner 1 got very confused when she first encountered it [78].

Table 35. Participant feedback on stopping sessions in Runtastic



Quotes Data source


Disliked sliding to stop

Negative [75] “I didn't like when I went inside to stop the run and it had sort of slide stop.”

Field testing


Sliding to stop is inconsis-tent with iOS slider to unlock

Negative, confusion

[77] “I wouldn’t say consistent with the iPhone. It’s really not at all. […] This Runtastic one wasn’t unlocking. It was the feature. It was a button. It was the pause button. Completely crazy.”

Field testing


Sliding to stop is inconsis-tent with iOS slider to unlock

Confusion [78] “That was a little bit confusing because I wasn't sure exactly what to do at first because I'm used to a slider being for when your phone's off and you're turning your phone back on, and not for when you're just trying to stop an app.”

Post-run interview

- 75 -

Endomondo mostly received more neutral or positive feedback (Table 37). Hobby Runner 2 did not experience any problems with stopping his run, and Hobby Runner 3 was comfortable with pause and stop buttons to finish the run, and even though there was no clear indication that stop will follow pause, he assumed that it would [79].

Health Runner 2 displayed a similar behaviour – she was not sure that pausing her run would be followed by stopping, but since it was the only available option on the screen she proceeded without a problem and was quite happy with a two-step interaction. She called it “a double-checking system” [80].

Table 36. Participant feedback on stopping sessions in Endomondo



Quotes Data source

Endomondo Hobby Runner 3

Assumed that stop would follow pause

Positive [79] “[…] it's good, I think. I just assumed that you paused and then stop would follow. […]Yes, I was comfortable with those buttons, they were good.”

Post-run interview


Assumed that stop would follow pause; good double-checking

Positive [80] “It’s probably good that they make you pause it first because, really, when you want to stop—the rush to stop is mostly just because you don’t want the extra time to go on while you’re walking or stopped, so I think it’s good that they have a pause first. It’s like a double-checking system. So you’re pausing it—

Post-run interview

- 76 -

it’s good. [Was it clear to you that if you press pause, it’s going to lead you to…] No, it wasn’t clear, but I assumed that it would. It wasn’t a sure thing, but it seemed to make more sense than anything else that was displayed on the interface.”

It is important to mention that two participants experienced problems with stopping a session in Endomondo, although slightly different and less severe than problems associated with Runtastic (Table 38). The first time she used Endomondo Health Runner 3 got confused by the unlabelled slider, which unlocks the screen [81].

Then in her diary Health Runner 3 reported an instance in which she was trying to stop the run but was faced with interface feedback delays and lagging, which resulted in pausing and restarting the run several times [82].

Health Runner 1 reported another issue – she was nervous that she would mess up stopping the run every time by hitting the wrong button because buttons were too close together and stop button was very small [83].

- 77 -

Table 37. Usability problems with stopping sessions in Endomondo



Quotes Data source


Unclear what the slider means

Confusion [81] “[…] so I slid the thing, but it didn't have any explanation at all. […] That was a bit odd”

Post-run interview


Lagging of interface feedback upon stopping session

Annoy-ance

[82] “[…] when I actually stop my run, I had to pause a couple of time because it was delayed, it was laggy and so I’m not really sure if I’d stop it and start it again, and then I have to pause it again then finally I stopped it […].”

Diary


Stop/ pause buttons ate too close together

Worry [83] “[…] it paused it, and then I noticed the stop was actually next to it, […] so that was a little bit strange, and I'll probably mess that up every time I use the app - pause it and then stop it, instead of just directly stopping it. Simply because it's a smaller button […] because you might want to pause it and accidentally hit stop because they are right next to each other.”

Post-run interview

- 78 -

Overall, most participants preferred interaction for stopping the run in Endomondo. Such pattern was independent of the order at which participants used applications in this study (Table 39).

For example, Hobby Runner 3 used Runtastic first and during our first run together was his first time using it and he had severe troubles with stopping the run. It was not clear that one needed to “slide” to stop the run, and he kept pressing around different buttons on the main run screen, and was directed to a map, and got very frustrated. Once he managed to stop the run, he was still unsure how did had done it [84]. Then having not ever used Endomondo at the time, he suggested changing Runtastic to provide the type of interaction currently offered by Endomondo – big clear pause button that needs to be clicked first before one can stop the run [85].

Health Runner 1, who had used Endomondo first said that it’s way of stopping the run is easier and safer than one in Runtastic [86].

Table 38. Comparative feedback on stopping sessions in applications



Quotes Data source


Severe troubles stopping session

Extreme confusion, extreme frustration

[84] “I was looking for a big button to say stop. I'm not quite sure – I can't remember now how I got to that stop button, I was pressing some icons that didn't particularly look like stop, and then a slide bar came up and said, “stop run.” I can understand the slide bar idea, because that means you don't stop it accidentally, by brushing it, but it wasn't immediately obvious, how to stop.

Field testing

- 79 -

We got to the end here, and it took probably another 20, 25 seconds to figure out. Obviously maybe I'll try it again and I'll learn how to do it quicker. […] you shouldn't really need to learn it, no.”


Sugges-tion to change stopping interact-tion to one in Endomondo

None [85] “I would suggest a big button saying, "Stop Run" or "End," "Finish," maybe even a “Pause Run.” Then once you pause, you can press it again to stop, something like that. But just a big button - not to take up the whole screen, but just a clear button that's probably bigger than the rest of the buttons on the screen and just clearly says "Pause," "Stop," something like that.”

Post-run interview


Pause and stop in Endomo-ndo is easier to use

None [86] “I think pause and stop was easier than that because that was slide and then choose from the three. […] I'm a little bit terrified the next time I go for a run I'll hit "stop and discard" instead of "stop and save," […] Yes, you slide first, but then something

Post-run interview

- 80 -

grabs your attention, its three little things, so it's like accidentally click the wrong button. […] I can see myself doing that, not paying attention, being on the run, you know.”

Lack of audio feedback on starting/pausing/resuming/stopping activities causes uncertainty and displeasure

One participant commented strongly on receiving audio feedback on session status change in both applications (Table 40).

Health Runner 3 typically starts the run and immediately pauses it until she has everything ready to go, and only then resumes it to start running [88]. In Runtastic she liked getting audio feedback on start and stop [87], but as there was no feedback on resume, she felt as if her “pausing” needs are special and not accounted for in application design [88].

Table 39. Participant feedback on audio feedback on manipulating sessions in Runtastic



Quotes Data source


Liked audio feedback on start and stop

Positive [87] “But then it just gives you the start and stop feedback, which was good.”

Post-run interview


Disliked absence of audio feedback on resume

Negative [88] “Then when we started running, I un-paused it using the resume button, and I didn’t get any feedback because

Post-run interview

- 81 -

it was on pause. I suppose they’re relying on me, on people, not doing what I did, which is start the run and then pause it before I’d even started.”

However, while using Endomondo, she repeatedly negatively commented on not having such audio feedback in both field testing sessions and her diary (Table 41). She got confused once because Endomondo did not update her whether her run had started [89], and indicated that she wanted to have feedback on starting sessions [90].

Table 40. Participant feedback on audio feedback on manipulating sessions in Endomondo



Quotes Data source


Disliked absence of audio feedback on start

Confusion [89] “Just a bit confused it didn't tell me when the run starts. Uhm, really, I really don't like that.”

Diary


Disliked absence of audio feedback on start

None [90] “I would have expected there to have some sort of feedback in the audio as well, just like I would have had for the start.”

Field testing

Few other participants also expressed uncertainty as to whether their applications had started tracking them, because they were not sure after pressing “start” during field testing sessions.

- 82 -

Phone positioning and sunlight can cause application visibility issues while running

I found that visibility and usability of main run screen was highly impacted by the way in which participants carried their phone while running, especially for those using iPhone sleeves (Tables 42 and 43).

For example, Health Runner 1 had her phone in a pouch on her left arm, and because of that duration timer was too far off on the screen for her to see it. She wanted it to be on the right-hand side, which would be closer to her sightline [91].

Hobby Runner 3 also had his iPhone in a pouch on his forearm, and as he was looking at Endomondo in a landscape orientation fashion, he would have preferred to be able to switch Endomondo orientation from portrait to landscape [92].

Table 41. Orientation-induced visibility problems in applications



Quotes Data source


Needs duration timer on the right-hand side

Decreased usability

[91] “While you're running, the screen that has the time is farther away from you. I attach my phone to my left arm, and so when I look, I expect it [duration] to be on the right-hand side, which is closer to my line of sight, and it's not. It's on the farther side, which is normally angled around my arm. It makes it difficult to see.”

Exit interview


Needs landscape orientation

Decreased usability

[92] “Also, the orientation, I would have preferred it to be landscape, rather than portrait,

Post-run interview

- 83 -

because I had it on my forearm, so when I was looking at it, I was looking at the screen in a landscape fashion, so that would have been better.”

Another pouch-related problem was reported by Health Runner 3 who encountered difficulties tapping buttons in screen corners of Runtastic because they were partly covered by the pouch rim [93].

Moreover, Hobby Runner 3 found dashboard in Runtastic hard to see while running because it was very sunny during his run, and his iPhone pouch added more to the glare so that to be able to read the information on the screen he had to tilt the phone and stay in the shade. He was annoyed because he was distracted from the running experience [94].

As per Endomondo screen visibility in sunlight, Health Runner 3 and Hobby Runner 3 reported it to be generally poor [95, 96].

Table 42. Phone sleeve-induced visibility problems in applications



Quotes Data source


Hard to reach buttons in corners

Decreased usability

[93] “These buttons the corners are really difficult […] especially when you run with it [iPhone pouch].”

Field testing



Annoy-ance; distracted from running

[94] “It was a bit hard to see the app today because it’s so sunny. And with the cover that I had on the phone as well pulled on the arm it reflect to the sun

Diary

- 84 -

quite a bit, so I just wanted to tilt it around and sort of trying to be in shade of it. So that was a bit annoying cause that would be distracted from the experience of running just to be have to check quickly.”



Annoy-ance

[95] “It’s very bright and had to […] see what the hell it was.”

Diary



Decreased usability

[96] “The app was hard to see in the sun.”

Diary

- 85 -

CHAPTER 4. GENERAL DISCUSSION

1.6. Design implications

This research explored user experience in mobile physical activity applications by conducting a comparative field usability testing of two example running applications, Runtastic and Endomondo. Applications were evaluated over a four-week study period, consisting of multiple participant observation sessions, interviews and a diary study. In this section I discuss usability testing findings and design implications for physical activity applications based on this research.

I observed seven cross-application usability issues and usage patterns (Table 44).

Table 43. List of all cross-application findings

1. Previous experience with physical activity applications affects usage

expectations and patterns

2. Need for correlation between elevation and pace

3. Lack of integrated music player negatively affects user experience

4. Preference of big buttons with distance between them

5. Need for audio feedback for starting, pausing, un-‐pausing and stopping

sessions

6. Positioning of the iPhone affects usability and visibility during running

7. Need for including motivation into features for Health Runners

- 86 -

In the following section I will discuss design implications of these and other strong findings that were observed during this study.

Previous experience with running applications affects usage expectations and patterns

I found that if participants have had experience using other running applications, they would expect application they use next to have their preferred functionality and features, and would not compromise. These expectations included having additional voice feedback information on how well user is currently doing compared to average pace, and the need for heat maps in post-run statistics.

This pattern suggests that when designing running applications user research for requirements should also include studying users’ current usage of applications, similarly to what has been done in current study to ensure accommodation of real user needs.

Need for correlation between elevation and pace

Both participants interested in elevation expressed a strong interested in having a correlation between elevation and their pace. However, before proposing integration of this feature into running application I suggest that it would be more rational to replicate this pattern with a bigger user sample.


All participants who typically listen to music during running wanted to have integrated music players in both Runtastic and Endomondo. Setting up music outside of the running application was cumbersome, uncomfortable and annoying. In fact, it appeared to be the biggest, most mentioned and constantly recurring annoyance for those participants. This showed that for those listening to music while running an integrated music player is a key element of user experience.

Preference of big buttons with distance between them

I found that users tend to lack attention on finishing their runs and have shaky hands and therefore interaction mode for stopping and saving sessions should be simplified as much as possible. “Sliding to stop” or pause is likely to produce user confusion and annoyance because it requires more attention and motor skills that

- 87 -

users have after running, it contradicts general iOS convention of sliding to unlock, and a s single-step process it is error-prone.

Findings of this study also suggest that a possibility of accidentally deleting session data when trying to save it (once session has been stopped) is an important concern of many users. To eliminate user confusion and potential frustration design of running applications should eliminate this possibility as much as possible.

I suggest that future designs of mobile running applications need to put a significant effort to reduce the motor skills load and reduce error possibility in stopping and saving sessions. This can be done in the following ways:

1. Interaction should use big clear and easy to tap buttons, and avoid using sliders for important actions

2. Paused session screen should be treated not as an end state of interaction but as an intermediate step, that does not require immediate input and minimises errors

Need for audio feedback for starting, pausing, un-pausing and stopping session

Feedback absence on starting or un-pausing sessions typically resulted in users worrying about tracking and having to check their iPhone screens until they were sure it was working correctly. While for those who tend to look at the screen while running or those who hold their phone in their hand it was a matter of added uncertainty, for those who disliked glancing at the screen while running or those who have phones in arm-sleeves it presented a significant disruption. Some however reported very frustrating incidents in their previous application usages when they had thought that tracking was on when it was not, and they did not record their data.

Similarly, absence of audio feedback on pausing and stopping sessions forced users to check their phone screen to ensure that they do not “spoil” their running statistics by mistakenly recording non-running time at the end of the session, which is something everyone tried to avoid as much as possible. Based on these findings I propose that running applications need to provide well-audible feedback on session status changes, such as:

1. start

2. pause

3. resume, and

4. stop

- 88 -

Positioning of the iPhone affects usability and visibility during running

I found that both Runtastic and Endomondo do not account for limitations induced by physical positioning of the phone during running, such as phone sleeves, orientation at which runners look at applications and sunlight. The following design implications may also apply for general physical activity applications used in dynamic outdoors activities:

1. No controls should be positioned in the corners of the screen because it is hard to reach them if the phone is in a pouch

2. Users should be able to customise application orientations: right or left handed and portrait or landscape

3. Visual design of applications should account for bright sunlight and glare-prone surfaces of phone sleeves to eliminate poor visibility issues

Need for including motivation into features for Health Runners

This study differentiated two types of runners: Health and Hobby Runners (Table 45). Key difference between them lies in reasons behind running. Health Runners run for health-related reasons and feel that they have to run. Hobby Runners on the other hand genuinely enjoy running and it has been integrated into their daily lives as a hobby, and while they do enjoy its health benefits, but more as a pleasant consequence than as a main reason.

Health Runners and Hobby Runners had different needs in running applications.

Hobby Runners only used applications as means of tracking and source of information. They needed accurate, informative and quick voice feedback on their performance, so that they can modify it without having to directly interact with their phone. In fact, Hobby Runners tried to minimise direct interactions with technology when running because it interrupted their pace and running-directed focus. They liked to “get [their] head down and not particularly look at the app” as summed up by Hobby Runner 3.

Health Runners on the other hand relied quite heavily on applications during running, and them voice feedback was strongly linked to motivation for running and improving their performance.

- 89 -

Table 44. Two types of runners and their needs in running applications

Hobby Runners Health Runners

Reasons for running Enjoy running, part of their lives

Health-related reasons

Reasons for using running applications

Tracking and information Motivation, tracking and information

Preferred amount of direct interaction with applications on the run

None Occasionally some

Preferred voice feedback Quick, clear and informative

Motivating, encouraging and informative

Other notes Need encouragement on finishing sessions and breaking personal records

Because there were overlaps in observed behaviours over the course of this four-week study, I propose looking at the two types of runners as a continuous spectrum rather than distinct categories (Figure 14). Hobby Runners, with behaviours described above belong to one end of the spectrum, and Health Runners are at another end. Runners with more blurred and less clearly defined behaviours would fall along the whole spectrum.

Figure 14. The runners’ continuous spectrum, Hobby Runners are on one end, and Health Runners on the other end of the spectrum

Designs of Runtastic and Endomondo currently provide more support for users that fall onto the left side of the spectrum. For a running application to successfully

Hobby Runners

Health Runners

- 90 -

target the whole of the spectrum it must provide an array of differentiated features that can be fully tailored to a specific need of user, whether he is a Hobby or a Health Runner, or somewhere in between.

It was suggested that too many features (so-called “creeping featurism” or “bloat”; McGrenere, 2000; Kaufman & Weed, 1998) can result in only a small amount of features used and can be overwhelming to users. As it would not be possible to satisfy the whole of runners’ spectrum by separate applications, careful design needs to ensure that it is the easy-to-do customization that is a key feature. Hobby Runners should be able to minimise the amount of information they receive from the application and maximise its immediacy; while Health Runners should be able to receive encouragement and motivation. The in-between users should be able to tailor the application to their specific needs.

In fact, despite lack of empirical literature on comparisons between static (non-customisable) and adaptable (allowing user-directed customisation) UIs, some researchers found that users in fact prefer adaptable interfaces (Findlater & McGrenere, 2004).

Other research on a similar topic of personalisation (i.e. customisation of appearance) shows that personalisation can have positive effects on user experience. As such, Monk and Bloom (2007) presented evidence that personalisation of appearance of PCs and mobile phones can improve various aspects of user experience, such as cognitive (e.g. ease of use and aesthetics), social (e.g. reflection of personal identity), and emotional (e.g. product feeling more personal, feelings of control and ownership, accommodating current emotional state, attachment to the system).

Such approach to design of mobile running applications will help to provide users with effective personalised user experience.

1.7. Discussion of methods

Combination of autoethnography, diary studies with media capturing, in situ testing, and interviews was used in this study. In this section I discuss each of the methods in detail.

Autoethnography

I used autoethnography to explore methodologies chosen for this study. It gave me first-hand experience of participating in the study and empathy with participants, and enabled me to see several methodological problems such as the need for diary

- 91 -

pointers, and it led to a decision to provide participants with a choice of two diary input methods.

Autoethnographical research is usually criticised for inherent problems associated with constructs such as reliability, validity, lack of analytics, and being too emotional and subjective (Holt, 2003; Ellis et al., 2011). However, participants’ data was also inherently subjective and tightly coupled with their emotions, personal backgrounds and past experiences in using mobile running applications. Although there are issues associated with it as a method, conducting autoethnography allowed a shared understanding of experiences, particularly negative ones during data analysis.

Diary study

After several modifications to the diary study method in this study I used iPhone application for note taking “Catch”. It was a lightweight mobile method, and participants could choose between making voice or text notes and add photographs and screenshots to their entries.

Format and punctuality of diaries

I borrowed the idea of the voice-mail diary developed by Palen and Salzman (2002). They showed that it was an easy to use method with a good return rate, and it eliminated much of the awkwardness of classical paper and text-based diaries, such as longer time need to make an entry.

Voice-notes were popular in this study as only two participants consistently chose text method, and another one produced mostly voice notes with occasional text entries; everyone else always made voice-notes. In total, 26 voice-note diaries and 18 text diaries were generated over the user study period. Voice note method also produced larger amount of data, which was more detailed: length of text diary entries ranged from 98 – 398 words, while transcripts of voice-note entries ranged from 275 – 750 words.

Researchers that had employed similar voice note diary study methodologies report that participants were more likely to complete a voice note entry than a text one (Swallow, Blythe, & Wright, 2005). It is a common practice for participants to pretend they are speaking on the phone when in fact they are recording a diary entry for a study (Swallow, Blythe, & Wright, 2005). It allows richer data collection and reduces awkwardness for participants, including Health Runner 1 who thought she might “look like a bit of an idiot on a street”.

- 92 -

Hobby Runner 2 was one of the participants who almost always produced text entries, and similarly to Health Runner 3 he mentioned that “it's awkward to talk to your phone when there's no one there”. Moreover, he went oh holiday and “didn't have [his] own space or anything, […] and people were able to listen in” when he recorded his voice note diary. He started writing his entries and found out that “writing isn't too bad”, and then he “never bothered to switch over”. He was “fairly fast at typing, so it wasn't a big problem”. In fact he quite like typing in his diaries because “it sort of gives you time to think about what you're typing as well, whereas talking, it's easy to end up with a lot of filler that doesn't really say anything.”

Although participants were asked to complete their diary entries within 30 minutes of running, that was not always the case. In total, 44 diary entries were made, of them 19 were made within 30 minutes of running, 20 at a later time and 5 entries were missing (Table 46). Delays for late diary entries ranged from 31 minutes to 20 hours 52 minutes, and on average were 1 hour 46 minutes. Average percentage of late and missed diary entries for participants using voice notes as their primary input method was 30.3%, and 95.5% for those who used primarily text (Hobby Runner 1 who used both methods equally is not included in this comparison).

- 93 -

Table 45. Overall statistics of diary entries made throughout the user study

Primary diary format

Total diary entries

Within 30 minutes

Later than 30 minutes

Missed entries

Health Runner 1

Voice notes 7 (100%) 4 (57%) 3 (43%) 0 (0%)

Health Runner 2

Voice notes 11(100%) 8 (72%) 2 (18%) 1 (10%)

Health Runner 3

Voice notes 5 (100%) 4 (80%) 1 (20%) 0 (0%)

Hobby Runner 1

Voice notes + text

4 (100%) 2 (50%) 2 (50%) 0 (0%)

Hobby Runner 2

Text 11 (100%) 1 (9%) 6 (55%) 4 (36%)

Hobby Runner 3

Text 6 (100%) 0 (0%) 6 (100%) 0 (0%)

Total 44 19 20 5

Participants reported various reasons behind making timely diary entries or not completing them on time. Interestingly, Hobby Runner 2 started “was kind of thinking about what [he] was going to write while [he] was running”, therefore submitting late diary entries was not much of a problem for him because he “had a clear picture, and [he] just needed to think back to what [he] was thinking”. Especially, “if [he] was just doing one run and then waited one day and didn't run in between, [he] was fine” in terms of recall. Health Runner 2 reported that leaving an entry in the fist 30 minutes after running was “an awkward time […] to leave a diary entry.” In fact, “[she] just did it, because [she] knew [she] had to get more compensation. […] It would have been better to do after [she]'d had dinner and showered”.

This shows that a diary study can induce some disruptions on participants’ routines, but variable compensation rates based on diary punctuality can be effective for encouraging some participants to complete diaries as directed.

- 94 -

Content of diaries

Qualitatively, diaries provided very rich data on general running behaviours and attitudes towards using technology in running, and usability. Previous research showed that diary studies were useful for gathering information about functions usage, but not as much for discovery of detailed usability problems (Kangas and Kinnunen, 2005), but allowed to discover behaviors in chronological order and rationales behind them (Palen & Salzman, 2002). However in current study participants generated a decent amount of specific usability reports. Out of 76 reports on usability issues discovered in both applications, 21 came from participants’ diaries, 29 from field testing, and 26 from interviews.

Moreover, usability issues reported in diaries were discovered in the target context during real-life application usage and were related strongly to participants’ usage preferences and running types. Answering question formulated by Hartson et al., (2001) to determine how effective a UEM is, “how well does the UEM help inspectors discover real […] usability problems?” (p. 383), diary study performed well on this dimension.

Using photos for deferred contextual interviews

Deferred contextual interviews in which participants went though photographs attached to their diaries added a little extra value to the diaries, which were detailed enough in terms of specific usability problems. However they provided additional in-depth information on rationales behind participant behaviours associated with those problems.

Typically photographs depicted places where participants ran or their faces or bodies or belongings (Figure 15). Only Hobby Runner 1 did not include any photos, for reasons he did not explain.

- 95 -

Most participants found taking photographs difficult and not very helpful in triggering memory. For example, Health Runner 3 did not find photographs helpful in eliciting memories and “some of them worked and some of them didn't, in terms of triggering memory. If you just take a picture of a tree in a park, it's not really [helping to remember]”. She also found taking pictures difficult because “there's not really much [memorable] around you”.

Health Runner 2 did not think that “the photos really help [her] remember much, other than where [she] was”. But application screenshots were more helpful in triggering memory “because [she] only took screenshots when there was a problem, or when there was a really interesting thing that happened. […] the summary would obviously remind [her] a lot more than a photo would of the run, because it would have all the details.” Similarly, Hobby Runner 3 indicated that “a screen grab of the map or the pace definitely helps you remember things.”

Health Runner 2 diary

Health Runner 1 diary

Hobby Runner 2 diary

Health Runner 3 diary Hobby Runner 3 diary

Figure 15. Examples of photographs included into participants’ diaries

- 96 -

Integration of photo-media capturing into the diary study in this research was based on Carter and Mankoff’s (2005) study that found that photographs led to more specific recall. Potential reason for application screenshots and run summaries being more helpful in triggering memories may be that they are a more meaningful media type in domain of running applications. I suggest that media-capturing methodology in research of mobile physical activity applications should be adjusted to substitute photographs with run summaries and application screenshots.

Field testing

During field testing I ran with participants and filmed their think-alouds and interactions with applications with a handheld video camera. In this section I will discuss methodological advantages of my approach and provide guidance for future research willing to use similar methods based on my reflections.

Holding video camera in my hand was flexible and effective for video-recording interactions at different angles: some participants would have their phone on their arm at the start of running but then take it off to hold it in their hand – and I was able to quickly adapt to those changes. However, audio recording with a handheld camera was not as effective due to high level of background noise as most runs held in Central London.

I suggest that field testing of mobile physical activity applications should combine handheld video cameras with small wearable audio-recording devices. Similar approaches were used in a field study done by van Elzakker et al., (2008) and Mark et al., (2001), and although their use of headsets and glasses may be disruptive in the context of dynamic physical activities, any small wearable device will control for environmental constraints placed on audio quality.

To conclude my discussion of equipment used in this study, I would like to mention that although wearable video camera used during pilot study had to be dismissed, it discovered the need for careful piloting of placement of a wearable camera, as users have specific postures of holding their phones. It is advisable to ask participants to run for a bit while using their phone and record that interaction, and then check the footage.

Thinking aloud during running did no cause any problems for participants, and in fact slower-paced runners engaged in deep discussions around running and technology, providing more data than they did in previous (or subsequent) pre- and post-run interviews. This confirmed notion that participant observation can yield richer data than other methods such as interviews (Becker & Geer, 1957).

Although most participants displayed natural behaviours, some were trying to fit applications into their runs more than they normally would so that we could test all the features. I overcame this limitation by reminding participants to keep their

- 97 -

behaviour natural, and that I was interested in studying their natural usage patterns rather than forced usage of every feature.

Further advantages of using naturalistic field testing was that application use cases that I observed were very situated, as described by Suchman (1987). At first participants would use applications in one way and say that this was the only interaction they were interested in, and then a few diary entries later they would be using it differently and have different problems and wishes. For example, Health Runner 2 indicated that she would not directly interact with running applications apart from listening to information provided by voice feedback. However, during the diary study period she tried using live map in Runtastic for navigation.

Furthermore, usability of both applications was highly dependent on the context. Similarly to the idea of multi-faceted context proposed by Dourish (2004), participants’ actions and usage behaviours were intertwined with environment and their personal goals for specific runs. For instance I discovered that it was difficult to tap buttons positioned in screen corners when phone was placed into a running phone sleeve, sliding to stop session in Runtastic was difficult because participants had shaky hands after intense running, and that both applications were hard to see in the sun. The same situated findings enabled me to reflect on how appropriate application interaction modes such as sliding to stop the run were in the target context rather than just their direct usability.

1.8. Limitations

The major limitation of this study was its limited participant sample of only six participants, which is much less than used in similar field testing research. For example, Goodman, Brewster, & Gray (2004) studied 56 participants in their field evaluation of mobile guides, and Brown et al. (2013) studied 15 participants in a study of in the wild mobile phone usage.

There were also several problems with data handling in this study. First, most of the audio was of very poor quality because it was taken outside and while running, and each transcript contained quite a few inaudible parts. Secondly, the amount of data was so great that it was almost unmanageable given the time constraints of the project and its exploratory nature. However given these limitations audio recordings were the only feasible method for capturing data during running sessions done by a single researcher, which provided the most amount of data in comparison to diaries and interviews.

For studies with clearer defined scope and availability of at least two researchers, I suggest combining collaboration and rapid ethnography (Millen, 2000) approach: one researcher can be running out, and live audio will be streamed to another researcher over voice over IP (VoIP) protocol, such as Skype. This way

- 98 -

the second researcher can only take notes of most necessary information, and narrow down the scope for faster field usability testing.

Finally, this study could have benefited from screen-capturing software such as developed by Brown et al., (2013) for field testing of mobile applications. As they are planning to provide it to other researchers it would mark a new evolutionary step in combining field and laboratory mobile testing, and would lead to easier and more precise data collection. Currently usability data was less accurate than it would be with a fixed camera in the lab, but such software will enable to merge advantages of in situ and laboratory testing of mobile physical activity applications.

- 99 -

CHAPTER 5. CONCLUSION

This research discovered that designs of physical activity applications need to account for various contextual factors, such as sunlight, outside noise, phone positioning, and motor and mental limitations imposed on users by performing a demanding physical activity, such as shaking and impaired attention. It also emphasised the need to conduct more ethnographic user research of users current used technologies because their previous application experiences impact their needs and preferences. In this study users fitness backgrounds and motivations framed their application-specific needs, and those needs differed significantly between users who engage into fitness for health-related reasons and those who treat exercise as hobby. These discrepancies in user needs showed the need of inclusion of motivational features and higher customisation levels into physical activity applications. It would be advisable for future research to focus on a larger user sample to explore motivation in physical activity and associated technology needs.

Field testing techniques used in this study helped gathering rich and detailed data about usability and general user needs, and it may most effectively serve needs concerned with requirements gathering, conceptualization or redesign of mobile physical activity applications. For research with clearer defined scope or limited time budget, these methodologies need to be adapted to become more lightweight and less time-consuming, for instance by employing several researchers and taking a rapid ethnography approach.

- 100 -

REFERENCES

Abowd, G. D., Hayes, G. R., Iachello, G., Kientz, J. A., Patel, S. N., Stevens, M. M., & Truong, K. N. (2005). Prototypes and paratypes: Designing mobile and ubiquitous computing applications. Pervasive Computing, IEEE, 4(4), 67-73.

Abras, C., Maloney-Krichmar, D., & Preece, J. (2004). User-centered design. Bainbridge, W. Encyclopedia of Human-Computer Interaction. Thousand Oaks: Sage Publications, 37(4), 445-56.

Ahtinen, A., Ramiah, S., Blom, J., & Isomursu, M. (2008, December). Design of mobile wellness applications: identifying cross-cultural factors. In Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat (pp. 164-171). ACM.

Amin, A., Townsend, S., Van Ossenbruggen, J., & Hardman, L. (2009). Fancy a drink in canary wharf?: A user study on location-based mobile search. In Human Computer Interaction–INTERACT 2009 (pp. 736-749). Springer Berlin Heidelberg.

Au, I., Boardman, R., Jeffries, R., Larvie, P., Pavese, A., Riegelsberger, J., Rodden, K., & Stevens, M. (2008, April). User experience at Google - focus on the user and all else will follow. In CHI'08 Extended Abstracts on Human Factors in Computing Systems (pp. 3681 3686). ACM.

Barkhuus, L., & Dey, A. K. (2003, July). Location-Based Services for Mobile Telephony: a Study of Users' Privacy Concerns. In INTERACT (Vol. 3, pp. 702-712).

Beck, E. T., Christiansen, M. K., Kjeldskov, J., Kolbe, N., & Stage, J. (2003, November). Experimental evaluation of techniques for usability testing of mobile systems in a laboratory setting. In proceedings of OzCHI (pp. 106-115).

- 101 -

Becker, H. S., & Geer, B. (1957). Participant observation and interviewing: A comparison. Human Organization, 16(3), 28-32.

Bennett, G., Lindgaard, G., Tsuji, B., Connelly, K. H., & Siek, K. A. (2006, April). Reality testing: HCI challenges in non-traditional environments. In CHI'06 Extended Abstracts on Human Factors in Computing Systems (pp. 1679-1682). ACM.

Brandt, J., Weiss, N., & Klemmer, S. R. (2007, April). txt 4 l8r: lowering the burden for diary studies under mobile conditions. In CHI'07 extended abstracts on Human factors in computing systems (pp. 2303-2308). ACM.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101.

Brewster, S. (2002). Overcoming the lack of screen space on mobile computers. Personal and Ubiquitous Computing, 6(3), 188-205.

Brown, B., McGregor, M., & Laurier, E. (2013, April). iPhone in vivo: video analysis of mobile device use. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1031-1040). ACM.

Carter, S., & Mankoff, J. (2005, April). When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 899-908). ACM.

Charmaz, K. (2000). Grounded theory. Objectivist and constructivist methods. In (Eds.), Handbook of qualitative research, Second edition (pp. 509-535). Thousand Oaks, CA: Sage Publications Inc.

- 102 -

Consolvo, S., Everitt, K., Smith, I., & Landay, J. A. (2006, April). Design requirements for technologies that encourage physical activity. In Proceedings of the SIGCHI conference on Human Factors in computing systems (pp. 457-466). ACM.

Consolvo, S., McDonald, D.W., Toscos, T., Chen, M.Y., Froehlich, J., Harrison, B., Klasnja, P., LaMarca, A., LeGrand, L., Libby, R., Smith, I., & Landay, J.A. 2008. Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2008). ACM, New York, NY, USA, 1797-1806.

Crabtree, A., Benford, S., Greenhalgh, C., Tennent, P., Chalmers, M., & Brown, B. (2006, June). Supporting ethnographic studies of ubiquitous computing in the wild. In Proceedings of the 6th conference on Designing Interactive systems (pp. 60-69). ACM.

Draper, S. (2013, June 22). The Hawthorne, Pygmalion, Placebo and other effects of expectation: some notes. Retrieved from http://www.psy.gla.ac.uk/~steve/hawth.html#Hawthorne%20overall

Duh, H. B. L., Tan, G. C., & Chen, V. H. H. (2006, September). Usability evaluation for mobile device: a comparison of laboratory and field tests. In Proceedings of the 8th conference on Human-computer interaction with mobile devices and services (pp. 181 186). ACM.

Duncan, M. (2004). Autoethnography: Critical appreciation of an emerging art. International Journal of Qualitative Methods, 3(4), 28-39.

Ednomondo Blog (October 2012). Facts. Retrieved from http://blog.endomondo.com/facts/

Ellis, C. S., & Bochner, A. (2000). Autoethnography, personal narrative, reflexivity: Researcher as subject. In (Eds.), Handbook of qualitative research, Second edition (pp. 733-768). Thousand Oaks, CA: Sage Publications Inc.

- 103 -

Ellis, C., Adams, T. E., & Bochner, A. P. (2011). Autoethnography: an overview. Historical Social Research/Historische Sozialforschung, 273-290.

Findlater, L., & McGrenere, J. (2004, April). A comparison of static, adaptive, and adaptable menus. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 89-96). ACM.

Furnham, A., Badmin, N., & Sneade, I. (2002). Body image dissatisfaction: Gender differences in eating attitudes, self-esteem, and reasons for exercise. The Journal of Psychology, 136(6), 581-596.

Goodman, J., Brewster, S., & Gray, P. (2004, September). Using field experiments to evaluate mobile guides. In Proceedings of HCI in Mobile Guides, workshop at Mobile HCI (Vol. 2004).

Gutwill, J. P. (2003). Gaining visitor consent for research II: Improving the posted‐sign method. Curator: The Museum Journal, 46(2), 228-235.

Holt, N. L. (2008). Representation, legitimation, and autoethnography: An autoethnographic writing story. International Journal of Qualitative Methods, 2(1), 18-28.

Hussain, Z., Slany, W., & Holzinger, A. (2009). Current state of agile user-centered design: A survey. In HCI and Usability for e-Inclusion (pp. 416-427). Springer Berlin Heidelberg.

Isomursu, M., Kuutti, K., & Väinämö, S. (2004, July). Experience clip: method for user participation and evaluation of mobile concepts. In Proceedings of the eighth conference on Participatory design: Artful integration: interweaving media, materials and practices-Volume 1 (pp. 83-92). ACM.

- 104 -

Jambon, F., & Meillon, B. (2009, April). User experience evaluation in the wild. In CHI'09 Extended Abstracts on Human Factors in Computing Systems (pp. 4069-4074). ACM.

Kallio, T., Kaikkonen, A., Kekalainen, A., Kankainen, A., & Cankar, M. (2005). Usability testing of mobile applications: A comparison between laboratory and field testing. Journal of usability studies, 1(4-16), 23-28.

Kangas, E., & Kinnunen, T. (2005). Applying user-centered design to mobile application development. Communications of the ACM, 48(7), 55-59.

Kaufman, L., & Weed, B. (1998, April). Too much of a good thing?: identifying and resolving bloat in the user interface. In CHI 98 Cconference Summary on Human Factors in Computing Systems (pp. 207-208). ACM.

Kellar, M., Watters, C., & Shepherd, M. (2007). A field study characterizing Web based information‐seeking tasks. Journal of the American Society for Information Science and Technology, 58(7), 999-1018.

Kjeldskov, J., & Graham, C. (2003). A review of mobile HCI research methods. In Human computer interaction with mobile devices and services (pp. 317-335). Springer Berlin Heidelberg.

Kjeldskov, J., Skov, M. B., Als, B. S., & Høegh, R. T. (2004). Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. In Mobile Human-Computer Interaction-MobileHCI 2004 (pp. 61-73). Springer Berlin Heidelberg.

Mackay, W. E., & Fayard, A. L. (1997, August). HCI, natural science and design: a framework for triangulation across disciplines. In Proceedings of the 2nd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 223-234). ACM.

- 105 -

Mancini, C., Thomas, K., Rogers, Y., Price, B. A., Jedrzejczyk, L., Bandara, A. K., Joinson, A., N., & Nuseibeh, B. (2009, September). From spaces to places: emerging contexts in mobile privacy. In Proceedings of the 11th international conference on Ubiquitous computing (pp. 1-10). ACM.

Mao, J. Y., Vredenburg, K., Smith, P. W., & Carey, T. (2005). The state of user centered design practice. Communications of the ACM, 48(3), 105-109.

Mark, G., Christensen, U., & Shafae, M. (2001). A methodology using a microcamera for studying mobile IT usage and person mobility. In CHI Workshop on Mobile Communications: Understanding Users, Adoption & Design, ACM CHI 01 Conf. on Human Factors in Computing Systems.

McGrenere, J. (2000, April). Bloat: the objective and subject dimensions. In CHI'00 extended abstracts on Human factors in computing systems (pp. 337-338). ACM.

Merchant, G. (2012). Mobile practices in everyday life: Popular digital technologies and schooling revisited. British Journal of Educational Technology, 43(5), 770-782.

Millen, D. R. (2000, August). Rapid ethnography: time deepening strategies for HCI field research. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 280-286). ACM.

mobiThinking (May 2013). Global mobile statistics 2013 Part A: Mobile subscribers; handset market share; mobile operators. 3) Mobile phone shipments. Retrieved from http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats/a#mobilephonepenetration

Monk, A. F., & Blom, J. O. (2007). A theory of personalisation of appearance: quantitative evaluation of qualitatively derived data. Behaviour & Information Technology, 26(3), 237–246. doi:10.1080/01449290500348168

- 106 -

Palen, L., & Salzman, M. (2002, November). Voice-mail diary studies for naturalistic data capture under mobile conditions. In Proceedings of the 2002 ACM conference on Computer supported cooperative work (pp. 87-95). ACM.

Palen, L., Salzman, M., & Youngs, E. (2000, December). Going wireless: behavior & practice of new mobile phone users. In Proceedings of the 2000 ACM conference on Computer supported cooperative work (pp. 201-210). ACM.

Raynsford, J. (12/08/13). Top 10 running apps. Retrieved from http://www.telegraph.co.uk/sponsored/sport/bupa-great-runs/10237554/top-10-running-apps.html

Riegelsberger, J., & Nakhimovsky, Y. (2008, April). Seeing the bigger picture: a multi method field trial of google maps for mobile. In CHI'08 Extended Abstracts on Human Factors in Computing Systems (pp. 2221-2228). ACM.

Rieman, J. (1993, May). The diary study: a workplace-oriented research tool to guide laboratory efforts. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems (pp. 321-326). ACM.

Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R. E., Hursey, J., & Toscos, T. (2007). Why it’s worth the hassle: The value of in-situ studies when designing UbiComp. In UbiComp 2007: Ubiquitous Computing (pp. 336-353). Springer Berlin Heidelberg.

Runtastic blog (16/05/2013). The Runtastic sprint continues: 30 million downloads & 25 million mobile users. Retrieved from http://blog.runtastic.com/en/blogs-and-reviews/runtastic-sprint-continues-30-million-downloads-25-million-mobile-users/

- 107 -

Saffer, D. (2010). Designing for interaction: creating innovative applications and devices (Second edition). Chapter 2: the Four Approaches to Interaction Design, pp. 31-46. New Riders: Berkley, CA.

Sohn, T., Li, K. A., Griswold, W. G., & Hollan, J. D. (2008, April). A diary study of mobile information needs. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 433-442). ACM.

Swallow, D., Blythe, M., & Wright, P. (2005, September). Grounding experience: relating theory and method to evaluate the user experience of smartphones. In Proceedings of the 2005 annual conference on European association of cognitive ergonomics (pp. 91-98). University of Athens.

van Elzakker, C. P., Delikostidis, I., & van Oosterom, P. J. (2008). Field-based usability evaluation methodology for mobile geo-applications. Cartographic Journal, The, 45(2), 139-149.

vom Lehn, D., Heath, C., & Hindmarsh, J. (2002). Video based field studies in museums and galleries. Visitor Studies Today, 5(3), 15-23.

Vredenburg, K., Mao, J. Y., Smith, P. W., & Carey, T. (2002, April). A survey of user centered design practice. In Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves (pp. 471 478). ACM.

Wauters, R. (31/12/12). RunKeeper aims to keep its 14 million users running with new app UI and improved photo sharing. Retrieved from http://thenextweb.com/apps/2012/12/31/runkeepers-aims-to-keep-runners-running-with-new-app-ui-photo-sharing-feature-and-more/

Wong, B. L., & Blandford, A. (2003, July). Field research in HCI: a case study. In Proceedings of the 4th Annual Conference of the ACM Special Interest Group on Computer-Human Interaction (pp. 69-74). ACM.

- 108 -

Zhang, D., & Adipat, B. (2005). Challenges, methodologies, and issues in the usability testing of mobile applications. International Journal of Human-Computer Interaction, 18(3), 293-308.

- 109 -

APPENDICES

Appendix 1. Online questionnaire

- 110 -

Outside 16 73%

On a treadmill in the gym 5 23%

On a treadmill at home 0 0%

Other 1 5%

Where do you run?

What do you usually do immediately (within 5-10 minutes) after you finishrunning? Give brief examples

Go to the next gym machine. Either ellipticals or the cycle Shower, go back to desk. Stretch,

drink water workout walk, get a drink (e.g. fizzy drink) n/a Walk the last 500m to wamr

down, listening to ipod and checknig run stats on Nike+. Then get home and drink water and

make some food - maybe protein shake. Stretch, other exercises, eat drink water Weights

at the gym I usually stretched, drank water, walked home walk slowly stretch cool-down

walk drink water drink water and stretch. Stretches for hamstrings, quads, calf, IT band, glutes

and hip abductors. stretching and eat something Stretching drink water, sit, browse the

internet, shower Shower stretch Drink water. Stretch. Watch TV. shower

If you were a participant in a study, what would you prefer: going home andmaking a dairy entry about your run later in the day,or making a diary entry onthe move right after the run, e.g. via text or voicemail? Why?

I'll make a diary entry right after the run via text. If I do it when I'm home I may forget vital pieces I

would have wanted to add about the run and I chose text because sometimes it's weird speaking

into a device and the sound may not be well-received by the device. Later in day. Have to work

after run Later, after a shower. right after, because otherwise i would forget home,

because i dont take things with me when i run n/a I always like making a dairy entry!

Probably straight after if it was simple questionnaire (not typing too much), later if it was more

lengthy. later in the day right after the run, quick though Right after via text when the

feeling is fresh. I'd prefer to do it right after the run because as soon as I get home I'm in a

much different mindset and all I want to do is sleep later in the day. more convenient

Immediately after -- I would trust the results more, then I won't be so bothered by it later as

well. I run right back to my front door, so I will always be home at the end of my run. I think I

would prefer text straight after a run as I could do it while stretching and it would be quicker. I

would not like voicemail however as I really dislike recording voicemails as you cannot edit

them. afterwards, would be too busy stretching and showering afterwards On the move on

- 111 -

Distance 16 32%

Calories expenditure 8 16%

Heart rate 3 6%

Speed 17 34%

I don't need performance info 3 6%

Other 3 6%

Smartphone app 11 41%

Physical devices, i.e. wristbands, heart rate monitors, etc 4 15%

Treadmill display 5 19%

the move via text. I would forget the details if I left it for later in the day Text Answering

questions via text right after the run: easy, I'm already using my phone, I need a breather, and I

enjoy doing a retrospective on my run later in the day

Do you keep track of performance info while you run?

Not in particular. My point is to work out, not compare performance Yes When running solo it

can help motivate. Yes sometimes No No Yes Yes Yes Yes No No

Yes Usually. Yes No Yes No Yes Yes Yes

What performance info is important to you about your runs?

How do you track performance information?

- 112 -

I do not track my performance 5 19%

Other 2 7%

If you use smartphone apps, which ones?

Nike plus endomondo n/a n/a Nike + Nike Running RunKeeper The nike one,

think it is called "Nike+ Running" Nike+ Running App MapMyRun Runkeeper Adidas

MiCoach

Why do you prefer that particular running app to others?

Quick and easy to use it's the only one i know n/a n/a its free and it works well and

gives you km by km updates on performance provides fun and useful information, gives me an

good overall overview of my overall runs It maps my run, gives me live updates on my pace,

lets me connect with friends It allows me to start and then pause the run before I have left the

house. I have had problems with others like runkeeper which only allow you to start a run when

you are outside and the gps is strong. I like to go out the house knowing my app is set up and I

only need to press one button to have it start tracking the nike app allows me to do this. It also

allows you to assign a run to a particular trainer which means you can keep track of when you

should get a new pair and it is kind of nice to know the millage of your footwear, I don't know why

but it is. Because it comes with a webpage that allows me to see and compare my performance

over a period of time GPS tracking Social platform is AMAZING for motivation. Decent UI

too. The ecosystem is pretty good, I can turn on coaching and select a training online and then

be coached according to the training. The website interface that comes with the app is really

helpful.

Have you had negative experiences with running apps? What happened? Did itaffect your usage the app?

Battery drains quickly on long runs. Prevents me from using all the time. no n/a n/a

They ask me how far I ran when that is what the app is for - I don't want to calibrate it myself!

The application showed a path I didnt take, crossing 2 miles in one minute. really bad, and made

me angry Sometimes the GPS would fail and not map my run, very frustrating. I quit using

those apps. Also sometimes I somehow accidentally paused the run midway which was equally

frustrating! Haven't had those issues with "RunKeeper"! :) Yes, one of the releases of the nike

app had a bug and sometimes when it was giving me verbal feedback during a run it would stop

playing my music and so I would have to fiddle with my iphone while running to try and get my

music back on. Once I ran straight into a really deep puddle because I was doing this. Very

embarrassing and rather uncomfortable for the rest of the run my trainers were soaking wet, I was

really annoyed with the app for that! App crashes right before the run is saved. It is

frustrating! no Laggy/crashy/slow. Stops me using them, I need them responsive if I'm meant

to use them on the go. No negative experience

Number of daily responses

- 113 -

Appendix 1. Latin square, order at which participants received applications

Participant 1st application 2nd application

1 Endomondo Runtastic

2 Runtastic Endomondo





user experience of mobile physical activity applications ... · an alternative approach, in situ...

Documents