Usability Studies with JapaneseChildren for Mobile Applications:

Conducting a Meta Analysis

Master‘s Thesis DefenceBernd Hollerit

Supervisor: Wolfgang SlanyExaminers: Keith Andrews, Andreas Holzinger

Bernd Hollerit 2

Schedule

• Motivation• Catrobat• Usability• Definition• Importance• Evaluation Techniques• Our Approach

• Catrobat Usability Evaluation• Evaluation of Surveys

• Identified Usability Problems• Catrobat Redesign• Paintroid Redesign• Suggestions for Japanese Market

• Conclusion• Outlook• Q&A

2013-11-28

Bernd Hollerit 3

Motivation

• IT important for most businesses• Not only IT staff need knowledge, but everyone working in an office• Most thought after IT skills nowadays are programming and app

development• IT sector is growing and is expected to keep thriving• 61% of IT executives announced plans to hire web- & app developers• Too few women in IT• Cell phone usage among children on the rise

2013-11-28

Bernd Hollerit 4

Catrobat

• Visual programming language• Inspired by Scratch• Developed for smartphones and

tablets• Targets children and teenagers• Goal: learn mathematical and

computational ideas, think creatively, reason systematically, and work collaboratively

2013-11-28

Bernd Hollerit 5

Usability – Definition

1. Effectiveness2. Learnability3. Efficiency4. Memorability5. Errors6. Satisfaction

2013-11-28

Bernd Hollerit 6

Usability – Importance

• If an application is difficult to use…• If the app fails to clearly convey what it has to offer…• If users get lost while navigating through the app…• If the purpose of an app is hard to figure out…• If the app doesn‘t fulfill a user‘s expectations…

2013-11-28

Bernd Hollerit 7

Usability – Importance

• If an application is difficult to use…• If the app fails to clearly convey what it has to offer…• If users get lost while navigating through the app…• If the purpose of an app is hard to figure out…• If the app doesn‘t fulfill a user‘s expectations…

…people will stop using it

• Usability is a necessary condition for survival

2013-11-28

Bernd Hollerit 8

Usability – Evaluation Techniques

• Early Focus on Users and Tasks• Empirical Measurement• Iterative Design• Parallel Design• GOMS (Goals, Operators,

Methods, Selection Rules)• Human Processor Model• Keystroke Level Modeling• Card Sorts

• Tree Tests• Ethnography• Heuristic Evaluation• Usability Inspection• Pluralistic Inspection• Consistency Inspection• Activity Analysis• Task Analysis

2013-11-28

Bernd Hollerit 9

Usability – Evaluation Techniques (cont.)• Focus Groups• Surveys/Questionnaires• Remote Usability Testing• Remote Usability Testing for

Mobile Devices• Thinking Aloud• RITE (Rapid Iterative Testing and

Evaluation) Method• Subjects-in-Tandem or Co-

Discovery

• Component-Based Usability Testing• Rapid Prototyping• Cognitive Walkthrough• Benchmarking• Persona• Meta-Analysis

2013-11-28

Bernd Hollerit 10

Usability – Our Approach

• Meta Analysis of a Focus Group, with Surveys/Questionnaires, Activity Analysis, Thinking Aloud, Cognitive Walkthrough and, to a lesser extent, GOMS and Task Analysis• Focus Group: Japanese teenagers• Survey before hands-on experience, questionnaire after• Start with Thinking Aloud• If children can‘t be encouraged to talk -> Activity Analysis, Cognitive

Walkthrough, GOMS, Task Analysis

2013-11-28

Bernd Hollerit 11

Catrobat Usability Evaluation

• 4 children aged 13-17 years• Tests lasted between 45 and 99 minutes• Facilitator creates timestamped log of activities

1. Survey before actual test2. Hands-on experience of the Catrobat software3. Questionnaire after using Catrobat

2013-11-28

Bernd Hollerit 12

Evaluation of Surveys

• Children agreed or strongly agreed that the usage of Catrobat is fun, interesting, cool and entertaining.• They disagreed or strongly

disagreed that Catrobat is frustrating and boring.• Half of the children agreed that

Catrobat was confusing, with the other half disagreeing.

• „Would you be interested in taking a computer science class in high school?”

2013-11-28

Before Catrobat Usage After Catrobat Usage

Maybe yes, maybe no Probably yes

Probably yes Definitely yes

Probably yes Probably yes

Maybe yes, maybe no Probably yes

Bernd Hollerit 13

Identified Usability Problems

• Children tap field for number input when they want to move a brick around• Children draw outside of boundaries of

Paintroid image• Children don't know the Paintroid tool “magic”• Children don't know the Paintroid tool “stamp”• Children exit Paintroid without saving• Images open on a very tiny zoom level• Localisation errors in Paintroid• Long-press actions aren't intuitive

• Interface is generally confusing• Inconsistent „Add“-

Buttons

-> Uniform Redesign

2013-11-28

Bernd Hollerit 14

Catrobat Redesign

2013-11-28

Bernd Hollerit 15

Catrobat Redesign

2013-11-28

Bernd Hollerit 16

Catrobat Redesign

2013-11-28

Bernd Hollerit 17

Catrobat Redesign

2013-11-28

Bernd Hollerit 18

Catrobat Redesign

2013-11-28

Bernd Hollerit 19

Catrobat Redesign

2013-11-28

Bernd Hollerit 20

Catrobat Redesign

2013-11-28

Bernd Hollerit 21

Paintroid Redesign

2013-11-28

Bernd Hollerit 22

Suggestions for Japanese Market

• Focus on features regarding drawing and animation• E.g. with templates, pre-set

images

• Localisation not complete, fixed missing translations• One younger participant (13

years old) was unfamiliar with pinch-to-zoom

2013-11-28

Bernd Hollerit 23

Conclusion

• Prepared, conducted and reviewed usability evaluation in order to improve the user experience of Catrobat• Performed Meta-Analysis consisting of Questionnaires/Surveys,

Activity Analysis, Cognitive Walkthrough, Thinking Aloud, Task Analysis and a Consistency Inspection with Japanese children• Suggested improvements and discussed resulting changes• Received Austrian national award for innovation for multimedia and

e-business• Promotion from Google in the Google Play Store (Education Section)

2013-11-28

Bernd Hollerit 24

Outlook

• Reevaluate the redesign• Assess actual user opinions• http://catrobat.org – Developer website hosting the latest builds of

Pocket Code and Pocket Paint• https://pocketcode.org – Hosts user-created apps that can be

downloaded and run by any Pocket Code version • http://blog.catroid.org – Catrobat Blog• https://github.com/Catrobat – Catrobat GitHub Project

2013-11-28

Thank you for your attention!

Q&A

Bernd Hollerit 26

Sources

• [ADAMS2009] Adams, Stephen. Children get first mobile phone at average age of eight. 2009 http://www.telegraph.co.uk/technology/news/4680507/Children-get-first-mobile-phone-at-average-age-of-eight.html

• [ANDREWS2013] Andrews, Keith. Human-Computer Interaction -- Course Notes. 2013 http://courses.iicm.tugraz.at/hci/hci.pdf• [BACON-SHONE2009] Bacon-Shone, John and Hui, Desmond. Design and construction of the Hong Kong Creativity Index.

Publications Office of the European Union. p. 91--102 2009 http://ec.europa.eu/education/lifelong-learning-policy/doc/creativity/report/hongkong.pdf

• [BURTSCHER2012] Burtscher, Daniel. Introduction of a Continuous Integration Process in an Open Source Project. 2012• [CARD1983] Card, Stuart. The psychology of human-computer interaction. L. Erlbaum Associates. 1983• [CARD1986] Card, S.K and Moran, T. P. and Newell, A.. Handbook of Perception and Human Performance. John Wiley and sons. p.

1-35 1986• [CROPLEY2000] Cropley, Arthur. Defining and measuring creativity: Are creativity tests worth using?. Routledge. p. 72--79 2000

http://www.informaworld.com/openurl?genre=article&doi=10.1080/02783190009554069&magic=crossref• [CSIKSZENTMIHALYI2009] Csikszentmihalyi, M.. Creativity: Flow and the Psychology of Discovery and. HarperCollins. 2009

http://books.google.co.jp/books?id=aci_Ea4c6woC• [DAVIDSON2012] Davidson, Danica. Manga grows in the heart of Europe. 2012 http://geekout.blogs.cnn.com/2012/01/26/manga-

in-the-heart-of-europe/• [FIDGEON2011] Fidgeon, Tim. Ethnography – when and how to use. 2011 http://www.spotlessinteractive.com/articles/usability-

research/ethnography-when-and-how-to-use.php

2013-11-28

Bernd Hollerit 27

Sources

• [FISHER2004] Fisher, R. and Williams, M.. Unlocking Creativity: A Teacher's Guide to Creativity Across the Curriculum. Taylor & Francis. 2004 http://books.google.co.jp/books?id=W6WOTKVVjCwC

• [FRANKEN2006] Franken, R.E.. Human Motivation. Thomson/Wadsworth. 2006 http://books.google.at/books?id=hvW6AAAACAAJ• [GARRETT2003] Garrett, Jesse James. The Elements of User Experience. David Dwyer. 2003• [GOULD1985] Gould, John D. and Lewis, Clayton. Designing for Usability: Key Principles and What Designers Think. Communications of

the ACM. 1985 http://dl.acm.org/citation.cfm?id=3170• [GREENSPAN2011] Greenspan, Daniel. Top IT Skills In Demand for 2012. 2011

http://www.itcareerfinder.com/brain-food/blog/entry/top-9-it-skills-in-demand-for-2012.html• [GRITSCHACHER2011] Gritschacher, Tobias. A Community Website for Interactive Mobile Content Created by Children and Teenagers.

2011• [GTB2008] Global Telecoms Business. A century of influence. 2008

http://www.globaltelecomsbusiness.com/Article/2199338/Search/Results/A-century-of-influence.html• [HOLLANDERS2009] Hollanders, Hugo and Van Cruysen, Adriana. Design and construction of the Hong Kong Creativity Index.

Publications Office of the European Union. p. 41--81 2009 http://ec.europa.eu/education/lifelong-learning-policy/doc/creativity/report/design.pdf

• [ISO1998] International Organization for Standardization. ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability. 1998 http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=16883

• [ISO2013] International Organization for Standardization. About ISO. 2013 http://www.iso.org/iso/about/about

2013-11-28

Bernd Hollerit 28

Sources

• [JASTRZEMBSKI2007] Jastrzembski, Tiffany S. and Charness, Neil. The Model Human Processor and the Older Adult: Parameter Estimation and Validation Within a Mobile Phone Task. Journal of Experimental Psychology: American Psychological Association. 2007

• [JOHNSON2013] Johnson, Daniel and Jones, Christian. Beyond the beat-em-up: video games are good for young people. 2013 http://pda.sciencealert.com.au/features/20133008-24746-2.html

• [KELLEHER2006] Kelleher, Caitlin. Motivating Programming: using storytelling to make computer programming attractive to middle school girls. 2006

• [KIERAS1993] Kieras, David. Using the Keystroke-Level Model to Estimate Execution Times. University of Michigan. 1993 http://www.pitt.edu/~cmlewis/KSM.pdf

• [KNAPITSCH2012] Knapitsch-Scarpatetti-Unterwegen, Ferdinand R.. Usability Testing of Mobile Applications for Children. 2012• [KRUG2000] Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability. Berkeley, CA: New Riders. 2000• [MARSHALL2009] Marshall, Andrew. Importance of IT in business. 2009 http://ezinearticles.com/?Importance-of-IT-in-

Business&id=3221599• [MAYER2012] Mayer, Stefan. Testen von mobilen HTML5 Web-Applikationen. 2012• [MEDLOCK2002] Medlock, Michael C. and Wixon, Dennis and Terrano, Mark and Romero, R. and Fulton, Bill. Using the RITE

method to improve products: A definition and a case study. 2002• [NIELSEN1995] Nielsen, Jakob. How to Conduct a Heuristic Evaluation. 1995 http://www.nngroup.com/articles/how-to-conduct-a-

heuristic-evaluation/

2013-11-28

Bernd Hollerit 29

Sources

• [NIELSEN1996] Nielsen, Jakob. Improving System Usability Through Parallel Design. 1996 http://www.nngroup.com/articles/parallel-design/

• [NIELSEN2000] Nielsen, Jakob. Why You Only Need to Test with 5 Users. 2000 http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

• [NIELSEN2001] Nielsen, Jakob and Snyder, Carolyn and Molich, Rolf and Farrell, Susan. E-commerce user experience. Nielsen Norman Group. 2001

• [NIELSEN2012A] Nielsen, Jakob. Usability 101: Introduction to Usability. 2012 http://www.nngroup.com/articles/usability-101-introduction-to-usability/

• [NIELSEN2012B] Nielsen, Jakob. Thinking Aloud: The #1 Usability Tool. 2012 http://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/

• [NYT1998] Richtel, Matt. Making web sites more `usable' is former Sun-engineer's goal. 1998 http://www.nytimes.com/library/tech/98/07/cyber/articles/13usability.html

• [RESNICK2013] Resnick, Mitch. Mitch Resnick: Let's teach kids to code. 2013 http://www.ted.com/talks/mitch_resnick_let_s_teach_kids_to_code.html

• [RUNCO2010] Runco, M.A.. Creativity: Theories and Themes: Research, Development, and Practice. Elsevier Science. 2010 http://books.google.at/books?id=nJlfKE-meqEC

2013-11-28

Bernd Hollerit 30

Sources

• [S2011] buzzle.com. Importance of Information Technology. 2011 http://www.buzzle.com/articles/importance-of-information-technology.html

• [SAWYER2006] Sawyer, R. Keith. Explaining creativity: The science of human innovation. Oxford University Press. 2006 http://ukcatalogue.oup.com/product/9780195304459.do?keyword=keith+sawyer&sortby=bestMatches

• [SCHELHOWE2012] . IDC '12: Proceedings of the 11th International Conference on Interaction Design and Children. ACM. 2012• [SCRATCH2013] Massachusetts Institute of Technology. Scratch. 2013 http://scratch.mit.edu/• [SPENCER2003] Spencer, Donna. Card-Based Classification Evaluation. 2003 http://boxesandarrows.com/card-based-

classification-evaluation/• [SPENCER2009] Spencer, D. and Garrett, J.J.. Card Sorting: Designing Usable Categories. Rosenfeld Media. 2009

http://books.google.at/books?id=_h4D9gqi5tsC• [STERNBERG1999] Sternberg, R.J.. Handbook of Creativity. Cambridge University Press. 1999 http://books.google.ca/books?

id=d1KTEQpQ6vsC• [STIERNA2009] Stierna, Johan and Villalba, Ernesto. Design and construction of the Hong Kong Creativity Index. Publications

Office of the European Union. p. 103--111 2009 http://ec.europa.eu/education/lifelong-learning-policy/doc/creativity/report/science.pdf

• [TED2006] TED Conferences, LLC. TED Talks: Ken Robinson says schools kill creativity. 2006 http://www.ted.com/talks/lang/en/ken_robinson_says_schools_kill_creativity.html

• [TED2013] TED Conferences, LLC. TEDx Events. 2013 https://www.ted.com/tedx

2013-11-28

Bernd Hollerit 31

Sources

• [TREITLER2012] Treitler, Peter. A generic approach towards smart self-rendering mobile user interfaces. 2012

• [TURKLE2011] Turkle, Sherry. Alone Together: Why We Expect More from Technology and Less from Each Other. Basic Books. 2011 http://books.google.at/books?id=_Dhf5xEZZD0C

• [UNITID2013] UNITiD and contributors. Android Patterns. 2013 http://www.androidpatterns.com/uap_pattern/pinch-spread

• [V2010] phonearena.com. No surprise—more children owning cell phones. 2010 http://www.phonearena.com/news/No-surprise—more-children-owning-cell-phones_id8622

• [VILLALBA2009] Villalba, E.. Measuring Creativity: Proceedings for the Conference ``Can Creativity be Measured?". Brussels, May 28-29, 2009. Publications Office of the European Union. 2009 http://books.google.at/books?id=DczbcQAACAAJ

• [WEBDESIGNERDEPOT2009] Apai, Walter. Interview with web usability guru, Jakob Nielsen. 2009 http://www.webdesignerdepot.com/2009/09/interview-with-web-usability-guru-jakob-nielsen/

• [WEISBERG1993] Weisberg, R.W.. Creativity: Beyond the Myth of Genius. Freeman. 1993 http://books.google.at/books?id=_VNwQgAACAAJ

• [WEISS2002] Weiss, Scott. Handheld usability. John Wiley and Sons, Ltd. 2002

2013-11-28