google sites usability test · 2014. 11. 18. · to build a website with google sites, one must...

Running Head: GOOGLE SITES USABILITY TEST

2014

Google Sites Usability Test

JILL HAYNES, FREDERICK KELLY, LINDSEY LEFFEW, KIMBERLY STAFFORD

THE UNIVERSITY OF TENNESSEE, IT521

NOVEMBER 12, 2014

GOOGLE SITES USABILITY TEST

1 | P a g e

Introduction

Before the advent of numerous web authoring tools, website creation was a convoluted process

better left to those with HTML coding knowledge and experience. These days, the process has

been simplified by the many "website creation tools" that exist on the Internet. These tools allow

anyone to build a site, whether they are a novice just starting out or someone who has had years

of experience.

The web authoring tool, Google Sites, allowed novices to create an easily navigable, tailored,

professional-looking website in less than one hour. To build a website with Google Sites, one

must have a Google account (which in turn provides the option of integrating your site with other

Google tools). The tool is ideal for people looking to build a basic, free website and offers

multiple themes geared toward everything from professional and business portfolios to teaching

and education to blogging and personal interests or hobbies.

Users also have the ability to create collaborative websites where multiple people can

collaborate, share files and edit content. In such a case, Google Sites offers three levels of

permissions for people to interact with a site: owners, editors, viewers. Owners have full

permissions and can manage the design and content of the whole website. Editors are only able

to change the text content of the site, not the design. Viewers, of course, may only view the site

and do not have permission to make any changes to text or design.

Google Sites is a middle-of-the-road tool, perfect for people with varying ranges of experience

building websites. Some familiarity with website building tools would be helpful, as would

experience using and navigating Google support (as it is a go-to document for assistance in


2 | P a g e

creating a Google Site). Extensions, such as Gadgets and HTML Box, are available for

experienced users who want to integrate custom CSS, JavaScript and HTML into their site.

Low-Cost Usability Testing Methods

As per Rubin and Chisnell (2008), we applied many of the basic elements of usability testing to

our project, including:

● The development of research questions

● The use of a representative sample of end users

● A representation of the actual work environment

● Any recommendation(s) of improvements to the design of the product (p. 25)

Prior to scheduling user testing, we developed a series of documents including a release form,

pre-questionnaire, a user task list, and a post-questionnaire,

which the reader can find in the appendices. The pre-

questionnaire elicited responses about our users’ previous

experience using computers, the Internet and Google Sites or

other website building tools. The user task list provided the

specific tasks we wanted users to perform, from the initial

creation of a website to the final saving and sharing of that site.

The post-questionnaire gathered responses about the ease of

each of task, the general usability of Google Sites, whether they would use the tool again and

what they might change about the tool.

Google Sites Landing Page


3 | P a g e

Once the forms were completed, we commenced with our low-cost testing method of scheduling

four volunteers time to work with Google Sites. Zero costs were incurred as the testing space,

video equipment and subsequent use of YouTube were available free of charge. The users ran the

gamut of computer experience, from novice to expert. The testing took place in a quiet room on

the University of Tennessee campus. We gave each user one hour in which to achieve the

checklist of tasks, provided below. By setting a time limit and specific goals, we were be able to

measure the effectiveness and “learnability” of the tool.

User Tasks:

● Create a Google account or log in with existing account

● Select a free theme to match your website theme (family, hobbies, news, special

interest, sports, business/professional)

● Add a title or name to your site

● Add a short description of your site

● Add a couple of pictures to your site, each one a different size.

● Add a second page to your site

● Add a picture or some text to your second page

● Make an edit to one of your pages

● Save and share your site


4 | P a g e

Group Expectations

Due to our earlier experiences using Google Sites for the first time, we anticipated some user

apprehension with the tool among our test subjects. Since, as Rubin and Chisnell (2008) explain,

“it is impossible to measure the usability of something. You can only measure how unusable it

is: how many problems people have using something, what the problems are and why” (p. 6), we

focused on the ease of use, effectiveness, and “learnability” of Google Sites according to our

participants.


5 | P a g e

We had four participants, who ranged in completed education from middle school to an M.D. Of

those four participants, one was 42 years old, did not use a computer for her job, and had a very

limited skill set consisting of performing word processing, launching a browser, and

finding/navigating web pages. Another participant, a 49-year-old, navigated the web regularly

for her job and had created a website over a decade before, using Microsoft FrontPage. The final

two participants were regular users of computers and the Internet. One was a 17-year-old male,

who is often online for school and entertainment purposes; the other was a 63-year-old man who

worked as a programmer for the banking industry for several years but whose current computer

use consists of using it at work and at home, for leisure. Neither of them had ever built a website

before and none of the four participants had any familiarity with Google Sites.

Google Sites is not particularly challenging, therefore, we expected three of these users to be

able to utilize the tool to construct a website within or under the time limit, even if it was not the

exact site they first desired. We predicted the 42-year-old participant with limited computer

experience, would likely be able to do some, but not all, of the assigned tasks within the one-

hour time limit.

According to Rubin and Chisnell (2008), “[t]o be usable, a product or service should be useful,

efficient, effective, satisfying, learnable, and accessible” (p. 4). In our own testing of Google

Sites we found that several of the features did not “behave in a way that we expect[ed].” Our

perception was that Google Sites was fairly easy to use, however, some of us found that we

could not accomplish the task of creating our own websites exactly the way we desired and we

expected our testers to have similar outcomes.


6 | P a g e

Usability Testing Results

User 1. The first step User 1 made was to confirm that she was proceeding correctly. Regarding

the setup process, she asked about the need for a Google Account, confirmed she had a Gmail

account and decided to create a new account to build her site. It might be good for Google Sites

to provide a more explicit statement, such as: “Google Sites uses your Google/Gmail login

information, if you already have an account.”

As User 1 created her account and began setting up her site, she spent several minutes very

deliberately reading statements and browsing templates and tips. As it turned out, the user found

the tips to be distracting. User 1 then set about completing the task list. She later complained

that Google’s template tips were “annoying,” and not what she expected. Specifically, she

considered them to be “rabbit holes.” The tips were not clear and she could not easily apply

them.

User 1 was ultimately able to complete her tasks within the allotted hour (despite problems with

the Internet connection, which resulted in Google Sites not loading quickly, accounting for some

of the lag time during her session). User 1 said that she might use Google Sites again, but she

was frustrated that she could not get a link she had added to work properly. She had also been

concerned that, upon closing the web browser window she had been using to edit to site, she

would no longer be able to find her site. She was, however, able to return to her site.

User 2. The 17-year-old participant, appeared confident and relaxed as he set out to begin his

tasks. He was able to finish all tasks without any difficulty and did so within 16 minutes. The

only questions he asked were for clarification. These included “do I need to open a new account

or use my existing one?” He used the existing one. He also asked if what he put as the title


7 | P a g e

mattered. The last question was whether he should create a link or just a second page. In the

post-testing interview, this participant said the only problem he experienced was due to not

reading the instructions (rather than it being difficult for him). He mentioned that Google Sites

functions much like Microsoft products that he was already very familiar with, therefore, he

found it quite intuitive.

User 3. User 3, who had very limited experience with computers and the Internet, first looked

through the initial steps for setting up her Google Site, then used an already-existing Google

account to log in and begin creating her site. At one point she inadvertently “backed out” of her

in-progress website, but navigated to her Google account and was able to find it again. The site

had saved her changes.

User 3 referred back to the templates tips multiple times. At 14 minutes into the test, she became

obviously frustrated. “I can’t figure out how to make a second page,” she said. After looking

through several of Google’s help pages, she found that in order to add more pages a user has to

first edit the sidebar. “That’s not intuitive,” she said, and then returned to Google’s help pages to

find how to edit the sidebar as required. After that she was able to add a second page to her site.

User 3 completed her tasks in just under 25 minutes. She had similar issues to User 1 when it

came to adding links to her Google site. “It’s difficult,” she said, but only because “it’s not like

anything I’ve ever done before”. User 3 said she would use the tool again, if only because it’s

now the only web authoring tool she’s had experience with to date.

User 4. The 63-year-old user was unsure where to begin and asked for guidance. He was told to

reference the open document, then shown where it was. Having no experience with this

particular laptop slowed this user down somewhat. Once he began, he worked on his task list


8 | P a g e

steadily. He mentioned that the name he had wanted for his site was unavailable, but was able to

continue without help. Later, he asked, “how do you do a final save?” but then proceeded

without help.

User 4 then seemed to have trouble creating a link and spent about five minutes on that task. In

the post-test stage, he was not sure how to get back to his Google Site to show the tester what he

did not like about it. When asked what he thought of his experience with Google Sites, he

expressed frustration that one must hover over Edit Page and New Page for them to be visible.

He asked why Edit Page and New Page were not showing all the time. His summary was that

creating a website with Google Sites was “not difficult, but not really easy.” He felt it was

somewhat counter-intuitive. This participant finished his tasks within twenty minutes. As he was

our fourth and final tester, once he finished, we were able to view and compile information on

the user experience of all of our participants.

Results of Post-Test Survey

The resulting impressions of all four users varied from neutral to very satisfied; while a few of

the participants experienced some difficulty executing a few of the tasks, this did not appear to

negatively affect their experience. Two of the participants had trouble adding links to their

Google sites, but User 3 “got around it by using cut/paste”. The other two participants enjoyed

the process of adding photos or pictures most of all. The required tasks were “all pretty boring,”

said User 2. “I didn’t really enjoy any [tasks] except for [those involving] pictures”. User 2 also

felt that he would prefer “templates with less on them” and User 4 agreed, adding that “adding

comment[s was] not enjoyable” because it was counter-intuitive.


9 | P a g e

Analysis

According to Rubin and Chisnell (2008), “[t]o be usable, a product or service should be useful,

efficient, effective, satisfying, learnable, and accessible” (p. 4). With our user tests, we were

very focused on the ease of use, effectiveness, and the “learnability” of the tool. After reviewing

each user’s video and post-questionnaire responses, we made the following observations.

Usefulness. Rubin and Chisnell state that “[u]sefulness concerns the degree to which a product

enables a user to achieve his or her goals….” (p. 4). After a few struggles, each with a separate

issue, but all of which were easily overcome, the users were ultimately able to complete all of

their tasks within the allotted time. Each user finished their site and while it might not have

looked entirely the way they intended, they seemed satisfied with what they had built. From our

perspective, the sites each user built appeared finished, ready to be viewed by the public. Thus,

Google Sites is useful.

Efficiency. Efficiency, Rubin and Chisnell describe as, “the quickness with which the user’s

goal can be accomplished accurately and completely and is usually a measure of time” (p. 4).

We gave our four participants a maximum of one hour to perform the list of tasks. All four

participants finished their tasks in significantly less than one hour. The shortest time taken was

16 minutes. The longest time a participant needed to finish was 52 minutes. However, this

length was due to the wireless Internet at UT being intermittent at that time. Again, all four users

finished their sites in less than one hour, even with complications by technology which had

nothing to do with Google Sites itself. Therefore, by Rubin and Chisnell’s definition, Google

Sites is efficient.


10 | P a g e

Satisfaction. According to the Handbook of Usability Testing, “[s]atisfaction refers to the user’s

perceptions, feelings, and opinions of the product.” (p. 4). As previously stated, the user reaction

ran from neutral to very satisfied. Even the users who faced challenges did not come away with a

negative opinion of the tool. While the users felt some parts of the tool could be reworked, such

as the clarity of the template tips, overall the users were satisfied with Google Sites.

Learnability. We understand that learnability refers to the “user’s ability to operate the system

to some defined level of competence after some predetermined amount and period of training”

(Rubin and Chisnell, 2008, p. 4). Learnability is part and parcel of effectiveness. With regard to

Google Sites, our group did not provide any training on the tool. Each user was merely instructed

to access the Google Site help pages if they faced any challenges. As each user was able to

complete all assigned tasks in building their website, within the allotted time and without

direction from the overseeing group member, we argue that Google Sites is very learnable. In

our experience, and based on the experience of our users, people need no introductory training to

build a website using Google Sites.

Effectiveness. As previously explored, effectiveness is what we most expected to measure.

Rubin and Chisnell (2008) explain that “[e]ffectiveness refers to the extent to which the product

behaves in the way that users expect it to and the ease with which users can use it to do what

they intend” (p. 4). In our own testing of Google Sites we found that several of the features did

not “behave in a way that we expect[ed].” Our perception of the ease of use was that Google

Sites was fairly easy to use, however, some of us found that we could not accomplish our task of

creating our own websites exactly the way we desired. For example, the sizes of images could

only be small, medium, large, or original size and could not be customized to any size other than

those options. Therefore, we did not expect all of our participants to report that Google Sites


11 | P a g e

behaved the way they expected. However, contrary to our expectations, all four of our

participants reported on their post-test questionnaires that Google Sites behaved as they

expected. As previously mentioned, all four of our participants were able to use Google Sites to

achieve the assigned tasks. These factors demonstrate that Google Sites is effective.

Accessibility. Within the broadest definition of accessibility, we found the site to be sufficiently

accessible for your average, able-bodied user. As previously mentioned, most of our users

encountered an issue that could have been prevented with some changes to the template tips and

clearer instructions on how to make additional pages. While our users found those issues to be

irritating, they were easily overcome with a little exploration.

With regard to the more narrow definition of accessibility, defined in Rubin and Chisnell (2008),

as "what makes products usable by people who have disabilities" (p. 5), we did not test for that.

None of our users disclosed a disability, whether temporary or permanent, so these accessibility

issues were not ones we directly faced.

Recommendations

We conclude that Google Sites meets all of the definitions of “usable” that we could observe,

including “useful, efficient, effective, satisfying, and learnable” (Rubin and Chisnell, 2008, p. 4).

We do have recommendations to Google Sites, however, based on the feedback of our testing

participants. The most common complaint was about adding a link: more than half of our users

found the process of creating a link to be very difficult. This process could be made more

obvious by using the word “Link” instead of the chain-link icon that is currently used. Adding a

second page was challenging for one participant as once you create a new page, you must edit

the site layout to ensure the page shows in your menu. User 3 specifically mentioned this in her


12 | P a g e

post-survey: “’Edit Sidebar [option was] confusing.” Having the option to “add the new page to

your menu” in the same area as users create a new page could simplify the process. Finally, our

participants reported that the template tips are very unclear. “The tips were annoying,” said User

1 and indicated that it would have been easier to launch “into the activity from the tip.” These

tips could be reworded in a more straightforward way, such as to not be perceived as a “rabbit

hole.”


13 | P a g e

Reference

Rubin, J. & Chisnell, D. (2008). Handbook of Usability Testing: How to plan, design, and

conduct effective tests (2nd ed.). Indianapolis, IN: Wiley Publishing.

IT 521, Usability Testing, RELEASE OF LIABILITY/IMAGES

APPENDIX A

Name ______________________________________________________________

I prefer to be called: ___________________________________________

Contact Information: Work (________) ____________-______________

Cell (________) ____________-______________

E-Mail ________________________________________________________

Preferred Method of Contact: Work Phone Cell E-mail

Emergency Contact: Name __________________________________________________

Number __________________________________________________

Release of Liability: For mutual consideration received, I, being of lawful age and sound mind,

release and hold harmless Kimberly Stafford and any agent of hers, from any and all actions,

expenses, damages, liability, claims, demands, compensation, loss of services, causes of action, and

the like, whatsoever, arising out of or related to any loss, damage, or injury, including death, that

may be sustained by the participant and/or the undersigned, during the course of this usability

testing, which will include any activity regardless of location, including but not limited to any

premises under the control and supervision of Kimberly Stafford; The University of Tennessee; and

to or from any of said premises required of the testing. I understand that this release also applies to

my heirs, executives, administrators, and assigns, and that it applies to any and all other persons,

firms, and corporations not specifically listed herein but that are part of the testing.

Release for Use of Photograph/Video: I hereby specifically give my permission to Kimberly Stafford

and The University of Tennessee to use any photos or video material taken of myself from the

classroom, off site, and/or my Google Sites Usability Testing participation. It is my understanding

that the photos and/or video material will only be used for presentation, research, training

purposes, and for the presentation of the research/training. As with all research consent, I may at

any time withdraw permission for photos or video footage of me to be used in this project, however

I understand that withdrawing my permission/consent must be in writing and will only be effective

from the date submitted forward.

IT 521, Usability Testing, RELEASE OF LIABILITY/IMAGES

APPENDIX A

Medical Emergency I understand that Kimberly Stafford will contact the person listed above in the

case of a medical emergency. If she is unable to reach said person, I hereby specifically give

Kimberly Stafford permission to seek necessary emergency medical treatment for me. I hereby

affirmatively state that I am in good physical and mental health, and that I do not have any known

physical/mental problems, restrictions, or condition that have not been disclosed, in writing, to

Kimberly Stafford. Should I need emergency medical treatment, I request that doctor/physician,

_________________________ (name) be called at __________________________ (telephone

number) and, if needed, that I be transported to ______________________ Hospital / Medical

Center.

I have read and understood all of the above and agree to the terms herein. I also acknowledge I have been given a unsigned copy of this Release. _____________________________________________________________ Date___/____/_____ Signature of Participant

IT 521, Google Sites Usability Testing, Participant Survey

APPENDIX B

Pre-Survey

Date: _________________________________

Name ______________________________________________________________ Your name will not be disclosed in the final report.

E-mail Address ______________________________________________________________ Your E-mail address will not be disclosed in the final report.

Gender Female Male Age __________

Occupation ______________________________________________________________ OPTIONAL

Highest Level of Education Completed (OPTIONAL):

Grade School Middle School High School 1-3 Years of College

Bachelor’s Degree Master’s Degree Doctorate Degree Post-Doctorate

Other: ______________________________

Average Estimated Computer Usage/Day:

Less than 30 minutes Three-Five Hours Nine-Ten Hours

One-Two Hours Six-Eight Hours More than Ten Hours

Percentage of Computer Usage/Day to total 100% (e.g., 30%, 70%):

Personal Use: __________ School / Work-Related Use: __________

Percentage of Internet Usage/Day to total 100%:

Personal Use: __________ School / Work-Related Use: __________

How often do you use the Internet for the following? (Please check the appropriate box.)

Never Once/Month Once/Week Once/Day More Than Once a Day

Check E-Mail

Pay bills online

Research/Search

Social Networking (e.g., Facebook)

Internet Gaming

Shopping

Do you have experience building a website? Yes No

If Yes, have you ever used Google Sites web authoring tool? Yes No

Disclaimer: Before you begin, please note that the IT 521 Usability Testing Team will not reimburse you for a paid Google Sites account. Please take care to select the FREE version from the onset and to select only a FREE template.

IT 521, Google Sites Usability Testing, Task List

APPENDIX C

Google Sites Usability Testing

Usability Test Task List

This document lists the tasks you will need to perform while using Google Sites. You will need a Google account to create a Google Site. You can sign up for an account here: Sign Up.

Once you’ve logged into your Google account, if you need to, you can access the Google Sites Help Center and click on “Beginners guide to creating a site.” This document will walk you through the creation and management of your site.

Google Sites Help Center https://support.google.com/sites/?hl=en#topic=1689606 Disclaimer: Before you begin, please note that the IT 521 Usability Testing Team will not reimburse you for a paid Google Sites account. Please take care to select the FREE version from the onset and to select only a FREE template. Once you have created your site, please refer to the task list below. Perform each of these tasks. Make note of any trouble you had with each task.

Create your account.

Select a FREE template to match your website theme (family, hobbies, news, special

interest, sports, business/professional, etc.).

Add a title or name.

Add a short description of your site.

Add a couple of pictures to your site, each one a different size.

Add a second page to your site.

Add at least one picture or some text to the second page of your site.

Make an edit to one of your pages by adding a link or Google gadget.

Save and share your site. (Send an E-mail to Kimberly Stafford: [email protected] with a

link to your site. Please note: There is no “d” in the E-mail address above for Stafford.)

NOTES:

https://accounts.google.com/SignUp

https://support.google.com/sites/?hl=en#topic=1689606

mailto:[email protected]

IT 521, Google Sites Usability Testing, Participant Follow-Up Survey

APPENDIX D

Name ______________________________________________________________ Your name will not be disclosed in the final report.

Please rate the level of ease with which you were able to do the following activities during your experience with Google Sites:

Very Difficult Difficult Neutral Easy Very Easy

Selecting a template

Adding a title or name

Adding a short description of your site

Adding a picture to your site

Adjusting the size of your picture

Adding a second page to your site

Adding a picture to the second page

Adding text to your second page

Adding a link or Google gadget

Saving your site

Sharing your site

Very Difficult Difficult Neutral Easy Very Easy

1. What would you change about Google Sites if you could?

2. Was there a task that you were not able to perform? If so, what obstacle(s) prevented you from

succeeding in your task?

3. Which task was most enjoyable for you to do?

PLEASE CONTINUE ON THE BACK PAGE.

IT 521, Google Sites Usability Testing, Participant Follow-Up Survey

APPENDIX D

4. Which task was least enjoyable for you to perform?

5. Did Google Sites behave the way you expected? If not, what was different from your

expectations?

6. If you were to create another website, how likely would you be to choose Google Sites as the

tool?

Very Unlikely Unlikely Neutral Likely Very Likely

Please Explain:

7. How satisfied are you with Google Sites overall?

Very Unsatisfied Unsatisfied Neutral Satisfied Very Satisfied

IT 521, Google Sites Usability Testing, Usability Testing Notes

APPENDIX E

Usability Testing Notes Things to look for while reviewing user test videos

Users Initial reaction to the tool? (Dive right in? Read FAQ? Etc.)

Did user seem to struggle at any point? With what?

Did user request help? For what?

Was user able to successfully complete all tasks?

google sites usability test · 2014. 11. 18. · to build a website with google sites, one must...

Documents