when too many is just enough: citizen engagement and federal government websites
TRANSCRIPT
![Page 1: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/1.jpg)
When Too Many Is Just Enough:Citizen Engagement and Federal Government Websites
Jeffrey Ryan Pass | IA Summit 2013
@jeffpass#ias2013
#UsabilityforGov
![Page 2: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/2.jpg)
I am:
It starts with a guy introducing himself…
Jeff Pass A first-time IA Summit speaker A “User Experience Consultant” Tenaciously fighting the good fight Here to talk about Citizen Engagement
through large-scale online IA-focused usability studies
Hi
![Page 3: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/3.jpg)
President Obama supports:
Next he talks about this guy…
Innovation Transparency Information Feedback We the People Portuguese Water Dogs
For all of these reasons (but one)…
Obama from “Change” to “Engage”
![Page 4: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/4.jpg)
On May 23, 2012 the White House, CIO and CTO released:
He commissioned a strategy & signed a memo…
Presidential Memorandum: Building a 21st Century Digital Government
Digital Government: Building a 21st Century Platform to Better Serve the American People
These are better known as the Digital Government Strategy
![Page 5: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/5.jpg)
The Presidential Memorandum:
The memo was a memo (and not very exciting)…
Introduced the Strategy Put departments/agencies on notice Established a 12-month roadmap
![Page 6: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/6.jpg)
Introduced by Federal CIO and CTO:
The Strategy was something altogether different…
“Federal Government must be able to deliver information and services to the American people anytime, anywhere and on any platform or device”
Four over-arching principles (wait for it…)
12-month agency milestones (keep waiting…)
Federal CIO, Steven VanRoekel Federal CTO, Todd Park
![Page 7: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/7.jpg)
Principle Addressed Through1 – “Information-Centric” Content Syndication
Data via APIsTaxonomy/Metadata
2 – “Shared Platform” Agency GovernanceAgency InventoryEvaluate GSA vehiclesShared CMS / Open Source
3 – “Customer-Centric” Modern UXSite/content consolidationSEOMobileMeasure satisfaction
4 – “Security and Privacy” FISMA complianceData securityPersonal Information Privacy
Four Over-Arching Principles*
* The first and third are most relevant to today’s chat
![Page 8: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/8.jpg)
12-Month Agency Milestones*
3 Months 8/2012 6 Months 11/2012 12 Months 5/2013 Identify 2 services for
APIs Identify 2 services for
Mobile Agency Governance Launch Digital
Strategy Page
Device Inventory Customer Satisfaction
Implemented
2 APIs Implemented 2 mobile services
Implemented New systems adhere Compliance
verification with GSA standards
Evaluate new GSA vehicles
ü ü* No real bearing on today’s chat but important to know
![Page 9: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/9.jpg)
Digital Content of, for, and by Citizens…*
Information-Centric includes: Presenting content “in the way that is most useful for the
consumer of that information”
Allowing “customers to shape, share and consume information, whenever and however they want it”
“Using modern tools and technologies [to gather] customer feedback to make improvements”
Customer-Centric includes:
* Not really, but bear with me
![Page 10: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/10.jpg)
Sounds Like a Job For…*
Me! Us: Information Architects Content Strategists Usability Specialists Other User Experience Professionals
* At least in some significant part
![Page 11: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/11.jpg)
So Much We Can Do…
Many aspects of IA/UX/Content Strategist work can execute on the Digital Government Strategy
Specifically, large-scale un-moderated usability tests (focused on IA) can contribute to “citizen engagement”
So let’s rummage through the IA toolbox…
![Page 12: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/12.jpg)
Rummaging Through the IA Toolbox…We have the technology (to perform large-scale usability testing and analysis)
Open Card Sorts (e.g. OptimalSort)
Closed Card Sorts (e.g. WebSort)
Reverse Card Sorts (Tree Sorts) (e.g. Treejack)
Un-Moderated Usability Tests (e.g. Usabilla)
One-Click Tests (First-Click) (e.g. ChalkMark)
Immediate Feedback Tests (e.g. FiveSecondTest)
I have this ultimate set of tools… I can fix it!
![Page 13: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/13.jpg)
Still Rummaging…We have other tools too(to compliment large-scale usability testing and analysis)
Surveys (e.g. SurveyMonkey) Page-based Feedback Mechanisms (e.g. Voice of Consumer) Customer Satisfaction Tools (e.g. ForeSee) Click Analysis Tools (e.g. CrazyEgg) Heat Mapping Tools (e.g. ClickHeat) User Research Tools (e.g. Ethnio) Crowdsourcing Feedback Tools (e.g. UserVoice)
![Page 14: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/14.jpg)
Case Study: Large-Scale Closed Card Sorts…
First the background:
IA design of a public-facing website for a government healthcare agency
Began with over 100 content collections Goal to end with no more than five domains under a
single, unified IA and residing in a WCMS Iterative testing as well as multiple rounds of wireframe
usability testing
![Page 15: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/15.jpg)
Case Study: Large-Scale Closed Card Sorts…
Next, card sorting basics:
Technique for organizing and validating IA Dates back more than 100 years Can be performed in person, remotely, or online Several types:
Open Card Sorts Closed Card Sorts Reverse Card Sorts
![Page 16: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/16.jpg)
Case Study: Large-Scale Closed Card Sorts…
An online (closed) card sorting looks like this:
![Page 17: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/17.jpg)
Case Study: Large-Scale Closed Card Sorts…
Now the conventional wisdom (card sorting in general, but closed card sorting specifically):
You need a minimum sampling for validity No value in samplings bigger than 25-35 participants
Creates more analysis and reporting work
So, how many participants should you have? There are many (well reasoned and documented) opinions…
![Page 18: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/18.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Freed (2012): 15 - 20
![Page 19: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/19.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Gaffney (2000): 4 - 6
![Page 20: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/20.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Nielsen (2004): 16
![Page 21: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/21.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Paul (2008): 6 - 12
![Page 22: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/22.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Robertson (2001): 4 - 8
![Page 23: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/23.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Spencer & Warfel (2004): 7 - 10
![Page 24: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/24.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Tullis & Wood (2004): 20 - 30
![Page 25: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/25.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Tullis & Wood (2005): 30 - 40
![Page 26: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/26.jpg)
Case Study: Large-Scale Closed Card Sorts…
Legend: Minimum Optimum
Wood & Wood (2004): 25 - 30
![Page 27: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/27.jpg)
Case Study: Large-Scale Closed Card Sorts…
So, how many participants did the case study have?
?
![Page 28: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/28.jpg)
Case Study: Large-Scale Closed Card Sorts…Multiple closed card sorts with 1,000+ participants!
![Page 29: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/29.jpg)
Case Study: Large-Scale Closed Card Sorts…
And how did we engage the participants? Directly.
Social media was our recruiterA blog post was our screener
OptimalSort and TreeJack were our vehicles
![Page 30: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/30.jpg)
Case Study: Large-Scale Closed Card Sorts…
And the result?
Gained valuable insight for IA improvements, plus confirmed that large-scale approach:
Serves as a user outreach/feedback mechanism Allows for qualitative data collection alongside
quantitative data (via free-text comment fields) Raises awareness of the contribution of usability studies to
the presentation and use of online content Supports the Digital Government Strategy Really doesn’t result in unnecessary analysis and reporting,
but…
![Page 31: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/31.jpg)
Challenges and Lessons Learned…
To avoid being crushed by the weight of data, analysis, and reporting you must:
Have a clear, well-established methodology Have a clearly defined goal and scope Use an online card sorting tool that can handle large-
scale participation
![Page 32: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/32.jpg)
So what do you think?
Share your thoughts and experiences about large-scale usability studies and direct user engagement
I’m listening…
![Page 33: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/33.jpg)
Thanks for your time and participation!
Jeffrey Ryan Pass Lead User Experience ConsultantAquilent (www.aquilent.com)
[email protected]@jeffpass
Didn’t get enough (I honestly cannot imagine)? Then check out our (with UserWorks colleague Weimin Hou) case study posters at #IAS2013!
Bye
![Page 34: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/34.jpg)
Shameless Poster Plugs…
![Page 35: When Too Many is Just Enough: Citizen Engagement and Federal Government Websites](https://reader035.vdocument.in/reader035/viewer/2022081511/55512850b4c905325d8b4794/html5/thumbnails/35.jpg)
Sources:Freed, E. (2012). How-To Guide for Intranet Card Sorting. The Social Intranet Blog (09/11/2012). Retrieved 03/12/2013 from http://www.thoughtfarmer.com/blog/2012/09/11/intranet-card-sorting/. Gaffney, G. (2000). What is Card Sorting? Information & Design, 2000. Retrieved 03/12/2013 from http://www.ida.liu.se/~TDDD26/material/CardSort.pdf. Nielsen, J. (2004). Card Sorting: How Many Users to Test. Jakob Nielsen’s Alertbox: July 19, 2004. Retrieved 12/21/2012 from http://www.nngroup.com/articles/card-sorting-how-many-users-to-test/. OptimalWorkshop (2011). How Many Participants Do I Need for My Survey? (And How Many Should I Invite?). Optimal Workshop Support Knowledge Base 11/14/2011. Retrieved 03/12/2013 from http://www.optimalworkshop.com/help/kb/remote-user-testing/how-many-participants-do-i-need-for-my-survey-and-how-many-should-i-invite.
Paul, C. L. (2008). A Modified Delphi Approach to a New Card Sorting Methodology. JUS Journal of Usability Studies, Volume 4, Issue 1, November 2008. Retrieved 03/12/2013 from http://www.academia.edu/150978/A_Modified_Delphi_Approach_to_a_New_Card_Sorting_Methodology. Robertson, J. (2001). Information Design Using Card Sorting. Step Two Designs, 02/19/2001. Retrieved 03/12/2013 from http://www.steptwo.com.au/papers/cardsorting/index.html. Sachs, J. (2002). Aristotle's Metaphysics. Green Lion Press, Santa Fe, NM.
Spencer, D., & Warfel, T. (2004). Card Sorting: A Definitive Guide. Boxes and Arrows 04/07/2004. Retrieved 03/12/2013 from http://www.boxesandarrows.com/view/card_sorting_a_definitive_guide. Tullis, T. S., & Wood, L. E. (2004). How Many Users Are Enough for a Card-Sorting Study? UPA 2004 Conference, Minneapolis, NM. Retrieved 12/21/2012 from http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf. Tullis, T. S., & Wood, L. E. (2005). How Can You Do a Card-sorting Study with LOTS of Cards? UPA 2005 Conference, Montreal, Quebec, Canada. Retrieved 12/21/2012 from http://www.eastonmass.net/tullis/presentations/Tullis&Wood-CardSorting.pdf. Wood, J. R., & Wood, L. E. (2008). Card Sorting: Current Practices and Beyond. Journal of Usability Studies, Volume 4, Issue 1, November 2008. Retrieved 03/12/2013 from http://www.upassoc.org/upa_publications/jus/2008november/wood3.html. UserZoom (2011). Online Card Sorting: What, How & Why? UserZoom 01/20/2011. Retrieved 03/12/2013 from http://www.userzoom.com/online-card-sorting-what-how-why/.
Note: The Digital Government Strategy was announced on 05/23/2012 in the Presidential Memorandum: Building a 21st Century Digital Government (http://www.whitehouse.gov/the-press-office/2012/05/23/presidential-memorandum-building-21st-century-digital-government) and detailed in the actual strategy document Digital Government: Building a 21st Century Platform to Better Serve the American People (http://www.whitehouse.gov/sites/default/files/omb/egov/digital-government/digital-government.html).