communications audit
DESCRIPTION
Presentation to library schools in Tallinn and StuttgartTRANSCRIPT
Demystifying the communications audit
Barbie E. KeiserUS Consulate – Stuttgart
November 17, 2008
2
Because the world changes so quickly, sooner or later we all
need to rebuild or refine …
• An intranet, extranet/public website, or e-newsletter
• The Library-related portions of an organization’s intranet or public website
• Intranet or extranet-delivered services and content• Communications programs (internal or external)• Web based client relationship management
AKA CLIENT-BASED CONTINUOUS IMPROVEMENT
3
What worked well last year may no longer be ideal
• Input may "happen our way," indicating that improvements are needed
• But it would be risky to wait for – or rely solely on such input!
• We need to engage in planned and well-executed audits
4
"Audit" is not a scary word
• But if it seems to be, just choose another name! Strategic Planning Review Resource Assessment Business Process Planning Communications Checkup
• Translation: Regularly repeated, systematic examination of current practices and future needs with the focus on the client
5
The Audit is supplemented by ongoing monitoring:
• Mechanisms to catch evidence of: Shifts in client needs due to changes in
internal operations, changes in their target markets, and developments in industries in which they operate
Opportunities created through the application of advanced technology
• The objective is to put in place a "360°" system so as never to be caught off guard
6
The methods we choose to employ in conducting the
Audit depend on:
• Existing client relationships
• Corporate culture• Time and money
tradeoffs Benefits of a
comprehensive effort
Options for a phased-approach
• Skill sets of staff involved
• Available time frame
• Complexity of what is being examined
7
Key consideration
• How can we engage informants in a non-burdensome way?
• How do we choose the right method for each target user group?
8
Tell-it-to-me … or Easter Eggs?
• Common techniques involve a mix: Interviews Discussion (focus) groups Surveys Spot checks
• But what people say isn’t always “the goods”
• We may need more “evidentiary” methods Logs (behavior trails) Stats (behavior trends) Observed behavior as-it-happens
9
All efforts are designed to help us:
• Understand where we need / want / are able to go in order to meet evolving client needs
• Build a business case for new or redirected efforts/investments
• Create a plan for how we get there from here
10
Focus for today
• Designing a process that will yield maximum insight into client perceptions and priorities
Involving the least effort - for us and for our clients
Resulting in the greatest degree of confidence in the findings
11
Case example 1
• The Corporate Library’s Intranet presence is not cutting it!
• It may once have been the cat’s meow, but time has gone by and “grafting” has resulted in a difficult-to-use site
12
We know some challenges, based on symptoms
• Usage has dropped off significantly
• Users are confused
• Navigation is not intuitive
• Users "miss" important announcements and new sources
• Users go elsewhere for information
13
But we need to dig deeper
• Our audit must not only point toward solutions for known concerns (for staff and users), but be designed to uncover details that may be "hiding" from us
14
What are the elements to be examined?
• What specific "defects" must we address?
• What content and services would be priority offerings in the eyes of our clientele?
• What functionality do clients consider essential vs. "nice to have"?
• Any "sacred cows" we can drop?
• What are "good" library intranets doing that we aren’t? How are those
intranets maintained and by whom?
• Are there good models we could use for inspiration?
15
Audit design preamble
• Balance what we already know and what we don't yet know - avoid skewing results
• Who can tell us? Who should find out? Who on our staff should be involved in the
process? What outside assistance will we need?
16
Typical process
• Review documented (web logs) & anecdotal evidence, and any marketing collateral distributed
• Identify "regular user" / fan base informants
• Identify "non-user who should be user" informants
• Identify individuals others view as role models
• Devise interaction structures Two-on-one
interviews Discussion/focus
groups• Devise direct
observation events • Test interview/survey
instruments• Set up logistics
17
A note on communication
• Must "ace" the invitation: WIIFM• Explain the why
• If possible, make the effort fun, not arduous
• Accommodate participants' schedules• Gently remind
• Demonstrate later that input got results
18
Tell me … or may I watch?
• Common techniques involve questions
• But what people say isn’t always "the goods"
• We may need other methods - such as observation of what people actually do in a specific situation
19
"Easter Eggs"
• A specific task answerable through the intranet site or other vehicle being examined
• Forcing respondents to not just say “looks nice” but to dive in and answer a specific question
• What path used to solve question?
• Where found answer - or abandoned?
20
At-the-elbow "Process Deconstruction"
An attempt to trace a logical decision path
• "Show me a typical task you need to accomplish"
• What do you do first? Why?• Stop – why did you look there?• Stop – then what made you decide to look
here?• Why did you not check here first?
21
Benefits of observation
• Discovery of "hidden knowledge" not documented and thus unavailable to new users
• Reality check to guard against owners' sense that "it's obvious"
• Insights useful in further communication and design
22
Case example 2
A Library Consortium’s Communications Program is not as effective as it could be
Too many vehicles are employed
Many services are under-utilized
23
Project design
Initial orientation meeting Survey of members Focus groups Needs assessment and usability-lite test
for the Consortium’s website Audit and benchmarking Synthesis of data
24
Survey Identified survey content,
designed survey, and coordinated Consortium’s review of draft survey
Identified survey pretest participants, completed survey pretest, and revised questionnaire
Developed and implemented Web-based survey – Addressed security &
privacy issues by observing professional protocols for information collection
– Attained buy-in and announced survey
Hosted and monitored Web-based collection tools and systems
Monitored survey completion and followed-up
Analyzed data– Overall (Example)– Type of library
(Example)– Portraits of _____
Library (Example)
25
Focus groups
Obtain information and clarification on:– Which issues are a priority– What features are important– Extent to which members will have influence– Ways to measure success– Role of interactivity
Conduct exercises to discern:– Awareness of the services offered to and valued
by the participants – Adequacy of communication about those
services– Availability and accessibility of services – Opportunities for improvement
Present results (to organization, participants, and members)
26
"Usability-lite" test of the Website
Does our website succeed in communicating clearly? Are users having trouble?
Usability-lite tests will help you determine:– Actuals / Optimals– Drivers / Incentives– Barriers / Potential solutions
A combination of telephone interviews (for pre-screening candidates) and in-person interviews in the participants’ normal work environment– Ask participants to “think aloud” as they
explore the website– Personae and specific situations– Ask some follow-up questions
27
Audit and benchmarking: Objective website review
Reviewed the site from a member’s perspective vis-à-vis stated goals
Analyzed extant data (e.g. logfile data) Evaluated typical navigation Examined usability/human factors Assessed the calls to action and flow of copy Identified interactive techniques Provided recommendations for:
– Navigation, technical, and usability functions– Marketing copy – Interactive techniques– Access to other information systems and
services
28
The result: User- and usage-centered design
29
Before…
30
After…
31
Tips for interviews and focus groups
Explain if/how an audio/video recording of the discussion will be utilized
Have a set of questions but be flexible Assure complete confidentiality Let them talk Be comfortable with silence, but ready to "prime
the pump“ Elicit elaboration by validating: Interesting, you
are not the first to say so Use "others-find" technique (you too?) Be aware of interpersonal dynamics and politics Recognize that some may not want to "look bad" &
may tailor comments to what is thought "correct"
32
Tips for surveys
Short - Fast – Easy – did we mention short! Clear, unambiguous questions in sections that flow Ranking of personal priorities (What means more
to you?) vs. "easy middle choice" "How much do you love us on a 1-10" yields less
valuable insight (no one wants to offend) Minimize the number of open-ended questions Do you agree with these statements made by your
peers? People shouldn’t agonize over responses, so
provide an "out" using Don’t know/Not sure options
33
Tips for the direct-observation portion of a web usability test
Explain that the findings from the evaluation will be used "for good purpose"- no need to be polite
Explain that you will be collecting data by taking written notes
Stress that the website is being tested, they aren't
Remind interviewees to articulate their thoughts
Stay neutral
Help users in distress
Ask if they have any questions before the interview begins
34
What do we have in hand when it's all done?
Ideally, the elements for our business case
The justification for action and investment
The credibility that "the users spoke"
35
Audit Report sets out (in one place)…
• Drivers for the Audit (why did we)• Goals (what looked for)• Methodology / Informants (how did we)• Findings (broken down by major topic area) –
factual, dispassionate• Conclusions that take the findings and group
them into themes• Recommendations flowing from the
findings (not just a crazy idea)• Business Case that includes a “how we will”
implement change & risks of not implementing
36
Some of what we uncovered may be similar to your own
experiences
Overall communications strategy: NSLS is responsive to member requests, but with some interesting effects
Target market: Probability of messages being opened, read, and acted upon increases dramatically as the targeting becomes more precise
Design/layout: Members normally find what they need on the NSLS website, but not easily, and they often forget where pages used in the past are located
Orientation/navigation: Consistency and clarity
37
Some of what we uncovered may be similar to your own
experiences (con’t)
Search: Search functionality provided by local Google and sitemap
Content: Targeted content; relationships established; sourcing of content
Networking: Member "control" over physical and virtual (listservs vs. CoPs)
Continuing education: Professional development opportunities with series of courses for career paths
38
Access to technology
Issues surrounding technology focused on convenience in accessing NSLS services rather than not having access to required technology to make full use of them
Self-described as not being “explorers,” merely accepting what is on the screen in the way in which it appears
Preference for accessing NSLS while working, and not at home (nights or weekends)
39
Moving from findings and conclusions to recommended
action
Document findings– Known challenges and potential solutions– Existing technical barriers and other constraints
Synthesize/interpret findings to draw conclusions– What works– Areas for improvement– Opportunities for growth
Make recommendations– Implement immediately– Implement in the near-term– Longer-term initiatives
40
What have we learned today?
Why organizations should conduct audits Options for soliciting client input The importance of communicating with
our clients (and especially those who give time to the audit process)
Alternative ways to say “no” Evolution, not revolution
– No surprises
41
Questions?
