building a crowdsourcing program at fda
TRANSCRIPT
BUILDING A CROWDSOURCING PROGRAM at FDA
FROM MARKET RESEARCH TO A SYSTEMATIC PROGRAM
Mark Ascione and Kim TaylorOffice of Strategic Programs (OSP)Center for Drug Evaluation and Research (CDER)Food & Drug Administration (FDA)
Here’s what we want you to learn from our story
Leveraging standardized tools for consistent results and increased bandwidth
Building excitement around and engagement in a crowdsourcing program
Increasing value through program improvements
2 31
Today’s agenda
Developing tools to support increased demand
1Leveraging early successes to establish best practices
Using learning from all of you to increase value
Introducing CDER Crowdsourcing
23
4Q&A5
Our program was designed to enhance intra-center communication and engagement
• Idea sparked by Partnership for Public Service’s “Effective Internal Communication Strategies Workshop”
• Crowdsourced the federal ideation community in designing program
• Operate as part-time crowdsourcers• Focus on short-term challenges (two
weeks) sponsored by senior leaders
Background
Execution
Goals• Drive ideas and information up to senior
leadership • Engage employees
Our early success was driven by thoughtful challenges,communications and incentives
E a r l y S u c c e s s
Example: Office Retreat Planning Challenge
• Objective: Better understand office issues to tee up discussion at upcoming all-office retreat
• Incentive: Office-wide one-hour early dismissal
• Participation rate: 80%
• Tangible results: New office newsletter, recognition process and recurring All-Hands events
• Engaged executive sponsors and moderators
• Focused topics with specific questions appropriate for crowdsourced conversation
• Interested target audiences
• Developed by communications professional• Consider target audience• Include multiple modes and touch points
• Targeted toward participation and/or quality idea generation
• Include creative approaches appropriate for federal space
Challenges
Communications
Incentives
Increased demand and limited resources required standardized tools for continued success
C o n t i n u e d S u c c e s s
Example: Pharmacy Preceptor Challenge
• Objective: Solicit ideas to enable more pharmacy preceptor students at CDER
• Incentive: Three four-hour time-off awards
• Evaluation criteria: Focus, timeliness, feasibility, popularity
• Tangible results: overhauled preceptor program process
• Outlines critical components of each challenge: challenge objective, audience, questions, incentive, etc.
• Informs planning conversations with executive sponsor and moderator
• Contains suggested communication touch-points customizable to challenge
• Spans pre and post challenge, including communication of any implementation plans
• Contains moderator best practices
Initiation Template
CommunicationPlan Template
Moderator Checklist
Going forward we plan to evaluate our program, connect with customers, and increase value
Value
“Stages” implementation and
enhanced community design
Customer Connection
Marketing segmentation and SharePoint
integration
Evaluation
Focus groups, interviews and surveys
We will leverage our core strength by evaluating CDER Crowdsourcing!
P E I S
Program Evaluation and Implementation Staff
• Consists of five expert evaluators (three of whom form Crowdsourcing team)
• Conducts and manages major internal evaluations related to key programs and initiatives
• Objective: Understand mindset of different user segments, how to drive engagement, and how to enhance CDER crowdsourcing
• Method: Surveys/Focus groups
• Objective: Learn what went well, what we can improve, and impact of implemented ideas
• Method: Surveys/Focus groups
• Objective: Discuss value of crowdsourcing, how to increase presence within CDER, and impact of implemented ideas
• Method: Interviews
Crowd
Moderators
Sponsors
We will utilize the “lurker” metric as we increase user engagement
U s e r s e g m e n t s
• Lurkers: Users who sign in to a challenge but do not participate
• Participants: Users who submit ideas, comment, or vote once or twice but no more
• Super-crowdsourcers: Users who log in repeatedly and participate more than twice
Engagement
Super crowdsourcer
Participant
Lurker
We will bring crowdsourcing to where staff does its work…SharePoint!
O p e n i d e a t i o n
• CDER’s large staff (5000) and bureaucratic culture not ripe for open ideation
• Offices are smaller (500-1000) and have staff dedicated to employee feedback
• Several offices are developing message board / suggestion box processes
• SharePoint integration brings power of IdeaScale right to users
IdeaScale
SharePoint
We plan to implement IdeaScale’s “Stages” to help ideas move past ideation
S t a g e s
• Ideate: Generate ideas and comments from staff
• Staff assessment: Adjudicate ideas through voting (currently combined with ideate)
• Sponsor assessment: Review and elevate top ideas using ReviewScale (currently done offline)
• Action: Choose ideas for implementation, consideration, and parking lot
Ideate
Staff Assessment
Sponsor Assessment
Action
We will create one area on our site for crowdsourcing resources
“ A b o u t u s ”
• Our crowdsourcing story
• Top-line statistics (# of challenges, # of ideas, etc.)
• Infographics(overview and how-to)
About us
Provides background on the program and includes infographics
Past challenges
Includes previous challenge questions, ideas, votes, and comments
How to participate
Describes how to submit an idea, vote, comment and participate confidentially
Host a challenge
Educates potential sponsors and moderators about how to host a challenge
Appendix
RESPOND
ENGAGE
Thank users for idea submissions (comments as warranted)
Ask for elaboration of ideas or comments on submissions that are vague or about which you are curious
Specify in a responding comment if an idea submission is already in practice MERGE + DIVERGE
Merge ideas that are similar
Diverge comments that contain new ideas into their own separate ideas
Send relevant ideas to SMEs and ask them to engage on the crowdsourcing site
SEND
MONITOR
NOTIFYNotify crowdsourcing team of any inappropriate or negative submissions
TRANSFERTransfer off-topic or irrelevant ideas to the “additional input” category
Version 2.0 | 6/1/2018
c he c k l i s t / / C D E R C R O W D S O U R C I N G
M O D E R A T O R S
CDER CROWDSOURCING CHALLENGEINITIATION TEMPLATE
SPONSOR
MODERATOR(S)
DATES
AUDIENCE
INCENTIVE
What senior leader sees the need for and will act as the face of this challenge?
Who will monitor submissions to make sure they are appropriate and respond to idea submissions?
When do you want to hold the challenge?
Which office(s) are you targeting?
Is there an incentive for this challenge (based on participation, idea quality, or some other factor)?
OBJECTIVES What do you plan to get out of this challenge? How will the results be used?
Insert your response here
Insert your response here
Insert your response here
Insert your response here
Insert your response here
This list should contain bullets of your overall goals for this challenge.
CDER CROWDSOURCING CHALLENGEINITIATION TEMPLATE
INTRO PARAGRAPH
DRAFT QUESTIONS
What is this challenge about? Why is it important?
Ask open-ended questions that require non-’yes’/’no’ answers
This paragraph will be posted on the home page of the crowdsourcing website and will be used to develop communications plan. Please keep your paragraph to 200 words or less.
Limit yourself to 3-5 questions and provide titles for each question.
CDER CROWDSOURCING CHALLENGEINITIATION TEMPLATE
OPTIONAL FEATURES Please highlight below the features you would like to utilize for your challenge.
- Executive sponsor photo on crowdsourcing site and in communications- Turn off voting- Turn off down voting- Close voting until the last week of challenge after ideas have been reviewed and selected- Add profile questions to track demographics- Incentive for challenge participation- Send final challenge report to participants- Using stages- Posting flyers for challenge promotion