leveraging the power of surveys - office of human resources · 2019-11-20 · 4/3/2018 leveraging...

Leveraging the Power of Surveys

Lillian ThomasAnalytics Manager (Section Chief)

HR Systems Analytics & Information Division (HR SAID)Office of Human Resources (OHR)National Institutes of Health (NIH)

Why Use Surveys?

Get more context Fill the ‘gaps’ of quantitative data Provide insight into employee opinions Understand issues and develop solutions Assess the effectiveness of programs Improve processes Track trends

4/3/2018 Leveraging the Power of Surveys 2

Kick-Off

Design

DevelopImplement

Analyze/Report

Take Action

Survey Life

Cycle


Kick-Off

Design Survey

Develop Online

Implement/Administer

Analyze Data/Report Findings

Take Action


Developing the OHR Pulse SurveyGet Leadership

Buy-In

Promote Survey- Leadership announcing survey

coming- Sharing previous survey

results/impact if applicable

Launch Survey- Monitor survey response rates- Continue to promote survey &

encourage participation

Determine Topics of Interests

Develop Survey- Questions development- Provide draft survey for

leadership review- Test survey

CASE EXAMPLE


Reporting for the OHR Pulse Survey

Analyze Results

Action Planning

Implement ChangesDebrief Leadership

Communicate Findings

CASE EXAMPLE

Survey Life Cycle Best Practices


Kick

-Off Discuss survey

objectives up front and make sure they align with questions and the analyses.

Desi

gn Evaluate survey questions and ensure they are free from errors and pitfalls.

Deve

lop Reduce survey-

taker burden.Ensure confidentiality.

Impl

emen

t Sponsor to advertise surveys.

Anal

yze/

Repo

rt Ensure data is free from errors.Conduct analyses appropriate to objectives.As reports mature, find ways to automate the report (or segments) for quicker dissemination.

Take

Act

ion Communicate

survey results.Take appropriate action.Communicate steps taken as a result of prior surveys.

What analyses can we do with engagement surveys? Basic summary report Significance testing Cross-tabulation Open-text analysis Trend analysis Cross analysis with other survey and HR data


Examples of Analysis


Significance Testing

Thematic/Open-Text Analysis

Trend Analysis

What is the difference in supervisor satisfaction between

employees likely to leave in a year and employees intending

to stay? Is that difference statistically significant or is that

due to chance?

What are people saying in their open

text responses? What can we extract from

the numerous lines of texts?

How is the organization doing compared to last

year? Are there any increases or decreases

worth noting?

Survey Data & Analytics


SURVEY RESULTS

o Pulse Surveyo NIH Exit

Surveyo FEVS

DATA/ANALYTICS

o Workforce Demographics

o Separation Data

OUTCOMES

Workforce PlanningRetention Strategies

Action Planning

Employee EngagementImproved Organizational Process

Data Driven Action PlanningInitiatives/Programs Resulting from Action Planning Based on OHR Pulse Survey


OHR Director’s Awards CeremonyOHR Training Program

HR Opportunity Program

ConnectMore Events

CASE EXAMPLE

Key Takeaways


Survey Participation

Advertise- Have someone important sponsor the

survey- Send reminder emails

Provide Confidentiality- Preserve confidentiality- Increase participation and valid

responses

Share Results- Share findings with participants

Take Action/Show Commitment- Show participant input is useful and

actions will be taken based on survey results

- Increase customer buy-in

Survey Validity

Keep It Simple- Use succinct questions- Reduce instrument bias to get more

valid data

Survey Utility

Incorporate Multiple Sources- Consider incorporating other sources

of data to gain further insights

Contact InformationLillian ThomasAnalytics Section ChiefNIH/OHR/HR [email protected]


mailto:[email protected]

Additional Resources: Types of Survey Questions


Demographic Questions

Select Best & Multi-Select Questions

Likert Scale Questions

Baseline/Overall Rating Questions

Open-Ended Questions



Demographic Questions• Questions that asks respondents their demographic

information • IC, Series, Grade, Pay Plan, Tenure

• Include demographic questions if they are needed for analyses

Please select the employee type that best describes you.o Intern/ FellowoNIH Employeeo Contractor



Select Best & Multi-Select Questions• Select Best: Respondents forced to select only one answer

Please select your preferred training delivery method.oClassroomoOnline

• Multi Select: Respondents can select multiple answers

What training areas are you interested in? Select all that apply.Project ManagementInformation SystemsHuman Resources Management



Likert Scale Questions• Rating questions

• Degree of agreement, satisfaction, likelihood, & etc.

Please rate your agreement with the following statement: The instructor communicated the objectives effectively.o Strongly Disagreeo Disagreeo Neutralo Agreeo Strongly Agree



Baseline/Overall Rating Questions

• Most commonly are likert scale questions• Ask the “overall experience

What is your overall satisfaction with the training session.oVery DissatisfiedoDissatisfiedoNeutraloSatisfiedoVery Satisfied



Open-Ended Questions• Provides respondents an opportunity to provide some general

comments.

Please provide any additional comments you have regarding the training session.

• Can be used to ask why respondents selected a specific answer.

You have indicated dissatisfaction with the statement above, please elaborate on your experience.

Additional Resources: Common Survey Question Issues/Pitfalls


Double/Multi-Barreled Questions

Biased/Leading Questions

Categories not Mutually Exclusive

Categories are not Exhaustive

Double Negative Questions

Open-Ended Question that can be answered with yes/no



Double/Multi-Barreled Questions

Bad example: Please rate how satisfied you were with the training content, instructor and materials.

Good example: Please rate your satisfaction with the following: Training content The instructor(s) Training materials provided

Tips on how to avoid error• Look for ‘and’, ‘or’ in the question• Break down the question and test whether you could answer each piece separately



Biased/Leading QuestionsBad example: On average, a hands-on training format is rated the most preferred. What type of training format do you prefer?

o Hands-Ono Lecture o Online

Good example: What type of training format do you prefer the most?o Hands-Ono Lecture o Online

Tips on how to avoid error

• Avoid including opinions and supporting information• Keep it simple; Only include the question at hand



Categories not Mutually Exclusive

Bad Example: Approximately how many in classroom trainings do you attend per year?

o 0-1

o 1-3

o 3-5

o 5 or more

Good Example: Approximately how many in classroom trainings do you attend per year?

o 0-1

o 2-3

o 4-5

o 6 or more

Tips on how to avoid error • Look for response options which overlap or repeat • Identify questions where a single response could fall into more than one response option/category; separate options into

independent categories



Categories are not ExhaustiveBad Example: How many years have you worked at NIH?

o 1-4 o 6-8o 10-12

Good example: How many years have you worked at NIH?

o Less than 1 yearo 1 year – Less than 5 years o 5 years – Less than 8 years o 8 years – Less than 12 years o 12 years or more

Tips on how to avoid error • Look for gaps between options• Look for limits which may exist in the range of response• Look for a potential to include an “Other” option• When applicable, add a text box (“Please Specify”) if interested in gathering more detailed information



Double Negative QuestionsBad example: Please rate the following statement:

The instructor was not effective in communicating the objectives. o Strongly Disagree Strongly Agree

Good example: Please rate the following statement:

The instructor was effective in communicating the objectives. o Strongly Disagree Strongly Agree

Tips on how to avoid error • Look for ‘not’ in the question• Look for a contradiction; combination of ‘not’ and a negative response option



Open-Ended Question that can be answered with yes/no

Bad Example: Do you have any other comments on your training experience?

Good example: Please provide any additional comments you have regarding your training experience.

Tips on how to avoid error • Test to see if “yes” or “no” will satisfy the question• Consider starting the question with “What,” “Please describe,” or “Please provide…” to

make it open-ended

leveraging the power of surveys - office of human resources · 2019-11-20 · 4/3/2018 leveraging...

Documents