How to Design Skill Certifications Like a Pro

Liberty J. Munson, PhD.Principal PsychometricianMicrosoft Worldwide Learning

Brought to you by

2

Shreya Sarkar-Barney, PhD.

CEO & Founder

Liberty J. Munson, PhD.Principal Psychometrician

3

Leadership

Excellence

Talent

Management

Excellence

US India

We help organizations achieve better outcomes through talent using

science, analytics, and empathy.

Drive business impact

through timely talent

actions

Evidence-based solutions

for measurable and

sustained improvements

in leadership

The Zoom Web Conferencing Platform

Raise Hand to comment by

voice

Chat for Tech

support or to

comment

To test/adjust

computer

audio

4

If you require HCG

or HRCI

recertification

credits for your

participation, please

be sure to join with

your full name.

18.52

44.44

40.74

a. Yes, they are required to perform certain

roles

b. Certifications are preferred but not required

c. Certifications are not required for any of the

roles

33.33

25.93

7.41

37.04

a.      Employees are encouraged to earn

industry recognized external certifications

b.      We develop role/skill specific

certifications in-house

c.      We use external and internally

developed certifications

d.      We have no formal requirements

around certifications

29.63

25.93

11.11

37.04

a.      Employees become certified after

demonstrating they can perform their role

b.      Employees become certified after completing

required training

c.      Employees must pass a test to become

certified

d.      Employees must pass a test and demonstrate

their skills on the job to become certified

29.63

25.93

11.11

37.04

a.      Employees become certified after demonstrating

they can perform their role

b.      Employees become certified after completing

required training

c.      Employees must pass a test to become certified

d.      Employees must pass a test and demonstrate

their skills on the job to become certified

29.63

14.81

3.70

22.22

33.33

a.      Technical roles (e.g. programmers, IT, cyber

security)

b.      Management roles (e.g. Project management)

c.      Finance (e.g. risk management)

d.      Sales (e.g. Product knowledge, sales skills)

e.      Functional roles (e.g. HR, Legal, marketing)

25.93

22.22

22.22

33.33

a.      Options for developing a certification program

and the tools required

b.      The strategic value of assessments and

certifications in a learning culture

c.      The relationship between certifications and role

readiness

d.      How to build and maintain an assessment

program to promote a learning culture

Data Driven: • Participate in hacks to better

understand learner and test taker behaviors

• Evaluate the effectiveness of initiatives to improve learning

• Work with teams to better surveys that will results in clear recommendations

Learner Obsessed: • Ensure validity, reliability,

psychometric soundness of all skills validation processes (exams, assessments, etc.)

• Ensure that we are creating quality content

• Maintaining and sustaining cloud-based exams

Driving Impact:• Reduced the costs of exam

development by more than 50%• Reduced development time by 30%• Defined process to keep cloud-

based exams up to date• Regularly monitor exam quality

(psychometrics) and passing rates to quickly identify issues before they affect candidates, partners, and trainers

• Identified approaches to drive closer alignment between exams and training

• Ensure exams are valid and reliable drives overall value of certification program

• Thought leader in certification industry—driving changes in the way industry thinks about certification

Invest in Innovation:• MOOC to cert pilot• Adaptive testing/learning pilot• Investigating how AI, ML, and bots

could fundamentally change exams and certification

• Innovating to develop content more quickly to support cloud-based exams and internal needs

Simplify: Developing repeatable and scalable processes to minimize internal resources to develop and sustain exams

Overview of Microsoft’s Certification Program

Enabling the next generation of innovators, entrepreneurs, and developers

Enable Customers

& Partners at Scale

Learning Credits

Badges

Microsoft Professional Degree

Machine Learning

Microsoft Azure Fundamentals

Programming in C#

MOC On Demand

Creative Coding through Games and Apps

Harvard/MSFT CS50.AP

C# for Absolute Beginners

Hour of Code

Minecraft

Drive Usage, Deployment& Consumption

Empower Students& Educators

Activate Cloud &

Productivity Partners

2

5

Adoption

Drive Deep Engagement

Advocacy

Enable Capability Mastery

Affinity

Spark Excitement

Total reach

New/Competitive Developers reached

Skilled technologists building on Microsoft platforms and services

Preference for Microsoft platforms & services

1 3 4

Drive adoption of Microsoft technologies through training and certification

Similar paths for Office:

• Microsoft Office Specialist

• Microsoft Office Expert

• Microsoft Office Master

Microsoft Certification Structure

CLOUD PLATFORM & INFRASTRUCTUREMCSECloud Platform & InfrastructureEarned: 2017

MCSA Windows

Server 2012

MCSA Linux on

Azure

MCSA Windows

Server 2016

MCSA Cloud

Platform

Elective

Choose two from:

532: Developing Microsoft Azure Solutions

533: Managing Microsoft Azure Infrastructure Solutions

534: Architecting Microsoft Azure Solutions

473: Designing and Implementing Cloud Data Platform Solutions

475: Designing and Implementing Big Data Analytics Solutions

LFCS:

Linux Foundation

Certified System Administrator

533: Managing

Microsoft Azure

Infrastructure Solutions

741:

Networking with

Windows Server 2016

740:

Installation, Storage, and

Compute with

Windows Server 2016

411:

Administering

Windows Server 2012

410:

Installing and Configuring

Windows Server 2012

412:

Configuring Advanced

Windows Server 2012

Services

742:

Identity with

Windows Server 2016

Exam 1

Exam 2

MCSA

MCSAElective Exam

MCSE/D

MCSE/D

Different Elective or

Retake Exam

MCSE/D + Year

Renewed

• Certifications include achievement date

• Signifies investment in staying up to date on the technology

• Every year, they can re-earn their MCSE certification by passing an additional exam from the list of electives

• Option to retake exams where technology is rapidly changing either to renew their MCSE or to simply show currency in the technology

Renewing Certifications & Retaking Exams

An Overview of MS Certification Exams

Computer administered

Test center deliveries or online proctoring

Global distribution

Ongoing delivery

Variety of item types (e.g., multiple choice, drag

and drop, active screen, hot area, case studies,

labs, text entry, code analysis, etc.)

22

Development Process

Objective

DomainBlueprint

Item Development

Research

Candidate

Manager

Alpha Review &Item Revision

Field Testing (a.k.a. Beta Exam)

Forms Assembly

LocalizedExamStandard

Setting

Item Selection

Publish Exam

Ongoing Psychometrics

Focus on “application” level questions rather than “remembering” or

“understanding” level questions

Keys to Successful Evaluation

• Application of knowledge, skills, and abilities, not recall, understanding, etc.

• Present a scenario describing a business problem or situation that requires application of skills in ways that are not directly covered in the training

• Ask about a real-world application of a concept covered in the training

• Require the learner to:

• Analyze and provide a solution for a complex problem

• Combine several concepts, ideas, or skills to solve the problem

• Identify and analyze patterns based on a limited amount of information or data

• Design a solution by drawing on multiple skills learned during the training, and/or

• Evaluate outcomes that may result from actions taken

• Assess ability to apply knowledge to real-world problems, and/or

• Ask about concepts, actions, and outcomes that require extrapolating beyond what is specifically covered in the training or linking information from multiple sources

Key Attributes of Well Designed Measurement

Continental Congress

A. John Hanson

B. John Adams

C. Thomas Jefferson

D. George Washington

Stem = Question

Options = Key +

Distracters

Distracters =

Wrong Answers

Key = Correct

Answer

Stem

Format

• Open

• The year that John

Adams was elected

President:

• Closed (Preferred)

• In what year was John

Adams elected

President?

Best Practices

• Succinct –remove unwanted

language

• Relevant and important

• Non-trivial

• Stem is NOT teaching

• Avoid using “Not” and “Except”

• Avoid using definitions in stem

• Avoid two questions at once

Quality Check

• Can you cover the options and

answer the question?

• What is the capital of France?

• A. Lyon

• B. Paris

• C. Normandy

• D. Orlean

Key: Best Practices

• Don’t be different from

the other options such as

the longest and technical

• Don’t use words that are

in the stem and repeated

only in the key = cueing

Distractors: Best

Practices

• Incorrect

• Plausible

• Common

misconceptions

• No overlap

• All options begin with the same part of speech (e.g., gerund, infinitive)

• Example - A) Jumping B)Ran C)To flee D) marathons

Formatting

• Alpha-numerical

• Example - A. 5.0% *B. 25.0% C. 6.5% D. 3.0%

Ordering

• A) 10-30lbs B) 20-60lbs C) 55-95lbs

Overlap

Proper spelling, grammar and punctuation

• Don’t use you

• Example – If you want to distribute shares, the statue you would be most concerned with is:

Use third person

• Do NOT use: “always,” “never,” “only,” “appropriate,” “proper,” “typically,”

“usually,” and “might”

Avoid Specific Determiners

• Unless you are asking “best” answer questions with clear documentation

supporting what is “best”, do not use “best,” “most,” and “least” because

they are subjective

Avoid Specific Modifiers

Do NOT use All or None of the Above

• Spell out acronyms first, unless they are understood by the professional

community

Acronyms

• Write using the fewest number of words to answer the question

• Example – The best part of junk food is how good it tastes. According to the FDA, how

much sugar should be consumed daily?

Be Succinct

• Place all common wording in the stem that appears in all the options

• A. Health Care Food Act

• B. Dietary Act

• C. American Health Eating Act

• D. Anti Junk Food Act

Eliminate Repetition

• Avoid specific cultural, gender, ethnic, religious, or personal pronoun references

• Example – A Japanese female decides to calculate how many calories a Christian eats. She

needs to determine the square feet immediately after the 9/11 tragedy for the entire Word

Trade Center including the chapel and mosque? What is the square feet of the Word Trade

Center that she needs to calculate if the dimensions are 10 x 10 feet?

Ensure Test Fairness

How Can This Apply in Your Organization?

Microsoft’s Technical Credentialing Program

Creating a Learning Culture

What is it?

•Set of organizational values,

processes, and practices that

encourage employees to increase

knowledge & competence

Built on a growth mindset

•Belief that potential is not fixed, that failure is on the

path to mastery, willing to take risks

•See gaps in their knowledge as opportunities to

learn

•See learning as a journey

How?

•Establish clear links between

learning and performance

•Reward appropriate risk taking

•Ensure that learning is applied to

the job

•Make employees more

accountable for their own learning

path

Overview Ensuring these employees

build and validate their

technical knowledge and

skills on an ongoing basis

and in conjunction with

broad readiness goals

1

Implementing technical

credentialing requirements

on a consistent,

predictable, annual rhythm

2

Presenting role-based

requirements to the field

and their managers that

are clear and actionable

3

1.00

1.20

1.40

1.60

1.80

2.00

Very Effective Effective Somewhat

effective

Not at all

Effective

0

500

1000

1500

2000

2500

3000

3500

4000

Azure certification Dynamics

certification

Office and

productivity

certification

Windows and

mobility

certification

All MSFT FY17 target audience

• 5,200+ technical sales professionals required to meet technical credentialing requirement to demonstrate readiness

• 76% find certification useful to job effectiveness

Since launch of first Azure initiative in FY15 over 20,000 certifications achieved

Interesting: Those who selected exams are more likely to believe that the

tech requirement has improved job effectiveness

An ExampleTi

mel

ine

Q1 Q2 Q3 Q4 Q1

FY18 FY19May 31Sept 1

Key Changes

• Exam requirement replaced with a combination of

exams and courses

• Pass 1 exam or 3 courses to fulfill your requirement

• Additional options (new exams, courses, and activities)

may be added as they are made available

Same as FY17

• Where: Requirement appears in My Learning

• Who: Applicable to those in role by September 1

• Recommended for those who enter role after that date

• When: Complete by May 31

Role Business Intelligence TSP Collaboration TSP

Course

Options

Complete any three of relevant Open EdX courses

• 6 available to date - Excel, Power BI, O365

• Additional courses/activities TBA throughout FY18

Complete any three of relevant Open EdX courses

• 23 available to date - Exchange, SharePoint, O365

• Additional courses/activities TBA throughout FY18

Exam Options • Pass any one of the following exams:• 70-346: Managing Office 365 Identities and

Requirements*

• TBA Power BI exam (late FY18)• NOTE: Practice exams and prep resources will likely not

be available for some time after release!

• Pass any one of the following exams:• 70-339: Managing Microsoft SharePoint 2016

• 70-345: Designing and Deploying Microsoft

Exchange Server 2016

• 70-346: Managing Office 365 Identities and

Requirements*

• 70-347: Enabling Office 365 Services*

OR OR

Characteristic Assessment Certification

Primary purpose Provide instruction/training to aid in

the acquisition of

knowledge/skills/competencies

(learning through instruction)

Assess knowledge/skills/

competencies that have already

been acquired (validation through

testing)

Purpose and scope of evaluation

process

Evaluate accomplishment of

intended learning outcomes of a

specific education/training program

Confirm mastery of the

knowledge/skills/competencies

required to effectively perform a job

function or occupational/

professional role

Duration of program Ends when certificate is awarded Ongoing; requirements must be met

on a routine basis to maintain

credential (recertification)

Recognition Appears on transcript; may be

related to badging but no official

acronym follows learner’s name

Recipient uses an acronym or letters

after his/her name to highlight

certified status

Formative vs. Summative Assessment: Driving a Learning Culture

Formative assessment: Allows

the learner to attempt problems

multiple times, often providing

hints, so s/he can identify areas

of weakness for ongoing

learning

Summative assessment:

Evaluation if test taker has the

skills, learned the course

objectives, etc.

Combining Learning with Evaluation: Moving Beyond Certificate to Certification

Formative assessments built

in the course through

knowledge checks,

homework, etc. that allow for

multiple attempts

Summative assessment is end

of course exam requires

application of skills attained

during the course; it is MORE

than an evaluation of learning

objective attainment

Skill set 1 validated

Course & Mini Exam #1

Course & Mini Exam #2

Work Project #1

Skill set 2 validated

Course & Mini Exam #3

Course & Mini Exam #4

Work Project #2

The Certification Journey in Your Organization Could Be…

Wednesday, April 25th, 2018

48

Thank You!For more information contact

Shreya Sarkar-Barney, Ph.D.

49

www.linkedin.com/company/human-capital-growth

@hcgtm

info@humancapitalgrowth.com

707.317. 7644

www.Humancapitalgrowth.com

Follow us

shreya@humanpitalgrowth.com

@sarkarbarney

https://www.linkedin.com/in/sarkarbarney/