informal learning measurement john r. mattox, ii, ph.d. director of research knowledgeadvisors
TRANSCRIPT
Informal Learning Measurement
John R. Mattox, II, Ph.D.Director of ResearchKnowledgeAdvisors
Our Promise to You
April 10, 2023 2© 2011, KnowledgeAdvisors
KnowledgeAdvisorswill improve
learning effectivenesswithin your organization.
KnowledgeAdvisors
3
Experts in metrics that
matter
Experts in metrics that
matter KnowledgeAdvisors is a human capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions.
April 10, 2023 © 2010, KnowledgeAdvisors
Features and Benefits
April 10, 2023 4© 2011, KnowledgeAdvisors
KnowledgeAdvisors Clients
5April 10, 2023 © 2011, KnowledgeAdvisors
Human Capital Process Model
6April 10, 2023 © 2011, KnowledgeAdvisors
Informal Learning Measurement Approaches
7April 10, 2023 © 2010, KnowledgeAdvisors
What is “Informal Learning” Research on Informal Learning tactics Measurement in Informal Learning
Informal Learning
8April 10, 2023 © 2010, KnowledgeAdvisors
Informal Learning: Knowledge transfer that occurs without the assistance of structured curriculum.
Source: KnowledgeAdvisors
What Informal Learning is Supported?
9April 10, 2023 © 2010, KnowledgeAdvisors
73% 56% 54% 46% 13%
% of respondents supporting each type of informal learning
# of
res
pond
ents
Source: KnowledgeAdvisors Informal Learning Research, 2009
How Much is Spent on Informal Learning?
10April 10, 2023 © 2010, KnowledgeAdvisors
% supporting each method
$70K $85K $170K$110KAvg spend
Source: KnowledgeAdvisors Informal Learning Research, 2009
9% of L&D budget on Informal Learning
Informal Learning Measurement
11April 10, 2023 © 2010, KnowledgeAdvisors
Are your Execs asking for quantitative measures?
Are you measuring business benefits?
While only 21% of respondents indicated that their Executives are asking for quantitative measures, twice that
number expect to be asked soon.
Source: KnowledgeAdvisors Informal Learning Research, 2009
Measurement Correlates to Budget
12April 10, 2023 © 2010, KnowledgeAdvisors
Measurement discipline
L&D Spend / person
COPs M&C VKS EPSS
Those who measure $958 $168,000 $84,000 $250,000 $250,000
Those who don’t measure $1004 $68,929 $82,679 $97,917 $129,286
Overall $996 $71,042 $83,026 $109,615 $167,750 % delta ( over ) -5% 143% 6% 155% 93%
• The group measuring Informal Learning spends less per person on L&D annually.
• However, their Informal Learning spend is notably higher than the non-measurement group in most areas.
• We might conclude that measuring impact of Informal Learning enables organizations to demonstrate outcomes and justify greater allocation of L&D $ to these critical areas.
Source: KnowledgeAdvisors Informal Learning Research, 2009
Informal Learning Measurement : When?
13April 10, 2023 © 2010, KnowledgeAdvisors
1. Become aware /
consider
3. Engage
4. Use, learn,
practice, contribute
5. Complete and/or
recommit
At each interaction
Periodically
At the conclusion
2. Search, access,
sign in or join
Source: KnowledgeAdvisors Informal Learning Measurement White Paper, 2010
Informal Learning Measurement : How?
14April 10, 2023 © 2010, KnowledgeAdvisors
Systems BasedSystems Based•Web analytics•Supporting system statistics•Performance data
Tools BasedTools Based•Needs analysis•Usability studies•Feedback links•Content ratings•Surveys•Assessment
People BasedPeople Based•Observation•Interviews•Focus groups•Case studies
High Tech High TouchSource: KnowledgeAdvisors Informal Learning Measurement White Paper, 2010
Match Method to Timing
15April 10, 2023 © 2010, KnowledgeAdvisors
Method At interaction points
Periodically At conclusion
Web analytics X
Web feedback X
Supporting system data X X
Surveys X X
Content ratings (or single question)
X
Usability studies X
Interviews (1-1) X X
Focus groups X
Source: KnowledgeAdvisors
Informal Learning in Metrics that Matter®
16April 10, 2023 © 2009, KnowledgeAdvisors
• Leverage Micropolls and Pop-up surveys for point of interaction
• on-demand resources• performance support systemsc• virtual knowledge networks
• Poll as learners consume the content just in time• Utilize thought leading standard surveys for mentoring, performance support, on-demand resource, and community of practice programs• Report results by filtering by program, delivery method, and content
Metrics that Matter
Easy wizard for administration of informal learning measurement
Metrics that Matter
Micro Poll Experience
Metrics that Matter
Pop Up Experience
Metrics that Matter
Standard Report on Informal Learning
Metrics that Matter
Standard Report on Informal Learning
Take-away Information Informal learning is complex Measure successfully by embracing three factors
April 10, 2023 22© 2009, KnowledgeAdvisors
Program Component
s
Measurement Tools
Appropriate Timing
Take-away Information
Evaluating Informal Learning is not simple Multiple components
Web content, CoPs, Coaching Multiple measurement tools
Micropolls, pop-up surveys, end-of- program evaluations, formal survey
Many points in timepoint of interaction, periodically, end-
of- study
April 10, 2023 23© 2009, KnowledgeAdvisors
Recommendations
Define the scope of the informal learning that needs to be measured
Select the right data collection tools Determine appropriate timing Leverage available systems (e.g., MTM) Seek guidance (e.g., from
KnowledgeAdvisors)
April 10, 2023 24© 2009, KnowledgeAdvisors
Contact InformationFor further information on the information contact:
John R. Mattox, II, Ph.D.Director of Research+1 615 591 [email protected]