using impact assessment to improve learner engagement john sener dr. joan d. mcmahon presented at...
TRANSCRIPT
Using Impact Assessment to Improve Using Impact Assessment to Improve Learner EngagementLearner Engagement
John Sener
Dr. Joan D. McMahon
Presented at the NUTN Conference, Park City, UT, June 10, 2008
June 10, 2008 Slide # 2
Session Outcomes
This presentation will Show examples of successful impact assessment in action Demonstrate how to re-configure online learning evaluations to reflect
retention and impact for skill-based and/or professional course, Demonstrate methods for collecting impact data on student learning, and Demonstrate how these strategies improve learner engagement.
Participants will have multiple opportunities to
Share and discuss examples of how to re-configure assessment questions to assess the impact of courses on student learning.
June 10, 2008 Slide # 3
Traditional Assessments
What are the traditional assessments you or your colleagues use?
June 10, 2008 Slide # 4
Focus of Traditional Learning Assessment
Most traditional learning assessment is focused on ‘internally’-oriented outcomes:
Grades, Course Completion Rates KSAs = What did they learn (to do, think/feel)? Use of results to improve courses; compare results Oriented toward institutional needs; data collection is primarily
extractive
This is not bad, but it is limited and limiting
June 10, 2008 Slide # 5
Why Impact Assessment in 2020?
Learner-centered pedagogy (LCP) will be more prominent in 2020
LCP needs learner-centered assessment Impact assessment is a form of LCA
June 10, 2008 Slide # 6
What is Impact Assessment?
Results vs. Impact = ‘What happened’ vs. The “So- What” Factor: Influence Relevancy to the real world: Force exerted by a new idea, concept, ideology Can use Quantifiable and Qualifiable Statistics Often Action Research Based Focused on effects outside the institution (learner, environment,
world) Insight mining (Argyris) Reflective practice (Schon)
June 10, 2008 Slide # 7
Assessing Impact: Example Framework
Kirkpatrick ScaleL1: Reaction (learner satisfaction, ‘smiley tests’)L2: Learning (learning effectiveness, perceived/real)L3: Behavior = application of learning to environmentL4: Results = effects on environment resulting from
learner performance Last two approaches are forms of assessing impact,
but they tend to be employer-focused, not learner- or societal-focused
June 10, 2008 Slide # 8
Impact Assessment and Learner Engagement
Traditional Assessment/Evaluation does not engage learners…
But it can:Lambert (NUTN 2005)Runyon (CSM IT course)
…thus using impact assessment can be even more engaging (demonstrate by example)
Impact assessment is reflective
June 10, 2008 Slide # 9
Impact Indicators
Internal Frame of ReferencePersonal, Emotional, Social, Intellectual,
Spiritual
External Frame of ReferenceAchievement, Recognition, Work itselfResponsibility, Advancement, Growth
June 10, 2008 Slide # 10
How is Impact Data Used?
Course Evaluations Justification for online offerings Proof of Student Learning Proof of Skill/Competency attainment Proof of Compliance to Standards
June 10, 2008 Slide # 11
Re-Configuring Traditional Assessments
Learner Focused Discussions: What
questions can you ask to get to “impact?”
How can these new questions become the mainstreamed practice in 2020?
June 10, 2008 Slide # 12
Falling Short on Impact Data Collection
Proof of Student Learning cut short.Proof of Student Learning cut short.
We give examples but fail to ask “so what”?We give examples but fail to ask “so what”? We go on field trips and then dismiss the class.We go on field trips and then dismiss the class. We ask for reports on a topic but don’t ask “so what”?We ask for reports on a topic but don’t ask “so what”? We give team assignments without team training.We give team assignments without team training. We have discussions but don’t develop communities.We have discussions but don’t develop communities.
June 10, 2008 Slide # 13
Reframing the questions
Formative Course Evaluations
How am I helping you learn?What is hindering your learning?What course corrections would you
suggest?
June 10, 2008 Slide # 14
Mid-term Questions
What are you discovering about your own learning-to-learn process?
How do you anticipate this will affect your learning in the future? How are you different as a learner in this medium? How are you experiencing this process? How did you change as a learner through your involvement with this
course? How is using this process clarifying your electronic personality? How have you revealed a part of yourself that has not been
revealed in other settings?
June 10, 2008 Slide # 15
Examples of Impact Data Collection
Proof of Student LearningProof of Student Learning What was most useful to you in your learning process?
The format of being able to answer the questions[about the text material] at any time and that the answers would be posted for all to see was motivating. I spent more time reflecting on my answers before writing than in a classroom discussion.
Moreover, I was able to focus more clearly on my classmates answers to the discussion board questions….
June 10, 2008 Slide # 16
Examples of Impact Data Collection
Proof of Student LearningProof of Student Learning
(“How does online learning help you become a lifelong learner?”).
I learned that I much prefer to work on assignments in pieces...I found that keeping a log made the writing of the case study much easier, but, I also found that I have problems being precise in my writing.
June 10, 2008 Slide # 17
Examples of Impact Data Collection
Proof of Skill/Competency AttainmentProof of Skill/Competency Attainment(“How does online learning help you become a lifelong learner?”).
The online team effort at developing the team depot and the tools researched have been very useful to me professionally. The structure of looking for analysis, interventions and evaluation tools opened a new field of professional thought for me.
As with self-directed learning, online learning requires that one must have superb time-management skills, if one does not (like me) the course becomes exceptionally challenging.
June 10, 2008 Slide # 18
End of Course Questions
How well has the course met your learning goals? How well do you feel you performed overall? What was most useful to you in your learning process? What was least
useful? Did you achieve your learning objectives in this course? If yes, what did you
achieve? If no, what got in the way of achieving those objectives? Do you feel that what you learned in and through this course will have
application in other areas of your life? If so, where will you apply this knowledge?
How well did you participate in this course? Were you satisfied with your level and quality of participation?
Did you see yourself as an active member of the group? Did you contribute adequately to collaborative assignments?
June 10, 2008 Slide # 19
Reframing the questions
Summative Course Evaluations
Given the mandate to offer this course solely through distance learning (online and phone in this case), how could difficult issues be more clearly explained?
After reading the syllabus, directions for the assignments and the corresponding grading rubric, how would you improve the clarity of the course requirements and evaluation procedures?
Online teachers are usually available more often than on-ground ones. How could the instructor have been more available to answer questions?
June 10, 2008 Slide # 20
Your Turn: Reframing Questions to Elicit Impact
Reframe the Question…The instructor encouraged critical thinking skills. Did this class challenge you? The instructor presented difficult issues clearly.The faculty encouraged interaction among
students.
June 10, 2008 Slide # 21
Your Turn: Change a Response into an Impact Question
What is the question?
To a large degree, I am an aural learner so that I missed that element of interaction. I also respond better to f2f situations as opposed to the solitude of the on-line set-up so that maintaining motivation was an ongoing issue for me.
June 10, 2008 Slide # 22
Re-Configuring Traditional Assessments
Learner Reflections/ Evaluations
Regardless of how you felt about learning online, what has been the impact of this course in your work or personal life? What new skills have you used to affect change?
June 10, 2008 Slide # 23
Certifier
How will impact data support your role as “certifier” of knowledge, skills, or attitudes?
June 10, 2008 Slide # 24
Societal Impact- Chart on Student Projects
Student Projects
Structured the disciplinary process for Structured the disciplinary process for his Division of his Division of Coca-ColaCoca-Cola by synthesizing/charting the actions supervisors must take when addressing the discipline of their employees. (Until this time, there had been no uniform system administered consistently.)
Created resource guide for managers Created resource guide for managers dealing with addictive behaviors in dealing with addictive behaviors in the workplacethe workplace
Impact on the Organization
Has already been implemented “as the ‘new’ progression of discipline” at one of the Coca-Cola facilities he represents.
Student presented it to HR director of Holy Cross Hospital for incorporation in the hospital’s management training program on substance abuse scheduled for late Fall.
June 10, 2008 Slide # 25
Personal Impact- Chart on Student Projects
Student Projects
Learned to Delegate using better Time Management strategies
Improved Communication Skills via Assertion, Dealing with Conflict, and assessing organizational change
Impact on the Organization
Promoted
Left a dysfunctional organization and took a higher paying job with a better fit.
June 10, 2008 Slide # 26
Benefits to Professional and Departmental Portfolios
Documentation of tangible effectiveness evidence for promotion & tenure processLearning effectivenessROI/VOI:
Student Satisfaction: Referrals to course Proof for accreditation purposes
June 10, 2008 Slide # 27
Collecting Impact Data
What additional ideas do you have in collecting impact data in your situation?
June 10, 2008 Slide # 28
Moving toward Impact Assessment in 2020
Two scenarios: 1) Impact assessment is mainstreamed into ‘traditional’ practice 2) Impact assessment as LCA spreads via LCP
Copy = Moving old evaluation forms to online rather than re-thinking (impact) assessment
TL = I taught it but did they learn it (old). TLR = I taught it but did they learn it and retain it (movement). TLRI = I taught it but did they learn it, retain it, and so what? (impact).
June 10, 2008 Slide # 29
Impact of this presentation
June 10, 2008 Slide # 30
Contact Us
John Sener, Founder and Chief LearnerSener Learning [email protected]
Joan D. McMahon, Professor EmeritusTowson University and CEO, Transforming Leaders, [email protected]