stages of concern during curriculum change: formative evaluation india broyles edd and mildred...
TRANSCRIPT
Stages of Concern During Curriculum Change:
Formative Evaluation
India Broyles EdD and Mildred Savidge PhD
University of New England, College of Osteopathic Medicine
11 Hills Beach Rd, Biddeford, ME 04005
What is the Difference in Formative and Summative Evaluation?
"When the cook tastes the soup, that’s formative.
When the guests taste the soup, that’s summative."
http://www.designedtoat.com/food2.shtml
The workshop will allow participants to Value formative evaluation during change Complete a sample of the SoC questionnaire Adjust the questionnaire to meet their own
innovation or curriculum change Score the questionnaire and to make
preliminary analyses of different types of profiles
Predict ways in which SoC data can facilitate the change process
Workshop ObjectivesLearning Outcomes
Agenda
• 10 minutes – Formative Evaluation of the change process
• 20 minutes – Complete the SoC questionnaire with ‘competency-based education’ as the innovation. Score and report.
• 15 minutes – Understanding the stages of concern and the SoCQ
• 15 minutes – What innovations are you implementing? Rewrite the questions. Exchange with a partner to review/edit the items. Questions?
• 20 minutes – Analysis of the data • 10 minutes – What does the college do with the
formative evaluation data?
Formative evaluation can serve as a guide for a college in the change process and improves the chances of a successful curriculum innovation that will be both well accepted and long lasting.
Formative EvaluationFormative Evaluation
Formative Evaluation
• Formative evaluation is a method of judging the worth of a program while the program activities are forming or happening.
• Formative evaluation focuses on the process.
– (Bhola 1990).
Formative Evaluation typically involves:
• gathering information during the early stages of your project or program
• a focus on finding out whether your efforts are unfolding as planned
• uncovering any obstacles, barriers or unexpected opportunities that may have emerged
• identifying mid-course adjustments and corrections which can help insure the success of your work (NW Regional Education Lab)
How do we do this?
The process of developing and implementing an innovation is
Change
So we look at the literature on educational change….
Individual Innovativeness TheoryCarl Rogers (1995)
• Innovators • Early Adopters • Early Majority • Late Majority • Laggards
Change is an ongoing process, not a short-term
event.
“Change requires ongoing support and resources and it takes time. Sometimes new users of an innovation get discouraged when they don't see immediate results. It is important to have realistic expectations about the time it will take to see significant progress and to make sure that other stakeholders in the community understand this as well. Failure to address key aspects of the change process can either add years to, or even prevent, successful implementation.”
Gene Hall, Shirley Hord, Archie George. Southwest Educational Development Lab
Change occurs in individuals first, then in organizations.
The best curriculum in the world will not succeed in your institution unless the people implementing it are ready and willing to make it a success. However, individual change is difficult if the organization is not supportive of the change.
Gene Hall, Shirley Hord, Archie George. Southwest Educational Development Lab
One Change in Medical Schools…
Competency-based Education with emphasis on national core competencies.
Please complete the questionnaire. Takes about 10 minutes. Score using quick-scoring guide. Create your profile. You will have 20 minutes total. Give the quick scoring guide to Millie to record. Be sure your name is on it.
Agenda
• 10 minutes – Formative Evaluation of the change process
• 20 minutes – Complete the SoC questionnaire with ‘competency-based education’ as the innovation. Score and report.
• 15 minutes – Understanding the stages of concern and the SoCQ
• 15 minutes – What innovations are you implementing? Rewrite the questions. Exchange with a partner to review/edit the items. Questions
• 20 minutes – Analysis of the data • 10 minutes – What does the college do with the
formative evaluation data?
What are Concerns?
• The world around us is complex. It is not humanly possible to focus at one one time on all of the many different issues. Certain aspects of our world are of higher priority.
• The composite representation of the feelings, preoccupation, thought and consideration to a particular issue or task is called a concern.
Concerns
• Depending on one’s closeness to and involvement with an innovation, one’s concerns will be different in type as well as in intensity.
• Many types or levels of concerns can be experienced concurrently.
• There is a predictable pattern to the movement of intensity of concern across types.
Based on studies at UTEXAS, Austin since 1974
• Stages have been identified with a developmental movement through these stages.
• Earlier concerns must be resolved before later concerns emerge.
• Timely provision of affective experiences and cognitive resources can facilitate resolution of early concerns and development of higher concerns.
• Personalized interventions are important.
Stage 0 -- Awareness. Users have little concern or involvement with the innovation.
Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners.
Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.
Stages of Concern
Concerns-Based Adoption ModelGene Hall, Shirley Hord, Archie George. Southwest Educational Development Lab
• Stages of Concern– Questionnaire– Interviews
• Levels of Use– Interviews
• Innovation Configuration
SoCQ Instrument
• Reliability– Coefficients of internal reliability– Test/Retest correlations
• Validity– Statistical – Content– Face
Coefficients of Internal ReliabilityN = 830
Stage 0 1 2 3 4 5 6
.64 .78 .83 .75 .76 .82 .71
Multiple studies: Coefficients of Internal Reliability for each stage
Authors Sample Size
0 1 2 3 4 5 6
Hall, George, & Rutherford
830 .64 .78 .83 .75 .76 .82 .71
Van Den Berg & Vandenberghe
1585 .77 .79 .86 .80 .84 .80 .76
Kolb 718 .75 .87 .72 .84 .79 .81 .82
Barucky 614 .60 .74 .81 .79 .81 .79 .72
Jordon-Marsh 214 .50 .78 .77 .82 .77 .81 .65
Martin 388 .78 .78 .73 .65 .71 .83 .76
Hall, Newlove, Rutherford, & Hall
750 .63 .86 .65 .73 .74 .79 .81
Test-Retest Correlations N=132
Stage 0 1 2 3 4 5 6
Pearson-R
.65 .86 .82 .81 .76 .84 .71
Validity Studies: Statistical
• Factor analysis – 7 stages explained 60% of variance– All had Eigen values above 1.0– Hypothesized scales matched factor scales– Each scale measured an independent construct
• Correlations between scale scores shows strong positive relationship between contiguous stages
Factor Analysis
SoC
StageVarimax Factor Scores
7 1 6 3 4 2 5
0 .83 -.36 .41 .04 .05 -.04 -.09
1 .46 .67 -.40 -.10 .22 -.35 .01
2 -.14 .49 .72 .36 .04 -.14 .26
3 .10 -.04 -.34 .91 .10 . 12 -.12
4 -.14 -.19 .00 .12 .96 -.02 -.07
5 .10 .37 .11 -.11 .11 .82 -.34
6 .16 -.05 -.17 -.02 07 .40 .88
Correlations between Scale Scores
Stages
Stages
2 3 4 5 6
1 .68 .47 .21 .21 .19
2 .78 .43 .37 .43
3 .45 .51 .59
4 .82 .80
5 .77
Validity: Comparison Studies
• Focus on accuracy in measuring highest and lowest concerns
• Interviews and open-ended responses used– Statements matched to scale scores/
profiles– Qualitative analysis showing greatest and
least concerns correlated with scale scores– External judges used to establish
relationships between interview analysis and scores
Validity: Longitudinal Studies
• Two 2-year longitudinal studies• Showed concerns changing over time
in accordance with concerns theory• Concerns measured by the
instrument moved from high Awareness, Informational and Personal concerns to lower concerns on all levels, or higher Consequence, Collaboration and Refocusing concerns
Use in clinical education
• Lewis and Watson (1997) measured the concerns of 57 nursing faculty about the use of computer technology. Their pre-post study results suggest that the primary concerns of the faculty were informational and that addressing these concerns through workshops increased interest in the innovation.
• Gwele (1996) measured the concerns of nurse educators (n=93) at four nursing colleges during the implementation of a major legislated curriculum reform. Concluded that when staff is required to adopt a major curriculum change the normal progression through the stages of concern is impeded. They suggested that in these situations it may be important to delay adoption until participants can come to terms with the need to adopt the new curriculum.
• Arwer, Harris & Dusold (2004) assessed the concerns of staff during the installation of a telemedicine system and to assure that concerns were addressed during system implementation. Survey findings were used successfully to modify the implementation and training phases of the program to better meet the needs of the staff.
Components
• Cover Letter / Introductory Page• 35 items• Demographics
Create a SoCQ for an innovation in your own institution. (15 minutes)
Interpretation of SoC Data
Look at your scores/profile on CBE…
• Peak Stage Score• 1st and 2nd High Stage Score• Profile Interpretation
Peak Stage Scores
Group Data
• Tally the number of individuals that are high on each stage. This gives a clear picture of the range of peak stage scores within a group (show of hands - how many at each stage).
• Aggregate individual data by developing a profile that presents the mean scores of each stage of the individuals.
Typical Non-User
0
10
20
30
40
50
60
70
80
AwarenesssInformational
Personal
ManagementConsequencesCollaborationRefocusing
Institutional Profile of Stages of Concern
Institutional Profile
0
10
20
30
40
50
60
70
80
90
100
0 1 2 3 4 5 6
Stages of Concern
Percentile
Typical Non-User Profile
0
10
20
30
40
50
60
70
80
90
100
0 1 2 3 4 5 6
Stages of Concern
Percentile
Profile Interpretation
• As individuals move from unawareness and nonuse of an innovation into beginning use and more highly sophisticated use, it is hypothesized that their concerns develop from early to late concerns
• Use clinical interpretation techniques• The total score is not generally used in
analyses because it does not have a unique meaning.
Institutional Profile, 2006
Stages of ConcernInstitutional Profile
0
10
20
30
40
50
60
70
80
90
100
Awareness Information
Personal
ManagementConsequenceCollaborationRefocusing
0 1 2 3 4 5 6
%ile
0102030405060708090
100
AwarenessInformation
Personal
ManagementConsequenceCollaborationRefocusing
0 1 2 3 4 5 6
Negative One –Two Split
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Negative One -Two Split with Tailing Up at Stage 6
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Stage 0 -- Awareness. Users have little concern or involvement with the innovation.
Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners.
Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.
Stages of Concern
Intense Management Concerns
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Consequence Concerns
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
High Collaboration and Consequence Concerns
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Single High Collaboration Concerns
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Stage 0 -- Awareness. Users have little concern or involvement with the innovation.
Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners.
Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.
Stages of Concern
High Refocusing Concerns
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
High Management Concerns With Ideas
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Stage 0 -- Awareness. Users have little concern or involvement with the innovation.
Stage 1 -- Informational. Users have a general interest in the innovation and would like to know more about it. Stage 2 -- Personal. Users want to learn about the personal ramifications of the innovation. They question how the innovation will affect them. Stage 3 -- Management. Users learn the processes and tasks of the innovation. They focus on information and resources. Stage 4 -- Consequence. Users focus on the innovation's impact on learners.
Stage 5 -- Collaboration. Users cooperate with other users in implementing the innovation. Stage 6 -- Refocusing. Users consider the benefits of the innovation and think of additional alternatives that might work even better.
Stages of Concern
Profile of Impact-Concerned User and Coordinator
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Unconcerned Innovation User
0
20
40
60
80
100
AwarenesssInformational
PersonalManagementConsequencesCollaboration
Refocusing
Guidelines
• Establish a holistic perspective• Look at High and Low scores• Look at individual item responses
• When scores are used in statistical analyses, developers recommend use of raw scores. Conversion to percentiles greatly affects the distribution of the scores.
Demographics
• In previous research, no outstanding relationships between standard demographics; age, gender, teaching experiences, etc.
• What demographics would you choose?
Responses from 8 basic scientists show high concerns for awareness, but very different concerns for other stages. All but two have tailing up at Stage 6.
Basic Scientist Profiles
Clinicians show more similar profiles at Awareness, Information, and Personal Concerns. Five clinicians have a high concern at refocusing. Although unusual for non-users, we are still in the development of several elements.
Clinical Faculty Profiles
Similar to clinicians in early concerns, as expected in non-users. Low management and consequence concerns may reflect non-classroom duties. The have strong concerns about collaboration.
Staff Profiles
STAGES OF
CONCERN
I MAY BE STUCK IF I AM SAYING…
I’M READY TO CHANGE AND AM
FOCUSED ON…
Awareness Everything is fine, so I am not interested. I am not concerned about it.
What is it? (re-active)
Information I don’t want to do it. I would like to know more about it. How does it work?
Personal I can’t do all that! How will using it affect me?
What’s my role in it?
Management I will try it, but I am not a believer. I spend all my time getting materials ready.
How can I master this? How can I fit it all in? What’s the minimum I must do?
Consequences I am not concerned that it is worth it.
Is it worth it? How is my use affecting learners? What’s the maximum potential of this?
Collaboration I have my own way of doing things.
How do others do this? How can I relate what I am doing to what others are doing?
Refocusing Everything is fine. Is there a better way? (pro-active). How can I refine it to have more impact? I have some ideas about something that would work even better.
Follow-up Interviews
Themes from interviews
Stage Stage 0 & 1 Awareness and Informational
Stage 2 Personal
Stage 3Management
Themes
Information Flow
Driving Force
Us versus Them
Lack of Attention
Curriculum Elements
Impact
•Don’t know
•Little Impact
•Strong Impact–Curriculum Elements
–Resistance
–Faculty Development
–Personal Time
Resources
Financial
Personnel
Faculty & Student Time
Assessment & Evaluation
Faculty & Staff Development
Leadership Issues
What we learned…
• The process of curriculum revisions needs to be systematic with timelines and responsibilities.
• Everyone needs to be kept informed of the process and the outcomes of deliberations with information coming in multiple formats – a website– hallway bulletin boards– written material– key descriptive and research articles.
• Care should be taken to release information in a logical fashion so that faculty and staff not directly involved in the development stages can understand the reasons for current plans and have an opportunity to respond if they are unhappy with plans at any given stage.
Additional resources will be needed…
• qualified assistance with effective use of WebCT™
• grant-writing support
• faculty coaching
• implementation of effective evaluation approaches
Understanding the stages of concern can result
• in more targeted strategies
• more relevant workshops
• directed planning to implement the new curriculum plan thereby creating successful, institutionalized change
We believe…
The importance of setting up a curriculum revision process based on both individual and institutional concerns has the potential for influencing the development and implementation of a new curriculum.