wrapping the gift? the role of the school library and evaluating the school library program
Post on 03-Jul-2015
264 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Wrapping the Giftbull The Role of the School Library
bull Evaluating the School Library
Program
LIB 620 Library ManagementFall 2014
2
What are school libraries for
The Mission of the School
Library
ndash The school library provides
information and ideas that are
fundamental to functioning
successfully in todayrsquos information and
knowledge-based society The school
library equips students with life-long learning
skills and develops the imagination enabling
them to live as responsible citizensndash IFLAUNESCO School Library Manifesto
2
3
AASL Mission
The mission of the school library
media program is to ensure that the
students and staff are effective users
of ideas and information The school
library media specialist empowers
students to be critical thinkers
enthusiastic readers skillful
researchers and ethical users of
information
ndash Empowering Learners (2009)
3
4
What about school librarians4
Roles and Responsibilities
of the School Library Media Specialist
ndash Leader
ndash Instructional Partner
ndash Information Specialist
ndash Teacher
ndash Program Administrator
5
More on the role of the LMS
Guiding principles
ndash Principle One School libraries have no
boundaries
ndash Principle Two Library and information
professionals should be flexible
ndash Principle Three Ensure that students are
effective users of ideas and information
ndash Principle Four Information is everywhere
essential and central bull Lowe C A The Role of the School Library Media
Specialist in the 21st Century ERIC Digest 2000-11-00
5
6
Another way to look at it6
7
School Librarian
Leadership
7
Library Media Early
Childhood Through
Young Adulthood
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
2
What are school libraries for
The Mission of the School
Library
ndash The school library provides
information and ideas that are
fundamental to functioning
successfully in todayrsquos information and
knowledge-based society The school
library equips students with life-long learning
skills and develops the imagination enabling
them to live as responsible citizensndash IFLAUNESCO School Library Manifesto
2
3
AASL Mission
The mission of the school library
media program is to ensure that the
students and staff are effective users
of ideas and information The school
library media specialist empowers
students to be critical thinkers
enthusiastic readers skillful
researchers and ethical users of
information
ndash Empowering Learners (2009)
3
4
What about school librarians4
Roles and Responsibilities
of the School Library Media Specialist
ndash Leader
ndash Instructional Partner
ndash Information Specialist
ndash Teacher
ndash Program Administrator
5
More on the role of the LMS
Guiding principles
ndash Principle One School libraries have no
boundaries
ndash Principle Two Library and information
professionals should be flexible
ndash Principle Three Ensure that students are
effective users of ideas and information
ndash Principle Four Information is everywhere
essential and central bull Lowe C A The Role of the School Library Media
Specialist in the 21st Century ERIC Digest 2000-11-00
5
6
Another way to look at it6
7
School Librarian
Leadership
7
Library Media Early
Childhood Through
Young Adulthood
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
3
AASL Mission
The mission of the school library
media program is to ensure that the
students and staff are effective users
of ideas and information The school
library media specialist empowers
students to be critical thinkers
enthusiastic readers skillful
researchers and ethical users of
information
ndash Empowering Learners (2009)
3
4
What about school librarians4
Roles and Responsibilities
of the School Library Media Specialist
ndash Leader
ndash Instructional Partner
ndash Information Specialist
ndash Teacher
ndash Program Administrator
5
More on the role of the LMS
Guiding principles
ndash Principle One School libraries have no
boundaries
ndash Principle Two Library and information
professionals should be flexible
ndash Principle Three Ensure that students are
effective users of ideas and information
ndash Principle Four Information is everywhere
essential and central bull Lowe C A The Role of the School Library Media
Specialist in the 21st Century ERIC Digest 2000-11-00
5
6
Another way to look at it6
7
School Librarian
Leadership
7
Library Media Early
Childhood Through
Young Adulthood
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
4
What about school librarians4
Roles and Responsibilities
of the School Library Media Specialist
ndash Leader
ndash Instructional Partner
ndash Information Specialist
ndash Teacher
ndash Program Administrator
5
More on the role of the LMS
Guiding principles
ndash Principle One School libraries have no
boundaries
ndash Principle Two Library and information
professionals should be flexible
ndash Principle Three Ensure that students are
effective users of ideas and information
ndash Principle Four Information is everywhere
essential and central bull Lowe C A The Role of the School Library Media
Specialist in the 21st Century ERIC Digest 2000-11-00
5
6
Another way to look at it6
7
School Librarian
Leadership
7
Library Media Early
Childhood Through
Young Adulthood
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
5
More on the role of the LMS
Guiding principles
ndash Principle One School libraries have no
boundaries
ndash Principle Two Library and information
professionals should be flexible
ndash Principle Three Ensure that students are
effective users of ideas and information
ndash Principle Four Information is everywhere
essential and central bull Lowe C A The Role of the School Library Media
Specialist in the 21st Century ERIC Digest 2000-11-00
5
6
Another way to look at it6
7
School Librarian
Leadership
7
Library Media Early
Childhood Through
Young Adulthood
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
6
Another way to look at it6
7
School Librarian
Leadership
7
Library Media Early
Childhood Through
Young Adulthood
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
7
School Librarian
Leadership
7
Library Media Early
Childhood Through
Young Adulthood
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
8
Leadership
Are there many
types of leaders
ndash Appointed Leaders
ndash Expert Leaders
ndash Interpersonal Leaders
ndash SocialInformal Leaders
bull Are you a ldquoborn leaderrdquo If not you can develop
skills to help you become an effective leader
Just as you become a better teacher through
experience youll become more comfortable as a
leader over time
bull The Teacher Librarian as Manager
8
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
99
Beyond Proficiency your libraryreg is a web-
based tool to guide Kentucky school media
librarians school administrators and SBDM
councils in creating and maintaining
effective library media programs for the 21st
century
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
10
Beyond Proficiency your
libraryreg supporting documents
ndash Beyond Proficiency your library Webcast
ndash Library Media Program Rubric
ndash Sample Evaluation Tool for School Media Librarians
Characteristics of Highly Effective Teaching and
Learning in the Library
10
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
11
Evaluation of School Library Programs
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
12
Preemptive Action
The busy librarianrsquos guide to
program evaluation
Based in part on
A workshop developed by the
Pennsylvania School Librarians Association
Professional Development Committee
(No longer available online)
indicates the slide was originally from the above presentation
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
13
httpevaluationtoolboxnetauindexphpoption=com_contentampview=articleampid=11ampItemid=17
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
14
What about school library program evaluation
Evaluation of
school library media centers
ndash When you examine something or
someone you examine the subject and
make a judgment about the quality
significance or condition of whatever is to
be evaluatedbull Emanuel T Prostano and Joyce S Prostano
The School Library Media Center 5th ed (Englewood CO
Libraries Unlimited 1999) 44
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
15
Purposes of evaluation
To find out what is right
To find out what is wrong
bull Summary of definition in Blanche Woolls The School
Library Media Manager 2nd ed (Westport CT Libraries
Unlimited 1999)
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
16
What can you evaluate
Everythingndash Collection
ndash Personnel
ndash Library Layout
ndash Programs
ndash Curriculum collaboration
ndash Reference servicecollection
bull Interview skills
bull Quality of the answers provided
bull Satisfaction of your patrons
ndash Students or teachersmdashor other school staff
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
17
What should you evaluate
That depends
ndash On your priorities
ndash On your economic needs and situation
ndash On the policies and procedures established by
your school school district andor state
ndash On how much time you have left over from your
other duties to devote to planning and executing
an evaluation project
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
18
Why evaluate
Because you have to
ndash Continuous assessment of services can be an ongoing requirement for accreditationfunding etc
Because you want to
ndash You want to find out how well yoursquore doing
ndash You want to find out how well others think yoursquore doing
ndash You want to find ways to improve your service
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
19
Why Evaluate
bull Schools are being evaluated by student
academic achievement in reading
writing and math
bull Recent studies show student
achievement correlates positively with
effective school library programs
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
20
Why Evaluate
Studies show the positive correlation with
student achievement occurs when there is an
effective school library program [and]
ndash School librarians are full time and
ndash Librarians collaborate with teachers on
instructional planning
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
21
Why Evaluate
Studies also show an effective school library program includes the following attributes
ndash Quality collections
ndash Increased hours of access beyond school day
ndash Professional development for teachers and librarians
ndash Student access to technology and
ndash Collaboration with other types of libraries
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
22
Why Evaluate Because
To improve the librarian must
have baseline data about the program
To be relevant the librarian must
know if the library services and
resources are aligned with the schoolrsquos
goals for student achievement
To build a case for better support the
librarian needs data that demonstrates
the value of the program
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
23
An evaluation enables you to
ndash Determine success in attaining program goals
ndash Determine studentsrsquo and teachersrsquo needs so they can be incorporated into the program
ndash Provide a basis for resource allocation
ndash Recognize strengths and accomplishments
ndash Examine the impact of the program on student learning
What Can an Evaluation
Do
bull Nancy Everhart Evaluating the School Library Media Center (1998)
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
24
It makes sensemdash
Focus data collecting and
evaluation on those components of a
school library program that the
research shows correlate with student
achievement
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
25
Types of Evaluation
Formal
Informal
External
Internal
Formative
During a program
Summative
After the program is completed
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
26
Evaluation requires standards
How do you know if the school library is ldquogoodrdquo ldquonot goodrdquo or even ldquogood enoughrdquo
ndash ldquoGood enoughrdquo for what For whom
Guidelines or rubrics or objectives for collections performance etc can be found in
ndash
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
27
Types of Data
Quantitative Qualitative
Number of OPAC
searches
Size of collection
Number of lessons planned with teachers
Studentsrsquo success rate in
locating appropriate
resources in OPAC searches
Collection supports the curriculum
Comprehensive and collaborative planning is in place
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
28
Standards require interpretation
Quantitative relatively easy
ndash
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
29
Qualitative more difficult ndash
Standards require interpretation
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
30
Interpretation means
operationalization
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
31
Interpretation means creating goals and
objectives
The purpose and study goals should
determine the types of methods and measures
you use to conduct the evaluation
Objectives will define your standard of
excellence--the minimum level of appropriate
service for your particular clientele
ndash Jo Bell Whitlach Evaluating Reference Services
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
32
Methods of Evaluation
Obtrusive
ndash People are aware of the evaluation
bull Self-evaluation
bull Surveys
bull Observation
Unobtrusive
ndash data collection that does not require intrusion into
the lives of participants by investigators
bull Glossary for Responsible Conduct in Data Management
ndash People are unaware of the evaluation
bull Unobtrusive measures of physical facilities
bull Use of proxiesmdashldquomystery patronsrdquo
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
33Obtrusive reference evaluations in a school
library context
Questionnaires or
interviews of students or teachers
Numbers gathering
ndash Reference question counts
ndash Numberstypes of reference books used
ndash Circulation statistics
Observation
ndash By external observer
ndash Self-observation Journal andor portfolio
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
34
Evaluation Model
1 Define the question
2 Collect data--determine needed data
and method of collection
3 Analyze the data
4 Formulate recommendations
5 Develop an action plan
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
35
Research Finding
Students whose school
librarian plays an
instructional role
tend to achieve higher
than average scores
What data are
needed to
determine whether
or not the school
librarian plays an
instructional role
in the school
The Question
Evaluation Model Step 1
Define the Question
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
36
Look at the rubric
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
37
Empowering Learners
Building collaborative partnerships
ndash Guideline
bull The school library media program
promotes collaboration among members of
the learning community and encourages
learners to be independent lifelong users
and producers of ideas and information
ndash Empowering learners p 20
37
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
38
Addressing multiple literacies
ndash Guideline
bull The school library media program
provides instruction that addresses
multiple literacies including
information literacy media literacy
visual literacy and technology literacy
ndash p 23
According to Empowering Learners
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
39
Determine the type needed
ndash QuantitativeQualitative
Determine the method of collection
ndash Existing statistic
ndash Survey questionnaire focus group observation
Evaluation Model Step 2
Collect data
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
40
ndash Identify areas of agreement and disagreement
among groups
ndash Identify areas of strength and areas that need
improvement
ndash Identify areas where more data is needed
How can the data be analyzed to determine
whether or not the librarian plays an effective
instructional role in the school
Evaluation Model Step 3
Analyze the data
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
41
Agreement Example--
Target Indicator Information Literacy Standards are integrated into content learning
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
12 Basic 88 Proficient ____ Exemplary
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
42
Disagreement Example--
Target Indicator
Curriculum development is modeled and
promoted
Library Staff Response
100 Basic _____ Proficient ____ ExemplaryAdministrative Response
____Basic 100 Proficient ____ ExemplaryTeaching Staff Response
82Basic 18 Proficient ____ Exemplary
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
43
Strength Area Example--
Target Indicator
Collaborative planning is
modeled and promoted
Library Staff Response
____Basic 100 Proficient ____ Exemplary
Administrative Response
____Basic 100 Proficient ____ Exemplary
Teaching Staff Response
____Basic 100 Proficient ____ Exemplary
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
44
Weakness Area Example--
Target Indicator
Students are engaged in reading writing
speaking viewing amp listening for enjoyment
enrichment amp understanding
Library Staff Response
____Basic 100 Proficient ____ ExemplaryAdministrative Response
100 Basic ____ Proficient ____ ExemplaryTeaching Staff Response
82 Basic 18 Proficient ____ Exemplary
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
45
Contradictory Data Examplemdash
Target IndicatorEffective teaching modeled amp promoted
Library Basic ____ Prof 100 Exemplary
Admin ____ Basic ____ Prof 100 Exemplary
Teacher Basic 60 Prof 40 Exemplary
Target Indicator Student achievement is assessed
Library 100 Basic Prof Exemplary
Admin 100 Basic ____ Prof Exemplary
Teacher 90 Basic 10 Prof Exemplary
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
46
Back to Step 2
Collecting More Data ndashFocus Groups
1 How are student products and performances
assessed in units involving use of the library
2 How do you determine the products and
performances to demonstrate mastery of content
in units involving use of the library
3 What is the role of reflection in student work
involving use of the library
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
47
bull Review the data formulate recommendations
bull For example ndash Because of the strong correlation between the
school librarianrsquos instructional role and student achievement the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric
Role of an Advisory Committee of stakeholdersmdash
Evaluation Model Step 4
Analyze the data
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
48Example
Recommendations for First Year
ndash Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis evaluation and inquiry
ndash Meet with teaching teams to solicit recommended revisions
ndash Convene a workshop for new teachers to review the standards
Standards for the 21st Century Learner
Adapted from
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
49Example
Recommendations for First Year
Collaborative Planning
ndash Establish regular common planning time for
teachers with the library media staff
Curriculum Development
ndash Review school policies to remove any barriers
that prevent librarians from participating in
building and district curriculum sessions
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
50Example
Recommendations for First Year
Reading Writing Speaking Viewing
ndash Provide a series of author workshops and develop plans with teachers for student reading writing speaking and viewing responses to the author workshops
ndash Involve Parents
Effective Teachingndash Clarify conflicting feedback through focus groups
targeting use of assessment differentiation amp inquiry in instruction
ndash Develop recommendations based on new data
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
51
Library staff develop an action plan for each
recommendation
The Advisory Committee of Stakeholders
reviews and approves the action plans
Evaluation Model Step 5
Develop an actionplan
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
52
Action Plan Example
Target Indicator
Curriculum development is modeled and promoted
Objective Activity Documentation Participants New
Resources
Completion
Date
Review
barriers
preventing
librarians
from
participating
in curriculum
sessions
Review
policies
School Council
minutes
School
Council amp
admin
None
projected
Sept 1
Secure
schedule
Post schedule in
lib amp faculty
rm
Principal None Sept 10
Participat
e in mtgs
Meeting
minutes
Librarians None
projected
June 1
Report
back
Faculty mtg
minutes
Librarians None Monthly
Written
material
In professional
library
Librarians None Within 5
days of mtg
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
53
This example of an evaluation
washellip
bull Type (which one)
ndash Formal
ndash Internal
ndash Formative
bull Data collection methods
ndash Research
ndash Rubric-based
questionnaire
ndash Focus Group
bull Data type
ndash Quantitative
ndash Qualitative
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
54
Unobtrusive evaluation in schools 1
Unobtrusive measures of physical facilities
ndash ldquoThe basic premise is that you can learn a
great deal by looking at how things wear
(lsquoerosionrsquo) how things are left in the building
(lsquotracesrsquo) and how things are rearranged
(lsquoadaptations for usersquo)rdquo
bull Nancy Everhart Evaluating the School Library
Media Center Analysis Techniques and
Research Practices Libraries Unlimited 1998
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
55
Unobtrusive evaluation in schools 2
Use of proxiesmdashldquomystery patronsrdquo
ndash Technique used more often in academic libraries and government documents reference services
bull Half-right reference the 55 rule
bull 5-minute rule
ndash Peter Hernon and Charles R McClure ldquoUnobtrusive Reference Testing The 55 Percent Rulerdquo Library JournalApril 15 1986 37-41
bull ldquoItrsquos not true and now we know why the so-called ldquo55 rulerdquo has never been tested against a truly representative field samplerdquo
ndash John V Richardson Jr ldquoReference Is Better Than We Thoughtrdquo Library Journal April 15 2002 41-42
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
56
Importance of a sophisticated model
Richardsonndash ldquoThe reference service performance model [that led to
the 55 rule] was overly simplistic samples were way too small and the test questions were not representative of real-world reference questionsrdquo
ndash ldquoStudents of reference service should learn about the existence of multiple performance outcomes (ie accuracy utility and satisfaction) and to recognize that each outcome is driven by different factorsrdquo
bull ldquoReference Is Better Than We Thoughtrdquo
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
57
Characteristics of good models
Measures of reference service must bendash Valid
bull They ldquoaccurately reflect the concept being studiedrdquo
ndash Reliablebull They ldquoare stable and dependable and provide
consistent results with each repeated userdquo
ndash Practicalbull They ldquorequire that data be relatively easy to collectrdquo
ndash Usefulbull They ldquoprovide information that can be used to
improve reference servicesrdquondash Whitlach Evaluating Reference Services
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
58
Another model
A 21st-Century Approach to
School Librarian Evaluation
ndash Workbook style prompts walk
school librarians through
suggested readings action tips
and evidence collection to help gauge
their current levels of achievement set goals
for progress and form plans for future
professional development
ndash In this archived webinar writer Patricia
Owen walks participants through the
workbookrsquos suggested readings action tips
and evidence collection
58
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
59
A model piloting in KY schools
Professional Growth
and Effectiveness System
ndash The vision for the Professional Growth
and Effectiveness System (PGES) is to
have every student taught by an effective
teacher and every school led by an effective
principal The goal is to create a fair and
equitable system to measure teacher and
leader effectiveness and act as a catalyst for
professional growth
bull PGES Background
bull TPGES for Teachers PPGES for
Principals SPGES for Superintendents
59
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
60
What about other professionals
Other Professionals Professional Growth
and Effectiveness System
ndash Other Professional Categories
bull School counselors ndash may be in summative
year
bull School level librarymedia specialists
bull School psychologists
bull School instructional specialistcurriculum
coaches
bull School speech therapists
ndash OPGES Pilot Overview
60
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
61
Problems of Evaluation in School Context
Often the school librarian is the only one
working in the library
An evaluation project can be time-
consuming to plan and to put into action
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
62
Solutions to evaluation problems
Depends on your local situation
ndash Your resources
bull Time money available warm bodies
ndash You and your creativity
Make evaluation part of your routine
ndash As you planprepare your program(s)
include an evaluation component
bull Appropriate affordable accessible
accountable
Seek collaborative partners whenever
possible
63
httpwwwanimationfactorycom
63
63
httpwwwanimationfactorycom
63
top related