september 18, 2014 special topics webinars: school improvement margaret buckton, partner susie...

September 18, 2014 Special Topics Webinars: School Improvement

Margaret Buckton, PartnerSusie Olesen, School Improvement Enthusiast

© Iowa School Finance Information Services, 2014 1

Assessment and Data“Data. Data. Data. I can’t make bricks without clay!” Sir Arthur Conan Doyle

• Update us with your email address sean.gibson@isfis.net• PowerPoint on ISFIS web site at

http://sites.google.com/site/iowaschoolfinance/Home/webinar-recordings

• Power Point on Skills Iowa web site at http://www.skillsiowa.org/?q=PL

• Use question pane to pose questions • Ask questions. We will find the answer if we don’t know it today. If

we don’t answer during the Webinar, we’ll get back to you.• This series of 9 webinars pairs with ISFIS conference for one credit

hour (15 hours of content) for administrator license renewal• This series of 8 webinars pairs with 1 day at the ISFIS office on

September 10 for one hour of admin. license renewal

Webinar Reminders

Dates, Topics and Links to Register

• Thursday, July 17, 2014, 9 AM – What’s happening in school with students?• Thursday July 31, 2014, 9 AM –What’s happening with teachers?• Thursday, August 14, 2014, 9AM – What’s happening with school leaders?• Thursday, August 28, 2014 9 AM – Professional Development• Thursday, September 4, 2014 9 AM – Collaboration

• Thursday, September 18, 2014, 9 AM – Assessment and Data

• Thursday, October 2, 2014, 9 AM – TLC Model• Thursday, October 16, 2014, 9 AM – TLC Model• Thursday, October 30, 2014, 9 AM – What’s next in my school?

3

Using Webinar Information Later

• PPT, Recording and related tools posted on the Webinar Page and also the Skills Iowa professional leaning page: http://www.skillsiowa.org/?q=PL

• Itemized list of contents is searchable. Find what you need when you need it via Google search box

• Use PPT or information with leadership teams or with PLCs or data teams to get the conversation going

• Content for school improvement meetings• Or shoot us an email and we’ll send you what you

need.

4

Agenda

• Why assess?• Uses of Data in the School Improvement Process

– How are students performing? – What is going on in classrooms?– Set goals for student performance– Select content for instructional improvement– Monitor ongoing student learning (formative)– Monitor implementation of instructional improvement

(implementation)– Program Evaluation (Summative Data)

• Other Uses of Data Related to Improvement5

Quotes on data brought to you by:

Web Analytics Action Hero, 31 Essential Quotes on Analytics and Datahttp://www.analyticshero.com/2012/10/25/31-essential-quotes-on-analytics-and-data/

The Data Science of Digital Marketing, Spinnakr Bloghttp://spinnakr.com/blog/data-2/2013/03/44-more-of-the-best-data-quotes/

• “Facts do not cease to exist because they are ignored.” Aldous Huxley

• “Anything that is measured and watched improves.” – Bob Parsons

• “We are drowning in information and starving for knowledge.” – Rutherford D. Rogers

• “True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information.” – Winston Churchill

Why do schools spend time and money testing students? Put some of your thoughts in the chat pane. . . . .

Why Assess?

The Nation and the State:• Confirm that all kids have access to a great education (insure

equity)• To prove to taxpayers and constituents that schools are doing

the job . . .or• To prove to taxpayers and constituents that schools aren’t doing

the job• To justify expenditures and direct resources• To change behavior and beliefs (prove examples of what’s

possible)• To comply with mandated data collection requirements so state

can draw down federal $$“In God we trust, all others must bring data.” W. Edwards Deming

Why Assess?

Districts and Schools• To comply with state and federal law• To get information to change behavior

– Change instruction based on what kids know and can do– Determine what teachers need to learn more about– Avoid wasting time– focus teaching on unmastered skills and

challenge students– Confirm that all kids have access to a great education (insure

equity)– Insure student success

“The price of light is less than the cost of darkness.” Arthur C Nielsen

Required Assessments• Iowa Tests as part of Iowa’s NCLB plan (3rd-11th grade; math, reading,

science) proficiency at 41st Percentile. – Current plan does not report student growth – Iowa is unlikely to obtain federal waiver since student achievement (test scores) are

not part of Iowa’s teacher evaluation system. – Final recommendation of Council on Educator Development are due November 2016

• District must have a second assessment as part of the NCLB compliance plan (included in CSIP)

• NAEP – some districts are chosen to participate in some grades• All districts must have some assessment for an Early Warning System for

early elementary (could be FAST)• Teaching Strategies GOLD Early Childhood Assessment to PK students and a

valid and reliable universal screening instrument, as prescribed by the DE, to every kindergarten student by Oct. 1

“A point of view can be a dangerous luxury when substituted for insight and understanding.” Marshall McLuhan, Canadian Communications Professor

Quotable

• “…(A)nother important way to understand the different uses and impacts of assessment is to see the assessment system as a structure which both provides information and influences what people do.”

The Teaching and Learning Research Programme

• “….(F)ormative and summative are not labels for different types or forms of assessment but rather describe how assessments are used.”

The Teaching and Learning Research Programme

12http://www.tlrp.org/

More…• “We need to get the word out to the nation’s teachers

that formative assessment is capable of triggering big boosts in students’ achievement—the educational equivalent of a cure for the common cold.”

James Popham, Professor Emeritus, UCLA

• "We have to think about accountability in a very different way. We have done a splendid job of holding nine-year-olds accountable. Let me suggest as a moral principle that we dare not hold kids any more accountable than we expect to hold ourselves."

Doug Reeves, Founder, Leadership and Learning Center

13

Where are we?Student Performance

• How are our students doing in various, specific areas?– State Tests– Interim Assessments– Common Formative Assessments– Attendance– Graduation rates– Surveys– Engagement

“Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein, Physicist

What kind of classroom environment are our students experiencing?

• What are students doing? – Reading? – Writing?– Interacting?– Working in groups?– Working alone?– On computers doing what?

• What are teachers doing?– Interacting with students around content?– Asking questions? What kind?– Monitoring student work?– Modeling expected learning/behavior?– Email ?– Lecturing?– Working at their desks?

• How are our classrooms organized; what do we see?– Books and other text for reading? – Student work displayed? – Places to meet together? Places to get individual work done?

• What are we teaching?– How long for various subjects?– Interdisciplinary work? – Higher order thinking and more concrete factual content?

• How are we assessing?

Set goals for student performance:• What is our biggest area of need? Be as specific as possible

(reading comprehension, math problem solving, informational writing, etc.)

• Look at data that helps everyone understand what’s possible.

• Set long term and short term goals related to student performance.

• Focus. Don’t try to do it all at once.• Determine how you’ll measure the goals. Remember, the

state test alone isn’t enough. Need multiple measures.“Errors using inadequate data are much less than those using no data at all.” Charles Babbage

Select content for improving instruction.

• Be specific and focused about where you want to see student improvement (writing informational text, reading comprehension, math problem solving, science heuristics, etc.)

• Scan the literature about what kind of instruction can change student performance in the area you wish to improve. (Ask 3-6 experts in the area you wish to improve to recommend 3 or 4 good studies related to your area of improvement. See if there are common recommendations.)

• Seek technical assistance. Schools can rarely do this alone. (AEAs, consultants you know and trust, etc.)

• Determine what new instruction you want everyone to learn. Be sure to be clear about: – The moves of the new instruction– The frequency and fidelity of implementation required for student learning

results.• Embed learning in on-going PD system that ensures the support needed

for implementation with fidelity

Monitor Ongoing Student Learning

• Determine how you will regularly monitor student learning and respond instructionally– CFAs

• Student Writing

– Daily work• PWIM words

– Interim assessments• Skills Iowa principal

“Never confuse motion with action.” Benjamin Franklin

Monitor Instructional Initiative• Observations by other teachers, administrators, coaches –

expert and peer– Video– Rehearsals in collaborative teams– With students– Other teachers– Administrators

• Self reports like logs that help guide the work• Walk throughs• Listening – discussions of how it went, problems encountered,

how students responded“Without big data analytics, companies are blind and deaf, wandering out onto the Web like deer on a freeway.” Geoffrey Moore, Author of Crossing the Chasm & Inside the Tornado.

Program Evaluation

• Summative (pre-program benchmark and post-program measure)

• Need multiple measures• Determination isn’t always an affirmative yes or no –

could be “Those who implemented with fidelity saw significant gains. Need to determine how to elevate fidelity and frequency for all staff.”

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.” – Jim Barksdale, former Netscape CEO

• Where do we go next?

Do any of these processes align to a focus area in your school? Which processes do you want to be sure to include in your future work?

Principal Use of Teacher Effectiveness Measures for Talent Management Decisions

Gates Study at Vanderbilthttp://principaldatause.org/assets/files/presentations/Gates_Convening-Opening_Presentation-201405.pdf

Questions Study Examined• How do principals perceive the quality of data

systems and data access that help them make talent management decisions?

• How do principals use teacher effectiveness data for talent management decisions?

• How can school systems support data use for talent management decision making?

• What training and supports do principals value to help them learn to use teacher effectiveness data?

Who participated in this nationwide study?

• 100 + central office people interviewed• 78 principals interviewed• 795 principal surveys, 82% return rate• 4 in depth case studies

Findings• Observation systems drive principals’ use of data• Strong, ongoing calibration of observation scores seems to

increase use of observation data• Value-added measures perceived as having many shortcoming• Principals rarely use stakeholder feedback surveys or past

teacher evaluation measures

Problems Principals See with Value Added

• Timing (i.e., results are not available in real time when decisions are made);

• Perceptions of validity (i.e., scores do not exist for untested subjects, students are taught by multiple teachers);

• Specificity (i.e., the scores do not provide a window into what teachers actually do that directly impacts students’ learning, the measures are not fine-grained and actionable);

• Transparency and complexity (i.e., lack of understanding for both school leaders and teachers about complicated statistical models negates comfort with the results and their interpretation. Users need clarity in how the measures were created).

“If you can’t explain it simply, you don’t understand it well enough.” Albert Einstein, Physicist

UEN Response to Value Added Assessment Critique, April 24, 2013

Hiring

• Few principals use multiple teacher effectiveness measures for hiring

• Individual principals—even within a single school system—vary in their awareness of available data that could point them to quality teacher candidates or of data sources for hiring.

“I never guess. It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Sir Arthur Conan Doyle

Recommendations:

• Screen applicants with a rubric aligned to evaluation framework

• Provide principals with access to internal transfers’ effectiveness data

• Require a demonstration lesson and feedback aligned with observation rubric

• Maintain hiring information to compare with future outcomes

Data Use for Teacher Assignment• Some principals use data to place teachers where

they can be most effective. • Many do not view assignments strategically or

think of data as informative in the assignment process.

• Many also feel constrained in their ability to effectively assign teachers.

• Principals use data to move less effective teachers into grade levels that aren’t tested

“If you torture the data long enough, it will confess.” Ronald Coase, Economist

Data Use for Teacher Support and PD

• Observation data help principals to have constructive teacher feedback conversations.

• Principals rely almost entirely on observation data for professional development decisions.

• Principals draw upon both school- and system-level resources to support teachers.

Recommendations• Develop teacher professional development resources that

align with specific indicators on the instructional framework or rubric.

• Monitor teachers’ use of professional development resources, ask participants for feedback, and track professional growth over time.

• Train and support principals to have honest, clear, and sometimes difficult conversations about teacher performance.

“Once we know something, we find it hard to imagine what it was like not to know it.” Chip and Dan Heath, Authors of Made to Stick, Switch

Data Use for Termination• The late timing with which teacher effectiveness data become

available impedes principals’ use of these data for contract decisions.

• An inability to gather enough documentation and lack of time to do so are challenges.

• A lack of central/home office support is a major barrier to dismissal.

Recommendations:• Train principals on what constitutes effective

documentation, how to avoid potential pitfalls in the process, and where to find resources to help them sort through evidence as they create the file for dismissal.

• Support principals in gathering the appropriate documentation that will hold up to official grievance processes. Streamline principals’ efforts in gathering a body of evidence for teacher dismissal by providing direct support with the process.

• Train principals to use multiple years of teacher effectiveness data in making renewal decisions.

Principals want….

Overall Recommendations…

Sum it Up

• “What gets measured gets managed.” Peter Drucker

• “War is 90% information” Napoleon Bonaparte

• “He uses statistics as a drunken man uses lamp posts – for support rather than for illumination.” Andrew Lang, Scottish Writer

Any insights related to data in how your district supports principals?

Assignment Details for Recertification Credit

It’s likely that you use some or many of the data practices discussed in this Webinar today, and also probable that you don’t use them all. (Warning - that’s just our guess, as we have no data to back that up.) • Write a paragraph identifying data practices in your district

that are useful and how they move your improvement work forward.

• Then write another paragraph on data practices you heard about today that your district doesn’t use. Could they help move your district forward? Why or why not? 38

Finish with these:

“Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital.” Aaron Levenstein, Business Prof at Baruch College“The alchemists in their search for gold discovered many other things of greater value.” Arthur Schopenhauer, German Philosopher“The greatest value of a picture is when it forces us to notice what we never expected to see.” John Tukey, American Mathematician“An intelligent person is never afraid or ashamed to find errors in his understanding of things.” – Bryant H. McGill

Additional Resource

Without an effective system of curriculum and assessment, we are all a little like Alice.“Would you tell me, please, which way I ought to go from here?”“That depends a good deal on where you want to get to.”“I don’t much care where –”“Then it doesn’t matter which way you go.”

-Lewis Carroll, Alice in Wonderland

Access the Booster on Assessment on the publications tab of the Skills Iowa Web site here: http://www.skillsiowa.org/?q=publications

Questions or Comments?

Margaret Buckton , ISFIS – PartnerCell: 515-201-3755

margaret.buckton@isfis.net

41

Susie Olesen, ISFIS School Improvement EnthusiastCell: 641-745-5284susie.olesen@isfis.net

Iowa School Finance Information Services

1201 63rd StreetDes Moines, IA 50311Office: 515-251-5970

www.isfis.net