Data Coaching Network A Collaboration Between The Office of Superintendent of Public Instruction & The Association of Education Service Districts Year Two Report 2013 Submitted by Sue Feldman, Education Service District 112 Additional edits by Sue Furth and Tara Richerson, OSPI

1

Report Summary

Data coaching is an example of how well the network of nine Education Service Districts

(ESD) can work collaboratively with the Office of Superintendent of Public Instruction

(OSPI) to provide training and new program implementation equitably throughout the

state. Each ESD participated in the data coaching institutes in year one and year two, which

resulted in interesting and unique approaches to data coaching in each ESD region. Given

the same training, each ESD team developed a unique and regionally relevant approach to

data coaching. Coming together to learn with and from each other was valued by each ESD

and provided necessary support data coaching implementation.

Data coaching is a good step toward improving data practices throughout the state. Each

ESD is providing some form of data coaching, primarily by integrating data coaching into

other ESD-funded services. This is a slow approach to meeting the actual needs of school

districts. If education is going to take advantage of the potential in data use, there continues

to be a serious need for data practice supports in most school districts. ESDs may be the

most efficient, effective, and economical way to provide these supports. Education is lagging

behind in the big data movement. While education generates local, state, and national data,

the system of education in Washington State has established few resources to use these

data. It is a rare school district with the software infrastructure to display and analyze data

and an even rarer school district with the analytic expertise to provide accurate, clear,

timely, and relevant analysis of data for decision making at the classroom, school, or school

district level. ESDs continue to be the best location for data coaches because ESDs can and

do provide economical and equitable access to education services. ESDs will need two key

resources to increase their data use support to school districts. First, ESDs and school

districts will need to establish data-sharing agreements. Second, ESDs will need funding to

hire quantitative expertise to work alongside data coaches who are currently prepared to

facilitate data practices, but are generally not prepared to conduct sophisticated analyses.

2

Current State of Data Use in Washington State’s System of Education Data use in education has become an established expectation while it continues to be an

unclear set of practices (Goren, 2012). In Washington State, teachers are expected to use

data for instructional purposes and school principals are expected to observe teachers’ uses

of data as a part of the new teacher and principal evaluation system. Student growth data is

also to be used as a part of the teacher and principal evaluation process. While these

expectations are in policy, educators are awaiting the implementation of a reliable and valid

assessment system that generates growth data, aligned to grade level expectations (Ingram,

Louis, & Schroeder, 2004; Supovitz & Klein, 2003).

After more than ten years of investments in testing apparatus and data infrastructure, it

remains a rare educator in Washington State who can develop a reliable or valid classroom

assessment, conduct an item analysis on test questions, or run an inferential analysis to find

a Z-score to compare test scores across unlike tests. While the lack of data tools and

technical expertise present a challenging situation for professional development, educators

in general are still drawn toward the promise of data; unfortunately, the Legislature has not

yet allocated resources to develop the data expertise necessary to realize this promise

(e.g.,Coburn & Talbert, 2006; Datnow & Park, 2009; Knapp, Copland, Swinnerton, &

Monpas-Huber, 2006).

In an effort to improve data use practices of teachers, school and school district leaders,

OSPI and the Washington School Information Processing Cooperative (WSIPC) garnered a

commitment from each of the nine Education Service Districts’ Assistant Superintendents

for Teaching and Learning to send an inter-program team to 12 days of data coaching

institutes. The institutes took place over a two-year period. ESD teams met for two days at a

time with trainers from Public Consulting Group (PCG) to support a network of data

coaching initiative and a common set of data coaching protocols for the network of nine

ESDs. A research initiative was implemented, in conjunction with the data coaching

institute, to document and analyze the process of the data coaching initiative. As a new

program intending to change practices, the data coaching initiative provided a good

opportunity to study program implementation and changes in statewide practice, in real

time. This is the report of the year two study of that effort.

3

Study Design Data coaching, sponsored originally by OSPI and WSIPC, in partnership with the network of

nine ESDs, moved into its second year of implementation in 2012–13. Data coaching

continued to expand its collaborative reach to include multiple departments at OSPI, school

districts, and other education-focused organizations including special education

cooperatives, early childhood education programs, and GATE: Graduation: A Team Effort.

Data coaching began with the development of a set of protocols to support school district

leadership teams to increase and improve their data practices. There is strong theoretical

support for the concept of data coaching, even if there is not a dedicated revenue stream to

support it (e.g, Halverson, Grigg, Prichett, Thomas, & Wisconsin-Madison, 2006; Kowalski,

Lasley, & Mahoney, 2008; Marsh, Pane, & Hamilton, 2006; Means, Padilla, DeBarger, &

Bakia, 2009).

Year Two Research Questions:

1. How, if at all, does data coaching support collaboration within and between OSPI,

ESDs, and school districts?

2. How, if at all, does studying the implementation of data coaching support the

implementation of data coaching?

Year one research questions will continue to be a focus of the study:

1. How, if at all, did the data coaching institute shape the work of the ESDs in year two

of implementation?

2. What, if anything, was the tangible outcome of ESDs participating in three data

coaching institutes during the 2011–2012 school year?

3. To what extent are ESDs ready to pursue data coaching as a service to their regions?

4. What more do ESD participants need in order to provide data coaching services in

their regions?

5. What are the supports and constraints on ESD resources for data coaching?

4

Study Purpose. This study was primarily designed to inform the implementation process. It

documented the second year of data coaching implementation and explored the emergent

data coaching activity within and between ESDs, school districts, departments at OSPI, and

WSIPC to understand how people participated in and developed capacity for data coaching

in Washington State during the 2012–2013 school year.

Study Method. This mixed methods (Cohen, Manion, & Morrison, 2007; Lodico, Spaulding,

& Voegtle, 2006) study used survey data collected after the final data coaching institute in

the spring of 2013, field observation data, interview data collected in semi-structured brief

interviews with data institute participants and project leaders, and documentary evidence

related to the project formation and implementation.

Data Collection. Interview data and documentary data were collected throughout the year.

Survey monkey was used to collect survey data. The first year survey produced better than

70 percent response rates. The response rate of actual data coaches was close to 100

percent. The second year survey had better than an 87 percent return. Additional data will

be collected throughout year two including data coaching institute field notes and

observations, document analysis and interviews with ESD data coaching team members,

school district data coaching participants, and OSPI leaders involved with data coaching.

Data Analysis. Field-notes and interview data and the qualitative survey responses were

analyzed (Miles, Huberman, & Saldana, 2013) using Nvivo software. A coding schema was

developed, based on the research questions, and free coding was used to capture themes

that emerged through the analysis process. Survey data were analyzed using descriptive

statistics to examine emerging trends in data coaching activity, influences of data coaching

on school district-based practices, and influences on data practices at OSPI and WSIPC.

5

Findings and Discussion Following the practices from the first year of the data coaching institutes, monthly phone

calls between OSPI leaders, interested ESD data coaches, and PCG facilitators were used to

monitor implementation and plan the data coaching institutes. The emerging interest of

data coaches seemed to be improving their data analysis skills. Therefore, the data coaching

institutes invited a variety of data experts, who work with different administrative data

sets, theoretically available to data coaches, to present to the data coaching teams. The

presentations were designed to introduce data coaches to the data sets and to analytic

practices that might be useful for making sense of the data. Here are three examples:

1) During the second data coaching institute, Paul McCold, the data analyst with the

state Office of Migrant and Bilingual Education, introduced data coaches to the data

sets he used to analyze the progress bilingual students were making in the

Washington State school system. He also taught data coaches how to conduct an

elaboration using inferential statistics.

2) The team working on the new longitudinal student data system for OSPI introduced

their data system and showed data coaches the types of questions that could be

answered by the system, which included teacher salaries and school district

revenues, making it easy to access two data sets that had previously been difficult to

locate.

3) At the school and district level, a reading specialist from one of the ESDs presented a

process she had invented to determine how each student in a school district was

meeting reading standards. Using formative and summative assessments and

aligning each assessment to a small set of standards and curriculum materials, she

demonstrated how to generate formative assessments for instructional decision-

making.

These three presentations, and multiple other presentations, were designed to improve

coaches’ data analysis skills; yet there is a long way still to go to prepare the data coaches in

the use of inferential statistics. The data analysis conducted through most of the data

coaching continues to be basic descriptive statistics. Survey results indicate ESDs are not

yet ready to offer data analysis using inferential statistics. Using inferential statistics was

the lowest rated service among all the data coaching work ESDs might do. Interviews

corroborated that all ESD data coaches continue to want more sophistication in the data

6

analysis skills. Data coaches are interested and willing to participate in training on inferential

statistics if it is done in the context of their districts’ data and the questions their districts are

trying to answer.

More coaching than data. The majority of the data coaches came into data coaching

confident and competent in the facilitation of socio-cultural practices. The data coaching

tool kit, developed during year one of the data coaching initiative, focused primarily on the

socio-cultural features of transforming district leadership teams into data leadership teams.

For example, the tool kit includes protocols to guide a conversation about what data the

team has and uses and for what purposes; protocol for facilitating a conversation about how

to increase the data sources used for decision making; facilitating data reviews; and

facilitating data informed decision making. Data coaches are comfortable leading these

processes that focus primarily on facilitation of productive interactions between people on

leadership teams. Data coaches are less confident in their ability to access and analyze

district data. While the data coaching institute did not expect competence in data analysis as

an outcome, by the end of the second year, the data coaches reported wanting to be more

competent in data analysis.

ESD data coaches want to respond to their districts’ needs for actual data analysis support.

Except in rare districts, schools and school districts do not have data analyst positions or

people with those skills and time to do data analysis. While there is a growing expectation

that teachers and administrators analyze data, there are actually few districts with the

staffing capacity to do this with rigor or consistency. ESD data coaches may be a good

solution to increasing districts’ data analysis capacity, but it will require hiring actual data

analysts at ESDs to work with the data coaches or training the data coaches as data analysts,

which was not the focus of the data coaching institutes.

Direct-coaching & certification. Data coaching certification was introduced during the

first year of data coaching as the focus of year two and the culminating event of the

institutes. Certification was a part of the original work plan between OSPI, WSIPC, and PCG,

but the data coaches questioned what the certification meant and the overall purpose or

value of the certification. A common perspective among data coaches was that certification

was superfluous. By virtue of being chosen to represent their ESDs at the data coaching

7

institute, they were already recognized as competent in the work and further certification

was not necessary. Given this questioning of the value and purpose of certification, PCG and

the lead OSPI facilitator of the data coaching institutes refocused the certification

framework not as a pass or fail process but instead as a tool to focus the direct-coaching

conversations.

Also in the original scope of PCG work, for year two, were two site visits made to each ESD.

These site visits were preceded by a pre-visit planning phone call. These site visits all took

place, although the original direct coaching was altered because no ESD data coaching team

implemented data coaching with a district leadership team using the data coaching tool kit as

designed. That is not to say that there was no data coaching conducted with district

leadership teams, but the majority of data coaching was conducted at the school level rather

than the district level. The majority of data coaching, in year two, did not focus on

developing a data leadership team, or transforming a leadership team into a data

management team; instead, data coaching was one activity that a leadership team may have

facilitated along with many other activities. This being the case, it was challenging for the

PCG coaches to provide direct coaching and it was challenging for the ESD data coaching

teams to use the certification framework. What the direct coaching did reveal was the

unique implementation of data coaching in each ESD. Each ESD team had a clear and

coherent plan and data projects they were leading. Each was uniquely constructed to utilize

the strengths and interests of the team members and leverage funding opportunities to

include data coaching, which was unfunded.

Data, data systems and data displays. Data coaching began as an initiative between OSPI

and the Washington Student Information Processing Cooperative (WSIPC). As WSIPC serves

as the most prevalent student data management system in the state, they were facing the

challenge of transforming the data system from a transactional system—designed primarily

for reporting data upstream—into a data system that could be easily used for decision

making at all levels of the system. Additionally, OSPI’s focus was the implementation and

sustainability of several large statewide initiatives including, but not limited to, Teacher and

Principal Evaluation (TPEP), Common Core State Standards (CCSS), the State Longitudinal

Data System (SLDS), and the Student Growth Model. It may be counter-intuitive that

teachers and principals have a hard time accessing student data, since student data

8

generally originates in the classroom and at the school level. The structure of the data

system, concerns about student privacy, and the lack of time allocated in the school day for

teachers, principals, and even district office directors to manage, sort, analyze, and use the

data put into the data system meant that the data were underutilized. Data sets improve

with use, and without constant use, the data were not high quality. Data were missing, and

of the data that were in the system, some of it was inaccurate. Inaccuracy is normal in data

sets that are not critically analyzed.

In a transactional data system the emphasis is on data input for upstream reporting. The

system in Washington State was fairly tight in terms of keeping the data moving up stream.

For example, many of the data coaches, who worked down the hall from the student

coordinator, were unfamiliar with the Skyward data system’s reporting capacity. It was not

unusual for data coaches to remark that they could not get access to the data for the school

they were supposed to be coaching. This issue of data access was not only a problem for

data coaches, it was a problem for teachers, principals, and district office directors

concerned with using data to make decisions. District-based data warehouse systems were

not the only data management systems that educators found difficult to access. The state

report card data is easy to view within a school or district view but limited for comparing

across schools and districts. State report card data was restricted to the school level so it

was impossible to analyze across third graders, for example. There were Excel spreadsheets

available for quick down loads of state report card data if an educator was inclined to work

in Excel or SPSS and conduct his/her own analyses. Even the data coaches were not

confident in their ability to conduct their own inferential statistics. The Query System was

another data system that most data coaches found difficult to access. It requires permission

from the district data coordinator, and while some data coaches were granted permission,

others did not get as far as identifying the right person in the district office to grant

permission. The data that were the most protected and often the most controversial were

the free and reduced-price lunch data. Some districts do not tell their teachers who qualifies

for free or reduced-price lunch and other districts did share those data if requested.

Complicating the data access and data use situation further, whether educators were

granted permission to have free and reduced-price lunch data or not, they tended to make

inferences about who did or did not qualify. In some cases, they made decisions based on

their impressions of students. The extent to which a district opens or closes their data

9

access to educators was a feature of the data environment that data coaches continuously

navigated with greater and less success. It is possible to make data requests to OSPI and at

least one data coach did request research data. The first request was fulfilled after six

months and the second request was fulfilled after three months and constant follow-up

requests urging the data office to prioritize the request. Transactional data systems were a

big step in the right direction in the early 2000s, but like all student information systems, by

the time they are debugged and ready to launch, they are outdated. By the time people learn

to use the systems they are even further behind. This is a problem not just in Washington

State but across the country (J. Wayman, 2003; J. C. Wayman, Brewer, & Stringfield, 2009; J.

Wayman & Chu, 2009; J. Wayman & Stringfield, 2006). For example, HOMEROOM is a data

display system designed primarily for teachers’ use as a response to the need to make a

commonly used transactional data system interactional. It was intended to replace the

conditionally formatted Excel spreadsheets that were emerging all over the state, developed

by teachers lucky to be trained in Excel. These teachers seemed to be few and far between

and rather than train the entire teaching cohort to use Excel, an entrepreneurial software

engineer developed a software option for teachers to view their student data at their

desktop. School Data Solutions (SDS), like most software developers, worked with a user

group from across the state to inform the development of the tool. There were no data

coaches in the development group, but data coaches took an active interest in the

development and potential implementation of HOMEROOM, keeping track of its

development and the implementation plan throughout 2012–13 school year.

As the SDS system HOMEROOM implementation began, new users immediately identified

new needs for the software, and new glitches in the use of the software. Some school

districts were using a variety of software for data display. For example, even within OSPI,

the School Success program was purchasing Data Director for the school success schools;

the Student Longitudinal Data System was being developed. The Education Research Center

was developing a web-based data display for education data, as was the Center for Research

on Education Data at UW Bothel. In some school districts, one school might be using

HOMEROOM, while the School Success program required them to use Data Director and

Indistar and they may have also been using SWISS for behavior data. None of these systems

work together. None of these data systems worked seamlessly and all of them required

more training than had first been expected, leading to these products being underutilized.

10

Additionally, as has been the case with much education software, about the time it is ready

for implementation, it has already been made obsolete by the early adopters who want it to

do more and better (Feldman & Tung, 2001; Goertz, Olah, & Riggan, 2010; Kerr, Marsh,

Ikemoto, Darilek, & Barney, 2006; Talbert, Scharff, & Lin, 2008). The promise of data

systems may be realized in the coming years as the developers continue to respond to the

users’ needs and the data coaches continue to support teachers learning to analyze student

data for instructional decision making.

Survey Results. The survey was conducted after the close of the final data coaching

institute. It was sent to all the ESD-based data coaches who participated in the year two

data coaching institutes, which made it a survey of the whole population of data coaches.

There was a 90 percent response rate on the survey. The full year two survey results are

included as an appendix of this report. Below is a summary and discussion of some of the

results.

Demographics. The majority of data coaches worked as directors or assistant

superintendents at ESDs. Each data coaching team included staff from a variety of programs

within their ESD, making these data coaching teams uniquely inter-program work groups.

Fifty-five (55) percent of the data coaches participated in at least eight days of training. And

only 10 percent of the data coaches participated in less than three days of training. The only

distinct difference in survey responses between the data coaches with more than eight days

of training and the data coaches with three or fewer days of training was that the data

coaches with less training were more sure, than their peers with more training, that their

ESDs were ready to provide all the data coaching services.

Discussion of team make-up. Due to program funding structures and tight funding

expectations, it is unusual for program staff to have the opportunity to work with staff from

outside of their area. This initiative was an opportunity for Educational Service District

(ESD) staff to work on an inter-program project. For example, the ESD112 data coaching

team had members who were student data coordinators from the Communication &

Information Management (CIM), special education, early childhood, prevention

intervention, research and assessment, and STEM. These inter-program teams were

constructed by design, although in some ESDs they also emerged by default. Given how busy

11

ESD staff are, team members may have been chosen because they had the flexibility to

participate, more than the inclination.

Data coaches rated the data coaching institutes as average or above average learning

experiences. The year one survey indicated that the majority of data coaches believed that

the data coaching institute did not provide new information or tools, but affirmed what they

already knew and already did. There also seemed to be further evidence of this tendency to

absorb data coaching into what the data coach already knew and did in year two. For

example, the most common data-coaching venue across ESDs, in year two, was with School

Success schools. The actual data practices reported tended to be the required/expected data

practices of the School Success program, which many of the data coaches had been

facilitating for multiple years before they were trained as data coaches. While there are

some similarities between the data practices in the School Success program and the data

practices in the data coaching tool kit including an inquiry cycle, problem definition, root

cause analysis, and action planning, the data coaching tool kit was intended to be used for

the establishment of data leadership/management teams which is a different focus than

school success. The research literature on implementing new practices in schools concurs

suggesting it is more likely for new practices to be absorbed into current practice, than to

change practice or replace current practice (J. P. Spillane, 2002; J. P. Spillane & L.K., 2002; J.

Spillane & Stein, 2005).

Data practices in schools and school districts. Just over half of the data coaches indicated

that it was uncommon for schools and school districts to ignore student achievement data.

All data coaches reported seeing some evidence of data coaching spreading in their regions,

but the majority of the data coaches (79 percent) also indicated schools and school district

data use focused primarily on students’ annual achievement test scores, which offer limited

use. Forty (40) percent of the data coaches reported that, while some school districts are

searching for data sources beyond student achievement data, only 21 percent of data

coaches reported that it was common across most school districts in their regions to search

for additional data sources for decision making.

Given the challenges generating, collecting, accessing, displaying, and analyzing data, it is

surprising that schools and school districts have not hired data expertise. Only 18 percent of

12

the data coaches reported schools or school districts hiring data expertise. The majority of

data coaches reported they had no knowledge of schools or school districts creating and

hiring a data expert position. It does seem reasonable that ESDs might be able to provide

data expertise for districts and schools given the challenges and the constraints that schools

and school districts face improving their data use.

ESD readiness to provide services. The majority of ESD team members reported, from the

start of this initiative, that they were ready to provide data coaching services through their

ESDs. Ninety (90) percent of the data coaches reported being ready to offer or already

providing analysis of student achievement data for a school and 76 percent were ready to

provide it for a whole school district. Over 83percent of the coaches reported being ready to

facilitate data analysis with teachers, principals, or central office administrators. That said,

73 percent of the data coaches reported that they were not yet ready to offer data analysis

using inferential statistics. The student achievement data analysis that ESDs are prepared to

provide falls short of inferential statistics. Descriptive statistics may be the norm and

considered adequate for students achievement test score analysis, but without knowing

how to do inferential statistics, it may be hard for the data coaches to assess what they are

missing. Seventy-six (76) percent of the data coaches reported being ready or already

offering training on the data coaching tool kit. Sixty six (66) percent of the data coaches

reported being ready or already offering planning for data coaching. Fifty-nine (59) percent

of the data coaches reported being ready or already offering support for developing

classroom assessment that show student growth. Overall, at least one data coach in each

ESD was ready or already offering all the data practices including generating data, collecting

data, displaying data, analyzing data, and facilitating decision making using data with

teachers, principals, or central office administrators. The data coaches are ready, and 47

percent of the data coaches reported that school districts were requesting some form of

data support from the ESDs, creating at least some possibility for data coaching to continue

to spread, even if it is not funded.

Data coaches indicated that the protocols in the data coaching tool kit were the same or

similar to protocols they already used. For example, the inquiry cycle at the center of data

coaching is similar to the cycle for school improvement planning, school success, and the

new teacher and principal evaluation process. The only service ESDs are already offering or

13

ready to provide is data analysis using inferential statistics. This service was not expected to

be an outcome of the data coaching institutes. The data coaching institute trained coaches to

facilitate meetings in which the district leadership teams were the experts on their own

data. The coaches were not expected to bring data, find data, or use the district data. Instead

the coaches were expected to facilitate the transition of district leadership teams to data-

focused leadership or data management teams. This situates data coaches as facilitators of

the social-cultural practices of leadership teams rather than data analysts or data

specialists.

Collaboration between ESD data coaching teams. The survey asked, to pursue data

coaching as a service to your districts, what supports would you want. The most commonly

reported support was to continue to meet with the other ESD teams to share emerging

practices. Of the data coaches who indicated further training, the training they want is in

data analysis skills.

14

Data Coaching Highlights From Each ESD During the data coaching institutes, each ESD presented on the data coaching work they had

accomplished between sessions. Unlike year one, when it was common for ESDs to report

no work on data coaching between institutes, in year two all the ESDs conducted some work

related to data coaching. It is worth highlighting at least one accomplishment of each ESD to

illustrate how each ESD took the data coaching training and tool kit and produced

something uniquely appropriate to their locally situated context. The work below highlights

only a small fraction of the varied data coaching work that emerged in each ESD during the

2012–13 school year1.

ESD 101, Walking our Talk. ESD 101 has a strong commitment to practicing internally

what they expect their districts to practice. To this end, the ESD developed an extensive

summer training for ESD staff and for a school district leadership team to learn the data

coaching cycle together. The district agreed to share their data and to model the cycle in a

fish bowl session. Each ESD department brought their staff to session and, after the district

modeled a stage of the cycle, the ESD teams practiced the stage internally. This process was

designed to span the full school year, and to replace some of the time spent in ESD

leadership team meetings.

ESD 105, Data Coaching Coop. ESD 105 has a strong culture of cooperative work with

their districts. They developed a cooperative of five districts all interested in increasing and

enhancing their data practices through data coaching. The original design included whole

co-op meetings for all the districts to the data coaching cycle together in one training. The

ESD differentiated training for each of the districts in the cooperative, and rather than

having the districts come to the ESD, the ESD went to the districts to provide personalized

training for each district. The flexibility and adaptability of ESD staff to the needs of the

school districts is a common and expected attribute of ESDs. Responding to the request for

differentiation was therefore viewed as a logical progression for learning.

1 North Central ESD is not included only because there was no opportunity for a timely interview.

15

ESD 112, Special Education. ESD 112 runs a large special education co-op. In 2012–13 the

longtime leader of the department retired and the new leader came in from a school district

where the district’s data were readily available for teachers and school and district leaders.

This was not the case for ESD 112 co-op teachers or directors. The data coaching team

focused on training the special education co-op directors to use data coaching as a tool to

increase and enhance their work with the general education teachers and school and

district leaders.

ESD 113, Early Childhood. ESD 113 leads a multi-county, long-standing, successful Head

Start program. The early childhood leadership team and the data coaching team worked

together to develop a training process to meet the growing demands for data practices in

Head Start programs. Drawing on inquiry cycle and some of the protocols in the data

coaching tool kit, the data coaching team designed a focused tool kit to train Head Start

leadership teams to meet the new data-use expectations in their programs. ESD 113 began

by training the local Head Start leaders working at ESD 113 and quickly found themselves

responding to requests for training throughout Washington State and far beyond

Washington State.

OESD 114, Inquiry Cycles. OESD 114 joined the data coaching institutes with confidence in

the data coaching cycle. They had been successfully leading cycles of inquiry in their math

and science partnership (MSP) work for many years. They were ready and able to take the

data coaching cycle into the majority of funded work including school success, MSP, and the

new teacher and principal evaluation project, which also relies on an inquiry cycle.

ESD 123 School Success. ESD 123, focused much of their data coaching work on their focus

and priority schools in the school success program, working to meet the goals for school

improvement. Data coaching made a nice match with the expectations within the school

success program helping districts collect, analyze, and use data for strategic planning and to

monitor changes toward improvement.

NWESD 189, HOMEROOM. NWESD 189 sits in the shadow of WSIPC. At the start of year

two, the lead data coach from the ESD became the leader of the ESD’s data center, working

closely with WSIPC. Given this proximity to WSIPC, NWESD 189 was the first ESD to support

16

the implementation of HOMEROOM, WSIPC’s data display system designed primarily for

teachers. The training success and challenges were worked out at NWESD 189 and the rest

of the ESDs benefited from their implementation trials. The ESD used the implementation of

HOMEROOM as an invitation for data coaching. Many of the opportunities to conduct

training on the tool kit were associated with HOMEROOM implementation. This was

perhaps the implementation example that was closest to what was originally conceived by

WSIPC.

Puget Sound ESD 121, Using data to determine cultural competence throughout the

early childhood program. The data coaching team at Puget Sound ESD 121 was led by the

early childhood department. This team focused on learning to use data to assess and adjust

their own cultural competence. This was work that was happening across all the programs

at the ESD, and the early childhood team had the advantage of time together to discuss their

data needs and uses for this assessment process.

17

Research Questions Answered in Brief

1. How, if at all, did the data coaching institute shape the work of the ESDs in year two

of implementation?

The assistant superintendents for teaching and learning at the ESDs have worked over the

past ten years to become a collaborative network. The data coaching institutes provided

another opportunity to further this interest. The data coaches from each ESD used the data

coaching institutes to share useful practices, and develop and share plans for new services.

Data coaches reported that they would like to continue meeting and learning from and with

each other. This collaborative approach of ESDs ensures that best practices spread across

ESDs. This means districts throughout the state can have access to the best practices that

emerge in a small district hundreds of miles away, equalizing access and opportunity across

all the districts in the state.

2. What, if anything, are tangible outcomes of ESDs participating in three data coaching

institutes during the 2011–2012 school year?

ESDs are all providing some form of data coaching services. Each ESD produced a creative

and locally-appropriate approach to data coaching.

3. To what extent are ESDs ready to pursue data coaching as a service to their regions?

All ESDs are ready to pursue all aspects of data coaching as designed. This means data

coaches are ready to facilitate district or school leadership teams becoming data-focused

leadership teams or data management teams. Across all the ESDs data coaches went beyond

the PCG approach to data coaching, and designed their own locally appropriate approach to

increasing and enhancing data practices in their region.

4. What more do ESD participants need in order to provide data coaching services in

their regions?

Data coaches want to learn basic inferential statistics to help their schools and school

districts. Data coaches indicated strong interest in continuing to meet to learn with and

18

from each other. They want to learn statistics in the context of the problems of practice

their schools and districts face using available data. The data coaching toolkit does not

require that data coaches know or use inferential statistics, but the majority of data coaches

indicated that statistics would be useful for data coaching. Data coaches were expressly

trained not to bring data, analyze data, or comment on the districts’ data, but instead to

facilitate the district leaders’ bringing data, analyzing data, and discussing data. Given that

most of the data coaches were not comfortable accessing or analyzing their districts’ data,

this helped them to be more comfortable in their role at the start. By the second year, the

data coaches wanted to know more about accessing data and analyzing data with and for

their districts. There is clearly an emerging need for data analysts throughout the states.

5. What are the supports and constraints on ESD resources for data coaching?

From one perspective, there seem to be no constraints on data coaching. Each is ready to

pursue data coaching and there is already evidence that each ESD developed a creative and

locally appropriate way to bring data coaching into the current work in their region. From

another perspective, without funding, data coaching will not be implemented as intended.

No ESD implemented data coaching as intended by PCG. Data coaching was largely absorbed

into the current practices of each ESD. This initiative showed that ESDs can and will find

creative and locally invited ways to bring new practices into their work.

19

Conclusion and Recommendations

What would a fully mature and productive data use practice look like for a state system of

education?

Integrated data systems with built in analytics. It is a rare school district and only one ESD

that has a full time researcher on staff, with responsibility for analyzing data sets to answer

pressing problems of practice in the field. For now, data-informed decision making is largely

rhetoric due to poor access to data and lack of personnel with data analysis skills. One

trained researcher, with a well-trained support person, could increase data use

substantially. ESDs now have data coaches ready to facilitate data conversations. What they

are not as ready to do is access and analyze the data.

Data access and analysis continues to be a problem. There are multiple sites (e.g., ERDC,

EDRC, OSPI Report Card, SLDS) where some education data and some data analytics can be

found and used. These are far from comprehensive and do not provide the flexibility

necessary for sophisticated analysis. It is possible to request administrative data from OSPI

and, with a data sharing agreement, a researcher can have student level data identified with

research numbers rather than names. These requests tend to take three to five months to

process. In context this is not as long as some school systems. Currently, Chicago tells

researchers to expect two years for data requests. All of this is much too slow for developing

robust data use practices in education.

There are additional obstacles to robust data practices. The data are strongly protected

from the student data coordinators. This is, of course, necessary for sensitive student data,

but it is also a challenge for ESD data coaches who need to access the data to facilitate data

analysis for the districts. This challenge can be overcome with data sharing agreements

between school districts and ESDs, but it can take up to a year for an ESD researcher to put

in place a data sharing agreement with each school district in his/her region. Then the data

needs to be accessed district-by-district, and comparisons between districts take much

more time than necessary, given the data could be rolled up into regional data sets directly

by WSIPC and OSPI. To be fair WSIPC has developed an administrative review that rolls up

data at the ESD level, but it is limited to the data that is put into the system at the school

20

level, which may not include state assessment data. Not only would ESDs benefit from a

definition of data access, small districts could benefit from sharing data to increase their

analysis population.

Working with sensitive data. Data use in education suffers from concerns about how data

might be misused. Some of these concerns are warranted. For example, individual students’

grades are private and must be protected. But from whom?

If the ESDs are going to support data-informed decision making, data analysis to help solve

pressing problems of practice, or data analysis to produce better understanding of the social

and political dynamics of their districts, data will have to be shared. As an example of just

how challenging the situation is currently, when HOMEROOM was being implemented in

pilot districts, it was set up for teachers to see only the students who were currently

enrolled in their classes. In middle school and high school, this made it impossible to

explore how their students were doing in other classes or to see if attendance, for example,

was a problem at any other time of day than their classes. Teachers, who are in a good

position to identifying pressing problems of practice, were extremely limited in their access

to their students’ administrative data. Even more difficult for teachers and administrators in

many districts is the restriction on analyzing free and reduced-price lunch data. When

schools are told they are failing their students who receive free and reduced-price lunches,

but they are not allowed to know who these students are, there is a serious limitation to

useful data analysis. It is perhaps time to re-evaluate what “businesses need to know”

stipulation in FERPA to include at least data analysis professionals who work in the interest

of the school district.

If data analysis is an important new approach to making decisions there are currently few

positions in schools, school districts, or at ESDs for data analysis. This clearly takes

specialized skills, generally developed in graduate programs with a research focus. It might

be necessary to allocate resources for these positions, as many school districts have across

the country.

21

The future of data use in education. While educators are both absorbing data into their

current practices and pressing for better tools (assessments, data displays, data analytics)

to increase the effectiveness of their data use, it is still unclear who (teachers, principals,

central office directors) needs to learn what in order to turn data into the powerful tool it

purports to be. It remains an empirical question what the actual benefits are of data use in

education (Goren, 2012). The promise of big data, as used by businesses like NIKE, and

Albertsons, to predict who will shop for what, in which store, on which day has little chance

of helping algebra teachers determine how to help their students learn functions. The

promise of geo-coded data and special analytics can provide the legislature with the exact

cost of the transportation routes for each students in the state; however, it cannot provide

the exact cost of educating each of those transported students.

The new Smarter Balanced assessments, nicely aligned to the new national

standards, will provide student achievement data; however, its promise is already

constrained by the challenges of forging data sharing agreements between school districts

and states. In other words, for all the promise of data, education has a long way to go to

predict, or accurately map, or describe and explain leading and learning in schools. As

wonderful as it would be to have an analytic data system to predict, describe and explain

the strengths and needs of each student, each teacher and each education leader remains far

from that vision.

Recommendations and Promising Possibilities. 1. Think big data. The promise of data-use reaches far beyond the narrow constraints

of current data practice in most school districts. The STARS geo-spatial data system,

developed and used by OSPI’s transportation group, is an example of the where

education data use might aim. Using geo-spatial data analytics and cartography tools

allows education analysts to map multiple layers of large data sets, across the state

landscape to display complex and under considered relationships important to

understanding complex and pressing problems of practices.

2. Think beyond student achievement data. These test score data are limited and often

useless in helping teachers and school leaders making consequential decisions. At

the same time, as big data analysis capacity is being developed, teachers and leaders

need data tools close at hand to support quick decisions including how to improve

instruction tomorrow. For these decisions video cameras are powerful. Collecting

classroom observation data with phone and tablet cameras provides a new source

22

of empirical data for teachers and principals to analyze actual classroom interaction.

No longer reliant on student achievement data as a proxy for learning, teachers and

principals can review and critically analyze learning in real time providing clear,

accurate, and meaningful evidence based feedback for teachers to use to perfect

their practice. Lack of trust constrains the use of video in many schools. The lack of

trust may be addressed as teachers and principals learn to collaboratively, analyze

classroom data from a growth prospective

Finally, this report is a product of a promising data practice. Data coaching began with a

research plan. Beyond evaluating the data coaching institutes, the leaders had wanted to

bring qualitative tools to document and study what happens when you bring all nine ESDs

together to collaborate on a new initiative. Using some of the tools from evaluation

including providing feedback on program implementation as it is emerging, the research

focused not on whether the data institute accomplished preset expectations but instead,

recognized that there were no preset expectations and it was therefore important to follow

what was emergent and new. Partnering with researchers to study the implementation of

new policy and programs is becoming more common, AIR has partnered with the Teacher

and Principals Evaluation Project (TPEP) to study the policy implementation and provide

feedback to the state steering committee throughout the implementation and providing

insight into the process, which is emergent and new. This form of researcher-practitioner

partnership holds significant promise for empowering education practitioners to learn from

their practice to continuously invent their practice in the local and highly complex,

education environments they lead.

23

References

Coburn, C., & Talbert, J.E. (2006). Conceptions of Evidence Use in School Districts: Mapping the Terrain. American Journal of Education, 112.

Cohen, L., Manion, L., & Morrison, K. . (2007). Research Methods in Education (Sixth ed.). New York: Routledge.

Datnow, A., & Park, V. (2009). School system strategies for supporting data use. Routledge, New York.

Feldman, Jay, & Tung, Rosann. (2001). Using Data-Based Inquiry and Decision Making To Improve Instruction. ERS Spectrum, 19(3), 10-19.

Goertz, M., Olah, L., & Riggan, M. (2010). From Testing to Teaching:The use of interim assessment in classroom teaching: Univeristy of Pensylvania.

Goren, P. (2012). Data, Data, and More Data- What's an Educator to Do? American Journal of Education, 118(2), 233-237.

Halverson, R., Grigg, J., Prichett, R., Thomas, C., & Wisconsin-Madison, University of. (2006). The New Instructional Leadership: Creating Data-Driven Instructional Systems in Schools. Journal of School Leadership.

Ingram, D., Louis, K., & Schroeder, R. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106, 1258-1287.

Kerr, K.A., Marsh, Julie A., Ikemoto, Gina Schuyler, Darilek, Hilary, & Barney, Heather. (2006). Strategies to Promote Data Use for Instructional Improvement: Actions, Outcomes, and Lessons from Three Urban Districts. American Journal of Education, 112(4), 496-520.

Knapp, Michael S., Copland, Michael A., Swinnerton, Juli A., & Monpas-Huber, Jack. (2006). Data-Informed Leadership in Education: Center for the Study of Teaching and Policy.

Kowalski, T., Lasley, T, & Mahoney, J. (2008). Data-Driven decisions and school leadership:Best practices for school improvement Boston: Pearson.

Lodico, M.G., Spaulding, D.T., & Voegtle, K.H. (2006). Methods in educational research: From theory to practice. San Francisco: Jossey-Bass.

Marsh, J. A., Pane, J. F. , & Hamilton, L. S. . (2006). Making Sense of Data-Driven Decision Making Sense of Data-Driven Decision

Making in Education: Evidence from Recent RAND Research. . Santa Monica, CA: RAND.

24

Means, B., Padilla, C., DeBarger, A., & Bakia, M. (2009). Implementing data-informed decision making in schools—teacher access, supports and use. In C. o. M. a. S. U.S. Department of Education (Ed.). Washington, DC: Evaluation, and Policy Development.

Miles, M.B., Huberman, A.M., & Saldana, J. (2013). Qualitative Data Analysis: An Expanded Source Book (Third ed.). Thousand Oaks, CA: Sage.

Spillane, J.P. (2002). Local theories of teacher change: The pedagogy of district policies and programs. Teachers College Record, 104(3), 377-420.

Spillane, J.P., & L.K., Seashore. (2002). School improvement process and practices: Professional learning for building instructional capacity. The LSS Review, 1(2), 12-13.

Spillane, James, & Stein, Mary Kay. (2005). What can researchers on educational leadership learn from research on teaching? Building a bridge. In W. Firestone & C. Riehl (Eds.), A New Agenda for Research in Education Leadership (pp. 28-45). New York: Teachers College Press.

Supovitz, J.A., & Klein, V. (2003). Mapping a course for improved student learning: How innovative schools use student performance data to guide improvement. Philadelphia, PA- Consortium for Policy Research in Education.

Talbert, Joan, Scharff, Nell, & Lin, Wendy. (2008). Leading School Improvement with Data:A Theory of Action to Extend the Sphere of Student Success. Palo Alto.

Wayman, J. (2003). Multiple imputation for missing data: What Is It and how can I use It? (C. F. S. O. O. Schools, Trans.). Johns Hopkins University.

Wayman, J. C., Brewer, C., & Stringfield, S. (2009). Leadership for effective data use. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.

Wayman, J., & Chu, V. (2009). Preparing educators to effectively use student data systems: Routledge, New York.

Wayman, J., & Stringfield, S. (2006). Technology-Supported Involvement of Entire Faculties in Examination of Student Data for Instructional Improvement. American Journal of Education, 112.

25