district data literacy survey analysis -...

District Data Literacy Survey AnalysisPrepared for the Arkansas Department of Education

September 2018

www.hanoverresearch.com

Table of Contents

2

Introduction ……………….. P 3

Recommendations ……………….. P 4

Key Findings ……………….. P 5

Section I: Respondent Overview ……………….. P 9

Section II: Data Use ……………….. P 12

Section III: Data Access, Analysis, and Guidance ……………….. P 19

Section IV: Training Needs ……………….. P 24

Section V: Additional Feedback ……………….. P 29

Introduction

3

This report presents the results of surveys administered by the Arkansas Department of Education (ADE) to school district test coordinators and similar district employees about the use of various data types and systems in their districts.

In total, 185 respondents completed at least part of the survey. Only one response per district was kept, leaving 143 respondents in the analysis.

Recommendations

Based on the results from this survey, Hanover recommends that the ADE:

Offer targeted trainings and guidance to support districts to use student growthdata, as well access and analyze data from sources with which district leadersare less commonly comfortable (e.g., Arkansas Student Intervention System,Cognos, SIS reports/Data Center, and studentGPS);

Support districts to use effectively the types of student achievement data thatare most predictive of student secondary and post-secondary success;

Provide trainings primarily through webinars or in-person conferences andworkshops, rather than self-paced courses; and

Consider how the needs and concerns of district leaders may vary by districtsize. While respondents report that district leaders often use studentperformance data similarly regardless of the size of their district, respondentsindicate that larger districts are more likely to use ELPA21 data and data fromunspecified sources, as well as use data to measure equity.

4

Key FindingsDistrict Data Use

▪ Respondents indicate that districts most frequently use studentperformance data as measured by ACT Aspire and K-12 interimassessments. Nearly all surveyed districts use ACT Aspire (99%) and K-2interim assessments (96%), while over 80 percent of districts also frequentlyuse student class performance, periodic assessment data, ACT/SAT, studentattendance, and growth data.

– Correspondingly, respondents indicate that these seven types of studentperformance data are also the most useful in their district's efforts towardsuccess. Likewise, respondents indicate that the least commonly used types ofstudent performance data (i.e., PSAT and NAEP results) are also the least useful insupporting student success.

▪ Nearly all (98%) of districts use student data to measure student progress,while approximately nine out of every 10 districts use student data tomeasure the effectiveness of instruction, plan professional and curriculumdevelopment, meet state and federal reporting requirements, and measureschool and/or program efficacy.

– Respondents report that districts are less likely to use student data to report tocommunities and stakeholders (79%), support schools and/or districts (69%), andmeasure equity (46%).

5

Key FindingsDistrict Data Use, By District Size

▪ Respondents report that, for the most part, districts frequently usestudent performance data from similar sources, regardless of district size.However, respondents at larger districts are significantly more likely toreport that their district frequently uses ELPA21 data and performance datafrom unspecified sources not listed in the survey.

– For example, 78 percent of districts with six or more schools and 73 percent ofthose with three to five schools use ELPA21 data, compared to only 40 percentof those with one to two schools that use this data.

▪ Accordingly, smaller districts are significantly less likely to considerELPA21 and alternate data sources (e.g., AAPA, MSAA, DLM) as useful.

– Only 30 percent of districts with one to two schools find ELPA21 a useful source,while only 25 percent find alternate assessments a useful source. Incomparison, 77 and 69 percent of districts with 6 or more schools that findELPA21 and alternate assessments useful, respectively.

▪ Larger districts are also more likely to focus on measuring equity. Indeed,74 percent of districts with six or more schools use data to measure equity,compared to 40 percent of districts with three to five schools and 38percent of districts with one to two schools that use data for this purpose.

6

Key FindingsDistrict Data Access, Analysis, and Guidance

Districts are typically more comfortable accessing and analyzing data fromsources for which the ADE provides guidance on their use:

▪ Respondents indicate that district leaders are more commonlycomfortable accessing and analyzing data from My School Info,Arkansased.gov, and various assessment portals. Over 60 percent ofrespondents indicate that their district's leaders are comfortable accessingand analyzing data from these sources. Similar percentages of respondentsalso indicate that the ADE provides clear guidance on how to effectivelyutilize these sources.

▪ Conversely, respondents indicate that district leaders are less commonlycomfortable accessing and analyzing data from sources including ArkansasStudent Intervention System, Cognos, SIS reports/Data Center, andstudentGPS. Forty-five percent of respondents or fewer indicate that theirdistrict's leaders are comfortable accessing and analyzing data from thesesources. Respondents correspondingly are less likely to report that the ADEprovides clear guidance related to utilizing these sources in comparisonwith other sources.

7

Key FindingsDistrict Training Capacity and Needs

▪ Respondents broadly report that their districts are equipped to supportschool and district leaders in the analysis and interpretation of studentdata. Indeed, three-quarters of respondents “agree” or “strongly agree”that their district is equipped to provide these supports.

– However, approximately half (48%) of respondents indicate that their districtwould like additional training related to the use of student growth data, whilebetween approximately a quarter and a third of respondents indicate that theirdistrict would like additional training related to topics including ACT Aspire, K-12interim and periodic assessments, various alternative assessments, the ACT/SAT,and student attendance.

▪ When seeking data literacy support, districts most often turn to co-ops(84%), colleagues (80%), and the ADE (78%). Relatively fewer respondentsindicated that their districts rely on professional organizations (24%) orvendors/consultants (23%) to provide this support.

– With regard to district leaders’ preferred delivery method for training arounddata literacy, respondents indicate that most leaders prefer to receive theirtraining through webinars (73%) or in-person conferences/workshops (61%).Conversely, only one-third of respondents indicate that leaders in their districtwould prefer self-paced online courses.

8

9

SECTION I: RESPONDENT OVERVIEW

Response Rate by District Size

10

District Size # of RespondentsTotal Districts

ContactedResponse Rate

1-2 Schools 55 113 49%

3-5 Schools 61 109 56%

6+ Schools 27 42 64%

Response Rate by District Size

Of the 264 districts that were invited to participate in this survey, 143 (54%)responded. As the figure below indicates, a higher percentage of districts withthree or more schools responded to the survey, in comparison with districtswith only one or two schools.

District and Respondent Breakdown

11

10%

90%

0% 20% 40% 60% 80% 100%

Other

District Test Coordinator

Which of the following best describes your position? (n=143)

19%

43%

38%

0% 20% 40% 60% 80% 100%

6+ Schools

3-5 Schools

1-2 Schools

District Size (n=143)

The majority (81%) of districts that responded to the survey have betweenone and five schools. Across these districts, district test coordinatorsresponded to this survey in nine out of 10 districts.

12

SECTION II: DATA USE

Performance Data Type

13

11%

20%

57%

59%

61%

67%

68%

81%

84%

84%

86%

88%

96%

99%

0% 20% 40% 60% 80% 100%

NAEP results

PSAT

Advanced Placement (AP)

Student discipline

ELPA21

Alternate assessments (e.g., AAPA, MSAA, DLM)

Graduation rates

Growth data

Student attendance

The ACT / SAT

Grades 3-10 periodic assessments (e.g., ACT Aspire Interim and Classroom)

Student class performance (e.g., grades, credits, GPA, transcripts)

K-2 interim assessments (i.e., Istation, Renaissance STAR, NWEA MAP)

ACT Aspire (summative assessments)

What student performance data does your district frequently use? (n=142)

Note: Percentages sum to more than 100 because respondents could select multiple options.

37%

11%

26%

59%

67%

78%

74%

59%

85%

85%

93%

81%

85%

96%

100%

20%

10%

22%

62%

55%

73%

68%

73%

88%

87%

83%

88%

88%

98%

97%

11%

11%

16%

51%

60%

40%

62%

65%

71%

80%

80%

85%

89%

93%

100%

0% 20% 40% 60% 80% 100%

Other*

NAEP results

PSAT


Student discipline

ELPA21*


Graduation rates

Growth data*

Student attendance

The ACT / SAT





What student performance data does your district frequently use?

1-2 Schools (n=55) 3-5 Schools (n=60) 6+ Schools (n=27)

Performance Data Type by District Size

14Note: An asterisk (*) indicates the presence of at least one statistically significant difference between district sizes.

Data Purposes

15

46%

69%

79%

89%

90%

91%

92%

92%

98%

0% 20% 40% 60% 80% 100%

To measure equity

To support schools and/or districts

To report to communities and stakeholders

To measure effectiveness of school and/or district programs

To meet state and federal reporting requirements

To guide curriculum development

To plan professional development

To measure effectiveness of instruction

To measure student progress by school and/or district

For what purpose(s) does your district use student data? (n=144)


Data Purposes by District Size

16

74%

85%

93%

89%

100%

89%

100%

96%

100%

40%

67%

85%

95%

90%

92%

93%

97%

98%

38%

64%

65%

84%

85%

91%

85%

85%

96%

0% 20% 40% 60% 80% 100%

To measure equity*

To support schools and/or districts

To report to communities and stakeholders*

To measure effectiveness of school and/or district programs

To meet state and federal reporting requirements

To guide curriculum development

To plan professional development

To measure effectiveness of instruction

To measure student progress by school and/or district

For what purpose(s) does your district use student data?

1=2 Schools (n=55) 3-5 Schools (n=60) 6+ Schools (n=27)

Note: An asterisk (*) indicates the presence of at least one statistically significant difference between district sizes.

Data Type Usefulness

17

35%

37%

5%

12%

32%

27%

18%

12%

10%

11%

24%

26%

31%

33%

27%

31%

24%

18%

18%

15%

13%

10%

9%

7%

7%

9%

30%

29%

31%

39%

42%

37%

42%

41%

39%

34%

37%

35%

17%

21%

20%

19%

29%

37%

38%

41%

46%

54%

53%

57%

0% 20% 40% 60% 80% 100%

NAEP results (n=114)

PSAT (n=115)

Alternate assessments (e.g., AAPA, MSAA, DLM) (n=131)

Advanced Placement (AP) (n=129)

ELPA21 (n=126)

Student discipline (n=133)

Graduation rates (n=130)

Grades 3-10 periodic assessments (e.g., ACT Aspire Interim and Classroom)(n=130)

Student class performance (e.g., grades, credits, GPA, transcripts) (n=132)

The ACT / SAT (n=130)

Student attendance (n=136)

K-2 interim assessments (i.e., Istation, Renaissance STAR, NWEA MAP) (n=135)

Growth data (n=133)

ACT Aspire (summative assessments) (n=135)

How useful do you consider the following in your district's efforts towards student success?

Not Useful at All Slightly Useful Somewhat Useful Quite Useful Very Useful

Data Type Usefulness by District Size

18

8%

17%

69%

58%

77%

62%

69%

65%

65%

85%

88%

92%

96%

100%

10%

10%

46%

51%

54%

54%

73%

78%

87%

82%

79%

93%

95%

95%

8%

7%

35%

46%

30%

61%

71%

74%

81%

80%

91%

81%

82%

85%

0% 20% 40% 60% 80% 100%

NAEP results

PSAT



ELPA21

Student discipline

Graduation rates



The ACT / SAT

Student attendance


Growth data


How useful do you consider the following in your district's efforts towards student success?(% Very Useful + % Quite Useful)

1-2 Schools (n=39-53) 3-5 Schools (n=50-57) 6+ Schools (n=24-26)

Note: An asterisk (*) indicates the presence of at least one statistically significant difference between district sizes.

19

SECTION III: DATA ACCESS, ANALYSIS, AND GUIDANCE

District Leader Comfort Accessing Data

20

36%

41%

41%

45%

52%

62%

62%

64%

66%

72%

0% 20% 40% 60% 80% 100%

Arkansas Student Intervention System (ASIS)

SIS reports/Data Center

studentGPS

Cognos

Accountability reports center

Assessment portal(s)

Arkansased.gov (e.g., ADE Assessment Summary Reports)

eSchool

Triand

My School Info (e.g., Report Card, ESSA Index, Act 1015 Compliance)

I feel that district leaders are comfortable accessing data from... (n=135)


District Leader Comfort Analyzing Data

21

35%

37%

37%

41%

52%

52%

53%

63%

68%

72%

0% 20% 40% 60% 80% 100%

studentGPS



Cognos

Triand


eSchool




I feel that district leaders are comfortable analyzing data from... (n=133)


ADE Guidance for Districts

22

44%

48%

50%

53%

56%

58%

59%

61%

74%

77%

0% 20% 40% 60% 80% 100%




Cognos

studentGPS

Triand


eSchool



The ADE provides clear guidance on how to effectively utilize... (n=133)


Guidance and Capabilities

23

66%

41%

41%

72%

64%

45%

62%

62%

36%

52%

52%

35%

37%

72%

53%

41%

68%

63%

37%

52%

58%

56%

50%

77%

61%

53%

74%

59%

48%

44%

0% 20% 40% 60% 80% 100%

Triand

studentGPS



eSchool

Cognos





District Leader Capabilities and ADE Guidance

ADE Guidance (n=133) Analysis Comfort (n=133) Access Comfort (n=135)

Note: Percentages sum to more than 100 because respondents could select multiple options. No significance testing was run between questions.Bars denoted by “ADE Guidance” indicate answers to the question: The ADE provides clear guidance on how to effectively utilize...Bars denoted by “Analysis Comfort” indicate answers to the question: I feel that district leaders are comfortable analyzing data from...Bars denoted by “Access Comfort” indicate answers to the question: I feel that district leaders are comfortable accessing data from...

24

SECTION IV: TRAINING NEEDS

Additional Training Topics

25

5%

7%

8%

10%

11%

11%

14%

24%

24%

25%

26%

29%

32%

48%

0% 20% 40% 60% 80% 100%

PSAT

NAEP results


ELPA21

Graduation rates


Student discipline

Student attendance

The ACT / SAT





Growth data

For which of the following student performance data topics would your district like additional training? (n=130)


Training Methods

26

33%

61%

73%

0% 20% 40% 60% 80% 100%

Self-paced online course

In-person Conference/Workshop

Webinar

How would leaders in your district prefer to receive training? (n=132)


District Capabilities in Training

27

14% 8% 57% 18%

0% 20% 40% 60% 80% 100%

My district is equipped to provide support to school and district administration for analyzing and interpreting student data. (n=132)

Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree

75%

Data Literacy Support Resources

28

23%

24%

78%

80%

84%

0% 20% 40% 60% 80% 100%

Vendors/consultants

Professional organizations (e.g., AAEA)

ADE

Colleagues

Co-op

When seeking data literacy support, which resource(s) does your district utilize? (n=132)


29

SECTION V: ADDITIONAL FEEDBACK

Open-Ended Responses

30

Please use the space provided to share any other comments you have about student performance/assessment data.

Changing K-2 assessments was helpful for our district. We are not able to look at useful data and make decisions about instruction, curriculum, and student needs. Please look at the timeline for school improvement plans and how it coincides with the release of student performance data. District and school instructional teams need time to dig deep into the data to provide realistic and meaningful strategies for improvement. Until this happens, the improvement plans will be more of a document for compliance and not a document for actual use and improvement.

Student performance and school performance is becoming a wide spectrum. Each district should be mandated to employ a person thatsolely concentrates on dissecting data for the schools and staff. I would encourage the Arkansas Department of Education to make this a priority if data is to be analyzed properly in the years to come.

We need more training to use these effectively. I attended one of the ambassador workshops last year, and I was surprised at all of the data/tools available through programs like My School Info and Triand. There is a lot there, but most educators don't even know about it. I brought what I learned back to our district and shared a few key things. If we had some recorded webinars, we could keep going back to them to share with our teachers. It is really difficult to schedule PD time throughout the year, because there are so many things that need to be covered.

I am beginning my third year as the DTC of my district, and I still feel I am inadequate in assessing some of the data resources. I am a hands-on learner. Webinars are great introduction tools for me, but I cannot always apply what I've learned after the webinar is over. Workshops are similar; however, I do get a chance to ask questions and speak with other DTCs which helps a great deal. I would like classes, so that I can immerse myself into the tasks and really learn them.

Sample of Representative Verbatim Open-Ended Responses

Garrett BrannContent Director

202-350-4754

[email protected]


Michael HammondRelationship Director

202-793-8848

[email protected]


district data literacy survey analysis -...

Documents