NAPLAN Workshop

Assessment for Better Learning using NAPLAN Data

Presented byPeter Congdon, Principal Consultant – KmetricsOn behalf of the VCAA

Workshop structure

• Main themes• How curriculum leaders and classroom

teachers can use their school-level data to analyse the impact of their school’s learning programs.

• How classroom teachers can use the responses to questions on the NAPLAN 2014 tests as a diagnostic tool to inform future teaching.

Workshop Content

• NAPLAN Data Service reports and functions• Methods of using the data and results for

monitoring and improvement purposes • Working with data, interpreting data,

describing data and developing an informed response to the data.

Context – Professional practiceUsing assessment data effectively has become embedded in teaching expectations and school improvement processes.

Use of data:•National Professional Standards for Teachers – Australian Institute for Teaching and School Leadership

Standard 5 - Assess, provide feedback and report on student learning

Context - To use student data to improve teaching practice.

Teachers need to be able to do the following:• Find the relevant pieces of data in the data system or display available to them (data location)• Understand what the data signify (data comprehension)• Figure out what the data mean (data interpretation)

– Substantive and contextual

• Select an instructional approach that addresses the situation identified through the data (instructional decision making)• Frame instructionally relevant questions that can be addressed by the data in the system (question posing)Teachers' Ability to Use Data to Inform Instruction: Challenges and Supports. U.S. Department of Education Office of Planning, Evaluation and Policy Development http://www2.ed.gov/rschstat/eval/data-to-inform-instruction/report.pdf

https://naplands.vcaa.vic.edu.auHelpdesk phone 1800 648 637

AIM results (2007 and earlier) are no longer available

Reference documents

• Assessment materials • Test performance & content summary guides• Reporting guides• Descriptive exemplars of marking guides• Analysis strategies• Online tutorial assistance for reports can be

accessed at http://usingassessmentdata.vcaa.vic.edu.au/naplan/index.aspx

All available within the NAPLAN Data Service to support use of the results

Box and Whisker Charts

Reference groups

National State

Focus group

School

Normal Distribution

90th percentile

10th percentile

25th percentile50th percentile75th percentile

Number of students if total = 100

10

15

25

25

15

10

Skewed Distribution

90th percentile

10th percentile

25th percentile50th percentile

75th percentile

Number of students per 100

10

15

25

25

15

10

NAPLAN Reporting Bands

Example NAPLAN Summary Year 7

What are the main features of these results?•Strongest in Writing and Spelling•Lower students not as low as State’s low students•Higher students not as high as State’s higher students in Reading, G&P and Numeracy•Is this a reflection of the school’s teaching program, and/or a feature of this cohort?•How much of these differences are due to imprecision?

NAPLAN Year 7 Summary

2013

2014

Year 7 results

• Usually, Year 7’s have only been at your school for a few months prior to testing.

• Results can reflect feeder school programs.• Consider grouping students by main feeder

schools and sharing results – network.

School Summary Exercise - 5 mins

• Review your School summary report(s)• Address the following

– Strongest in;– Lower students compared to State’s low students;– Higher students compared to State’s higher students;– Major influence on results include; – Strategies to consider;

Trend Data

• Find evidence of the impact of change over five years– Shows the range of student achievement levels,

Box and Whiskers– Plots the mean student achievement level;

Main sources of variability:• Different students

• Work ethic• Behaviour• Home support

• Measurement imprecision• Test properties• Equating• Group size

• School• Leadership• Resources

• Teacher• Effectiveness

• Program• Content• Alignment

Five Year Trend Exercise – 5 mins Review your Five Year Trend report(s)Address the following• Compare performance relative to State group

– High = Top 25% v State Top 25%, – Medium = Middle 50% v State Middle 50% – Low = Bottom 25% v State Bottom 25%

• Reading• Writing• Spelling• Grammar & Punctuation• Numeracy

• Identify Influencers on results– Cohort– School– Teacher– Programs

Main sources of variability:• Different students

• Work ethic• Behaviour• Home support

• Measurement imprecision• Test properties• Equating• Group size

• School• Leadership• Resources

• Teacher• Effectiveness

• Program• Content• Alignment

Chooseone

Group Summary Report

How do our groups stack up against the State groups?

What does this tell us about;Cohort,School,Teachers,Programs.

Group Summary Exercise – 5 minutesReview your Group summary report(s)Address the following• Compare the performance of each group relative to State group in one domain

– High = Top 25% v State Top 25%, – Medium = Middle 50% v State Middle 50% – Low = Bottom 25% v State Bottom 25%

• Girls V State Girls • Boys V State Boys• LBOTE V State LBOTE• ATSI V State ATSI

• Influencers on results– Cohort– School– Teacher– Programs

Main sources of variability:• Different students

• Work ethic• Behaviour• Home support

• Measurement imprecision• Test properties• Equating• Group size

• School• Leadership• Resources

• Teacher• Effectiveness

• Program• Content• Alignment

Assessment Area Report

Percentage of items answered correctly in short answer questions

Number of items

Raw score average, State (36 items @ 60% correct) = 36*.60 => 21.6

Raw score average, School (36 items @ 52% correct) = 36*.52 => 18.7

Assessment Area Exercise – 2 minutes

• Identify if any dimensions have been flagged as significantly different from the State

• Calculate or estimate the Raw score difference between your students and the State on one or more dimensions

Writing Criteria Report

Writing Criteria Exercise – 2 mins.Compare modal scores.

Modal score = most common scoreWhich criterion are you relatively strongest on?

School State

3-4 4

2 3

Item Analysis Report

Finding skills of relative strength or weakness – Graphical format

Understanding student weaknesses – Table format

How classroom teachers can use the responses to questions on the NAPLAN 2014 tests as a diagnostic tool to inform future teaching.

Test and group detailsLink to some test detailsLink to test itemsItem Analysis Report

Item Analysis Report – GraphFinding skills of relative strength or weakness

Harder than State =

Easier than State =

Item AnalysisUnderstanding student weaknesses

Item AnalysisUnderstanding student weaknesses

50 is half of 100 , 22 is half of 44

44+50 = 94

A D

Half of the group could do it – BRemainder had problems. Were they related toFormat, language, concept, process, knowledge, skill,...?

One quarter of the group could do it – option ARemainder had problems. Were they related toFormat, language, concept, process, knowledge, skill, opportunity to learn,...?

Student Responses - Individuals

Peter Congdon - Kmetrics

Item level Diagnostics• By comparing your students’ success at the item level, to that of all other

students in the state, you can..• Look for relative differences as the test progresses

– Did our students answer all the items?– Were they consistently above, below or similar to the rest of the

state?• What do any difference represent? Put it into context• Is language or vocabulary an issue?• Is test taking an issue – format, motivation, terminology?

– Are there items that represent areas of the curriculum not yet introduced?

• How does this compare to the rest of the state?– Are there areas taught, but not as well as expected?– Consider what is happening at other Year levels – curriculum mapping.– Where are the relative strengths – how can you learn from them?

Item Analysis Exercise – 2 minutes

• Identify items indicating relative strengths and weaknesses

• Follow up (for class room teachers)Understand and capitalise on strengthsInvestigate relative weaknesses and develop a plan in responseDiscuss and share with colleagues

Zone of Proximal DevelopmentItem Number 14 30 29 25 13 8 28 11 27 9 12 20 26 19 22 21 18 24 7 5 10 6 23 16 3 4 17 2 15 1

Answer Key # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

Dimension S S S S S S S S S S S S S S S S S S S S S S S S S S S S S S raw score

ANDERSEN, HANAPHI

# # # # # # √ # # # # # # # # # # # # # # # # # # # # # √ √

3BIDDELL, RILEY # # # # # # √ # # # # # # # # # # # √ √ # # # √ # # √ # # #

5BRENTON, HAMISH - # - - # # # # # # # # - # - √ # # # - # # - - # √ √ √ √ #

5BEATTY, BENJAMIN # - # # √ # # # # # √ # - # - - # # # # # # # - # - √ √ √ √

6BLAZEY, MAXWELL # # # # # # # # # # √ √ # # # # # # # # # # # # # √ √ # √ √

6BELL, MICHAEL - # # # # # # # # # # # √ # # # # # # √ √ √ √ # # √ - √ √ √

9BRADSHAW, REBECCA

# # # # # # # # # # # # # # # # # # # √ √ √ √ √ # √ √ √ # √

9BUCKLER, ROCKY # # # # # # # √ # # # # # # # # # √ √ # √ # # # √ √ √ √ √ √

10ANDERSON, EMILY # # # # # # # # # # # √ √ √ # √ # # # # # √ # √ √ √ √ √ # √

11ANGUERRE, CHARLES

# # # # # # # # √ # # √ # # # # # # # √ # # √ √ √ √ √ √ √ √

11AL MALIKI, MITCHELL

# # # # √ √ # # # # √ # √ # # # √ # √ # √ # # √ # √ √ √ √ √

13CAITHNESS, JOSHUA

- # # # # # - √ √ √ - √ # - - - √ √ # # √ √ - √ √ √ - √ # √

13BENATSKY, GEORGIA

# # # # # # # # √ √ √ √ √ # √ # # √ # √ # # √ √ √ √ √ √ √ √

16BOTHAM, NATHAN # # # # # √ √ # √ √ # √ √ # √ # √ √ √ √ √ √ √ √ √ √ √ √ √ √

21BALLA, ZANE # # # # # √ √ √ √ √ √ # √ √ √ √ # √ √ √ √ √ √ √ √ √ √ # √ √

22BEAUPEURT, MARK # √ √ √ √ √ √ √ √ √ # √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ # √

27BEAVIS, JORDAN √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √

30

Zone of Proximal Development:

What students a

re capable of learning with the guidance

and support from teachers a

nd peers.

What students can already do independently.

Level of potential after other steps have been made.

Vygotsky and other educational professionals believed education's role was to give children experiences that were within their zones of proximal development, thereby encouraging and advancing their individual learning

Substantive descriptions of achievement levels

Using the NAPLAN items to identify skills, knowledge, procedures...National bands – parent reports skill descriptors, www.nap.edu.au

Item Number 14 30 29 25 13 8 28 11 27 9 12 20 26 19 22 21 18 24 7 5 10 6 23 16 3 4 17 2 15 1

Answer Key # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

Dimension S S S S S S S S S S S S S S S S S S S S S S S S S S S S S S raw score

ANDERSEN, HANAPHI

# # # # # # √ # # # # # # # # # # # # # # # # # # # # # √ √

3BIDDELL, RILEY # # # # # # √ # # # # # # # # # # # √ √ # # # √ # # √ # # #

5BRENTON, HAMISH - # - - # # # # # # # # - # - √ # # # - # # - - # √ √ √ √ #

5BEATTY, BENJAMIN # - # # √ # # # # # √ # - # - - # # # # # # # - # - √ √ √ √

6BLAZEY, MAXWELL # # # # # # # # # # √ √ # # # # # # # # # # # # # √ √ # √ √

6BELL, MICHAEL - # # # # # # # # # # # √ # # # # # # √ √ √ √ # # √ - √ √ √

9BRADSHAW, REBECCA

# # # # # # # # # # # # # # # # # # # √ √ √ √ √ # √ √ √ # √

9BUCKLER, ROCKY # # # # # # # √ # # # # # # # # # √ √ # √ # # # √ √ √ √ √ √

10ANDERSON, EMILY # # # # # # # # # # # √ √ √ # √ # # # # # √ # √ √ √ √ √ # √

11ANGUERRE, CHARLES

# # # # # # # # √ # # √ # # # # # # # √ # # √ √ √ √ √ √ √ √

11AL MALIKI, MITCHELL

# # # # √ √ # # # # √ # √ # # # √ # √ # √ # # √ # √ √ √ √ √

13CAITHNESS, JOSHUA

- # # # # # - √ √ √ - √ # - - - √ √ # # √ √ - √ √ √ - √ # √

13BENATSKY, GEORGIA

# # # # # # # # √ √ √ √ √ # √ # # √ # √ # # √ √ √ √ √ √ √ √

16BOTHAM, NATHAN # # # # # √ √ # √ √ # √ √ # √ # √ √ √ √ √ √ √ √ √ √ √ √ √ √

21BALLA, ZANE # # # # # √ √ √ √ √ √ # √ √ √ √ # √ √ √ √ √ √ √ √ √ √ # √ √

22BEAUPEURT, MARK # √ √ √ √ √ √ √ √ √ # √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ # √

27BEAVIS, JORDAN √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √

30

Easier

Harder

Describing Zone of Proximal Development – for most raw scores of 5-6

drafting

blizzard

tertels

Seriuslyorkwardly

Relative Growth• How is relative growth defined? • Each student’s level of relative growth is determined by comparing

their current year NAPLAN result to the results of the group of all ‘similar’ Victorian students. ‘Similar’ students are defined as those that had the same NAPLAN score two years ago.

• Compared to these similar students, if a student’s current NAPLAN score is in the : – highest 25%, their growth level is categorised as ‘High’. (Green) – middle 50%, their growth level is categorised as ‘Medium’ (Yellow), and – lowest 25%, their growth level is categorised as ‘Low’ (Red).

• Note that the percentages within each category will vary from school to school.

Relative Growth

+138

+15

+18

-30

Relative Growth Exercise – 2 minutes• Identify domains with low relative growth

greater than 25%• And or• Identify domains with high relative growth

greater than 25%• Was the relative growth even across the starting

Bands• Follow up (Co-ordinators and class room

teachers) implementation and effectiveness of differentiation

Working with NAPLAN Data• Principal

• Analyse Summary and Trend results• Conduct program evaluations• Facilitate staff access to work with results

• Co-ordinator• Map the relatively high and low performances against

the delivery of the curriculum• Look for class/group differences• Work with colleagues .

• Class room teacher• Diagnose misconceptions, weaknesses & strengths.• Develop teaching plans in response.

Peter Congdon - Kmetrics

Reporting Back• Summative position• Overall location of the students and subgroups

– Shape of the distributions• Are students being left behind?• How spread out are they?• Is there too much focus on the low achieving students - top students held back?

– How would describe the location of your students against the state?– How spread out are your students in the different dimensions?

• Trends– How have your results changed over time?

• Consider both the location and the shape of the distribution.– Are there identifiable factors that may be contributing?

• Student level effects: motivation, engagement, home• Teacher level effects: method, style, experience, support, workload• School level effects: leadership, resources, programs

• Growth,– Are students maintaining their position relative to the state?

• Is the top growing as fast as the bottom relative to the state?• Is growth even across the dimensions?

• Curriculum Mapping– Item level Diagnostics

• Can you find relative strengths and weaknesses?• What can you and the school adjust based on these findings?• Link to curriculum scope and sequence• Link to programs and pedagogy

Thank you for attending

Please use the rest of this time to go over your results, and clarify your interpretations.

Peter CongdonPrincipal ConsultantMobile: 0434 000 561email: peter@kmetrics.com.auweb: www.kmetrics.com.au

PowerPoint presentation available at www.kmetrics.com.au

Further help is available by contacting me directly