Frequently Asked Questions About ICAS Results

Nicholas Marosszeky

Rassoul Sadeghi, PhD

Educational Assessment Australia

LEARNING OBJECTIVES

• Review – Box Plots

• Report User questions (Part 1)

• Frequently asked questions

• Report User questions (Part 2)

2

Review – Use Box Plots

• Another way to show the distribution of grouped data

– Box Plots

3 SOURCE: NZCER

Review – Use Box Plots

• Online Statistics Education: A Multimedia Course of Study

• http://onlinestatbook.com/

• Project Leader: David M. Lane, Rice University.

4

Report User Questions (Part 1)

• Is there a way to see the “common scale” mark in numeric form, rather than only a place on an unscaled continuum, in order to be able to compare different cohorts?

– See Sections 1, 2.7, 2.8

– Visual presentation of the information

5

Report User Questions (Part 1)

• ICAS results can’t be accessed using an IPAD at this time, because IPAD don’t support flash, which is required for the ICAS results site. With the increasing usage of such technology, are there any plans to make the results site flash-free?

– We are working to resolve this issue

6

Report User Questions (Part 1)

• In terms of using ICAS data to drive a year-level school improvement agenda, I am interested in your suggestions as some good starting points with the data. – Use all available test and teaching data

– Look at individual students

– Conduct a pattern analysis

– Strengths and weakness (Page 2 – Student Report)

– Set individual learning goals – maintenance or improvement

7

FAQ - Question 1

• How are you able to compare the performance of students across different year levels? Couldn’t the test be more difficult in some years than in others?

– Vertical equating / link items

– Adjusted by historical equating / link items

8

FAQ - Question 2

• Why does the average score of students usually seem to increase as the year level becomes higher? Does this mean that the students are getting smarter?

– The test has been developed and designed to do this (also use vertical equating)

– Reflects greater ability in Maths

9

FAQ - Question 3

• What does standard deviation mean? What figure would this usually be, approximately? Is it better for it to be low or high?

– Description of the group / sample

– Describes the spread of scores

– Depends on purpose

– Use for comparison purposes

10

FAQ - Question 4

• On the graphs of student results, why are the top 10% and bottom 10% of students not included? Doesn’t this make the data less reliable?

– Describes the distribution of student data

– Removes extreme ends and outliers

11

FAQ - Question 5

• How can I use reports to measure a student’s progress across years? What are the things I should look for to know whether or not they are improving? • Common Scale

• Group Level Comparisons (See Sections 1, 2.7, 2.8)

• Individual Level Comparisons (See Section 2.9 and Page 1 Student Report)

• Tap School Reports only

12

FAQ - Question 6

• What are distracters? Why should I pay attention to them when interpreting reports?

– Response Options in MC questions

– Further Pattern Analysis

– Strengths and Weaknesses (see Sections 2.3, 2.4 and 2.6)

– Assists with teaching to a class

13

FAQ - Question 7

• How can I use my ICAS data to identify gifted and talented students and students who may require additional support?

– Use more than one source of evidence

14

FAQ - Question 8

• How can I use my students’ ICAS results to help inform my teaching programs? For instance, identifying areas of strength and weakness in students; comparing performance across classes to see if there are some classes who have performed better in particular skills, then investigating what methods their teachers used to teach these skills.

15

FAQ - Question 8

• Strengths and Weaknesses (see Sections 2.3, 2.4 and 2.6) – Pattern Analysis

• School Year Cohort comparisons (see Sections 2.7 and 2.8) – Compare the current Year Cohort with the Locale over time

– Compare to the previous Year Cohorts

– NB: Students in the School Cohort / Locale Cohort may change

– (Experimental method - Independent Variable versus Dependent Variable; Control Group)

16

FAQ - Question 9

• How can I use reports to compare the performance of a year group this year, with the same year group in previous year? For instance, this year’s year 6 with last year’s.

– See Sections 2.7 and 2.8

– NB: Students in the School Cohort / Locale Cohort may change

17

FAQ - Question 10

• How can I compare my students’ ICAS and NAPLAN results? Why do they all seem to have performed better in NAPLAN than in ICAS? Does this mean the ICAS test is too hard?

– Different Tests with Different purposes

– Look at the pattern for individual students and the class

18

FAQ - Question 10

19

NAPLAN ICAS

FAQ - Question 10

20

FAQ - Question 11

• Do you find that, from looking at the data, in some years, there are groups who are just stronger than others? Why is this? – Natural variation

– Chance factors

– Different cohorts

– Impact of different teaching methods and curriculum changes

– Need to look at representative samples

21

Report User Questions (Part 2)

• Any questions ?

22