evaluation of principal info seminars final report · the seminars also allowed ccea to access...

February 2010

InCAS & Annual ReportEvaluation of Primary PrincipalInformation Seminars on InCAS andthe Annual Report

Prepared by Research and Statistics ND6068/0910 1

Contents Page

Section

Page

1.0 Executive Summary

2

2.0 Introduction

5

3.0 Method

6

4.0 Results

7

5.0 Conclusion

23


1.0 Executive Summary

Rationale

This academic year (09/10) CCEA held a series of seminars providing Primary

Principals with information on InCAS and the Annual Report.

All Principals who attended the seminars (n = 740) were given the opportunity to

provide feedback via an evaluation questionnaire.

This report outlines the findings from the questionnaire. Feedback gathered from this

phase of evaluation will be used to advise CCEA whether schools require further

training, support or resources regarding InCAS. Feedback will also inform CCEA of

the usefulness of such sessions in raising awareness of InCAS and the Annual

Report.

Key findings

Feedback indicates that the majority of Principals felt the aims of the seminars had

been met. Respondents felt the seminars:

Provided useful information regarding the range of data and feedback

available from InCAS (99.5%, n = 421);

Informed Principals of the Department of Education updates to InCAS

(98.8%, n = 421) and Pupil Reporting Regulations (98.3%, n = 415);

Were useful in informing Principals about the updated CEM website

(94.8%, n = 382);

Illustrated how stakeholder feedback from previous evaluations has been

used to inform the phased implementations of InCAS and the Annual Report

(93.0%, n = 361); and

Provided an informative update of InCAS software developments

(88.2%, n = 344).


A large number of respondents indicated that they intend to use the optional

assessments within InCAS:

82.9% (n = 282) intend to use the Spelling assessment;

80.6% (n = 250) intend to administer Mental Arithmetic;

78.8% (n = 253) intend to administer Developed Ability; and

68.5% of teachers (n = 178) intend to administer the Attitudes assessment.

Principals illustrated that they intend to use InCAS data at school management level

for a number of purposes:

Tracking pupil progress (83.4%, n = 361);

Target setting (75.8%, n = 328);

Identifying areas for improvement (70.2%, n = 304); and

Incorporating into the school development plan (63.5%, n = 275).

94.9% of Principals (n = 385) indicated that overall they were satisfied with the

information seminars.

Recommendations

Just under a third of respondents (29.5%, n = 87) indicated that they would welcome

further information or support on InCAS and the Annual Report. In particular, 27

respondents commented that they require further training on the interpretation of

InCAS data.

Several Principals made specific recommendations and expressed some concerns

they had regarding InCAS and the Annual Report. Such Principals stated that:

Schools require additional time (71 comments);

o To analyse InCAS outcomes (31 comments);

o To conduct InCAS assessments (17 comments);


Schools require more computers (30 comments); and

A small number of Principals expressed concerns regarding the reliability of

InCAS outcomes (21 comments);

o Particularly regarding the accuracy of Year 4 InCAS outcomes

(7 comments).

It should be noted that the Principal Information seminars were facilitated at the time

it became apparent that an error in General Maths had led to incorrect results being

produced for a number of Primary Schools. Although this error was quickly corrected

and accurate information was provided to all schools affected, nevertheless this error

may have impacted on Principals’ confidence in the reliability of InCAS outcomes.


2.0 Introduction

Between the 7th and 23rd October 2009, CCEA invited all Primary Principals in

Northern Ireland to attend an information session on InCAS and the Annual Report.

The learning intentions for the sessions are outlined below:

To inform Principals of the Department of Education updates to InCAS and

Pupil Reporting Regulations;

Demonstrate the effective use of InCAS data with key stakeholders;

Provide an overview of feedback from the evaluations on InCAS and the

Annual Report conducted to date;

Provide information on the developments made to InCAS software;

Highlight the help and support available for schools; and

Provide advice on sharing good practice.

The seminars also allowed CCEA to access Principals’ views on InCAS and the

Annual Report.

All Principals who attended the seminars were given the opportunity to provide

feedback via a questionnaire. Principals rated the usefulness of the seminars and

provided information such as how they plan to use InCAS feedback within their

schools.

Information obtained from this study will be used to advise CCEA if schools require

further training, support or resources regarding InCAS. The feedback will also inform

CCEA of the usefulness of such information sessions in raising awareness of InCAS

and the Annual Report.

This evaluation is part of a programme of work evaluating InCAS and the Annual

Report this academic year (2009 / 2010).


3.0 Method

3.1 Aim

The aim of this evaluation was to ascertain if schools require further training, support

or resources regarding InCAS, as well as informing CCEA of the usefulness of such

information sessions in raising awareness of InCAS and the Annual Report.

3.2 Research Instruments / Tools

Data was collected via a questionnaire. The questionnaire comprised of 16 questions

which were split into two parts; Part A required quantitative answers and Part B

required qualitative comments.

3.3 Data Analysis

SPSS 17.0 was used to analyse quantitative data and thematic content coding was

used to analyse qualitative data.

3.4 Participants

740 Principals attended the information seminars. 433 Principals responded to the

evaluation, representing a response rate of 58.5%. The response to this evaluation

represents almost half of Primary Principals (47.5%) in Northern Ireland.


4.0 Results This section presents results for each question from the questionnaire. Results are

based on questionnaires received from 433 respondents. Percentages relate to the

number of respondents who answered each question, which may be fewer than the

overall total of 433. A number of Principals reiterated comments throughout their

questionnaire; as a result some comments are repeated throughout this section of

the report.

Figure 1: Did today’s seminar inform you of the Department of Education

updates to InCAS?

The majority of

Principals

(98.8%, n=421)

agreed the

seminars informed

them of the

Department of

Education updates

to InCAS. Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.

Nine respondents emphasised that the sessions provided them with clarification on

the Department of Education updates to InCAS.

98.8

1.2

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No


Figure 2: Did today’s seminar inform you of the Department of Education

updates to Pupil Reporting Regulations?

98.3% of

respondents

(n=415) agreed the

seminars informed

them of the

Department of

Education updates

to Pupil Reporting

Regulations.

Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.

Seven Principals were positive that the seminars clearly outlined Department of

Education updates to Pupil Reporting Regulations.

A small number of respondents expressed concerns regarding DE updates to Pupil

Reporting Regulations. Eight Principals commented that the DE circular was

received too late. Six respondents also felt that the information from the DE was

ambiguous and commented that further clarification is required.

98.3

1.7

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No


Figure 3: How useful was today’s session in informing you about the range of

data and feedback available from InCAS?

The majority of

respondents

(99.5%, n=421) felt

the sessions were

useful in informing

them about the

range of data and

feedback available

from InCAS.


Almost 60% of respondents (59.1%, n = 250) found the sessions to be very useful.

16 respondents emphasised that the seminars were particularly useful in outlining

the various types of data and feedback available from InCAS.

However, ten Principals commented on the need for time to allow schools to analyse

the range of feedback options available from InCAS.

59.1

40.4

0.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very useful Useful Not useful


Figure 4: Please indicate which type of chart(s) or table(s) you think are

particularly useful to you at a school management level?

55

43.249 46.9

33.735.3

64.2

41.1

73

4.8

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Standard feedback table

Scores Chart

Longitude Chart

Pupil Scores table

Scores tableDifference table

Age comparison chart

Age standardised scores table

Pupil progress chart

Develop own form

at


Principals indicated which charts and tables they believe to be particularly useful at a

school management level. It should be noted that respondents were given the option

of selecting more than one chart or table.

Figure 4 illustrates that respondents rated the Pupil Progress chart (73%, n = 316)

and Age Comparison chart (64.2%, n = 278) as most useful at a school management

level.

54 Principals commented that the Pupil Progress Chart is particularly useful for

reporting InCAS outcomes to parents. One such respondent commented:

“All information required is clearly illustrated for parents.”


Figure 5: Do you intend to use any of the optional assessments?

82.9 80.6 78.8

68.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Spelling MentalArithmetic

DevelopedAbility

Attitudes


82.9% of Principals (n = 282) indicated they would use the Spelling assessment.

80.6% of respondents (n = 250) intended to administer the Mental Arithmetic

assessment, 78.8% of respondents (n = 253) intended to administer Developed

Ability and 68.5% of teachers (n = 178) indicated they would use the Attitudes

assessment.

Principals commented on their responses. 12 respondents commented that they

would administer Developed Ability in order to determine whether pupils are

performing to their full potential. Eight Principals felt the Spelling assessment would

provide them with a broader overview of their pupils’ performance.


Figure 6: Do you intend to use any of the optional (Irish Medium)

assessments?

Figure 6 illustrates

the optional

assessments that

Principals of Irish

Medium schools

and units intend to

use.


10.6% of Principals (n = 12) indicated they would use the Mental Arithmetic

assessment. 9.1% of respondents (n = 10) intended to administer the Spelling

assessment and 9% of respondents (n = 9) intended to administer the Attitudes

assessment.

It should be noted that the above percentages are in relation to the whole sample of

Principals who responded to the questionnaire (n = 433), rather than the number of

Principals from Irish Medium schools and units in attendance at the training.

9.1 10.6 9

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Spelling (IrishMedium)

Mental Arithmetic(Irish Medium)

Attitudes (IrishMedium)


Figure 7: How do you intend to make use of the InCAS data at a school

management level?

63.5

75.883.4

70.2

5.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Incorporate into school development plan

Target setting

Track progress

Identify areas for improvem

ent

Other


Figure 7 illustrates how Principals intend to use InCAS data at a school management

level. It should be noted that several respondents indicated that they would use the

data for a number of purposes.

83.4% of respondents (n = 361) indicated they would use the data to track pupil

progress. 12 Principals commented they would use the data alongside other

assessments and their own observations.

Seven respondents commented that InCAS data provides useful information at a

school management level. Two such Principals commented:

“I feel the results will be extremely useful [in] identify[ing] strengths and

weaknesses in class.”

“InCAS data [is] useful to help inform planning and target setting.”


However, nine Principals expressed concerns regarding the reliability of InCAS

outcomes. In particular, five respondents queried the reliability of InCAS outcomes in

relation to those from other standardised assessments.

Figure 8: Did today’s information seminar illustrate how stakeholder feedback

from previous evaluations has been used to inform the phased

implementation of InCAS and the Annual Report?


Respondents (93%, n = 361) were positive that the seminars illustrated how

stakeholder feedback has been used to inform the phased implementation of InCAS

and the Annual Report.

However, eight Principals expressed frustration that the Department of Education

had not taken on board stakeholders’ views. A further eight respondents commented

on the need for DE representation at future information seminars.

93

7

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No


Figure 9: Did today’s session provide you with an informative update of InCAS

software developments?

88.2% of

respondents

(n=344) felt the

sessions provided

an informative

update of InCAS

software

developments.

Chart does not precisely reflect the data. For exact figures, see percentage above individual bars. Figure 10: How useful was today’s session in informing you about the updated

CEM website (www.incasproject.org)?

94.8% of

respondents

(n=382) felt the

sessions were

useful in informing

them about the

updated CEM

website.

Chart does not precisely reflect the data. For exact figures, see percentage above individual bars. Eight respondents commented on the usefulness of the sessions in encouraging

familiarisation with the CEM website.

88.2

11.8

0

20

40

60

80

100

Yes No

29.3

65.5

5.2

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very useful Useful Not useful


Figure 11: Subsequent to today’s session, do you intend to make use of the

menus within the CEM website?

Figure 11

illustrates the

menus within the

CEM website that

Principals intend to

use.


The highest percentage of Principals (90.2%, n = 294) indicated that they intend to

use the Administration menu within the website. Seven respondents commented that

they will use the menus if the need arises.

Support and Resources

Principals indicated how useful they found the sessions in informing them about the

various support and resources available for schools. Respondents felt the sessions

were useful in informing them about:

Support / resources Frequency Percentage

The InCAS and Annual Report resource pack 397 99.0

CCEA Helpdesk (telephone support) 361 94.7

Online materials at www.nicurriculum.org.uk 353 93.4

InCAS and Annual Report online course 323 92.8

InCAS preparation letter (Principal’s letter 09/10) 330 89.7

2009 Parent Leaflet 321 87.7

InCAS SEN Guidance for teachers 305 86.6

90.281

74

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Administration Support manualdownloads

Help (e.g. FAQs)


Figure 12: Are there any other aspects of InCAS or the Annual Report in which

you would like further information or support?

29.5% of Principals

(n=87) agreed there

were aspects of

InCAS or the Annual

Report in which they

would like further

information or

support.*


Respondents commented on what information and support would be beneficial. The

most common responses are outlined below:

Training on the interpretation of InCAS data (27 comments);

“Further training [is required] on the use of data interpretation.”

“Interpretation of data at school management level.”

Clarity on what to include in Annual Reports (12 comments); and

Additional time (13 comments);


o To compile Annual Reports (3 comments); and

o To conduct InCAS assessments (2 comments).

*The above percentage relates to the number of Principals who answered the question

(n = 295), rather than the total number of respondents to the evaluation (n = 433).

29.5

70.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No


Figure 13: Do you have a nominated person / coordinator for the preparation of

InCAS within your school?

77.8% of Principals

(n=312) indicated

that they have a

nominated person /

coordinator within

their school for the

preparation of

InCAS.

Chart does not precisely reflect the data. For exact figures, see percentage above individual bars. Principals were asked to specify the member of staff given this responsibility. The

most common responses are outlined below:

Assessment coordinator (129 comments);

ICT coordinator (59 comments); and

Principal (55 comments).

88.2% of Principals (n = 268) indicated that members of staff with responsibility for

the preparation of InCAS have received training on InCAS and the Annual Report.

However, over two thirds of such Principals (69%, n = 171) believed the nominated

person would benefit from further training. Such respondents felt that coordinators

would benefit from information on any updates to InCAS (n = 31) and further training

on the analysis of data (n = 25).

77.8

22.2

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No


How many classroom assistants do you have?

Respondents were asked to provide information on the number of classroom

assistants they have in school. This information is summarised below:

Number of classroom assistants Frequency

1 - 3 157

4 - 6 104

7 - 9 52

10 - 15 56

Figure 14: Are any of your classroom assistants involved in supporting pupils

to complete InCAS?

Just over half of

respondents

(50.5%, n=202)

indicated that their

classroom assistants

are involved in

supporting pupils to

complete InCAS.


Principals provided information on how CCEA could best support these classroom

assistants. 123 respondents commented that classroom assistants require InCAS

training. One such Principal commented:

“Provide awareness raising training. Allow [classroom assistants] to trial

assessments. Provide guidelines for support.”

50.5 49.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No


Figure 15: Do you have any pupils in your school with English as an additional

language?

47.4% of Principals

(n=194) have pupils

in their school with

English as an

additional language.


Principals specified the first language of pupils with English as an additional

language. Responses are outlined below:

Polish (n = 112);

Lithuanian (n = 35);

Latvian (n = 17);

Russian (n = 16);

Chinese (n = 15);

Slovak (n = 15);

Portuguese (n = 12); and

Hungarian (n = 10).

47.452.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No


Figure 16: Overall, how satisfied are you with today’s information seminar?

Respondents

indicated their

overall satisfaction

with the information

seminars.


The majority of Principals (94.9%, n = 385) indicated high overall satisfaction with the

information seminars.

20 Principals commended the facilitators for their helpful, informative style

throughout the seminars. 11 Principals also commented on the usefulness of the

sessions. One such respondent provided the following comment:

“An informative seminar. Feeling much more confident returning to school to

complete parent meetings and using data for school improvement.”

However, nine Principals expressed the need for representation by the Department

of Education at future information seminars.

41.9

53

4.70.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very satisfied Satisfied Not satisfied Not at allsatisfied


Additional Comments

Principals were given the option of presenting any additional comments at the end of

the questionnaire.

A large number of respondents took this opportunity to make recommendations or

express any concerns regarding InCAS and the Annual Report. Such comments are

outlined below:

Schools require additional time (71 comments);


o To conduct InCAS assessments (17 comments);

Schools require more computers (30 comments);

Have concerns with recording pupil accreditation in Annual Reports

(29 comments);

o Clarification is required on what needs to be recorded (15 comments);

o Disagree with the need for recording pupil accreditation in Annual

Reports (14 comments);

Query the reliability of InCAS outcomes (21 comments);

o With particular concerns regarding the accuracy of Year 4 InCAS

outcomes (7 comments);

InCAS outcomes are inconsistent with other standardised assessments

(12 comments); and

Dissatisfied with administering InCAS to SEN pupils (11 comments).


5.0 Conclusion

In conclusion, findings from the evaluation indicate that Principals were very satisfied

with the InCAS and Annual Report Information Seminars facilitated by CCEA.

The majority of Principals indicated that they found the sessions useful in informing

them about the updates to InCAS software developments and in illustrating how

stakeholder feedback from previous evaluations has been used to inform the phased

implementations of InCAS and the Annual Report. The majority of respondents also

rated the sessions as useful in informing them about the support and resources

currently available for schools.

Several Principals indicated that they intend to use the optional assessments within

InCAS, with the highest number of respondents (82.9% n = 282) intending to use the

Spelling assessment. Respondents also illustrated that they intend to use InCAS

data at school management level for a number of purposes, such as to track pupil

progress and inform target setting.

Just over half of respondents (50.5%, n = 202) indicated that their classroom

assistants are involved in support pupils to complete InCAS. 28.4% of such

Principals (n = 123) commented that classroom assistants require InCAS training.

Feedback also demonstrated that almost half of Principals (47.4%, n = 194) have

pupils with English as an additional language within their school. Pupils’ first

languages include: Polish (n = 112); Lithuanian (n = 35); and Latvian (n = 17).

Several Principals took the opportunity to communicate concerns they had regarding

InCAS and make suggestions for improvement, at the end of the questionnaire. Such

respondents commented on the need for additional time to conduct InCAS

assessments and analyse outcomes. Respondents also recommended the provision

of additional computers to schools for the administration of InCAS. A small number

of Principals also questioned the accuracy of InCAS outcomes.


Just under a third of respondents (29.5%, n = 87) indicated that they would like

further information or support on InCAS and the Annual Report. In particular, 27

respondents commented that they require further training on the interpretation of

InCAS data.

Although Principals outlined some concerns and recommendations for improvement,

overall, feedback indicates that the aims of the seminars were met. Respondents

were positive that the sessions informed them of the Department of Education

updates to InCAS and Pupil Reporting Regulations, as well as providing useful

information on the range of data and feedback available from InCAS.

evaluation of principal info seminars final report · the seminars also allowed ccea to access...

Documents