Umalusi Moderation & Release of the 2010 Standardisation

Decisions

National Consultative Forum Meeting07 May 2011

WHAT CAN WE LEARN FROM THE NSC RESULTS?

Context, principles, approaches and processes

Reasons for Disclosure• Unprecedented interest in the role of

Umalusi• Request: Access to Information• Uninformed comments• Credibility of the NSC• Credibility of Umalusi

Legal Mandate• Chapter 2A of GENFETQA Act 58 of 2001 (as

amended in 2008)

Section17 of the GENFETQA Act(1) The Council must assure the quality of assessment at exit points

(2)(a) The Council must develop policy for the accreditation of assessment bodies other than departments of education and must submit it to the Minister for approval

(b) The Minister must make regulations in which the policy for accreditation is set out

(c) The Council must accredit an assessment body in accordance with the regulations contemplated in paragraph (b)

(3) The Council must perform the external moderation of assessment of all assessment bodies and education institutions

(4) The Council may adjust raw marks during the standardisation process.

Section17 of the GENFETQA Act(5) The Council must, with the concurrence of the Director-General and

after consultation with the relevant assessment body or education

institution, approve the publication of the results of learners if the

Council is satisfied that the assessment body or education institution

has—

(i) conducted the assessment free from any irregularity that may

jeopardise the integrity of the assessment or its outcomes;

(ii) complied with the requirements prescribed by the Council for

conducting assessments;

(iii) applied the standards prescribed by the Council which a learner is

required to comply with in order to obtain a certificate; and

(iv) complied with every other condition determined by the Council.

(6) The Council must issue certificates to learners who have achieved qualifications or part qualifications

Framework for QA of Learner Achievement Based on established and existing practices in

assessment for certification

• Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment

• Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

Framework for Quality Assurance of Assessment Evaluation and /or accreditation of assessment bodies

Periodic inspection of assessment systems

Ongoing monitoring of assessment systems

Quality assurance of external examinations through:

Moderation of examination question papers

Monitoring and moderation of SBA

Monitoring the “State of readiness

Monitoring the conduct of examinations

Moderation of marking

Standardization of assessment outcomes

Approval of Results• Compliance with minimum requirements

• Shift from Moderation to Verification (Moderation for verification)

• Emphasis on Assessment Body reports The monitoring of the conduct of examinations and the

moderation of marking as well as the monitoring of the implementation of internal assessment and the moderation of internal assessment is undertaken by Umalusi to verify the veracity of the assessment body reports in respect of these process.

• Broadened definition of irregularities (including assessment system processes)

Why Umalusi standardises results, and how

• Provision of GENFETQA – Council may adjust raw marks.

• International practice – large scale assessment systems

• Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance.

• Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

Objectives for Standardisation1) To ensure that a cohort of learners is not

advantaged or disadvantaged by extraneous factors other than their knowledge of the subject, abilities and their aptitude.

2) To achieve comparability and consistency from one year to the next.

Why Umalusi standardises results, and how

• Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably

• Process of standardisation

Moderation of question papers

Review of learner performance against historical performance of candidates in each subject.

Historical average (norm) constructed using past 3 to 5 years data.

Pairs analyses provides further comparisons of raw means

Statistical moderation of Internal assessment

Why Umalusi standardises results, and how

• Qualitative input meetings

Reports (moderator, Chief marker and Internal Moderator)

Umalusi research (maintaining standards &post exam analysis)

• Responsibility of Assessment Standards Committee

Committee of Council

Responsible for setting and maintaining standards

• Observers (SAQA, HESA, Teacher Unions)

Process for Standardisation• Continuation of JMB and SAFCERT model

• Assessment Standards Committee

• Qualitative Reports

• Pre-standardisation and Standardisation meetings.

• Standardisation booklets (data) – subject raw mark distributions (external written component only) of entire cohort.

• Subjects are standardised individually, in a linear and non- iterative manner

General principles applied in the standardisation of examination marks

1. In general no adjustment should exceed 10% or the historical average.

2. Adjustments in excess of 10% could be considered at the upper end to increase the number of distinctions in a subject.

3. In the case of the individual candidate, the adjustment effected should not exceed 50%of the raw mark obtained by the candidate.

4. If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to the limitations.

General principles applied in the standardisation of examination marks (cont.)

5. If the distribution of the raw marks is above the historical average, the marks may be adjusted downwards subject to the limitations.

6. The computer adjusted mark is calculated based on these principles.

7. For those subjects with a practical component of 50%, raw marks could be accepted.

8. Umalusi however retains the right to amend these principles where and when deemed to be necessary based on sound educational principles.

Main instruments of standardisation

1.Historical average (HA) of raw scores over 3-5 years

2. Distribution of raw and adjusted marks in 10% intervals

3. Graph plot showing cumulative percentage of candidates at each mark

4. Raw mark distribution including the number and cumulative percentage of candidates at each raw mark

Main instruments of standardisation

5. Pairs analyses

6. Qualitative inputs: examiners’, moderators’ and examining bodies’ reports.

7. Qualitative and quantitative inputs: Umalusi’s ‘research group’.

Guiding Principles of Standardisation

1. Apply both positive and negative adjustments

2. No adjustments of more than 10% in either direction

3. Positive adjustments should generally not push the results above the HA (norm)

4. Marks that are well beyond the HA/norm should be adjusted downwards, but not necessarily as far as the norm

Guiding Principles of Standardisation

5. Raw marks which are close to the norm should not be adjusted

6. Take account of reports by examiners, moderators and research group

7. Any adjustment – no change in rank order 8. Small subjects generally not adjusted - raw 

Specific issues / debates• Does Umalusi meet with government and look at the

pass rates on raw exam marks?

• Does Umalusi adjust marks to achieve a particular pass rate?

• Is Umalusi subjected to political pressure to achieve a particular pass rate?

WHAT CAN WE LEARN FROM THE NSC RESULTS?

STANDARDISATION DECISIONS

ERCO NSC 2010

SUBJECT ADJUSTMENT

Accounting 0 at 0 scale to +18 at 72, block 18 at 222 and scale to 0 at 300

Afrikaans First Additional Language RawAfrikaans Home Language RawAgricultural Management Practice RawAgricultural Science RawBusiness Studies Raw

Civil Technology (DBE)

0 at 0; scale to -15 at 30, scale to -8 at 248, scale to 0 at 300

SUBJECT ADJUSTMENTComputer Application Tech. Raw

Consumer Studies (DBE)

0 at 0, scale to +4 at 8, block +4 from 8 to 296, scale to 0 at 300

Dramatic Arts(DBE) Raw

Economics Raw

Electrical Technology RawEngineering Graphics and Design RawEnglish First Additional Lang.

0 at 0, scale to +6 at 234 and scale back to 0 to 300

English Home Language0 at 0 , scale up to +9 at 111, scale down to 0 at 300

SUBJECT ADJUSTMENTFrench 2nd Additional Language (IEB)

0 at 0 , +20 at 70 scale back to 0 at 300

Geography (DBE)

0 at 0, scale to +8 at 41, block +8 at 86 ; scale to 0 at 154, and then raw.

History (DBE) RawHospitality Studies RawInformation Technology (DBE) RawIsiZulu First Additional (DBE) RawLife Orientation Raw

SUBJECT ADJUSTMENTLife Sciences Raw

Mathematics

0 at 0, scale to +27 at 63, scale to +12 at 228 and scale to 0 at 300

Mathematical Literacy RawMathematics P3 RawMechanical Technology (DBE) Raw

Physical Science

0 at 0, scale to +12 at 78 and scale to +2 at 238 and 0 at 300

Portuguese 1st Additional Language (IEB) Raw

SUBJECT ADJUSTMENT

Religion Studies RawSesotho 1st Additional Language (DBE) Raw

Tourism Raw

Visual Arts (DBE) Raw

WHAT CAN WE LEARN FROM THE NSC RESULTS?

STANDARDISATION DECISIONS

IEB NSC 2010

SUBJECT ADJUSTMENT

Accounting0 at 90 scale to at -7 to 247; Scale back to 0 at 300

Afrikaans First Additional Language

0 at 0, scale to +6 at 84, block +6 from 84 to 234, scale down to 0 at 300

Afrikaans Home Language Raw

Business Studies

0 at 0, scale up +6 at 84, scale to +9 at 231, scale back to 0 at 300

Computer Application Technology Raw

Consumer Studies Raw

Design Raw

SUBJECT ADJUSTMENT

Dramatic Arts0 at 200 scale -4 at 244 and scale back to 0 at 300

Economics Raw

Engineering Graphics and Design

0 at 0 + 12 at 78 scale to 0 at 200, scale to -10 at 250, scale back to 0 at 300

English First Additional Lang Raw

English Home Language Raw

Geography Raw

History Raw

Information Technology Raw

SUBJECT ADJUSTMENTIsiZulu First Additional Language Raw

Life Orientation Raw

Life Sciences Raw

Mathematical Literacy

0 at 0 to -30 at 63, block -30 from 63 to 281, scale back to 0 at 300

Mathematics

0 at 0 to +6 at 105, block +6 from 105 to 144, scale back to 0 at 300

Mathematics P3 Raw

Physical Science

0 at 90 scale to +6 at 144, block of +6 from 144 to 234, scale back to 0 at 300

SUBJECT ADJUSTMENT

Tourism Raw

Visual Arts0 at 200 + 2 at 238, scale back to 0 at 300

Dance Studies Raw

Hospitality Studies Raw

Music RawSepedi First Additional Language RawSesotho First Additional Language RawSetswana First Additional Language Raw

SUBJECT ADJUSTMENTSiswati First Additional Language Raw

Siswati Home Language RawIsiXhosa First Additional language Raw

IsiZulu Home Language Raw

Arabic 2nd Additional Lang0 at 0, scale to +20 at 70 scale back to 0 at 300

French 2nd Additional Lang0 at 0, scale to +20 at 70 scale back to 0 at 300

German Home Language RawGerman 2nd Additional Language

0 at 200,scale to -7 at 247, scale back to 0 at 300

SUBJECT ADJUSTMENTGujurati 2nd Additional Lang RawHebrew 2nd Additional Lang

0 at 200, scale to -7 at 247, scale to 0 at 300

Hindi 2nd Additional Lang Raw

Italian 2nd Additional Lang Raw

Latin 2nd Additional Lang RawModern Greek 2nd Additional Lang RawPortuguese 1st Additional Lang RawPortuguese 2nd Additional Lang Raw

SUBJECT ADJUSTMENTSpanish 2nd Additional Lang RawTamil 2nd Additional Lang RawTelegu 2nd Additional RawUrdu Home Language Raw

Advanced Programme Mathematics

0 at 0, scale to +15 at 150, block +15 to 240, scale to 0 at 300

Equine Studies Raw

Sport and Exercise Science

0 at 0, scale to +15 at 30, block +15 at 200, scale to 0 at 300

WHAT CAN WE LEARN FROM THE NSC RESULTS?

STANDARDISATION DECISIONS

DBE NSC 2010

SUBJECT ADJUSTMENT

Accounting0 at 0, scale to +23 at 67, scale to +6 at 234, scale to 0 at 300

Afrikaans FAL Raw

Afrikaans HL Raw

Afrikaans SAL Raw

Agricultural Management Practice

Raw

Agricultural Science

0 at 0, scale to -9 at 18, block -9 from 18 to 220, scale down to 0 at 240 and raw to 300

SUBJECT ADJUSTMENT

Agricultural Technology Raw

Business Studies Raw

Civil Technology 0 at 0; scale to -15 at 30, scale to -8 at 248, scale to 0 at 300

Computer Application Tech Raw

Consumer Studies0 at 0, scale to +4 at 8, block +4 from 8 to 296, scale to 0 at 300

Dance Studies Raw

Design Raw

Dramatic Arts Raw

SUBJECT ADJUSTMENT

Economics Raw

Electrical Technology0 at 0, scale -6 at 46, block -6 to 226, scale to 0 at 240 and raw up to 300

Engineering Graphics and Design

Raw

English HL0 at 0, scale to +3 at 117, scale back to 0 at 171, thereafter raw to 300

English First Additional Lang FAL

0 at 0, scale to +3 at 6, block +3 from 6 to 297, scale to 0 at 300

SUBJECT ADJUSTMENTEnglish Second Additional Lang

Raw

Geography

0 at 0, scale to +8 at 41, block +8 at 86 ; scale to 0 at 154, and then raw to 300.

History Raw

Hospitality Studies RawInformation Technology Raw

IsiNdebele HL0 at 0, scale to -6 at 12, block of -6 from 12 to 300

IsiZulu HL0 at 0, scale to -6 at 12, block of -6 from 12 to 300

IsiZulu FAL Raw

SUBJECT ADJUSTMENT

IsiZulu SAL Raw

IsiXhosa HL Raw

IsiXhosa FAL0 at 0, scale to -6 at 12, block of -6 from 12 to 300

IsiXhosa SAL Raw

Life Orientation Raw

Life Sciences0 at 0 ; scale to -9 at 23; block -9 from 23 to 286 and scale down to 0 to 300

Mathematics 0 at 0, scale to +22 at 68, scale down to +15 at 225, scale to 0 at 300

SUBJECT ADJUSTMENT

Mathematical Literacy Raw

Mathematics P3 Raw

Mechanical Technology Raw

Music Raw

Physical Science Raw

Religion Studies0 at 0 scale to -12 at 24; block -12 to 291 and scale down to 0 at 300

Sepedi HL Raw

Sepedi FAL Raw

SUBJECT ADJUSTMENT

Sesotho HL 0 at 210, scale to +4 at 236, scale down to 0 at 300

Sesotho FAL Raw

Sesotho SAL Raw

Setswana HLRaw up to 210; scale to +7 at 233; scale to 0 at 300

Setswana FAL Raw

Setswana SAL Raw

Siswati HL0 at 0, scale to -6 at 12, block of -6 from 12 to 300

Siswati FAL Raw

SUBJECT ADJUSTMENT

Tshivenda HL 0 at 210 scale to +7 at 233 and scale to 0 at 300

Tshivenda FAL Raw

Tourism Raw

Visual Arts Raw

Xitsonga HL0 at 120 scale to -6 at 148 and block of -6 to 272 and scale to 0 at 300

Xitsonga FAL Raw

Nautical Science Raw

Maritime Economics Raw