Comparison of two Normalization Procedures

Results of some statistical analysis

Normalization procedure 1

• Get percentile (P) of a student on the basis of 5-subject aggregate scores in board.

• Determine the JEE-Main aggregate marks corresponding to percentile P at the All-India level. Regard this as the normalized board score of the student (B1).

• Get JEE-Main aggregate marks (A0) of the student.• The composite marks used for drawing merit list is

A1 = 0.6 A0 + 0.4 B1.

Normalization procedure 2• Get percentile (P) of a student on the basis of 5-subject

aggregate scores in board.

• Determine the JEE-Main aggregate marks corresponding to percentile P from the set of aggregate marks obtained by different students of that board in JEE-Main. Regard this as the normalized board score of the student (B2).

• Get JEE-Main aggregate marks (A0) of the student.

• The composite marks used for drawing merit list is

A2 = 0.6 A0 + 0.4 B2.

Common assumptions• Common assumptions

• aggregate scores of students in a board examination are in the order of their general merit within that board (scholastic achievement indicates ability).

• aggregate scores of the students in JEE-Main are in the order of their merit in respect of engineering admissions (achievement indicates aptitude).

• Assumptions may be violated if • a student has a poor exam,• A student has a special advantage.

• If these effects are confounded with merit, violations become difficult to rectify.

• No fairer assumption is possible.

Exclusive assumptions

• Procedure 1• General merit distribution does not vary from one

board to another.• Procedure 2

• Merit distribution varies from board to board; this difference is captured by differences in performance patterns of students of various boards in JEE-Main .

Data for analysisOne data common data set:• Aggregate marks in AIEEE 2012 and the corresponding percentiles

computed from the set of all students taking the examination.Three sets of data in respect of six boards:• Aggregate marks in board examinations 2012 (expressed as

percentage) and the corresponding percentiles for all students taking the examination.

• Aggregate marks in AIEEE 2012 and the corresponding percentiles computed from the set of students from the chosen board.

• Matched pair of AIEEE aggregate marks and board aggregate percentage.

Issue 1: Extent of ‘compensation’ for board percentile ‘disparity’

0 10 20 30 40 50 60 70 80 90 100

-100

-50

0

50

100

150

200

250

300

350

CBSE AM

JH MR

MZ UK

Percentile in board (P)

Tran

sfor

med

boa

rd sc

ore

( B2)

‘Compensation’ for ‘disparity’: some specific examples

• Transformed score of a student at the 50th percentile of CBSE is more than that of– a student at the 80th percentile of Maharashtra Board,– a student at the 85th percentile of Jharkhand Board.

• Transformed board scores of top five students of three boards: – CBSE: 346,– Maharashtra board : 331,– Jharkhand board : 274.

Amount of compensation (by Procedure 2) is LARGE!

What is being ‘compensated’?• Poor performance of students of some boards in AIEEE may

be due to– Poor merit/ability (not a tenable explanation for large boards),– Non-alignment of board examination pattern with AIEEE

(rank correlations indicate this),– Lack of instruction in English and Hindi

(only available languages for AIEEE/JEE-Main),– Less access to coaching,– Load of an extra subject in board (for some boards).

• All these effects are confounded.• If confounding factors are ignored, solution may be worse

than the problem.

Issue 2: Performance pattern of different boards (distribution of students in different percentile ranges)

Assam

CBSE

Jharkhand

Maharash

tra

Mizoram

Uttarakhan

d0%

20%

40%

60%

80%

100%

95 <= P175 <= P1 < 9550 <= P1 < 75P1 < 50

Assam

CBSE

Jharkhand

Maharash

tra

Mizoram

Uttarakhan

d0%

20%

40%

60%

80%

100%

95 <= P275 <= P2 < 9550 <= P2 < 75P2 < 50

Assam

CBSE

Jharkhand

Maharash

tra

Mizoram

Uttarakhan

d0%

20%

40%

60%

80%

100%

95 <= P075 <= P0 < 9550 <= P0 < 75P0 < 50

Current procedure

Procedure 1

Procedure 2

Disparity among the distributions is the least for Procedure 1.

Issue 3: The question of fairness• In 2012, five students (4 from CBSE, 1 from Maharashtra

board) were tied at– AIEEE aggregate marks of 130 ,– board percentile of 93.1.

• According to Procedure 1, they shared rank 20,352.• According to Procedure 2

– CBSE students had shared rank 18,077,– Maharashtra board students had rank 34,175.

• The differential treatment is entirely due to performance of other students of the two boards in AIEEE.

• This goes against the principle of fairness.

Issue 4: Sampling bias• Procedure 2 requires the

assumption• Board to board

difference in merit distribution is captured by differences in performance patterns of students of various boards in JEE-Main .

 Proportion of AIEEE candidates in percentile range (board aggregate)

Board 0-20 20-50 50-80 Above 80 TotalAssam 0% 0% 2% 98% 100%

CBSE 29% 15% 19% 37% 100%

Jharkhand 16% 5% 5% 74% 100%

Maharashtra 11% 8% 11% 69% 100%

Mizoram 1% 1% 2% 96% 100%

Uttarakhand 14% 3% 4% 79% 100%

• For this assumption to hold, sampling of AIEEE candidates from the boards should follow the same pattern.

• However, 2012 data shows that AIEEE attracts different profiles of students from different boards.

In summary• Students of different boards face different types/levels of difficulty in AIEEE.• The presence of these factors confound any presumed board-to-board

difference in merit distributions.• Board-to-board difference in performance patterns in AIEEE is vast.• Use of AIEEE marks for ‘tracking’ would produce drastic amount of

compensation.• Such levels of unequal treatment of percentiles of different boards, based on

presumptions, is unfair.• Procedure 2 also penalizes a student for poor performance of peers from

same board.• For large boards, there is a strong reason to presume a common merit

distribution.• Procedure 1 produces a more equitable pattern of performance of students

of different boards.

Criteria that are NOT appropriate for choosing between procedures

• Correlation of new and old ranks.– How much correlation is ideal?– The more the merrier?

• Movement of students from one merit list to another.– How much movement is ideal?– The less the merrier?

A benchmark for rank correlationsRank correlation between board and AIEEE marks

BoardSpearman rank correlation

with AIEEE aggregate marksAssam 0.4124

CBSE 0.6891

Jharkhand 0.1926

Maharashtra 0.4322

Mizoram 0.2919

Uttarakhand 0.3235

All matched samples 0.4841

Observations

• No rank correlation is more than 0.7.

• Board examinations are different from AIEEE (measure something else).

• CBSE marks have highest rank correlation with AIEEE:– Similarity of examination

pattern,– Same agency administers

CBSE exams and AIEEE.

Correlation of new and old ranks

Board

Rank correlation between A0 and A1

Rank correlation

between A0

and A2

Assam 0.7283 0.8347

CBSE 0.9464 0.9302Jharkhand 0.5782 0.7163

Maharashtra 0.7691 0.8337

Mizoram 0.5936 0.7025

Uttarakhand 0.5996 0.7435

All matched samples 0.8559 0.9106

• Not much difference overall, and for CBSE.

• For some boards, Procedure 1 produces lower correlation .

• This is in line with observed low correlation between board and AIEEE marks.

• Why should large correlation be a good thing?

Movement from old to new rank list

• Procedure 1 produces greater movement.

• This is in line with observed disparity between board and AIEEE marks.

Some questions of interest to CBSE

1. What do the correlation patterns and rank movements indicate about the fate of CBSE students?

2. Will the CBSE students be at a disadvantage under Procedure 1?

3. Will they get a better deal under Procedure 2?

4. Will smaller boards gain unfairly from Procedure 1?

5. Is there a possibility of ‘board hopping’ if Procedure 1 is used?

Q1: Correlation patterns and rank movements

• Correlation and movement patterns for the two procedures are SIMILAR.

• Correlations indicate degree of association/alignment between new and old ranks.

• Correlations do not reveal ANYTHING about the relative advantage or disadvantage to CBSE students.

• If all CBSE students are pushed down 10,000 places in AIEEE merit list, their rank correlation with board scores would stay the same.

• Scatter-plots of rank movement show that students exchange fate with one another . – no clue about advantage/disadvantage to CBSE students.

New simulations to answer Q2-Q5

• Need simulated data for ALL the boards• Simulation conditions

– 10% Random samples chosen from AIEEE 2012 scores data.

– Presumed correlation between AIEEE and board scores:

• Observed correlations for AS, CBSE, JK, MR, MZ, UK• 0.4 for other boards.

– 100 separate simulation runs.

Actual performance of CBSE students in AIEEE 2012

• 30.1% of all AIEEE candidates are CBSE students.

• However, this groups constitutes– 66 of the top 100 AIEEE rand holders,– 608 of the top 1,000 AIEEE rand holders,– 5,977 of the top 10,000 AIEEE rand holders,– 54,430 of the top 100,000 AIEEE rand holders.

Share of boards in different percentile ranges of AIEEE 2012

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CBSEMR AP BR UP RJ All other

How Procedure 1 or 2 would change the shares

• Procedure 1 reduces marginally the dominance of CBSE students in merit list ,

• Procedure 2 increases drastically that dominance.

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

20%

40%

60%

80%

100%

AIEEE 2012 - Present scenario

CBSEMR AP BR UP RJ All other

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

20%

40%

60%

80%

100%AIEEE 2012 - Procedure 1

CBSEMR AP BR RJUPAll other

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

20%

40%

60%

80%

100%AIEEE 2012 - Procedure 2

UPCBSEMR AP BR RJAll other

…and the numbers behind the charts Procedure 1: Share of boards in different percentile ranges of AIEEE 2012Board 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100CBSE 15% 18% 21% 24% 26% 29% 33% 38% 44% 53%MR 17% 16% 14% 13% 12% 11% 10% 8% 7% 5%AP 7% 8% 8% 8% 8% 8% 9% 9% 10% 13%BR 10% 9% 9% 8% 8% 7% 6% 6% 5% 3%All other 51% 49% 48% 47% 46% 44% 42% 39% 35% 26%

Procedure 2: Share of boards in different percentile ranges of AIEEE 2012Board 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100CBSE 11% 14% 17% 21% 25% 29% 35% 42% 48% 59%MR 20% 18% 16% 14% 12% 11% 9% 7% 5% 4%AP 6% 6% 7% 8% 8% 9% 9% 10% 11% 13%BR 11% 10% 10% 9% 8% 7% 6% 5% 4% 2%All other 53% 51% 50% 49% 47% 44% 41% 36% 31% 22%

Current scenario: share of boards in different percentile ranges of AIEEE 2012Board 0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100 OverallCBSE 14% 17% 20% 22% 26% 28% 33% 39% 46% 54% 30%MR 17% 17% 15% 14% 12% 11% 9% 8% 6% 5% 11%AP 7% 8% 8% 8% 8% 8% 9% 9% 10% 13% 9%BR 10% 9% 9% 8% 8% 7% 6% 5% 4% 3% 7%All other 51% 49% 48% 48% 46% 45% 42% 38% 33% 25% 43%

Findings from simulation

Present scenario Procedure 1 Procedure 2Top 0.01% 65.1% 61.9% 82.7%

Top 0.1% 60.5% 60.4% 80.1%

Top 1% 60.0% 60.0% 69.6%

Top 10% 54.4% 53.4% 58.8%

• Procedure 1 produces a marginal reduction in CBSE’s share at the top; 2% reduction at the top 0.01%.

• Procedure 2 brings a great windfall to CBSE students.• CBSE students gain more and more from Procedure 2 as one goes up

the merit list (share at top 1% and top 0.1% increases by 10% and 20%, respectively ).

• If interest of top CBSE students had been the criterion, Procedure 2 would be a clear winner!

How does Procedure 2 dramatically increase CBSE’s share of top-rankers?

• It does so by practically turning the board score into another AIEEE score.

• The composite score A2 for each board has same mean as A0, but has much smaller spread.

• Each set of board scores get more concentrated around their respective means.

• In effect, every students gets a second chance to perform at the same mean level.

• As a result, CBSE students gain substantially.

Why does Procedure 1 have only a mild effect on CBSE’s share of top-rankers?

• For CBSE, the students who do well in AIEEE are generally those who do well in board exam.– They mostly hold on to their position in merit list.

• For boards with less rank correlation with AIEEE, this is not the case.– Many AIEEE top-rankers from other boards move towards

middle in terms of composite score.

– If their board exams had been better aligned with AIEEE, they would have gained more from Procedure 1.

Answers to Q2 and Q3

• Q1: Will the CBSE students be at a disadvantage under Procedure 1?

– Yes, but marginally so.

• Q2: Will they get a better deal under Procedure 2?

– Yes, the top third of the merit list gets PACKED with CBSE students.

– It is not just a better deal, it is a great deal.

– It would not be fair to other boards!

Q4: Will smaller boards gain unfairly from Procedure 1?

• A small board may not have same merit distribution as others. It may have – Stronger students, or

– Weaker students.

• To say that they gain more (from Procedure 1) than larger boards is to PRESUME that they have weaker students.

• Percentiles generally correct for board size.

• Here, percentile of a student is defined as percentage of candidates strictly below that student.

• Toppers of a smaller board will always have a smaller percentile than the toppers of a larger board.

• Students of smaller boards are also more vulnerable to variations of all kinds.

Q5: Will Procedure 1 encourage board hopping?

• Yes, but TOWARDS CBSE (continuation of current trend) – CBSE’s loss and other boards’ gain is marginal.– This loss is only in relation to the present scenario; 2013 students cannot

hop into 2012 system.– Even after this minor loss, CBSE students will have continued advantage

over students of most other boards in terms of• alignment of examination pattern• alignment of language• access to coaching• a well administered examination system (less uncertainty)

– If a CBSE student hops to a smaller board, good marks are not guaranteed (exam is likely to be of a different type)

• Confounding advantages in AIEEE should not give them false confidence.

Composition of different percentile groups in 2012 by Procedure 1

0-1010-20

20-3030-40

40-5050-60

60-7070-80

80-90

90 to 99

99 to 99.9

99.9 to 99.99

above

99.990%

10%20%30%40%50%60%70%80%90%

100%

CBSEMR AP BR RJUPAll other