Assessing Confidence in the Chinese Learner

Stephen Bruce

Napier University, Edinburgh

Assessing Confidence in the Chinese Learner

Chinese students at Napier

Rationale for confidence-based assessment

Student responses to the test

Academic self-concept, confidence and achievement research

Future work

English Foundation Programme

Set up in 2001 to prepare Chinese students for entry to Napier University, both in terms of linguistic and study skills.

The needs of these students were complex, the “learning curve” for them was extremely steep, and that the demands they faced on a linguistically complex and culturally unfamiliar one-year Masters would be considerable, even given the requisite IELTS pass.

Proportion of Chinese students on Napier Business Masters programmes has increased in recent years, which has resulted in staff and the institution facing a very different cohort with different strengths, and also needs, to their predecessors.

Presents a major challenge to lecturers as they struggle to adapt their teaching approaches to suit both the increased number of overseas students and also the home students on their modules.

Dr Lesley Gourlay, Napier University (2004). Crossing Boundaries: A Case Study Masters Level Chinese Students, LTSN (In Press).

Linguistic and Cultural Issues

Challenges for Students

LANGUAGE Extreme difficulties in understanding / taking notes in lectures Difficulties with tutorial participation due to lack of confidence in English Some difficulties with tutorial tasks due to slow reading speeds Fear of failing exams due to difficulties in writing English in a time limit

EDUCATIONAL CULTURE Feeling “lost” in semester 1 Some students unclear about expectations in UK-style coursework &

reading Experience extremely stressful for some

Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)

Linguistic and Cultural Issues

Challenges for Staff

LANGUAGE Unsure if students are understanding lectures Required to spend extra time explaining lecture content at the end Some difficulties with tutorial participation and integration

EDUCATIONAL CULTURE Some staff faced with numerous examples of plagiarism in coursework Some difficulties with tasks involving critical appraisal Some students struggled with application of theory to practice

Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)

Potential responses by the university

Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)

LECTURE COMPREHENSION & NOTETAKING:

SPECIFIC SESSIONS FOR INTERNATIONAL

STUDENTS? SUPPORT & IDEAS FOR

LECTURERS?

TUTORI AL PARTI CIPATION

ISSUES:

SPECIFIC SESSIONS FOR INTERNATIONAL

STUDENTS? MORE OPPORTUNITIES

FOR STUDENT PREPARATION?

PLAGI ARI SM AS LANGUAGE COPI NG

STRATEGY:

SPECIFIC SESSIONS FOR INTERNATIONAL

STUDENTS? MORE “USER-FRIENDLY” PLAGIARISM AVOIDANCE

MATERIALS? GROUP

ROTE-LEARNING AS LANGUAGE COPI NG

STRATEGY:

REDUCTION IN EXAM PREDICTABLITY?

CLEAR POLICY ABOUT GRAMMAR ERRORS AND

MARKS?

STUDENT ISOLATION &

ANXIETY:

MORE “BUDDYING” AND INTEGRATION INITIATIVES?

MORE INTERNATIONAL STUDENT

COUNSELLING?

INCREASED WORKLOAD FOR

STAFF:

EXTRA TIME GRANTED IN WORK

ALLOCATIONS? INTERNATIONAL

ACADEMIC SUPPORT OFFICERS?

Confidence and achievement

Learner self-esteem has been found to be positively associated with academic achievement (Brookover et al., 1964; Prendergast & Binder, 1975; Song & Hattie, 1984)

Comparative studies support a view that the self-esteem of young Chinese students is lower than UK and American (Chan, 2000).

– self-effacing and modest values in Chinese culture, strongly influenced by the Confucian tradition of a ‘humble’ character

– traditional authoritarian style of education or the highly competitive pressures created by schools, families and society

– gap in living standards in the UK and Chinese cultures

Recent advances in academic self-concept and achievement research

Assessing Confidence in the Chinese Learner

This short pilot study is interested in two questions:

1. Would the purported reticence of the Chinese student be reflected in confidence-based assessment ?

2. Would their confidence levels in the test bear any relation to their academic self-concept ?

Rationale for confidence-based MCQ

A student’s ability in answering may fall into a number of categories– I know it– I’m not quite sure, but I think I know it– Perhaps I can identify the answer by a deductive process on the distracters– If I guess I’ve a 25% chance of being correct (for 4 answer choices)– I really haven’t a clue

and perhaps worst of all– I really know it …… what do you mean I’ve got it wrong !! Davies (2002)

The standard MCQ cannot distinguish between the above– the student certain of their knowledge should be rewarded– the student should not be rewarded for guesswork

Would the student use this knowledge to make a decision or perform an action (usable knowledge) ? Hassmen and Hunt (1994)

Confidence-based MCQ

Select an answer in the usual way for a multiple choice question

C=1 (low) C=2 (mid) C=3 (high)

Gardner-Medwin and Gahan (2003)

Indicate your confidence that you are correct.

Scoring confidence-based MCQ

What is my score if my answer is or ?

Score when

Score when

2

-1

Confidence levelC = 2(Mid)

1

0

C = 1(Low)

C = 3(High)

3

- 4

More severe penalties for True/False format

Gardner-Medwin and Gahan (2003)

Encouraging good confidence judgement

If you are sure that your answer is correct ….

If your are unsure (or guessing !) ….

o select C=2 (mid) or C=3 (high) confidence level

get the marks your confidence deserves!

o select C=1 (low) confidence level

don’t lose marks through misplaced confidence!

Good confidence judgement means more marks !

Confidence-based MCQ feedback

A 100% confidence-based score is 40 answers at C=2 (mid) confidence…. so that 40 correct at C=3 would be 150% Gardner-Medwin and Gahan (2003)

A summary of confidence levels….

Confidence-based MCQ feedback

Is the confidence-based score higher than the % correct score?

1. Check the instances of INCORRECT answers at C=3 (high) and C=2 (mid). These can indicate areas of knowledge where you are misinformed. A large number of these perhaps indicates general overconfidence.

2. Check your number of CORRECT answers at C=1 (low). A large number indicates that you know more than you are willing to admit and are perhaps underconfident.

No

Confidence-based MCQ feedback

Is the confidence-based score higher than the % correct score?

You have shown good confidence judgement!

You are willing to express an appropriate level of confidence when considering your knowledge.

This is important when making decisions and performing actions based on that knowledge.

Yes

The student group

31 Chinese students (54% sample)o 18 male, 13 femaleo 24 postgraduate and 7 undergraduateo Ages: 18-25 (19), 26-32 (8), 33-40 (4)

Current on English as a Foreign Language (EFL) Foundation Programmeo 35 weeks of 20 hours of EFL studyo Gateway and Starter routes dependant upon English proficiency

Students recently completed their IELTS exam (International English Language Testing System). o Entry on to a Napier Programme conditional on a student’s IELTS score o At the time the students did not know their results

Students complete a confidence-based academic vocabulary test (40 Qs)

Chinese student responses to confidence-based MCQ test

No indications that Chinese students are reluctant to select high confidence levels (C=3).

On average, C=3 (high) confidence was selected most often

No significant difference between genders observed.

Proportion of confidence levels

37.7

33.4

29.0

0.0

10.0

20.0

30.0

40.0

C = 3 C = 2 C = 1

%

Chinese student responses to confidence-based MCQ test

Instances of C=3 (high) confidence increased as the question difficulty decreased.

‘Difficult’ language comprehension test as some small grammatical differences separated answers and distracters.

C = 3 (high)

R2 = 0.3137

0

5

10

15

20

25

30

0.0 0.2 0.4 0.6 0.8 1.0

Question difficulty

No

. of

stu

de

nts

Harder Easier

Correlating confidence-based test and final exam (IELTS)

Achievement in test scores

% correct IELTS exam

Confidence-based score 0.91 (p=0.000) insignificant

IELTS exam insignificant -

only 7 students (of 31) improved on their % correct score with good confidence judgement. o 6 of these students were the highest test scorers in the class.

no apparent correlation between IELTS exam and confidence-based testo IELTS exam a true measure of achievemento several randomised confidence-based tests required for any

correlation to identifiedo Students will become more proficient with practice

Student opinions of the test

1. Strongly agree

2. Agree

3. Neutral

4. Disagree

5. Strongly disagree

It is appropriate for a test to measure the confidence I have in my knowledge

0

3

6

9

12

15

18

1 2 3 4 5

Relative agreement

No

. o

f stu

den

ts

I sometimes took a risk by selecting a higher level of confidence than I really felt

0

2

4

6

8

10

12

1 2 3 4 5

Relative agreement

No

. of

stu

den

ts

I thought the confidence-based test gave me valuable feedback about my knowledge

0

3

6

9

12

15

18

1 2 3 4 5

Relative agreement

No.

of s

tude

nts

Student comments

“It is a real good interesting test and I can receive different aspects of my knowledge. However, it will cost lots of time to finish this exam. As a result, I suggest we can do the exam at a regular time such an once a week.”

“It is an interesting test and I would like to do it at my university. I think that is obviously fair for students.”

“It is interesting and a little difficult to understand the result. I think I should get a higher confidence-based score because I chose (c3) nine times, and the number answered correct is 7. Maybe I have not understood this well.”

“I think it is quite interesting and helpful.it is also a good way to show me the link between confidence and academic study.”

Confidence and academic self-concept

This system asks learners to rate their confidence. Therefore learners are being assessed on their perceived confidence.

Does it matter in practice ? perhaps not important for assessing a student’s usable knowledge

important for informed interventions for enhancing academic self-concepto new research making use of recent advances in theory Craven (1996)o scope for applying confidence-based feedback in primary/early secondary

educationo exploit the reciprocal effects of academic self-concept and achievement

It may be of interest to examine the aspects of self-perception (self-concept) and how they relate to academic achievement.

As a staff developer encouraging uptake, it would be useful for me to give some indication of the factors that might be influencing a student’s decision to select a confidence level.o some non-academic self-concept ?o general academic self-concept ?o subject specific self-concept ?

A model for self-concept

Shavelson, Hubner and Stanton (1976)

Numerous studies now based or extended from this model Hattie (1992)

Posited to be multifaceted and hierarchical in nature

Self-concept and academic achievement

From the literature…

Support for the multifaceted nature….o academic self-concept and academic achievement strongly correlated

Marsh, Byrne and Shavelson (1988), Hattie (1992)

o little correlation between social self-concepts and academic achievementSong & Hattie (1984), Waugh (1999)

Support for the hierarchical nature less clear cut…. o more support for a hierarchical model for adolescents and

a unitary structure for younger children Hattie (1992)

o achievement and academic self-concept deserve special interpretations at the level of specific subjects Marsh (1990)

o standard self-description questionnaires based on the Shavelson model for preadolescents, adolescents and late adolescents Marsh (1992a, b, c)

Measuring academic self-concept

1. Your perceptions of your ability and achievementadapted from Marsh (1992), Song & Hattie (1984) and Waugh (2001)

indicate your relative agreement with each of the 20 statements:All the time, or nearly all the timeMost of the timeSome of the timeNone of the time, or almost none of the time

Questionnaire feedbackFor your general university experience, your responses total 18 points (30 maximum).For your English Language classes experience, your responses total 19 points (30 maximum).

1st 10 general university experience eg. I am capable of getting good marks at university

I am proud of my achievements at university

2nd 10 experience of English Language classes only eg. I am sure of myself in English Language classes

I am achieving at a high level in English Language classes

Measuring academic self-concept

2. Your attitudes to academic lifelocus of control inventory adapted from Trice (1985)

select True or False to indicate your agreement with each statement:

eg. My academic marks most often reflect the effort I put into classes. I came to university because it was expected of me

The closer your score is to 0, the more you believe that your academic experience is determined by your own abilities, efforts and attitudes (internal factors).

The closer your score is to 28, the more you believe that your academic experience is determined by external factors such as chance, other people, fate or luck.

Questionnaire feedback

Correlating academic self-concept, confidence and achievement

Questionnaire responses

Perception of ability & achievement

General EFL

EFL 0.61 (p=0.000) -

Locus of control - 0.54 (p=0.002) - 0.61 (p=0.000)

perception of ability and achievement scores (general and EFL scales), increase in tandem.

as locus scores become increasingly internal (towards 0), perception of ability and achievement scores increase accordingly

o General perception 19 ± 4.6o EFL perception 19 ± 4.7o Locus of control 12.2 ± 4.2

Gender observations in the questionnaire responses

Scores from questionnaire:

Females tended to respond with lower scores than males for the perception of ability and achievement (general and EFL) scales.

The responses indicating the most internal of locus of control scores, were male students.

Perception of ability & achievement - Generally

0

2

4

6

8

10

10-14 15-19 20-24 25-30

Range of scores (max is 30)

No.

of s

tude

nts

Male

Female

Perception of ability & achievement - EFL

0

2

4

6

8

10

10-14 15-19 20-24 25-30

Range of scores (max is 30)

No. o

f stu

dent

s

Male

Female

Locus of control

0

1

2

3

4

5

6

0-7 8-11 12-14 15-18 19-22

Range of scores(0 <------ internal : external ------> 28)

No.

of s

tude

nts

Male

Female

Correlating academic self-concept, confidence and achievement

Questionnaire response v. IELTS exam

Perception of ability & achievement

Locus of controls General EFL

insignificant insignificant 0.43 (p=0.024)

Student’s perception of ability and achievement in EFL shows a moderate correlation with their IELTS exam result.o in line with academic self-concept and achievement research – strongly

subject based.

No significant gender differences in the confidence-based test scores.

Females attained higher scores in the IELTS exam.

Correlating academic self-concept, confidence and achievement

Questionnaire responses v. confidence profile

Perception of EFL ability

Perception of General ability

Locus of control

C=3 (high)

C=3 (high) and

0.38 (p=0.035) 0.35 (p=0.052) insignificant

insignificantC=2 (mid)

C=2 (mid) and insignificant

-0.39 (p=0.032)

-0.37 (p=0.038)

C=1 (low)

C=1 (low) and insignificant

Perception of EFL ability & achievement scores are moderately correlated with the number of high confidence levels selected.

These figures may indicate a trend, but may be noise

o one confidence-based test is insufficient to identify genuine correlationso there is little correlation between the confidence-based test and the

IELTS results (true measure of achievement)

Future work

Provide a series of confidence-based tests for EFL studentso Deliver a system of numerous randomised tests (numerical and reading) o Provide a visual ‘confidence profile’

Enhancing self-concept may positively affect other desirable academic behaviours such as persistence on academic tasks, academic striving behaviours, self-worth, self-efficacy, self-attributions in success and failure situations and academic achievement.Craven (1996)

Examine academic self-concept , achievement and confidence-based assessmento Focus on younger learners who may enjoy a game perspectiveo Harness the confidence-based feedback for academic self-concept

enhancement interventionso The testing and feedback may dovetail with the reciprocal relationship

between academic self-concept and positive academic behaviours.

Bibliography

1. Brookover, W.B., Thomas, S. and Paterson A. (1964) Self-concept of ability and school achievement, Sociology of Education, 37, pp. 271-279.

2. Craven, R. (1996) Enhancing Academic Self-Concept: A Large-Scale Longitudinal Study in an Educational Setting, PhD thesis, University of Sydney

3. Davies, P. (2002) There’s no confidence in multiple-choice testing, Proceedings of the 6th International CAA conference, Loughborough, pp. 119-130.

4. Echternacht, G.J. (1972) The use of confidence testing in objective tests, Review of Educational Research, 42:2, pp. 217-237.

5. Gardner-Medwin, A.R. and Gahan M. (2003) Formative and summative confidence-based assessment, Proceedings of the 7th International CAA conference, Loughborough, pp. 147-155.

6. Gourlay, L. (2004) Crossing Boundaries: A Case Study Masters Level Chinese Students, LTSN (In Press)

7. Hassmen, P. and Hunt, D.P. (1994) Human self-assessment in multiple-choice testing, Journal of Educational Measurement, 31, pp. 149-160.

Bibliography

8. Hattie, J. (1992) Self-Concept, Lawrence Erlbaum Associates, New Jersey

9. Khan, K.S., Davies, D.A. and Gupta, J.K. (2001) Formative self assessment using multiple true-false questions on the Internet: feedback according to confidence about correct knowledge. Medical Teacher, 23, pp. 158-163.

10. Marsh, H.W (1992a) Self-Description Questionnaire (SDQ)III: A theoretical and empirical basis for the measurement of multiple dimensions of preadolescent self-concept: A test manual and research monograph, MacArthur, New South Wales, Australia, University of Western Sydney, Faculty of Education.

11. Marsh H.W., Byrne, B.M. and Shavelson, R.J. (1988) A multifaceted academic self-concept; Its hierarchical structure and its relation to academic achievement. Journal of Educational Psychology, 80, 366-380.

12. Marsh H.W. (1990b) The structure of academic self-concept: The Marsh/Shavelson model, Journal of Educational Psychology, 82, 623-636.

13. Prendergast, M.A. and Binder D.M. (1975) Relationships of selected self-concept and academic achievement measures, Measurement and Evaluation Guidance, 8, pp. 92-95.

Bibliography

14. Song, I.S. and Hattie, J. (1984) Home environment, self-concept and academic achievement measures, Journal of Educational Psychology, 76, pp. 1269-1281.

15. Waugh, R.F. (2001) Measuring ideal and real self-concept on the same scale, based on a multifaceted, hierarchical model of self-concept, Educational and Psychological Measurement, 61:1, pp. 85-101.