1 bacon – t. a. webinar – 7 march 2012 transforming assessment with adaptive questions dick...

1Bacon – T. A. Webinar – 7 March 2012

Transforming Assessment with Adaptive Questions

Dick BaconDepartment of PhysicsUniversity of Surrey

[email protected]

Using assessments to encourage learning



Effective feedback

Adaptive questions

Question design

‘Help’ design

Results and discussion

Conclusions

Student opinions

Context


Context

Four level 1 electronics tests containing 27 questions, that are set as coursework

The questions have been used in SToMP assessment systems (this is the fourth) since 1995.

The latest system SToMP-II is QTI v2.1 compliant, and this paper discusses the deployment of 15 of these questions that have been modified to improve the effectiveness of their feedback.


www.stomp.ac.uk


To be effective, feedback must be

This can all be achieved within one QTI v2.1 adaptive question

► targeted( use mal rules )

► relevant to student’s probable understanding( use history )

► attended to( give it immediately but randomise the data )

► applied( offer another try )

freely adapted from Gibbs and Simpson (2004)



v2 includes support for adaptive questions.

A specification for the transfer of assessment questions between different testing systems.

The most widely implemented version is v1.2.1, which was published in March 2003.

v2.0 was published in May 2005. The structure is similar, but the detail is completely different.

Support for numerical questions is much improved in v2, including randomisation and the testing of both accuracy and precision.

QTI : Question Test Interoperability


Adaptive questions allow a dialogue to be supported

► question text can be changed

► interactions can be changed

► feedback can be changed

When the submit button is pressed:

► further tries can be offered

!! There is no limit to the complexity !!



Show question

Show feedback for right, approximate or precision error.Adjust mark.Set score = mark.

Show feedbackaddressing mal-rule mistake.Reduce mark.

right?mal-rule?

Show feedback for unrecognised error.Reduce mark.

mark > 3yesyes no

no

Show final ‘wrong’ feedback.Set score = 0

no

yes

Combine these for a multiple numeric response question . .

Single numeric response question

. . blanking the correct responses.

Set mark = 10



Help was somehow to be provided for the student who did not know how to start the problem.

For clarity, this was offered as a separate entity.

Giving help

We also wanted to give the student some control over the amount of detail that was to be given, i.e. the level of help that was required.




Q 3.2


Describe how to tackle the problem [-2 marks] . .

Show how a solution can be obtained [-4 marks] . .

Split the problem into smaller parts [-6 marks] . .

discusses the principles involved in obtaining a solution

describes in more detail a process by which a solution can be obtained

breaks the problem down, getting the student to solve the components until the solution is obtained. Then it usually re-poses the original problem.



no

yes

yes

Explain nth part and pose question.

Right ? 1st time ?Explain in simpler terms and usually give a numeric expression for the answer

Address the student’s inability to answer the question, but leave remediation to the student. Then continue to next part.

Dealing with feedback for the problem-parts help.

no



In a five option MCQ, the probability of getting it right in three tries and thus gaining some marks without knowing the subject at all, is 0.7.

Thus, except for cases where there are many more options, a “don’t know” option can be the more attractive way of encouraging a student to engage.

Feedback can be made more detailed, or can offer a different viewpoint, each time ‘don’t know’ is selected. As the amount of help increases, the final mark is reduced.

Other question types




Two questions used the “don’t know” model(one single choice, one multiple selection)

Between them,59 right7 part right2 part wrong2 wrong2 “don’t know”s, then wrong



Q1.2



Two questions used the “don’t know” model(one single choice, one multiple selection)

Between them,59 right7 part right2 part wrong2 wrong2 “don’t know”s, then wrong

One randomised single choice (from 6) offered multiple tries,

31 right5 right second try2 right third try3 wrong third try



Q4.4


32 right

4 mal right

4 wrong right

3 wrong wrong right

11 wrong wrong wrong wrong

4 wrong wrong

2 mal1 mal1/2 wrong wrong

2 wrong mal wrong wrong

1 mal

Q 4.4.a/b Voltage gain in LR filter




Q 3.2


16 right

2 mal-rule right

2 mal-rule mal-rule right

1 wrong right

2 wrong wrong right

1 mal-rule mal-rule wrong

1 wrong wrong wrong

1 wrong wrong help 1 right

1 wrong help 1 right

1 help 1 mal-rule wrong right

1 wrong wrong help 2 wrong

1 help 2 wrong wrong

1 help 3 right

1 help 3 mal-rule right

Charges on two capacitors in series

Multiple answers, with mal-rules and feedback



questions with mal-rule feedback (+332 right)

24%

22%

10%

8%

15%

21% mal-to-right

wrong-to-right

multi-wrong-right

multi-mix-right

multi-mix-wrong

multi-wrong

ended right (196)

ended wrong (109)This chart

refers only to those who would have scored zero.



questions with mal-rule and help fback (+71 right)

30%

6%

13%11%

13%

6%

1%

20%wrong-to-right

mal-to-right

multi-mix-right

help-to-right

help-mix-right

help-mix-wrong

multi-mix-wrong

multi-wrong

ended right (57)

ended wrong (22)This chart

refers only to those who would have scored zero.




effects of mal-rule and wrong feedback, and help

0

10

20

30

40

50

60

70

tran

siti

on

co

un

t -

per

cen

tag

e

mal to right

mal to wrong

wrong to right

wrong to wrong

help to right

help to wrong

There were 23 instances of help being requested,156 mal-rule instances and 316 wrong instances.

156 316 23


Student comments:

“Really liked the opportunity to have another go at questions with hints if you were unsuccessful in the first attempt.”

Best feature ?

“Giving hints if the answer entered is wrong.”

“The hints given as to how to do the question if you got it wrong in the first instance, and the chance to have another go, albeit for less marks.”

“the hints - they often nudge my attention to a key fact”

“the overall presentation was clear and easy to understand what was required of us”

“the ability to see your mark immediately afterwards “



Student comments:

Worst feature ?

“Difficult to know how to enter data with powers.”

“would like a marks summary”

“The inability to review closed questions.”

“sometimes the hints when you got it wrong were totally irrelevant”

“Not very reliable - we experienced a few crashes and disruptments with the system over the year”




With multiple randomised variables, two different mal rules may well have very similar values, and the condition used for mal rule testing only checked within 5% (to cater for precision and rounding effects).

I have addressed this by checking for all mal rules when marking, and presenting all appropriate feedback for the mal rules that have matching values.




Otherwise, the feedback may indeed be badly written – so as to be unrecognisable by students with particular misconceptions, for example.

This will need a detailed analysis of what a student needs to know to be able to solve the problem (concept mapping).

It might be addressed by starting a dialogue (within the feedback) to establish what it is that each student needs to be told.



Conclusions:

► looking for more mal-rules

► improving some of the mal-rule feedback text

► making ‘wrong’ feedback progressive, or interactive

2. That out of a total of 758 student responses, 205 were corrected by a combination of targeted and general feedback and online help, leaving only 111 as incorrect.

1. The questions need further development, specifically . .

► improving some of the help text

► providing more mark information for users ?



The end.

If you are interested in getting involved in creating or using these sorts of questions, then do contact me: [email protected]

If you are interested in seeing what is available in QTI v2.1 systems,Then have a look at the JISC project that has been collecting resources and helping finalise the current specification,

http://cloud.niallbarr.me.uk/ .

To find out more about the SToMP system, look athttp://www.stomp.ac.uk .

To locate e-questions for a variety of assessment systems (including QTI v1 & 2) look at the Physical Sciences Question Bank,

http://www.heacademy.ac.uk/physsci/ (follow Q for Question bank) or (better) from a link in the SToMP site.

(The HEA site is a legacy site, but the SToMP site is maintained)


1 bacon – t. a. webinar – 7 march 2012 transforming assessment with adaptive questions dick...

Documents