blueprinting: deciding what to measure, liz norman 2014
DESCRIPTION
Blueprinting: deciding what to measure, Institute of Veterinary, Animal and Biomedical Sciences Teaching and Learning Workshop, July 3, 2014, Massey University, Palmerston North, New ZealandTRANSCRIPT
![Page 1: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/1.jpg)
Blueprinting
Liz Norman
Massey University
![Page 2: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/2.jpg)
Blueprinting: planning the sample
Breadth or scope
Depth or level
Controlling difficulty and demand
![Page 3: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/3.jpg)
What is a blueprint?
"blueprint, n.". Something which acts as a plan,
model, or template
OED Online http://www.oed.com
Railway and harbour report, Vancouver, B.C.
by City of Vancouver Archives, Attribution License
![Page 4: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/4.jpg)
What is a blueprint?
Specifies what’s going to be in the exam
– Content/topics – breadth of the exam
– Level - depth of the exam
Railway and harbour report, Vancouver, B.C.
by City of Vancouver Archives, Attribution License
![Page 5: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/5.jpg)
Why we need to blueprint
• Its important that we sample representatively
from the content domain (all that it is possible to
examine)
• This is so we can extrapolate performance to the
whole content domain
![Page 6: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/6.jpg)
Purpose of blueprinting
To document the sampling.
To show that a particular examination is representative ofall that could be examined and is sufficient.
![Page 7: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/7.jpg)
Breadth by learning outcome
Lit review
assignment
Group
presentation
Written
examination
Practical
examination
LO1 X X X
LO2 X X
LO3 X X X
LO4 X
![Page 8: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/8.jpg)
Breadth by topic
PathophysiologyInvestigation and
diagnosis
Treatment and
management
Gastrointestinal P1Q1 P1Q1, P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
![Page 9: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/9.jpg)
Breadth across species
species number of Qs percent
small animal 15 52%
farm 5 17%
horse 4 14%
exotic 3 10%
lab 1 3%
all 1 3%
![Page 10: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/10.jpg)
Level - depth
• Surface – deep
• Fact recall – applied
• Blooms taxonomy
• SOLO taxonomy
![Page 11: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/11.jpg)
Fact recall vs applied
Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)
Applied (higher order)Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact
From: Peitzman et al. (1990). Academic Medicine, 65(9), S59-60.
![Page 12: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/12.jpg)
create
evaluate
analyse
apply
understand
rememberRecall
Application
Blooms taxonomy
![Page 13: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/13.jpg)
Blooms instructional verbs
• Create: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare.
• Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate
• Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise.
• Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch.
• Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell
• Remember: define, repeat, record, list, recall, name, relate, underline.
![Page 14: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/14.jpg)
SOLO taxonomy
http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/
![Page 15: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/15.jpg)
Prestructural Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149
![Page 16: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/16.jpg)
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
Quantitative change
Qualitative change
![Page 17: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/17.jpg)
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
surface
deep
![Page 18: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/18.jpg)
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
recall
application
![Page 19: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/19.jpg)
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple
instructions, calculate, reproduce, arrange, recognise, find, note,
seek, sketch, pick
Multistructural Combine, classify, structure, describe, enumerate, list, do
algorithm, apply method, account for execute, formulate, solve,
conduct, prove, complete, illustrate, express, characterise
Relational Analyse, compare, contrast, integrate, relate, explain causes,
apply theory (to its own domain), argue, implement, plan,
summarize, construct, design, interpret (some senses), structure,
conclude, substantiate, exemplify, derive, adapt
Extended
abstract
Theorise, generalise, hypothesise, predict, judge, transfer theory
(to new domain), assess, evaluate, interpret (some senses),
critically reflect, predict, criticise, reason
Potter & Kustra (2012). http://www1.uwindsor.ca/ctl/system/files/PRIMER-on-Learning-Outcomes.pdf
![Page 20: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/20.jpg)
Topic 130%
Topic 230%
Topic 340%
total
Knowledge of terms 2 5 5 12
Comprehension of principles 4 3 4 11
Application of principles 3 3 3 9
Analysis of situations 3 2 5 10
Evaluation of solutions 3 2 3 8
Total questions 15 15 20 50
In this example
• Topic 1 represents 30% of the whole curriculum being assessed
so 15 of 50 questions are to address it.
• Four questions are to address comprehension of the principles
of Topic 1.
Scoring office, Michigan State University, Writing Test Items, http://scoring.msu.edu/writitem.html
![Page 21: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/21.jpg)
Blueprints that include depth
PathophysiologyInvestigation and
diagnosis
Treatment and
management
recallhigher
orderrecall
higher
orderrecall
higher
order
Gastrointestinal P1Q1P1Q1,
P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
NervousP1Q2,
P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
![Page 22: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/22.jpg)
Blueprint
Enables you to plan and check that an exam, or
set of exams, covers the content it is meant to
cover, at an appropriate level.
Without this you can’t extrapolate performance on
this exam to performance in the whole domain
= one component of validity
![Page 23: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/23.jpg)
Controlling difficulty and demand
![Page 24: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/24.jpg)
Should questions be difficult?
Norm-referenced vs
standards-based
grading
![Page 25: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/25.jpg)
Should questions be difficult?
Appropriate demand
vs irrelevant difficulty
![Page 26: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/26.jpg)
What makes questions difficult?
• Familiarity – novelty
• Complexity – components, links
• Scale – components, links
• Resources utilised – provided, generated
• Abstraction – concrete experience, concepts, ideas, principles, projection in time
• Task strategy – simple, stepwise, integrated
• Response strategy – simple, stepwise, integrated
• Guidance
Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the
demands of GCSE and A level exam questions. British Educational Research Association
conference, Aug 27-30 1998, Belfast.
![Page 27: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/27.jpg)
Question 1
List 4 clinical signs of diabetes mellitus in dogs.
Question 2
List 4 clinical signs of feline hypersomatotropism.
Question 3
Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism, in dogs and in cats.
![Page 28: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/28.jpg)
What makes questions difficult?
• Familiarity – novelty
• Complexity – components, links
• Scale – components, links
• Resources utilised – provided, generated
• Abstraction – concrete experience, concepts, ideas, principles, projection in time
• Task strategy – simple, stepwise, integrated
• Response strategy – simple, stepwise, integrated
• Guidance
Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the
demands of GCSE and A level exam questions. British Educational Research Association
conference, Aug 27-30 1998, Belfast.
![Page 29: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/29.jpg)
Describe the skull. (30 marks)
![Page 30: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/30.jpg)
What makes questions difficult?
• Familiarity – novelty
• Complexity – components, links
• Scale – components, links
• Resources utilised – provided, generated
• Abstraction – concrete experience, concepts, ideas, principles, projection in time
• Task strategy – simple, stepwise, integrated
• Response strategy – simple, stepwise, integrated
• Guidance
Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the
demands of GCSE and A level exam questions. British Educational Research Association
conference, Aug 27-30 1998, Belfast.
![Page 31: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/31.jpg)
Question 1
Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)
Question 2
a) Describe the advantages and disadvantages of insulin therapy for diabetes mellitus in cats (10 marks)
b) Indicate the dose and frequency of administration of insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)
c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).
![Page 32: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/32.jpg)
What makes questions difficult?
• Familiarity – novelty
• Complexity – components, links
• Scale – components, links
• Resources utilised – provided, generated
• Abstraction – concrete experience, concepts, ideas, principles, projection in time
• Task strategy – simple, stepwise, integrated
• Response strategy – simple, stepwise, integrated
• Guidance
Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the
demands of GCSE and A level exam questions. British Educational Research Association
conference, Aug 27-30 1998, Belfast.
![Page 33: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/33.jpg)
You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)
Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.
![Page 34: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/34.jpg)
What makes questions difficult?
• Familiarity – novelty
• Complexity – components, links
• Scale – components, links
• Resources utilised – provided, generated
• Abstraction – concrete experience, concepts, ideas, principles, projection in time
• Task strategy – simple, stepwise, integrated
• Response strategy – simple, stepwise, integrated
• Guidance
Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the
demands of GCSE and A level exam questions. British Educational Research Association
conference, Aug 27-30 1998, Belfast.
![Page 35: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/35.jpg)
Question 1
A 10 kg diabetic dog is being treated with 5 units of Caninsulin at 8 am each morning. It is fed Purina Dog Chow at the same time as the insulin and again at 4pm. A glucose curve is performed.
a) Examine the curve and determine the blood glucose nadir and the duration of action of the insulin.
b) Comment on whether you would expect glycosuria to be present in this dog and explain why.
(5 marks)
![Page 36: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/36.jpg)
Question 2
Illustrate the general shape of an ideal blood glucose curve for a stable diabetic dog being treated with insulin. Indicate on the curve each of the following:
a) The ideal highest and lowest points of the curve
b) The time of feeding
c) The time of insulin injection
d) The duration of action of the insulin
e) The value above which glucose would be present in the urine
(5 marks)
![Page 37: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/37.jpg)
What makes questions difficult?
• Familiarity – novelty
• Complexity – components, links
• Scale – components, links
• Resources utilised – provided, generated
• Abstraction – concrete experience, concepts, ideas, principles, projection in time
• Task strategy – simple, stepwise, integrated
• Response strategy – simple, stepwise, integrated
• Guidance
Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the
demands of GCSE and A level exam questions. British Educational Research Association
conference, Aug 27-30 1998, Belfast.
![Page 38: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/38.jpg)
What makes questions difficult?
• Familiarity – novelty
• Complexity – components, links
• Scale – components, links
• Resources utilised – provided, generated
• Abstraction – concrete experience, concepts, ideas, principles, projection in time
• Task strategy – simple, stepwise, integrated
• Response strategy – simple, stepwise, integrated
• Guidance
Adapted from Hughes, Pollitt, & Ahmed (1998) The development of a tool for gauging the
demands of GCSE and A level exam questions. British Educational Research Association
conference, Aug 27-30 1998, Belfast.
![Page 39: Blueprinting: deciding what to measure, Liz Norman 2014](https://reader034.vdocument.in/reader034/viewer/2022051516/55981f991a28abb6648b465d/html5/thumbnails/39.jpg)
http://www.slideshare.net/liznorman