assessing quantitative reasoning in student writing: a quirky experience
Post on 12-Jan-2016
30 Views
Preview:
DESCRIPTION
TRANSCRIPT
Assessing Quantitative Assessing Quantitative Reasoning in Student Reasoning in Student
Writing: A QuIRKy ExperienceWriting: A QuIRKy Experience
Nathan D. GraweNathan D. GraweCarleton CollegeCarleton College
Serc.carleton.edu/quirkSerc.carleton.edu/quirk
With support from the US Department of Education’s Fund for the Improvement of Post-With support from the US Department of Education’s Fund for the Improvement of Post-Secondary Education, the National Science Foundation, and the WM Keck Foundation.Secondary Education, the National Science Foundation, and the WM Keck Foundation.
What is QR?What is QR?
What is QR?What is QR?
The habit of mind to consider the The habit of mind to consider the power and limitations of quantitative power and limitations of quantitative evidence in the evaluation, evidence in the evaluation, construction, and communication of construction, and communication of arguments in public, professional, arguments in public, professional, and personal life.and personal life.
What is QR?What is QR?
Four facets of QR:Four facets of QR:
1) 1) QR requires a QR requires a basic skill setbasic skill set
What is QR?What is QR?
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) 2) QR demandsQR demands application in context application in context
"The test of numeracy, as of any literacy, is "The test of numeracy, as of any literacy, is whether a person naturally uses appropriate whether a person naturally uses appropriate
skills in many different contexts" skills in many different contexts"
-National Council on Education and the -National Council on Education and the Disciplines (2001) Disciplines (2001)
What is QR?What is QR?
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) QR demands2) QR demands application in context application in context
3) 3) QR involves QR involves argumentargument
What is QR?What is QR?
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) QR demands2) QR demands application in context application in context
3) 3) QR involves QR involves argumentargument
““Deploying numbers skillfully is as Deploying numbers skillfully is as important to communication as important to communication as deploying verbs.”deploying verbs.”
-Max Frankel, -Max Frankel, The New The New York Times MagazineYork Times Magazine
What is QR?What is QR?
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) QR demands2) QR demands application in context application in context
3) 3) QR involves QR involves argumentargument
““Numbers [are] the principal language Numbers [are] the principal language of public argument.” of public argument.”
-BBC Program -BBC Program More or More or LessLess
What is QR?What is QR?
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) QR demands2) QR demands application in context application in context
3) QR involves 3) QR involves argumentargument
4) 4) QR is a QR is a habit of mindhabit of mind
““[QR] is not a discipline but a way of [QR] is not a discipline but a way of thinking….” thinking….” -Lynn -Lynn SteenSteen
Achieving Quantitative LiteracyAchieving Quantitative Literacy
What do the numbers show?What do the numbers show?
What do the numbers show?What do the numbers show?
How representative is that? How representative is that?
Compared to what?Compared to what?
How was the variable How was the variable operationalized?operationalized?
Who is in the Who is in the measurement sample?measurement sample?
Registered Voters
Eligible Voters
2006 Midterm Elections
Is the outcome statistically Is the outcome statistically significant?significant?
What’s the effect size?What’s the effect size?
What’s the effect size?What’s the effect size?
Consider two statements:Consider two statements: A) A) With a p<.001 With a p<.001 we can conclude that whites we can conclude that whites
in MN, WI, and IA were in MN, WI, and IA were more likelymore likely to vote for to vote for Obama than whites in LA, MS, and ALObama than whites in LA, MS, and AL
B) The fraction of whites who voted for B) The fraction of whites who voted for Obama in MN, WI, and IA was Obama in MN, WI, and IA was at least 35 at least 35 percentage points higherpercentage points higher than that in LA, MS, than that in LA, MS, and AL (<15% vs. >50%) –forget the p value!and AL (<15% vs. >50%) –forget the p value!
What’s the research design What’s the research design (correllational or (correllational or experimental)?experimental)?
Fact: Those who work with Fact: Those who work with computers earn 15-20% more than computers earn 15-20% more than others.others.
““Thus, computer training may, at Thus, computer training may, at least in the short run, be a profitable least in the short run, be a profitable investment for public and private job investment for public and private job training programs.”training programs.”
What’s the research design What’s the research design (correllational or (correllational or experimental)?experimental)?
Other interesting returns:Other interesting returns:
Calculator = 12.8%Calculator = 12.8%
Telephone = 11.4%Telephone = 11.4%
Pencil/Pen = 11.2%Pencil/Pen = 11.2%
Work while sitting = 10.1%Work while sitting = 10.1%
Controlling for what?Controlling for what?
Controlling for what?Controlling for what?
• About 50% of the differential has to do with About 50% of the differential has to do with different career choices….different career choices….
• 25% percent involves greater time women 25% percent involves greater time women spend on care-taking…. spend on care-taking….
• The other 25 percent is due to bias and The other 25 percent is due to bias and prejudice….”prejudice….”
What is QR?What is QR?
“…“…sophisticated reasoning with sophisticated reasoning with elementary mathematics more than elementary mathematics more than elementary reasoning with elementary reasoning with sophisticated mathematics.”sophisticated mathematics.”
-Lynn Steen-Lynn Steen
Achieving Quantitative LiteracyAchieving Quantitative Literacy
Implications for AssessmentImplications for Assessment
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) QR demands2) QR demands application in context application in context
3) QR involves 3) QR involves communicationcommunication
4) QR is a 4) QR is a habit of mindhabit of mind
Implications for AssessmentImplications for Assessment
““[QR] is largely absent from our [QR] is largely absent from our current systems of assessment and current systems of assessment and accountability.” accountability.”
-NCED (2001)-NCED (2001)
Implications for AssessmentImplications for Assessment
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) QR demands2) QR demands application in context application in context
3) QR involves 3) QR involves communicationcommunication
4) QR is a 4) QR is a habit of mindhabit of mind
Implications for AssessmentImplications for Assessment
Four facets of QR:Four facets of QR:
1) QR requires a 1) QR requires a basic skill setbasic skill set
2) QR demands2) QR demands application in context application in context
3) QR involves 3) QR involves communicationcommunication
4) QR is a 4) QR is a habit of mindhabit of mind
Implications for AssessmentImplications for Assessment
Application in context:Application in context:
““Standardized conditions are decontextualized Standardized conditions are decontextualized by design"by design"
-Grant Wiggins-Grant Wiggins
“ “'Get Real!': 'Get Real!': Assessing Assessing for Quantitative for Quantitative Literacy ”Literacy ”
Implications for AssessmentImplications for Assessment
Communication:Communication:
While traditional assessment tools While traditional assessment tools effectively measure comprehension, effectively measure comprehension, the ability to read others’ QR the ability to read others’ QR exposition does not guarantee the exposition does not guarantee the ability to engage in the creation of ability to engage in the creation of QR arguments.QR arguments.
Implications for AssessmentImplications for Assessment
Habit of mind:Habit of mind:
““As in book literacy, evidence of students’ ability As in book literacy, evidence of students’ ability to play the messy game of the [QR] discipline to play the messy game of the [QR] discipline depends on seeing whether they can handle tasks depends on seeing whether they can handle tasks without specific cues, prompts, or simplifying without specific cues, prompts, or simplifying scaffoldsscaffolds from the teacher-coach or test designer.” from the teacher-coach or test designer.”
-Grant -Grant WigginsWiggins “ “'Get Real!': 'Get Real!': Assessing Assessing for for Quantitative Literacy”Quantitative Literacy”
Implications for AssessmentImplications for Assessment
"[QR] requires creativity in "[QR] requires creativity in assessment, since neither course assessment, since neither course grades nor test scores provide a grades nor test scores provide a reliable surrogate." reliable surrogate."
-Lynn -Lynn SteenSteen
Achieving Quantitative LiteracyAchieving Quantitative Literacy
Implications for AssessmentImplications for Assessment
"The interdisciplinary and contextual nature "The interdisciplinary and contextual nature of [QR] cries out for a cross cutting of [QR] cries out for a cross cutting approach." approach."
-Lynn Steen-Lynn SteenAchieving Quantitative LiteracyAchieving Quantitative Literacy
““We want to regularly assess student work with We want to regularly assess student work with numbers and numerical ideas numbers and numerical ideas in the field….in the field….””
-Grant Wiggins-Grant Wiggins““'Get Real!': Assessing 'Get Real!': Assessing
for Quantitative Literacy”for Quantitative Literacy”
The QuIRK RubricThe QuIRK Rubric
QuIRK’s idea:QuIRK’s idea:
Measure QR in the “natural” context Measure QR in the “natural” context of papers written in courses across of papers written in courses across the curriculum—papers written for the curriculum—papers written for “authentic” purposes.“authentic” purposes.
The QuIRK RubricThe QuIRK Rubric
Sophomore Writing PortfolioSophomore Writing Portfolio
3-5 papers plus reflective essay written in 3-5 papers plus reflective essay written in 2 of the 4 college divisions 2 of the 4 college divisions
-observation-observation
-analysis-analysis
-interpretation-interpretation
-documented sources -documented sources
-thesis-driven argument -thesis-driven argument
The QuIRK RubricThe QuIRK Rubric
Writing requirement assessment:Writing requirement assessment:30 faculty members, 3 days, 450 30 faculty members, 3 days, 450 portfoliosportfolios
QR assessment:QR assessment:6-8 faculty members, 3 days, random 6-8 faculty members, 3 days, random sample of portfolios, 1 paper each sample of portfolios, 1 paper each drawn from analysis, interpretation, or drawn from analysis, interpretation, or observation categoriesobservation categories
The QuIRK RubricThe QuIRK Rubric
1. A little book-keeping1. A little book-keeping
The QuIRK RubricThe QuIRK Rubric
2. Did the student take the paper in 2. Did the student take the paper in a QR-relevant direction?a QR-relevant direction?
The QuIRK RubricThe QuIRK Rubric
Central Use:Central Use:
Use of numbers to address a central Use of numbers to address a central question, issue, or themequestion, issue, or theme
Peripheral Use:Peripheral Use:
Use of numbers to provide useful Use of numbers to provide useful detail, enrich descriptions, present detail, enrich descriptions, present background, or establish frames of background, or establish frames of referencereference
The QuIRK RubricThe QuIRK Rubric
The importance of the periphery:The importance of the periphery:““Even for works that are not inherently Even for works that are not inherently quantitative, one or two numeric facts quantitative, one or two numeric facts can help convey the importance or can help convey the importance or context of your topic.”context of your topic.”
-Jane -Jane Miller Miller
The Chicago Guide to The Chicago Guide to Writing About NumbersWriting About Numbers
The QuIRK RubricThe QuIRK Rubric
3. If relevant, to what extent did the 3. If relevant, to what extent did the student actually use QR?student actually use QR?
The QuIRK RubricThe QuIRK Rubric
4. If present, rate the holistic quality 4. If present, rate the holistic quality of the QR.of the QR.
The QuIRK RubricThe QuIRK Rubric
5. Spotlight on several recurring 5. Spotlight on several recurring problems:problems:
The QuIRK RubricThe QuIRK Rubric
6. Does assignment explicitly call for 6. Does assignment explicitly call for QR?QR?
The QuIRK RubricThe QuIRK Rubric
Inter-Rater Agreement StatisticsInter-Rater Agreement Statistics
QR Relevance:QR Relevance: 75%75%
QR Extent:QR Extent: 82%82%
QR Quality:QR Quality: 67%67%
Problem Characteristics:Problem Characteristics:68%68%
Examples from Student WorkExamples from Student Work
Example of QR-irrelevant paper:Example of QR-irrelevant paper:
““The Maiden who Needs No The Maiden who Needs No Saving”—an analysis of Keat’s Saving”—an analysis of Keat’s treatment of helplessness and power treatment of helplessness and power in “La Belle Dame Sans Merci”in “La Belle Dame Sans Merci”
Examples from Student WorkExamples from Student Work
Example of ineffective peripheral QR Example of ineffective peripheral QR paper:paper:
Title: UntitledTitle: Untitled
Thesis: A synthesis of 3 alternative Thesis: A synthesis of 3 alternative views on poverty provides a more views on poverty provides a more complete picture than any 1 complete picture than any 1 perspective alone perspective alone
Examples from Student WritingExamples from Student Writing
$192 billion $60 billion1/65th of GNP 1/230th of GNP
Examples from Student WorkExamples from Student Work
Example of effective peripheral QR Example of effective peripheral QR paper:paper:
Title: “Les Banlieues d’Islam”Title: “Les Banlieues d’Islam”
Thesis: Thesis:
Examples from Student WorkExamples from Student Work
Example of ineffective centrally QR Example of ineffective centrally QR paper:paper:
Title: UntitledTitle: Untitled
Thesis: Based on data from a time Thesis: Based on data from a time diary and other sources, the American diary and other sources, the American quality of life is better than that in quality of life is better than that in less-developed societies.less-developed societies.
Examples from Student WorkExamples from Student Work
Example of ineffective centrally QR Example of ineffective centrally QR paper:paper:
““Day Care and Development”Day Care and Development”
Topic: Literature review of effects of Topic: Literature review of effects of day care participation on child day care participation on child developmentdevelopment
Examples from Student WorkExamples from Student Work
Even though quantitative evidence is Even though quantitative evidence is referenced extensively, student referenced extensively, student doesn’t appreciate the real power of doesn’t appreciate the real power of the actual numbers:the actual numbers:
Ex: “The amount of time that Ex: “The amount of time that children spent in day-care was children spent in day-care was positively correlated with reports of positively correlated with reports of how many friends they had.”how many friends they had.”
Examples from Student WorkExamples from Student Work
Example of effective centrally QR paper:Example of effective centrally QR paper:
Title: “Modeling the Implicit Learning of Title: “Modeling the Implicit Learning of Language Acquisition”Language Acquisition”
Thesis: Even if people aren’t aware of Thesis: Even if people aren’t aware of the systems undergirding language, the systems undergirding language, they implicitly learn the system through they implicitly learn the system through experience.experience.
Examples from Student WorkExamples from Student Work
““In phase two, the mean number of correct string In phase two, the mean number of correct string classifications the subjects gave was 17.3 out of 24; all classifications the subjects gave was 17.3 out of 24; all but two participants correctly identified more than 12 but two participants correctly identified more than 12 strings, and the mode number of correct responses strings, and the mode number of correct responses was 18….”was 18….”
[Discussion of statistical test.][Discussion of statistical test.]
““It was expected that the participants would be able to It was expected that the participants would be able to correctly classify more than 12 of the 24 strings in correctly classify more than 12 of the 24 strings in phase two, so the result is not especially surprising. phase two, so the result is not especially surprising. Still, the data provide a striking example of implicit Still, the data provide a striking example of implicit learning. ”learning. ”
Infusing QR Across the Infusing QR Across the CurriculumCurriculum
“…“…numeracy is not something mastered in a numeracy is not something mastered in a single course….Thus quantitative material needs single course….Thus quantitative material needs to permeate the curriculum, … so that students to permeate the curriculum, … so that students have opportunities to practice their skills and see have opportunities to practice their skills and see how useful they can be in understanding a wide how useful they can be in understanding a wide range of problems.” range of problems.”
-Derek Bok (2006)-Derek Bok (2006)
“…“…authentic and enduring learning…can rarely authentic and enduring learning…can rarely succeed one course at a time.”succeed one course at a time.”
-Lee Shulman (1997)-Lee Shulman (1997)
Infusing QR Across the Infusing QR Across the CurriculumCurriculum
““If [QR] remains the responsibility If [QR] remains the responsibility solely of mathematics departments—solely of mathematics departments—especially if it is caged into a single especially if it is caged into a single course such as ‘Math for Liberal course such as ‘Math for Liberal Arts’—students will continue to see Arts’—students will continue to see [QR] as something that happens only [QR] as something that happens only in the mathematics classroom.”in the mathematics classroom.”
-Lynn -Lynn SteenSteen
Achieving Quantitative LiteracyAchieving Quantitative Literacy
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are centrallycentrally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio) LegendGreen = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are centrallycentrally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio) LegendGreen = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are centrallycentrally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio) LegendGreen = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are centrallycentrally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio) LegendGreen = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are centrallycentrally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio) LegendGreen = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are peripherallyperipherally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio)Legend
Green = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are peripherallyperipherally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio)Legend
Green = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are peripherallyperipherally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio)Legend
Green = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are peripherallyperipherally relevant papers relevant papers
written? written? ((Population: All papers submitted to writing
portfolio)Legend
Green = Math & Natural SciencesBlue = Social SciencesRed = Other
Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant
Infusing QR Across the Infusing QR Across the CurriculumCurriculum
• 30% of gen ed papers are centrally 30% of gen ed papers are centrally QR relevantQR relevant• Of these, nearly 50% are outside natural Of these, nearly 50% are outside natural
sciencessciences
• Another 25% are peripherally soAnother 25% are peripherally so• Of these, 90+% are Of these, 90+% are outsideoutside natural natural
sciences and more than half are in Arts, sciences and more than half are in Arts, Lit, & HumanitiesLit, & Humanities
Infusing QR Across the Infusing QR Across the CurriculumCurriculum
By QR relevance, what fraction of By QR relevance, what fraction of papers use QR little/none, some, and papers use QR little/none, some, and extensively?extensively?
Centrally Relevant
Peripherally Relevant Either
Little/None 19.5% 94.7% 52.1%
Some 17.1% 5.3% 12.0%
Extensive 63.4% 0.0% 35.9%
Infusing QR Across the Infusing QR Across the CurriculumCurriculum
““Weasely” examples:Weasely” examples:A) “The downtown areas of large cities A) “The downtown areas of large cities are often home to copious franchises.”are often home to copious franchises.”
B) Recall Daycare PaperB) Recall Daycare Paper
Our Curricular ResponseOur Curricular Response
Our Curricular ResponseOur Curricular Response
Our Curricular ResponseOur Curricular Response
Our Curricular ResponseOur Curricular Response
Our Curricular ResponseOur Curricular ResponseDisciplines Currently Disciplines Currently RepresentedRepresented::
Our Curricular ResponseOur Curricular ResponseComing Soon!:Coming Soon!:
Our Curricular ResponseOur Curricular ResponseExample 1: An Ill-Structured Physics LabExample 1: An Ill-Structured Physics Lab
Example 2: Intro to Lintina/o Studies PaperExample 2: Intro to Lintina/o Studies Paper
Example 3: History Paper on India’s Example 3: History Paper on India’s Colonial CensusColonial Census
Example 4: Exploring and Architectural Example 4: Exploring and Architectural RemodelRemodel
Our Curricular ResponseOur Curricular ResponseExample 5: Environmental History PaperExample 5: Environmental History Paper
Example 6: Development Economics Example 6: Development Economics PaperPaper
Example 7: Native-American Literature Example 7: Native-American Literature AnalysisAnalysis
Our Curricular ResponseOur Curricular Response
top related