1 student reflections as data dr. allyson fiona hadwin university of victoria (epls & ltc)...
Post on 20-Dec-2015
219 views
TRANSCRIPT
11
Student Reflections as DataStudent Reflections as Data
Dr. Allyson Fiona HadwinDr. Allyson Fiona Hadwin
University of Victoria University of Victoria
(EPLS & LTC)(EPLS & LTC)
[email protected]@uvic.caUBC Institute for the Scholarship of Teaching and Learning
22
Examining and Interpreting Examining and Interpreting Data Collected over TimeData Collected over Time
Student reflection dataStudent reflection data
Student interview dataStudent interview data
33
Student Reflection DataStudent Reflection Data
JournalsJournals
PortfoliosPortfolios
Reflection assignmentsReflection assignments
44
Why do you want to look at Why do you want to look at this data in the first place?this data in the first place?
What is your research question?What is your research question?
55
There are different kinds of There are different kinds of research questionsresearch questions
Introducing the language so you know Introducing the language so you know where to look beyond this coursewhere to look beyond this course
Covered in many introductory research Covered in many introductory research methods texts in educationmethods texts in education
Creswell, J. W. (2005). Educational Research: Planning, Creswell, J. W. (2005). Educational Research: Planning, conducting and evaluating quantitative and qualitative conducting and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson-Merrill Prentice research. Upper Saddle River, NJ: Pearson-Merrill Prentice Hall.Hall.
66
Develop an understanding of what students gained through the experience (THEORY GENERATION)
What are some possible models to explain student learning in this program?
INDUCTIVE
?
77
Examine students' reflections according to theoretically driven constructs
(THEORY DRIVEN)
What kinds of sensitivities and awareness developed with respect to each of these
issues: X, Y, Z?
Theory verification
?
88
Understand the essence of student experiences of X.
What is the meaning of this learning experience for students?
Phenomenology
?
99
Compare students to one another
How were these students’ experiences similar or different?
Cross case study
?
1010
Compare students to themselves at different points in time
How did students change over the course of this instructional experience?
Within case study
?
1111
What kind of question do you What kind of question do you have?have?
Bottom Up
No idea what to expect
Looking for something
Top Down
1212
What is the question?What is the question?
A characteristic of reflection data:A characteristic of reflection data:
Usually interested in higher order, complex Usually interested in higher order, complex thinkingthinking
Critical thinkingCritical thinking
Community awarenessCommunity awareness
Self-regulated learningSelf-regulated learning
Difficult to operationalize and measureDifficult to operationalize and measure
1313
CautionCaution
Data analysis and interpretation are Data analysis and interpretation are limited (and afforded) by:limited (and afforded) by:
– the kind of data you collectedthe kind of data you collected– how you collected the datahow you collected the data– your own biases and roles in the processyour own biases and roles in the process
1414
Who is doing the analysis and Who is doing the analysis and interpretation?interpretation?
ResearcherResearcher
The student The student
Collaboration between researcher and Collaboration between researcher and studentstudent
1515
Beginning steps in the Beginning steps in the analysis processanalysis process
1.1. What is the unit of analysisWhat is the unit of analysis
2.2. Organize data according to unit of Organize data according to unit of analysisanalysis
3.3. Choose an analysis approach that is Choose an analysis approach that is consistent with your research consistent with your research questionquestion
1616
Unit of AnalysisUnit of Analysis
Individual
Responses to questions or themes Timing of events
Key Events Key Processes
1717
Data Sets Data Sets
1.1. Reflections on specific questions Reflections on specific questions collected by Dr. Dana Damian (University collected by Dr. Dana Damian (University of Victoria)of Victoria)
2.2. Reflections about studying and learning Reflections about studying and learning over 3 study sessions using computer over 3 study sessions using computer based technologies for learning based technologies for learning (collected by Allyson Hadwin)(collected by Allyson Hadwin)
1818
Data setsData sets
What unit of analysis might you What unit of analysis might you choose for each data set?choose for each data set?
Explain why?Explain why?
1919
Organize dataOrganize data
Make 2 copies of everythingMake 2 copies of everything
Make sure identifier information is on Make sure identifier information is on each document (and unit)each document (and unit)
Group data into the desired unit of Group data into the desired unit of analysisanalysis
Choose an appropriate analytic Choose an appropriate analytic approachapproach
2020
Semi-structured or Semi-structured or Loosely structured?Loosely structured?
Semi-structuredSemi-structured– Responses to specific questionsResponses to specific questions– Reflections on particular themesReflections on particular themes
Loosely structuredLoosely structured– Open discussionOpen discussion– Reflections about learning without specific Reflections about learning without specific
criteriacriteria
2121
Which data set is:Which data set is:
Semi structured?Semi structured?
Loosely structured?Loosely structured?
2222
Analytic approachesAnalytic approaches
Content analysisContent analysis
Case studyCase study
Inductive analysisInductive analysis
Narrative analysisNarrative analysis
Phenomenological analysisPhenomenological analysis
Grounded theory analysisGrounded theory analysis
2323
Combining analytic Combining analytic approachesapproaches
Often these analytic approaches are Often these analytic approaches are combined.combined.
I describe each separately to give you a I describe each separately to give you a sense of the main focussense of the main focus
2424
Content AnalysisContent AnalysisCode (from theory or data)Code (from theory or data)Pull apartPull apartRe-construct according to categories and Re-construct according to categories and themesthemesSummarized numerically & with quotesSummarized numerically & with quotesBuild theory and explanationBuild theory and explanationNot very sensitive to context of statements Not very sensitive to context of statements and ideasand ideas
2525
Narrative AnalysisNarrative Analysis
Preserves the wholePreserves the wholerepresents the perspective of the “teller”represents the perspective of the “teller”Themes in contextThemes in contextResearcher voice appears in explanation Researcher voice appears in explanation and representationand representationMany of the participants words includedMany of the participants words included
2626
Case Study AnalysisCase Study Analysis
Assemble all sources data for one case Assemble all sources data for one case (reflection, interviews, etc)(reflection, interviews, etc)Construct a case record –classifying and Construct a case record –classifying and editing raw data into an accessible packageediting raw data into an accessible packageWrite a case narrative (organized Write a case narrative (organized chronologically or thematically, or both)chronologically or thematically, or both)Pattern matching (comparing theoretically Pattern matching (comparing theoretically predicted patterns to data based patterns)predicted patterns to data based patterns)Explanation building (theory building)Explanation building (theory building)Time series analysis –trends over timeTime series analysis –trends over time
2727
Inductive analysisInductive analysis
Themes emerging from the dataThemes emerging from the data
Also top down sensitizing from the Also top down sensitizing from the researcherresearcher
Movement toward typologiesMovement toward typologies
Synthesis and explanation of typologiesSynthesis and explanation of typologies
2828
Grounded Theory analysisGrounded Theory analysis
Fine-grained analysis of small units (sentences or Fine-grained analysis of small units (sentences or phrases)phrases)
description
conceptual ordering
theorizing is interplay between data and theory
asking questions and making comparisons throughout
layers of coding from micro-analysis to modelling
iterative inductive cycle with specific stages
Constant comparative approach
Saturation
2929
Phenomenological analysisPhenomenological analysis
• -examining or uncovering the essence of a experience or behaviour
• -involves exploration of personal biases• -looking for what is "real” & teasing apart judgment• -bracketing or identifying data in its pure form
1. Epochè -look inside at personal bias2. Phenomenological reduction -bracketing from the
world and all presuppositions3. Horizontalizing -data spread out for interpretation4. Structural synthesis -deeper meaning or essence of
phenomenon is revealed
3030
Important elements of analysisImportant elements of analysis
Systematic approach (open about and Systematic approach (open about and describe the approach)describe the approach)
Coding dataCoding dataDescriptionDescriptionConceptual OrderingConceptual OrderingHigher order analysis or theorizingHigher order analysis or theorizing
Strauss and Corbin (Grounded theory)Strauss and Corbin (Grounded theory)
3131
Steps for your analysisSteps for your analysis
1.1. Focus your analysisFocus your analysis
2.2. Organize your dataOrganize your data
3.3. Decide upon and justify the Decide upon and justify the appropriate way to attack or analyze appropriate way to attack or analyze the data?the data?
4.4. (coding)(coding)
5.5. Conceptually organize data (try 2 Conceptually organize data (try 2 different ways of presenting findings different ways of presenting findings to bring meaning to themes)to bring meaning to themes)
3232
CodingCoding
Marking segments of data with symbols, Marking segments of data with symbols, descriptive words, labels or namesdescriptive words, labels or namesKeep a master list of codes, their meaning, Keep a master list of codes, their meaning, examples, & non-examplesexamples, & non-examples
Theory drivenTheory drivenData drivenData driven
3333
DescriptionDescription
Emerges from coding the dataEmerges from coding the data
What does the data sayWhat does the data say
What is importantWhat is important
What are the patterns?What are the patterns?
What are basic storiesWhat are basic stories
3434
DescriptionDescription
3535
Numerical descriptionNumerical description
3636
Conceptual Ordering -logical Conceptual Ordering -logical analysisanalysis
• Creating categories, finding means for identifying and Creating categories, finding means for identifying and displaying themesdisplaying themes
• Keeping an eye toward interpretationKeeping an eye toward interpretation
• Must be careful - once you start organizing data Must be careful - once you start organizing data according to a frame it becomes difficult to let go and according to a frame it becomes difficult to let go and try something different when it is not workingtry something different when it is not working
• MatrixesMatrixes• Concept mapsConcept maps• Summary tablesSummary tables• DiagramsDiagrams
3737
Conceptual orderingConceptual ordering
3838
Data may be organized:Data may be organized:
• Time orderedTime ordered• Role orderedRole ordered• Role by time orderedRole by time ordered• Role by groupRole by group• Conceptually clusteredConceptually clustered• Site dynamics Site dynamics • Predictor - outcomePredictor - outcome• Process -outcomeProcess -outcome• see Miles & Huberman (1984) for detailssee Miles & Huberman (1984) for details
3939
Numerical analysisNumerical analysis
4040
Interpretation -TheoryInterpretation -Theory
• Focus is illumination, extrapolation, and Focus is illumination, extrapolation, and understandingunderstanding
• Does not focus on cause and Does not focus on cause and consequenceconsequence
• Going beyond the data to connect with Going beyond the data to connect with theory or to generate theory/explanationtheory or to generate theory/explanation
• Dealing with rival explanations, Dealing with rival explanations, disconfirming cases, and data irregularities disconfirming cases, and data irregularities
4141
Higher order theorizingHigher order theorizing
4242
Group activityGroup activity
What have students learned What have students learned about their learning through the about their learning through the computer based learning activitycomputer based learning activity
4343
Analyzing data activityAnalyzing data activity
Focus your analysisFocus your analysis
Organize your dataOrganize your data
Decide upon and justify the appropriate Decide upon and justify the appropriate way to attack or analyze the data?way to attack or analyze the data?
(coding)(coding)
Conceptually organize data (try 2 Conceptually organize data (try 2 different ways of presenting findings to different ways of presenting findings to bring meaning to themes)bring meaning to themes)
4444
When a theoretical frame When a theoretical frame informs your analysisinforms your analysis
Coding scheme comes from the theoryCoding scheme comes from the theory
For example, you can use a rubric to For example, you can use a rubric to analyze (and assess) deeper level thinkinganalyze (and assess) deeper level thinking
Top Down
4545
Working with theory driven Working with theory driven constructsconstructs
Defining your constructDefining your construct– From the data (inductive)From the data (inductive)– From the theory (lens for the data)From the theory (lens for the data)
RubricsRubrics
Content analysisContent analysis
4646
An Example from Critical An Example from Critical ThinkingThinking
VanGyn, G., Ford, C., & Associates VanGyn, G., Ford, C., & Associates (2005). Teaching Critical Thinking. (2005). Teaching Critical Thinking. Unpublished manuscript.Unpublished manuscript.
4747
Critical Thinking RubricCritical Thinking Rubric
Interviewed facultyInterviewed faculty
Examined Examined critical thinking critical thinking assignmentsassignments
looking for ways to define and assess looking for ways to define and assess critical thinkingcritical thinking
4848
Key Dimensions of critical Key Dimensions of critical thinkingthinking
Intellectual habitsIntellectual habits
Intellectual deliberationsIntellectual deliberations
Reflexive DispositionReflexive Disposition
4949
Intellectual habitsIntellectual habits
Intellectual curiosityIntellectual curiosity
Respect for truth and reasonRespect for truth and reason
Fair and open mindednessFair and open mindedness
Tolerance for ambiguityTolerance for ambiguity
Intellectual courage to take a positionIntellectual courage to take a position
Intellectual work ethicIntellectual work ethic
Willingness to engage in collaborative Willingness to engage in collaborative inquiryinquiry
5050
Intellectual deliberationsIntellectual deliberations
Identify the challenge, situation or taskIdentify the challenge, situation or taskGather, interpret and understand Gather, interpret and understand background information & evidencebackground information & evidenceApply thinking strategies relevant to the Apply thinking strategies relevant to the type of inquiry relevant to the challengetype of inquiry relevant to the challengeGenerate or select alternativesGenerate or select alternativesMake evaluative judgments among Make evaluative judgments among alternatives based on criteriaalternatives based on criteriaProvide justification for the conclusionProvide justification for the conclusion
5151
Reflexive dispositionReflexive disposition
Plan ahead for critical thinkingPlan ahead for critical thinking
Monitor its quality throughoutMonitor its quality throughout
Reflect on the strengths and limitations of Reflect on the strengths and limitations of the use of intellectual habits and the use of intellectual habits and intellectual deliberations in making a intellectual deliberations in making a judgmentjudgment
5252
Quality, rigor, credibilityQuality, rigor, credibility
Qualitative analysis is judged by different criteria Qualitative analysis is judged by different criteria largely governed by:largely governed by:
– Design chosenDesign chosen– Philosophical position (often represented Philosophical position (often represented
in designin design
Identify/reference the framework for credibility you Identify/reference the framework for credibility you are using *** Think of your writing as an opportunity are using *** Think of your writing as an opportunity to educate the reader about rigor in qualitative to educate the reader about rigor in qualitative researchresearch
5353
Internal validityInternal validity
Triangulation (data, perspective, theory)Triangulation (data, perspective, theory)
Member checksMember checks
Long-term observationLong-term observation
Peer examinationPeer examination
Participatory modelsParticipatory models
Exposing researcher biasesExposing researcher biases
5454
Fossey, et al. (2002)Fossey, et al. (2002)CongruenceCongruence
Responsiveness to social contextResponsiveness to social context
– Emergent research designEmergent research design– Sampling, data collection, analysisSampling, data collection, analysis
AppropriatenessAppropriateness
– SamplingSampling– Data collectionData collection
AdequacyAdequacy
– SamplingSampling– Data gathering and analysisData gathering and analysis– WritingWriting
5555
TransparencyTransparencyData collection and analysisData collection and analysis
AuthenticityAuthenticityPresentation of findings and interpretationsPresentation of findings and interpretations
CoherenceCoherencePresentation of findings and interpretationsPresentation of findings and interpretations
ReciprocityReciprocityAnalysis, findings and interpretationsAnalysis, findings and interpretations
TypicalityTypicalityWritten reportWritten report
PermeabilityPermeabilityFindings and interpretationsFindings and interpretationsWritten reportWritten report
5656
ReferencesReferencesCreswell, J. W. (2005). Educational Research: Planning, conducting and evaluating quantitative and Creswell, J. W. (2005). Educational Research: Planning, conducting and evaluating quantitative and
qualitative research. Upper Saddle River, NJ: Pearson-Merrill Prentice Hall.qualitative research. Upper Saddle River, NJ: Pearson-Merrill Prentice Hall.
Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). Understanding and evaluating Fossey, E., Harvey, C., McDermott, F., & Davidson, L. (2002). Understanding and evaluating qualitative reesarch. Australian and New Zealand Journal of Psychiatry, 36, 717-732.qualitative reesarch. Australian and New Zealand Journal of Psychiatry, 36, 717-732.
Hadwin, A. F., Hadwin, A. F., Boutara, L., Knoetze, T., & Thompson, SBoutara, L., Knoetze, T., & Thompson, S. (2004). Cross case study of self-regulation . (2004). Cross case study of self-regulation as a series of events. Educational Research and Evaluation, 10, 365-418.as a series of events. Educational Research and Evaluation, 10, 365-418.
Hadwin, A. F., Hadwin, A. F., Wozney, L., & Pontin, OWozney, L., & Pontin, O. (in press). Scaffolding the appropriation of self-regulatory . (in press). Scaffolding the appropriation of self-regulatory activity: A social constructivist analysis of changes in student-teacher discourse about a graduate activity: A social constructivist analysis of changes in student-teacher discourse about a graduate student portfolio. student portfolio. Special Issue ofSpecial Issue of Instructional Science.Instructional Science.
Hadwin, A. F., Hadwin, A. F., Wozney, L.., & Venkatesh, VWozney, L.., & Venkatesh, V. (2003, April). . (2003, April). A narrative analysis of the dynamic A narrative analysis of the dynamic interplay between students’ emerging task understanding and instructional scaffoldsinterplay between students’ emerging task understanding and instructional scaffolds . Paper . Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL, presented at the Annual Meeting of the American Educational Research Association, Chicago, IL, USA. USA.
Krathwohl, D. (1998). Methods of educational and social Krathwohl, D. (1998). Methods of educational and social researchresearch: An integrated approach. New : An integrated approach. New York: LongmanYork: Longman
Miles, M. B., & Huberman, A. M (1994). Qualitative Data Analysis. Thousand Oaks, CA: Sage, 1994 Miles, M. B., & Huberman, A. M (1994). Qualitative Data Analysis. Thousand Oaks, CA: Sage, 1994
Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, Calif.: Sage.Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, Calif.: Sage.
Strauss, A. L., & Corbin, J. M. (1990). Basics of qualitative research: grounded theory procedures and Strauss, A. L., & Corbin, J. M. (1990). Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage Publicationstechniques. Newbury Park, CA: Sage Publications
VanGyn, G., Ford, C., & Associates (2005). Teaching Critical Thinking. Unpublished manuscript.VanGyn, G., Ford, C., & Associates (2005). Teaching Critical Thinking. Unpublished manuscript.