using mixed method approaches in institutional research rick kroc university of arizona richard...
TRANSCRIPT
Using Mixed Method Approaches in Institutional Research
Rick Kroc
University of Arizona
Richard Howard
University of Minnesota
Arizona Association for Institutional ResearchAnnual Meeting
Tucson, Arizona March 2007
Purpose of Decision Support
is
to reduce the risk to the decision maker
Structure and Subjectivity
Research Process
Ways of Knowing
Two Classic Paradigms
Positivist Constructivist
there are multiple realities and that truth is ever changing, dependent on context and the individual (subject and researcher)
there is a single reality or truth across time and contexts, and that this truth can be understood through the objective study of independent variables
Conceptual Model of the Constructivist Research Process
After a literature review and pilot study, define the Study:
ProblemPurposeQuestion
Data Collection & Management
Data Restructuring/Coding & Analyzing/Read-Theme
Data Verification with Respondents:
FACTUAL ERROR
Analysis Verification with Reviewers:
Interpretation Error
COMMUNICATENew Theory/Model
TEST Theory/Model
Iterative Process
Delimited by PP&Q
Conceptual Model of the Positivist Research Process
After a literature review and pilot study define the project
ProblemPurposeQuestion
Operationalize theory or model constructs
Data Collection and Management
Data restructuring and analysis
(Interpretation of Results)
Communication of new
knowledge
For internal decision support- test meaning of the findings
For external contexts- test/replicate for generalization
RESEARCH PARADIGMS
APPROACHES
Constructivist Positivist
QuantitativeQualitative
Consumer Generalizes Researcher Generalizes
Mixed Methods
Combines the collection, analysis, and interpretation of qualitative and
quantitative data.
Often used in studies that require an understanding of not only “what” is
observed but also “why” the observed behavior occurred.
Such as in assessment, evaluation, action research, and other decision support studies.
Use of Mixed Method Approaches in Institutional Research
Provides answers to two questions:
What - What happened and what was the outcome? Answers tend to be
arrived at using quantitative methods
Why - Why did people behave the way they did? Answers tend to be arrived at using qualitative methods
What - goals are - how measured – Supports Accountability (External)
Why – goals were met or not – people issues – Often the focus of Assessment/Evaluation (Internal)
So What?
Ways of Knowing
A Third Paradigm
Positivist Constructivistthere are multiple realities and that truth is ever changing, dependent on context and the individual (subject and researcher)
there is a single reality or truth across time and contexts, and that this truth can be understood through the objective study of independent variables
PragmatismUse what ever approach and methodology that will best answer the question.
Charge from President Likins“As Chair of the NCAA Division I Presidential Task Force on the Future of Intercollegiate Athletics, I have both the opportunity and the responsibility to examine the culture of higher education athletics programs in America. My responsibilities begin at home. My task force experience as well as events at colleges and universities around the country led me to conclude that it is appropriate to examine the environment defined by the culture of high-level athletic competition here at The University of Arizona. At the national level a serious movement to strengthen the academic experience of student-athletes is well underway. We must examine the presence of any cultural influences here that have the potential of distorting the shared values of academics and athletics in higher education.”
Focus of the Study
The study was divided into two phases. Phase I, mostly quantitative in nature, was designed to look somewhat broadly at the two fundamental study areas:– Student athletes’ academic experience; and– Student athletes’ culture and environment.
Using qualitative methods, Phase II was designed to dig more deeply into the critical issues identified from Phase I. Detailed, targeted recommendations would be developed from these two phases.
Domain of the Study
Athlete’s academic experience
• Academic preparation• Progress to degree• Student engagement• Academic support• Post-eligibility academic
experience• Other academic issues
Athlete’s culture and environment
• Alcohol and drugs• Dishonesty and cheating• Gambling• Racism• Sexism• Sexual behavior• Violence• Weapons
Phase 1—Quantitative • Board of Regents and NCAA reports on athletes’ grades
and graduation rates (ten year history).• Institutional student records data.• Health and Wellness Survey, administered annually to
assess student alcohol, drug, and sexual behaviors (seven year history).
• Dean of Students Code of Conduct and Code of Academic Integrity violations (five year history).
• Coalition on Intercollegiate Athletics (COIA) report.• Annual University Survey of Graduating Seniors, assessing
student engagement, satisfaction, and other academic issues (five year history).
• NCAA GOALS survey (student-athletes and student non-athletes).
• Additional UA-specific survey questions appended to GOALS survey (student-athletes and student non-athletes).
Phase II—Qualitative
• Focus groups• Interviews• Follow-up surveys• Populations:
– Students (athletes and non-athletes)– Coaches– Faculty– Staff– Advisors
The Value of a Mixed Methods Approach
• Athletic culture spans a spectrum of areas, some of which (alcohol use, for example) have become fairly well understood over the years from quantitative methods, others of which (physical violence, weapons use) need the deeper exploration and understanding provided by qualititative methods.
The Value of a Mixed Methods Approach
• The formative and summative aspects of this study, which is more like program evaluation in many ways than like a traditional research study, lend themselves to a mixed methods approach.
The Value of a Mixed Methods Approach
• Audiences and stakeholders consist of some people who respond best to systematic, inferentially rigorous, quantitative data, but also others who want to hear the complex, richly detailed stories derived from the individuals who are immersed in the culture. Different audiences resonate to different approaches.
The Value of a Mixed Methods Approach
• Mixed methods also means that multiple methods have been employed, which is essential to using triangulation as a means to establish greater validity of the findings. Any single approach used in the messy world of social science research will be flawed. If a wide array of methods leads to consistent conclusions, then analysts and stakeholders feel more confident about conclusions and recommendations.
Questions??