Research Designs : Research on Instructional Strategies

By: Fatma & Wenshan

Research Designs

Research on

Instructional

Strategies

Strategies

Research on

Educational

Technologies

Technologies

Research on

Models for

Instructional

Design

Models

Research on

Design and

Development

Design and

Development

A quick Review on research methods Short Break Research on instructional strategies Activity Implication for Research approaches

Agenda

Introduction: A Quick Review on Research Methods Get A Research Method Quantitative Research

› Is the collection and analysis of numerical data to describe, explain, predict, or control phenomena of interest.

Qualitative Research is the collection, analysis, and interpretation of

comprehensive narrative and visual (i.e., non-numerical) data to gain insights into a particular phenomena of interest.

Mixed-methods research› Studies that combine qualitative and quantitative

data collection methods.

Overview of qualitative an quantitative research characteristics

Quantitative Research Qualitative Research

Type of Data collected Numerical data Nonnumerical data

Research Problem Hypothesis and research procedures stated before beginning the study

Research problems and methods evolve as understanding of topic deeper

Manipulation of context Yes No

Sample size Large Smaller

Research Procedures Relies on statistical procedures

Relies on categorizing organizing data into patterns to produce a descriptive synthesis.

Participant interaction Little interaction Extensive interaction

Underlying belief We live in a stable and predictable world that we can measure, understand, and generalize about.

Meaning is situated in a particular perspective or context that is different for people and groups, therefore, the world has many meanings.

An example› What are the most significant differences

in the ways Grade 3 students at K.G school interact with inert media such as books, and with active media such as computer software for learning Mandarin vocabulary?

Research on Instructional Strategies ( IS)

The rise ,fall & likely Resurgence of Experimentation on Instructional Strategies

1. The Rise What gave experimental design initial

impetus for researching IS ? Field’s behavioural roots. Experimentation as acceptable research in the

field. Promotion & tenure. Research journals preference.

Hannafin (1986)

Research on Instructional Strategies ( IS)

2. The Fall ( Hsieh et al.2005)

Intervention studies published in Educational psychology journals in 1983 & 1995-2004:

To What factors can this trend be attributed ? interest & acceptance of qualitative studies. Cost & complexity Resources & time

1983 •47%

1995 •34%

2004 •26%

Hsieh et al.2005

Research on Instructional Strategies ( IS)

3.The likely Resurgence :

- No Child Left Behind legislation (2001) What works in education.

scientific research is done as a jump off from a peer reviewed journal and applies rigorous, scientific methodologies as its focus.

Rigorous Studies

Randomized ExperimentsRigorous Regression Discontinuity Designs

Rigorous quasi-experiments

Ross and Morrison

A Trend Analysis of ETR&D

A Trend Analysis of ETR&D

BREAK TIME

Types of experimental designs

› Randomized experiment

› Rigorous quasi-experiment

Key words (2)

› Experiment A design that involves the comparison of one treatment to

another, using two or more different groups.

› Randomized Experiment A design that involves the comparison of one treatment to

another, using two or more different groups. In randomized experiment design, participants are randomly

assigned to treatment

› Quasi – experiment A design that involves the comparison of one treatment to

another, using two or more different groups. In Quasi-experimental design, participants are not randomly

assigned to treatment.Research Methods

Quantitative ApproachDescriptive Study Experimental StudyRandomized Experimental

DesignQuasi-Experimental Design

Qualitative Approach

“The most powerful design for maximizing internal validity is the true experiment (Slavin, 2002).

An example of randomized experiment

› Does the “Bio-Sym”, a software package that allows simulated experiments , improve the grade 11 students’ learning in Biology 20?

(adopted from Using Experimental Methods in Higher Education Research Ross and Marrison, 2005)

An example of quasi-experiment

› Does “Bio-Sym”, a software package that allows simulated experiments , improve Grade 11 students’ learning in Biology 20?

(adopted from Using Experimental Methods in Higher Education Research Ross and Marrison, 2005)

Experimental design trend

Instructional strategies

Definition : The prescribed sequences & methods of

instruction to achieve an objective ( Morrison et al.,2007,p.150)

Factors that determined which IS should be used:

Learning theories.Type of content.Desired performance.Theoretical paradigm ( student charac. ,

Resources, allotted time)

Research & Instructional Methods

Research Methods

Micro- Instructional strategies

Macro-Instructional Strategies

Message design strategies

Focus on teaching Focus on classroom-oriented Focus onSpecific content info. Strategies e.g. cooperative perception issues

e.g.e.g. facts & concepts learning, problem-based learning pictures, colors & legibility.

Randomized experimental ? Quasi-experimental?

› Rummel et al. (2003) compared usage of a visual mnemonic to free study of a list of theorists’ names and accomplishments. Participants were randomly assigned to either a visual mnemonic or a free study treatment. The visual mnemonic showed an image of one or more individuals with objects to provide the mnemonic device and delayed post-test and a measure of attitude toward the effectiveness of the method. Result showed that participants in the mnemonic group remembered more of the targeted information than the free study group. For designers, this suggests that visual mnemonics are beneficial in helping learners recall important information.

Randomized experiment? Quasi-experiment?

› The study by Cavalier et. Al. (1995) of cooperative learning in a technical training course was conducted. Intact groups were assigned to either a cooperative learning treatment or control (conventional instruction) treatment, both of which were taught by the same instructor. Assessments included a post-test, attitude survey, and group interaction behaviors. Findings showed that practice in a cooperative group environment produced superior achievement as well higher levels of social and cognitive interactions.

Small Group TaskTime allocation: 10-15 minutes

Pick a study topic.

Discuss whether you would choose true experimental design or quasi-experimental design to your study, and Why?

Briefly identify what your research process would be.

Factors Impacting the Rigor of Experiments

on Instructional Strategies

Internal & External Validity Guala (2002) defines: “an experimental result is internally valid,

if the experimenter attributes the production of an effect B to the factor A, and A really is the cause of B in the experimental setup E .

The result externally valid , if A causes B not only in E, but also in a set of other circumstances of interest F,G,H, etc.”

Validity ThreatsFactors has the potential to bias results.

Knowledge of validity threats, why is it important ?

Validity Threats

most commonly referenced threats to validity

• Selection participants in groups may be unlike in some way

• Teacher one group receive exposure to a better teacher than the other

Mortality Participants drop put of the test, making the groups unequivalent.

Interaction Two or more threats can interact

Maturation Change due to aging or development

Experimenter Bias Expectations of an outcome may inadvertently influence participant or cause the experimenter to view data in a different way

History Unexpected events occur between the pre- and posttest, affecting the dependent variable

Contemporary Issue in Instructional strategies Experimentation

1 . Balancing Internal & External Validity:

Internal validity means eliminating sources of extraneous variance in testing treatment effects; orients researchers to design experiments in which treatment manipulations can be tightly controlled.

Educational psychology researchers

External validity: - interaction of medium and method . - Realistic media applications need to be established. Educational Technology

researchers

Contemporary Issues in Instructional strategies

Experimentation

1.2 Basic-Applied Design Replications Basic designs demand high degree of control to proved valid tests of

principles of instruction and learning; once principle has been tested with consistent results, study is replicated in real-world.

Levin’s Stage Model (2004) - Model for educational intervention research . - Consist of 3 stages:

Stage 1 Formulating ideas & hypotheses + initial observation.

Stage 2 Controlled laboratory experiments on strategies of interest ( high internal validity).

Stage 3 Randomized field trials (RFTs) At student level. At classroom level. At school level.

Conducting RFTS

At the Student Level

At the Classroom Level

At the School Level

Units of Random Assignment

Implication for Research approaches

Design Research/design-based research (DBR): Brown (1992) & Collins (1992)

Definition : a systematic but flexible methodology aimed to improve

educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real-world settings, and leading to contextually-sensitive design principles and theories (Wang and Hannafin, (2005) p.6 )

Some common characteristics : Interventionist :The research aims at developing an

intervention in the real world. Iterative : The research incorporates a cyclic approach of

design, evaluation, and revision. Process oriented: the focus is on understanding and

improving interventions.

Some common characteristics :

• Utility oriented : The merit of the design is measured, in part, by its practicality for users in real contexts .

• Theory oriented: The design is based on theoretical propositions,

and field testing of the design contributes to theory building.

How does DBR differ from other approaches? Design- based research vs. Laboratory experimental

research (Allan Collins 1992)

Design-based research vs. Action research (Reeves, Herrington, & Oliver 2005; Wang & Hannafin, 2005)

How do I get started with Design-Based Research:

• Begin with a meaningful problem• Collaborate with practitioners • Integrate robust theory about learning and teaching • Conduct literature review, needs analysis, etc. to

generate research questions • Design an Educational Intervention • Develop, implement, and revise the design intervention • Evaluate the impact of the intervention • Iterate the process • Report DBR

Example of DBR

Trends in Design & development Research

Trends in Product & Tool Research: Focused on non-instructional products caused

expand of the definition of the design &development research.

e.g. Nguyen(2005) EPSS needs assessment :oops, I forgot how to that! Perform. Improv.

Focus on media and technology use in the field . e.g. Lewis (1993) The effect of computer simulations on introductory thermodynamics understanding.

Focus on distance learning products &Web-based tools. e.g. Koszalka (2001) Designing synchronous distance education: a demonstration project

Instructional Technologies:A Brief History

Instructional Technology: Looking Backward, Thinking Forward

References:› Van

Richey, R. C., Klein, J. D., and Nelson, W. A. (2004). Developmental research: studies of instructional design and development. In Handbook of Research on Educational Communications and Technology, 2nd ed., edited by D. H. Jonassen, pp. 1099–1130. Mahwah, NJ: Lawrence Erlbaum Associates.

Ross, S. M., Morrison, G. R., and Lowther, D. (2005). Using experimental methods in higher education Research. J. Comput. Higher Educ., 16(2), 39–64.*

The Rise of Behaviorism--The Psychology of Pavlov, Watson, and Skinner By Kendra Van Wagner, About.com

Hsieh, P., Acee, T., Chung, W., Hsieh, Y., Kim, H., Thomas, G., Levin, J. R., and Robinson, D. H. (2005). Is educational intervention research on the decline? J. Educ. Psychol., 97(4), 523–529.

Levin, J. R. (2004). Random thoughts on the (in)credibility of educational–psychological intervention research. Educ. Psychol., 39(3), 173–184.* http://bobhall.tamu.edu/epsy602/Topics/Articles/Levin.pdf

Thank You !