School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
INFO4990: IT Research MethodsExperimental Research(Joseph Davis) 9/5/2005
Outline:• Strategy of experimental research (Platt)• What are experiments and how to do experimental
research,• Different kinds of experimental designs.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Experimental Research Strategy(Platt)
• Systematic and precise method of scientific thinking• Accumulative method of inductive inference• Can contribute to rapid scientific progress
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Steps in Experimental Research1. Devise alternative hypotheses (Existing theory)2. Devise crucial experiments with alternative possible
outcomes, each of which exclude one or more possible hypotheses, (Experiment)
3. Conduct the experiment, get a clean result, (Outcome)
4. Back to step 1, making sub-hypotheses, or sequential hypotheses to refine the possibilities, exclusion and induction (exclusion and building the inductive (logical) tree)
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Origins of the experimental approachFrancis Bacon (interconnecting theory and experiment) building
the conditional inductive tree, consecutive inductive inferences)
Karl Popper, falsificationism, there is no such thing as proof in science, science advances through a series of disproofs or falsification. Assertions in science have to be falsifiable,
“.. it must be possible for all empirical scientific systems to refuted by experience”.
Fisher’ s work in the 1930’s and 40’s in the area of statistical inference.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Experimental Research Methods
What is an experiment?Manipulation of one or more variables by the experimenter to
determine the effect of this manipulation on another variable.Carefully designed and executed plan for data collection and
analysis to test specific hypotheses.Examples of hypotheses?
Well-designed experiments can permit causal inferences to be made.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Experiment : Features
• Typically held at the discretion of the researcher.• Ability to use various controls to isolate sources of
variation.• Ability to explore cause-effect relationships.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Phases in Experimental Studies
1. Formulation of one or more hypotheses.- Usually deductions or derivations from theoretical
explanations (of the behavioural phenomenon) or strong hunches/speculations.
2. Translation of the hypotheses into a set of treatment conditions and appropriate experimental design.
3. Conduct the experiment, collect the data4. Statistical analysis of the data, interpretation of the results
and writing up.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Experimental Design
• Independent Variables:The variable(s) manipulated in the experiment.(also called manipulated variable, treatment variable or
factor).Typically nominal (categorical) variable.• Dependent Variable(s)Measure(s) that capture (performance) of the
phenomenon
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Control or Nuisance Variables
Undesired sources of variation in an experiment that affect the dependent variable measurement,
Typically of three types:-organismic-environmental-experimental task.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Approaches to control the nuisance variable
• Random assignment of subjects to treatment groups,• Holding the (pre-identified) nuisance variable
constant for all subjects,• Statistical control using Analysis of Covariance
(ANCOVA).
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Sources of Bias
• Experimenter cues• Demand characteristics• Evaluation apprehension• Hawthorne Effect• Negativistic subject
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Experiments - Advantages
• Possibility of a variety of manipulative and statistical controls,
• Random assignment of subjects – greater precision and higher confidence in specifying and testing causal relationships,
• Manipulation Checks possible.• May help identify issues and problems previously
unrecognised.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Experiments - Disadvantages
• Problems associated with lab settings,• Some phenomenon cannot be studied under
controlled conditions,• Limitations imposed by moral concerns.
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Procedures1. Getting Ethics Committee Approval2. Cover Story – description and purpose3. Recruiting participants- Sample selection- Reference to criterion population- Remuneration and motivation3. Training the participants4. Preparing the setting5. Controlled manipulation of independent variable(s)6. Manipulation checks7. Precise measurement of dependent variable(s).
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Reviewing a Paper that used Experimental Research Method
• Theoretical Background • Key Hypotheses• Experimental Design
- Treatment (Independent) Variable(s)- Dependent Variables- Nuisance Effects- Manipulation checks
• Results – robustness and value• Alternative Research Design – how would you do it differently?
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Types of experimental Designs• Completely Randomised Designs (CR-p) with one treatment.(single treatment with p levels)• Randomised Block Designs (RB-p)Isolating the effects of nuisance variables by a blocking
procedure whereby subjects who are relatively homogeneous w.r.t to the nuisance variable are assigned to the same block
• Completely Randomised Factorial Designs (two or more treatments evaluated simultaneously
• Repeated Designs
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Quasi-experimental Designs
• Used when it is impossible or difficult to perform true, controlled experiments,
Particularly in organisational settings• Essentially compromise designs• Used when time ,cost, and practicality are critical.1. One shot design2. One group pretest-posttest design3. Static Group Design
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Observational Methods
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Survey Methods
Typically questionnaire surveysStrengths:Quantitative dataTransparency and some level of objectivitySuccinct and easily understoodComparability/reproducibilityCan deal with complex problems/issues
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Survey Methods
• Surveys:Interviewer completionRespondent completionTypes of surveys1. Household2. Telephone3. Mail4. Customer5. Captive Group6. Organisation7. Web-based
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Survey Methods
Issues:
Response ratesBiases and errors
School of Information TechnologiesFaculty of Science, College of Sciences and Technology
The University of Sydney
Survey Methods
• Types of QuestionsOpen ended and pre-coded
Measurement of Attitudes and opinions:Likert ScalesAttitude StatementsSemantic Differential