project design and data collection methods: a quick and dirty introduction to classroom research...
Post on 22-Dec-2015
215 views
TRANSCRIPT
Project Design and Data Collection Methods:
A quick and dirty introduction to classroom research
Margaret Waterman
September 21, 2005
SoTL Fellows 2005-2006
Where are we inthe Action Research Cycle?
Identify Innovation
Gather Data
Analyze DataInterpret Data
Develop Action Plan
☻
Decide when and how to collect data
Design vs. Method Design is about when and from whom you are
going to collect data so that the data can be interpreted.
On Tuesday
With Colonel Mustard
Methods are about where and how you are going to collect that data.
In the Library
With a candlestick
With an interview
Designs: Experimental Experimental: strongest for drawing
inferences about cause and effect. Control vs. experimental groups, randomly
assigned, one variable at a time Diagram as follows:
O1 X1 O2
O1 O2
O = observation, X = treatment; Each row is a group
Experimental designs
Problems: Not always possible in education to have
two or more matched groups,
Random assignment usually not feasible,
May not be suitable for small-scale projects
Usually not for classroom research problems
Designs: Pre-Experimental Weakest designs: Post test only
X1 O1
Slightly stronger: Post-test with comparison group
X1 O1
O1
NOT RECOMMENDED
Pre-Experimental: Post Test OnlyProblems:
Too many other reasons for observed conditions Cannot draw valid inferences about causality
because you don’t know what pre-treatment condition is.
In comparison group: may have differences in incoming ability in the two groups
Good for: preliminary observation and development of hypotheses, not showing cause and effect
Designs: Pre-Experimental One group, Pre-test Post-test
O1 X1 O2
Uses: Can make some inferences about cause and effect, but only when discuss possible intervening variables.
Useful in classrooms when trying new techniques or materials.
Pre-Experimental: Pre-test, Post-test Problems:
Can’t discount all intervening variables: e.g., history, awareness of being in an experiment, selection effects on group membership.
Generalizability is limited May sensitize students with the pre-test
Designs: Quasi-Experimental
Uses an experimental design, but groups are not randomly assigned.
O1 X1 O2
O1 O2 Effect of treatment isolated
X1 O2 Use two more groups to
O2 isolate effect of pre-test.
Can make inferences with more confidence than with one-group designs
Problem: intervening variables again.
Design: Time Series
One group, multiple observations before and after treatment. Can be two groups or two treatments.
O1 O2 O3 O4 X1 O5 O6 O7 O8
OR
O1 O2 O3 O4 X1 O5 O6 O7 X2 O8 O9 O10
Look for BIG differences after treatment as compared to before treatment.
Problem: The intervening variables
Linking Design and Methods
O1 X1 O2
Suppose the design is one-group, pre-test, post-test:
How, where, and exactly when will the data be gathered at the two points of observation?
Action Research Often Uses These Kinds of Methods to Gather Data
Surveys
Observations
Interviews
Artifact Analysis
Methods of Gathering Data Make observations
Structured observations
Unstructured observations
Ask questions
Surveys
Interviews
Examine and Score Artifacts
Tests, portfolios, treatment plans, student comments
Must design scoring guide
Structured Observations Examples:
Use a class map to keep track of who asks questions, answers questions, makes comments.
Have a list of behaviors. Make simple hash marks for each type of behavior when it occurs.
Make a list, from a recorded class discussion, of the kinds of questions you asked.
Methods: Unstructured Observations Examples
A journal that a teacher keeps to record personal impressions of how a class is going.
A written remembrance of interactions with one or a few students that are being tracked over time.
A written set of impressions made while watching a videotape of a class.
Methods: Unstructured Observations Think of these as similar to ethnographic “field
notes,” a place to record reactions, impressions, planned actions, responses and thoughts.
Can be useful to see trends in how YOU are responding to the innovation you are attempting.
Can be useful when studying one or a few students, again, over time to see trends.
Methods of Gathering Data Make observations
Structured observations
Unstructured observations
Ask questions
Surveys
Interviews
Examine and Score Artifacts
Tests, portfolios, treatment plans, student comments
Must design scoring guide
Methods: Questioning (survey/interview)
Uses a prepared set of questions. May have open-ended or close-ended
questions, or a combination. Good for getting background info. Can be used pre and post treatment. Can address things that may not be visible,
such as attitudes, motivation. Use with individuals or groups.
Methods: Questioning with Surveys Examples:
A survey of attitudes toward science
Rankings of importance
Student ratings of instruction
May include open-ended questions, e.g., what element of this course most helped you to learn?
May include some content, but if entirely content, it’s a test and is an artifact of the course.
Sample from an Survey
Methods: Questioning with Interviews Advantage over surveys: can ask follow-ups,
more personal contact. Examples:
Sort readings by usefulness, telling why Solve a genetics problem aloud, explaining
thinking A focus group Interview of team members in small groups
Sample from Interview
Methods of Gathering Data Make observations
Structured observations
Unstructured observations
Ask questions
Surveys
Interviews
Examine and Score Artifacts
Tests, portfolios, treatment plans, student comments
Must design scoring guide
Method: Examine and Score Artifacts Examples of artifacts:
Diagrams of cells before and after instruction Wear on computer keys to see which are hit most Answers to a test question Portfolios Term papers Case analyses Management plans Treatment plans
Methods: Artifact Analysis
Decide what you kinds of materials you want to collect
Justify why the artifact you are choosing is a good choice given your research question
Create a scoring rubric (guide) to assign points
Good for pre-post designs
A KEY TO SUCCESS: Pilot Test Your Instruments
Give a small group of people (not in your class if you can) your survey or interview
Collect sample artifacts to see if your grading scheme works
Try out your observation scheme to see if it needs to be tweaked.
Designs and Methods: Summary Many of these approaches are familiar to we
who teach Much of this is used in research in many
disciplines. Talk about your design and instruments with
Teaching Associates, Fellows, other colleagues.
Keep your data collection focused. Try to keep the project reasonable for the
time you have available.
To create your own plans answer the questions below: From whom are you gathering data?
More than one class, subgroups?
When will you gather data?
All one term, over several courses, in three weeks?
How will you gather data?
Questions, artifacts, observations?
Where will you gather data?Classroom, online forum, mailed survey, telephone?