Research Seminars inIT in Education(MIT6003)

Research Methodology I

Dr Jacky Pow

Agenda

Method evaluation: critical review of research procedures

Typical research procedures

Formulation of the research problem– An emerging issue

Review of the literature– What has been done and what need to be done

Data collection– How to collect the necessary data to answer the

research question

Typical research procedures

Data analysis– How to analyze the and organize the data

Discussion – To discuss the implication of the data and see

whether the research questions can be answered– Future research agenda

Conclusion– What this research has achieved

Class activity

What questions would you ask if you were to evaluate the appropriateness of the research method in a study?

How will you answer the questions you identified?

Issues in educational research

Reliability (Trochim, 2002)

– In research context, the term reliability means repeatability or consistency. A measure (of the variables) is considered reliable if it would give us the same result over and over again (assuming that what we are measuring is not changing!)

Types of reliability

Inter-Rater/Inter-observer Reliability– Used to assess the degree to which different

raters/observers give consistent estimates of the same phenomenon

Test-Retest reliability– Used to assess the consistency of a measure

from one time to another– Administer the same test to the same (or a

similar) sample on two different occasions

Types of reliability Parallel-Forms Reliability

– Used to assess the consistency of the results of two-tests constructed in the same way from the same content domain

– Administer both instruments to the same sample

Internal Consistency reliability– Used to assess the consistency of results across

items within a test– Calculate the average correlation of the items

splitting into half [e.g., Cronbach’s alpha (α)]

Cronbach’s alpha (α)

Trochim, 2002

Step 1

Cronbach’s alpha (α)

Trochim, 2002

Step 2

Issues in educational research

Validity– Refers to what we measure is what we intend to

measure (highly related to the research question)

– We have to make sure that the tools we use to measure the variables are valid

Internal validity– The degree to which correct conclusions about

causal relations can be drawn

Reliability and validity

Trochim, 2002

Class activity

Discuss with your group members to see whether the issues facing by quantitative and qualitative research are the same in terms of reliability and validity

Issues in educational research

Applicability– To be of any use educational research must

generalize from the sample to the population– The findings of the research should be able to

inform teaching practice or policy formulation

Issues in educational research

Feasibility– There may be some excellent ideas in

educational research but due to some reasons, the research may not be feasible

– These reasons may include resource, time or ethical considerations

Issues in educational research

Significance– Educational research should have impact or

acceptable level of importance to the field– Significance = the value of the research

Issues in educational research

Ethics– The research we conduct should not violate the

ethical standard– For instance, it is not ethical to segregate a

group of students during their programme in order to test out a new teaching technique

Common errors made in research

Selective observation Inaccurate observation Overgeneralization Made-up information Ex post facto hypothesizing Illogical reasoning Ego involvement in understanding Premature closure of inquiry

Selective observation

It happens when attention is drawn to observations or answers that confirm pre-existing beliefs

A few ways you can try to avoid selective observation in your research: – Literature review– Decide your research approach beforehand – Take thorough notes – Watch for "disconfirmatory" information – Consider both "sides" of your study

Inaccurate observation

It occurs when we “misremember” or “misrecord” data

To increase your accuracy you have to plan your observation:– Using forms – Doing time/area sampling – Writing down as much information as you can– To practice observing and recording your

observations before actually doing it for real

Overgeneralization

Overgeneralization is generalizing to others who are different from one's research population

To avoid overgeneralizing:– Replicate one's study – Support many tests of the same theory – Attempt to use representative samples – Recognize the limitations of one's research

Made-up information It happens when one fills in details without

a scientific basis for doing so Instead of making up information, you

could do a variety of things:– Ask someone who knows what's going on for

their opinion– Do a thorough literature review to help you

understand possible outcomes and unique customs

– Rely on prior research studies and/or theory to guide your own interpretation of what was going on

Ex post facto hypothesizing

It occurs when a researcher decides what happened after it happened and after the study was done

In scientific research, we have to decide what will happen before we do our study, not after

Illogical reasoning

Illogical reasoning is just that: illogical to avoid illogical reasoning in your own

research:– Base your decisions on prior research and

theory – Make extensive use of peer review

Ego involvement in understanding

It happens when people let the human side of them dictate their findings and how they view findings by other researchers

To avoid ego involvement:– To try to remain neutral – To stay away from topics about which you can't

be neutral– To use the team approach

Premature closure of inquiry

It occurs when we decide that we know enough about a topic and decide that it no longer warrants future study

To deal with premature closure:– To keep looking for answers, even if it involves

using different approaches – To find non-controversial ways to do it as far as

possible

Mystification

It happens when we attribute results to the supernatural

To avoid mystification:– Keep looking for answers when you feel

tempted to rely on enchanted explanations – Peer review will prevent you from

embarrassing yourself

End of lesson 9