Educational Research:Educational Research:Post-analysis Considerations, Post-analysis Considerations, Preparing and Evaluating a Research Preparing and Evaluating a Research ReportReport
EDU 8603
Educational Research
Richard M. Jacobs, OSA, Ph.D.
Completing a research study is not the terminus of the research process……data must be verified and stored…results must be interpreted…the research report must be written
In particular, computer analyses must be checked very carefully…
…inputting mistakes lead to erroneous results
Verifying the data...Verifying the data...
The data must be double-checked, coding procedures verified, accuracy of computations examined, and the reasonableness of the findings evaluated
…the research needs to find errors before reviewers, editors, and readers of scholarly journals do
Storing the data...Storing the data...
After verifying the accuracy of the data, the researcher must label, organize and file the data in a safe place…the original data may be subsequently needed by
the researcher or other researchers who may wish to examine the data utilizing different statistical techniques
Interpreting research results...Interpreting research results...
The results of statistical analyses need to be interpreted in terms of…
…the purpose of the study…the original research hypothesis…and with respect to other studies
that have been conducted in the same area of research
hypothesized resultshypothesized results......…the researcher must describe
adequately what statistics indicate about null hypothesis
…the issue concerns statistical statistical significancesignificance and why the researcher accepts/rejects the null hypothesis
unhypothesized resultsunhypothesized results......…during the research process an
apparent relationship may emerge that was not previously hypothesized
…collect and analyze data on these unforeseen relationships; present the results, but do not change or add to the original hypothesis
…instead, use the unhypothesized results as the basis for a later study designed specifically to test the new hypothesis
statistical issuesstatistical issues......…valid inferences require that the
sample be representative of the population
…valid inferences also require that the conditions (or assumptions) underlying the statistical procedure are met
…otherwise, bias enters into the statistics used and the research findings and, hence, the generalizations from them are weakened if not rendered meaningless
methodological issuesmethodological issues......…ignoring measurement error…ignoring low statistical power…performing multiple comparisons
measurement errormeasurement error......…measurements are seldom error-free…large amounts of measurement error
hamper the ability to find statistically significant research results
statistical powerstatistical power......…the probability that a Type II error is
avoided
…however, lowering statistical power increases the likelihood that the researcher will overlook or miss the outcome desired
statistical power depends upon four interrelated factors…
…the significance level selected
…the direction of the significance test
…the sample size
…the effect size (i.e., the degree of the departure from the null hypothesis)
strategies for estimating the effect size….
…select a cut-off score below which an effect size is judge unimportant
…choose conventional, generally agreed upon definitions of small, medium, and large effect sizes
…use the effect sizes of studies of the same phenomenon
to increase power….
…(a priori) increase the significance level
…(a priori) increase the sample size
the strongest support for a research hypothesis comes from replication…
…as the study is repeated with different participants in the same or different settings
…which increases the generalizability of the findings
The matter of “significance”...The matter of “significance”...
The fact of statistical significancestatistical significance does not automatically mean that a study’s results have any practical practical significancesignificance
statistical significancestatistical significance......…the results of the study are likely to
occur by chance a certain percentage of the time
…the observed statistical relationship or differences is probably a real difference, but not necessarily an important one
practical significancepractical significance......…the results of the study are able to
applied or used by practitioners or clinicians in their particular setting
Preparing a research report...Preparing a research report...
After verifying, storing, and interpreting the data, the researcher begins the process of writing the final report……which requires outlining and
providing details for the general sections
Elements of a research report…Elements of a research report…
1. preliminary pages2. main body
3. appendices
1) preliminary pages…
…table of contents…list of tables and figures…abstract
…title page…acknowledgements page
title pagetitle page......…indicates the title of the report, the
author’s name, the requirement being fulfilled, the name and location of the institution, the date of submission of the report
…states the purpose of the study as succinctly as possible
acknowledgements pageacknowledgements page......…expresses an author’s appreciation
to persons who have contributed significantly to the completion of the report
table of contentstable of contents......…provides an outline indicating on
which page each major section (or chapter) and subsection begins
list of tables and figureslist of tables and figures......…gives the number and title of each
table and figure and the page on which it can be found
abstractabstract......…a statement summarizing the basic purposes
of the study, including: the problem investigated, types of participants and instruments, the design, the procedures, the major results, as well as the major conclusions
2) main body…
…results…discussion…references
…introduction…method
introductionintroduction......…a well-written description of the
problem, a review of related literature, and a definition of terms
…should lead logically to the statement of the hypothesis
methodmethod......…a description of the participants,
instruments, design, procedures, assumptions, and limitations
resultsresults......…describes the statistical techniques
or qualitative interpretations that were applied to the data and the results of the analysis
…information about the process applied during data analysis should be provided
discussiondiscussion......…identifies the agreement or disagreement
of the findings with previous results obtained by other researchers in other studies or the hypotheses stated at the start of the study
…identifies the theoretical and practical implications of the findings and makes recommendations for future research or future action
referencesreferences......…a listing of all sources, alphabetically
by authors’ last names, directly used in writing the report
3) appendices……information and data that are
pertinent to the study that either are not important enough to be included in the main body of the report or are too lengthy
Some general rules for writing…...format according to APA Publication Manual (5th edition)
…use clear, simple, straight-forward style, including correct spelling, grammar, and punctuation
…learn from the corrections provided on your 8603 exercises
Evaluating research reports…Evaluating research reports…
requires the development of a set of skills that enable an individual to differentiate the quality and value of research reports……by accurately identifying their
strengths and weaknesses
General evaluative criteria…General evaluative criteria…
1. introduction2. method
3. results
4. discussion
5. abstract or summary
1) introduction…
…hypotheses
…problem…review of literature
2) method…
…design and procedure
…participants…instruments
3) results…4) discussion (including conclusions
and recommendations)…
5) abstract or summary…
Type-specific evaluative criteria…Type-specific evaluative criteria…
1. qualitative research2. descriptive research (including questionnaire, interview, and observation studies)
3. correlational research (including relationship and prediction studies)
4. causal-comparative research
5. experimental research
Mini-Quiz…Mini-Quiz…
True and false…
…the larger the denominator in a parametric test of significance, the larger the numerator must be to attain significance
True
True and false…
…computer analyses of data always provide accurate results of the data input into the program
True
True and false…
…the product of a test of significance is a number
True
True and false…
…what the test of significant means requires interpretation
True
True and false…
…virtually any difference can be made significant if the sample is large enough
True
True and false…
…the higher the level of significance at which the null hypothesis will be rejected, the more powerful the test
True
True and false…
…the failure to reject the null hypothesis renders a study insignificant
False
True and false…
…a true test of a hypothesis comes from its ability to explain and predict what is happening
False
True and false…
…the lack of randomized samples can introduce bias into a study and limit its usefulness
True
True and false…
…although the researcher is likely to know the significance level of the study and its sample size, the researcher is not likely to know the effect size
True
True and false…
…smaller sample sizes mean less power
True
True and false…
…statistical and practical significance are synonymous
False
True and false…
…a greater mean difference is probably required to reject a null hypothesis
True
True and false…
…repeating a study with different participants in the same or different settings increases the generalizability of the findings
True
True and false…
…the most common flaw in a research study is the lack of information provided about validity and reliability
True
True and false…
…whether the most appropriate design was used in a study, given the problem, involves a degree of subjective judgment
True
Fill in the blank…
…the process of double-checking, cleaning, and organization of the data as well as evaluating the research conclusion
verification
Fill in the blank…
…the probability of avoiding a Type II error
power
Fill in the blank…
…the results of a study are likely to occur by chance a certain percentage of the time
statistical significance
Fill in the blank…
…the strongest research evidence supporting a research hypothesis
replication
This module has focused on...This module has focused on...
post-analysis considerations, preparingand evaluating research reports
...the procedures for checking and storing all data in an organized manner and the general guidelines for reporting a study’s findings