multidisciplinary methods in educational technology research...

HAMK

Multidisciplinary Methods in Educational Technology Research and Development

Justus J. Randolph

Over the past thirty years, there has been much dialogue, and debate, about the conduct of educational technology research and development. In this brief volume, Justus Randolph helps clarify that dialogue by theoretically and empirically charting the research methods used in the field and provides much practical information on how to conduct educational technology rese-arch. Within this text, readers can expect to find answers to the following questions:

• What are the methodological factors that need to be taken into consideration when designing and conducting educational technology research?

• What types of research questions do educational technology researchers tend to ask?

• How do educational technology researchers tend to conduct research? – What approaches do they use? What variables do they examine? What types of measures do they use? How do they report their research?

• How can the state of educational technology research be improved?

In addition to answering the questions above, the author, a research methodologist, provides practical information on how to conduct educational technology research – from formulating research questions, to collecting and analyzing data, to writing up the research report – in each of the major quantitative and qualitative traditions. Unlike other books of this kind, the author address some of research approaches used less commonly in educational technology research, but which, nonetheless, have much potential for creating new insights about educational pheno-mena – approaches such as single-participant research, quantitative content analysis, ethno-graphy, narrative research, phenomenology, and others.

Multidisciplinary Methods in Educational Technology Research and Development is an excellent text for educational technology research methods courses, a useful guide for those conducting (or supervising) research, and a rich source of empirical information on the art and science of educational technology research.

Randolph: Multidisciplinary M

ethods in Educational Technology Research and Development

ISBN 978-951-784-453-6ISSN 1795-4231

HAMKin julkaisuja 12/2007

HAMK


Justus J. Randolph

Over the past thirty years, there has been much dialogue, and debate, about the conduct of educational technology research and development. In this brief volume, Justus Randolph helps clarify that dialogue by theoretically and empirically charting the research methods used in the field and provides much practical information on how to conduct educational technology rese-arch. Within this text, readers can expect to find answers to the following questions:

• What are the methodological factors that need to be taken into consideration when designing and conducting educational technology research?

• What types of research questions do educational technology researchers tend to ask?

• How do educational technology researchers tend to conduct research? – What approaches do they use? What variables do they examine? What types of measures do they use? How do they report their research?

• How can the state of educational technology research be improved?

In addition to answering the questions above, the author, a research methodologist, provides practical information on how to conduct educational technology research – from formulating research questions, to collecting and analyzing data, to writing up the research report – in each of the major quantitative and qualitative traditions. Unlike other books of this kind, the author address some of research approaches used less commonly in educational technology research, but which, nonetheless, have much potential for creating new insights about educational pheno-mena – approaches such as single-participant research, quantitative content analysis, ethno-graphy, narrative research, phenomenology, and others.

Multidisciplinary Methods in Educational Technology Research and Development is an excellent text for educational technology research methods courses, a useful guide for those conducting (or supervising) research, and a rich source of empirical information on the art and science of educational technology research.

Randolph: Multidisciplinary M

ethods in Educational Technology Research and Development

ISBN 978-951-784-456-7 (PDF)ISSN 1795-424X

HAMKin e-julkaisuja 1/2008


by

Justus J. Randolph

Digital Learning Lab

HAMK University of Applied Sciences

Justus J. RandolphMultidisciplinary Methods in Educational Technology Research and Development

ISBN 978-951-784-456-7 (PDF) ISSN 1795-424X

HAMKin e-julkaisuja 1/2008

© HAMK University of Applied Sciences and the author

JULKAISIJA – PUBLISHERHAMK University of Applied SciencesPO Box 230FI-13101 HÄMEENLINNA, FINLANDtel. +358 3 6461fax +358 3 646 [email protected]/julkaisut

Book layout: HAMK Publications

Hämeenlinna, December 2008

Contents

Preface ...................................................................................................................................... 5

Chapter 1Methodological Factors in Educational Technology Research and Development ...................13

Chapter 2Types of Research Approaches in Educational Technology Research and Development ............................................................................................................................41

Chapter 3A Meta-Synthesis of Methodological Reviews of Educational Technology Research ............. 57

Chapter 4Data Collection in Educational Technology Research............................................................. 73

Chapter 5The Analysis of Educational Technology Research Data......................................................... 83

Chapter 6Reporting Educational Technology Research ...........................................................................91

Chapter 7Conclusions about Methods Choice in Educational Technology Research and Development ......................................................................................................................... 103

References ............................................................................................................................. 105

AppendixKey Questions in Educational Technology Methods Choice ................................................... 117

List of Tables

Table 1. The Question Topics in K-12 Computing Education Program Evaluations ..................................... 26

Table 2. Major Categories of Research Questions from FIE (1997 – 1999) .................................................. 27

Table 4. Major Categories of Research Questions from HCI (1995 – 1999) ................................................. 28

Table 5. The Five Major Categories of Educational Technology Research Questions.................................. 29

Table 6. Research Questions in Past Educational Technology Methodological Reviews............................. 59

Table 7. Characteristics of Educational Technology Reviews Included in the Quantitative Synthesis......... 60

Table 8. The Composition of Educational Technology Metacategories ....................................................... 61

Table 9. Comparison of the Proportion of Human Participants Articles in EducationalTechnology and Education Proper ................................................................................................................ 64

Table 10. Comparison of the Type of Methods Used in Educational Technology and Education Research Proper ........................................................................................................................... 65

Table 11. Research Approaches of ICALT Articles Dealing with Human Participants .................................. 66

Table 12. Experimental or Quasi-Experimental Designs in ICALT papers ................................................... 66

Table 13. Measures Used in ICALT Papers.................................................................................................... 67

List of Figures

Figure 1. Strategic framework for the Digital Learning Lab research project ............................................... 5

Figure 2. Merging of design and research processes into the integrative learning design framework ..... 21

Figure 3. Example of a research map. By F. Girardin, 2007......................................................................... 24

Figure 4. Scatter plot of a strong positive correlation, (r = .93)................................................................. 45

Figure 5. Scatter plot of a strong negative correlation, (r = -.91) .............................................................. 45

Figure 6. Scatter plot when there is practically no correlation, (r = -.02).................................................. 46

Figure 7. Example of a single-participant graph ......................................................................................... 51

Figure 8. Proportion of types of articles in educational technology journals ............................................. 62

Figure 9. Proportions of types of educational technology articles by forum............................................... 62

Figure 10. Proportions of types of educational technology articles by time period.................................... 63

Figure 11. Virre: A computer-assisted self-interview tool ........................................................................... 75

Figure 12. A flowchart for the typical process of content analysis............................................................... 87

Figure 13. The data analysis spiral............................................................................................................... 88

5

Preface

In 2004, HAMK University of Applied Sciences began a research and development project whose goals were (a) to produce theoretical and empirical information about e-learning, (b) to develop innovative e-learning applications, and (c) to foster the growth of the e-learning enterprise in the Kanta-Häme province of Finland. To achieve those goals, a nine-element research agenda was devised. Figure 1 shows

6 Multidisciplinary Methods in Educational Technology Research and Development

Figure 1. Strategic framework for the Digital Learning Lab research project.

The purpose of the eighth element (Multidisciplinary research methods in educa-tional technology), and the purpose of this book, is to chart the multidisciplinary methods used in educational technology research and, from that charting, to pro-duce information that can help foster improved research methods in educational technology. The improved research methods are, in turn, intended to bring about improved theoretical and empirical information about technologies for improving education.

This book is based on numerous disciplines. They include education, psychology, sociology, media studies, computing, program and policy evaluation, software en-

educational technology research and development.

Organization

-

following:

The research problem;

The purposes and frameworks for the research;

The state of and types of the previous research;

The type of research act implied by the research question;

The level of generalizability needed;

The level of accuracy needed;

The feasibility of carrying out an investigation;

The propriety of an investigation;

The utility of an investigation;

Whether a quantitative, qualitative, or mixed-methods tradition is being adopted;

The degree to which stakeholders participate in the research process; and

The degree to which the researcher becomes a participant in the intervention or setting of the investigation.

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

7

-logy research and development.

-ches (survey research, causal-comparative and longitudinal research, correlational

case study research, grounded theory research, and ethnography).

-tional technology research by synthesizing and analyzing the results of several pre-vious methodological reviews. The questions that the overview answers are listed below:

What are the meta-categories that can be used to subsume the research cate-gories in other methodological reviews of educational technology research?

What are the proportions of quantitative, qualitative, and mixed methods research that educational technology researchers have tended to use?

How do those proportions differ over time periods and publication forums?

-tion research proper?

In what proportions do educational technology researchers choose (a) rese-arch methods, (b) experimental research designs, and (c) measures?

How do educational technology researchers tend to report educational technology studies?

What suggestions are given for improving educational technology research?

and refer the reader to the seminal books for each form of data analysis. I discuss the quantitative analysis of quantitative data, the quantitative analysis of qualita-tive data, the qualitative analysis of qualitative data, and the qualitative analysis of quantitative data.

educational technology investigations. I provide information on writing up conven-tional quantitative and qualitative reports, discuss alternative styles of reporting,

1.

2.

3.

4.

5.

6.

7.

Preface


technology research. In the Appendix, I have included a list of key questions to con-sider when planning educational technology research and development projects.

The Target Audiences and How They Will Benefit from this Book

The target audiences for this book are primarily educational technology students, their supervisors, and educational technology researchers. However, because of the multidisciplinarity of this text, I expect that students, instructors, and researchers

groups.

-

them with the basic information needed to make informed decisions about what methods to choose. The third chapter will help students understand what types of methodological choices practicing educational technology researchers tend to make. The remaining chapters are intended to help students make informed decisi-ons about data collection, analysis, and reporting and to refer them to the seminal resources for carrying out those activities. The appendix, Key Questions in Educa-

issues that are critical when choosing methods in educational technology research and development.

-ers of educational technology students. With hope, this book, in whole or in part, can serve as a course text in an educational technology research methods class and as a catalyst, focusing tool, and source of common vocabulary for academic dialo-gue between supervisors and their students.

-

serve as a starting point for clarifying the methods-choice debates in educational technology. Also, because the third chapter of this book synthesizes the research about the practices of educational technology researchers, it is my hope that it can

– questions such as:

How do educational technology researchers and developers tend to conduct

What methods do they tend to use?

What methods do they tend not to use?

How do the observed practices in educational technology research differ from what is suggested as best practice, and why?

•

•

•

•

9

How do the research practices in educational technology research differ from

In addition, the answering of these questions is intended to help educational technology researchers understand the prevalent epistemological and ontological

Using this Book in the Classroom

As noted earlier, one of the purposes of this book is to serve as a tool for instruc-ting students in educational technology research methods. For example, having students read chapters one and two and then having them think through the list of key questions in educational technology methods choice at the end of this book would be a good way to familiarize students with the factors that are important in choosing appropriate methods. The key questions in methods choice at the end of this book might also serve as an intellectual organizer for supervisors and their students who are beginning to plan theses, dissertations, or projects. The third chapter can be used to introduce students to the practice of educational technolo-

analysis, and reporting; can be used to familiarize students with those issues and to refer them to the essential texts in those areas. At the end of each chapter, I have included a Questions to Consider section that can be used as a catalyst for group or online discussions.

Positioning Myself

In the tradition of qualitative research, I think that it is worthwhile to provide the reader with some information about the author. For the past four years I have been involved with educational technology research and evaluation in various programs at the University of Joensuu and HAMK University of Applied Sciences. My main area of interest is how scientists, particularly educational technology researchers, conduct and report their research. I also am interested in research about and meta-analysis of educational interventions. My doctoral training; at Utah State Univer-

degree was in English, art history, and philosophy. Although my doctoral studies were mostly quantitative in nature, I have spent the past few years developing my toolbox of qualitative methods.

Besides working for HAMK University of Applied Sciences and the University of Joensuu, I have worked as an evaluator or researcher for organizations such as the

school principal.

•

Preface


I was reared and educated in the United States, but I have been living in Europe for most of my adult life. I originally came to Finland, where I still reside, to do a Fulbright-sponsored research and evaluation internship.

Key Sources

Although I drew on many sources when putting together this book, I tended to draw Qualitative Inquiry and Research

Design: Choosing Among Five Traditions was particularly useful for providing the information on the sections about the qualitative traditions and their reporting.

Experimental and Quasi-Experimental De-signs for Generalized Causal Inference was the source of much of the information

The User Friendly Handbook for Project Evaluation was drawn on heavily for the summary of data collection methods.

Some Notes on the Terms Used

Before going on, I think that it is important to clarify what I mean by some of the terms I use, because different authors use them differently. I use the term research method to refer to the set of steps that a researcher goes through to conduct an investigation, from designing an investigation to reporting the results of that in-vestigation. By research approach I mean the approach that underlies the research method. Examples of the research approaches I discuss in this book are survey research, causal-comparative research, correlational research, experimental rese-arch, case-study research, narrative research, grounded theory research, ethno-graphic research, and phenomenological research. Finally by the term research act,I mean the types of intellectual activities that are characteristic of a certain rese-arch approach. For example, quantitative description is a characteristic intellectual activity of survey research; explanation is a characteristic activity of many types of

Acknowledgments

-ments on this text. Thanks to Ilkka Jormanainen for his work on recreating several

support on this project.

volume, Modern Approaches to Digital Learning: DLL Project’s Results, published

11

was adapted with permission.

Preface

13

Chapter 1

Methodological Factors in Educational TechnologyResearch and Development

The methods-choice debate is one that resurfaces with regular frequency in the education research community. This regular resurfacing is not surprising, though,

are affected by the political, economical, and social currents of the times (Greene, Lipsey, Schwandt, Smith, & Tharp, 2007). The methods-choice debate helps de-termine what the research community, the media, government agencies, program funders, and the public accept as convincing evidence. And, among many other

-temological, ontological, and axiological positions.

--

luation in which laboratory research methods were favored for informing policy.

Ten years later the debate resurfaced as what are referred to now as the paradigm wars -cating qualitative methods -digm wars waned as the mixed-methods paradigm gained increasing acceptance. In its latest form, the methodological debate has resurfaced in response to the U.S.

adopts formal random sampling and experimental designs.

these reviews). Other high and low points in the history of the educational techno-

almost the entire body of previous research on distance learning because of its met--


-

about the desirability of generating evidence or about the need to consider the re-lative value of different methodologies. Instead, the debate is primarily over when, or under what circumstances, various methodologies provide the most useful, or

forward in the methods-choice debate is not to try to resolve the controversy, be-cause the controversy involves deeply-rooted disagreements that are not likely to go

It is with that piece of advice in mind that I put forth the goal of this chapter: to clarify the issues, I identify and describe some of the factors that are particularly important to consider when choosing methods for educational technology research and development. To make these factors more easily understood I break them into two categories, both of which are critical to understanding methods choice: factors

and how a research question is answeredthe research question are listed below:

The research problem.

The purposes of research and their corresponding traditions.

The state of the previous research.

-low:

The methods used in the previous research.

The research act implied in the research question.

The feasibility of the research.

Safeguards for propriety.

The degree of utility needed.

The degree of accuracy needed.

The degree and kind of generalizability needed.

•

•

•

•

•

•

•

•

•

•

15

The degree of stakeholder participation in the research process.

The degree of researcher participation in the research setting.

-arch questions in educational technology research and development. With hope, identifying and describing these factors and research questions will help improve the productivity of the dialogue about methods choice in educational technology re-search within and between researchers, funders, policy makers, and practitioners.

What I do not provide here is a concrete set of rules for determining what research approach to use, what data collection methods to use, what analysis methods, or what reporting methods to use over a large set of research situations. One reason is that what may constitute the best methods choices is somewhat subjective – hence, the deep-seated disagreements about methods choice that are not likely to go away. The other reason is that while I believe that there are probably some general guideli-nes that apply across cases, methodological choices are heavily context-dependent. The methods that bring about actionable evidence in one setting may not bring about actionable evidence in another. Methods choice involves a careful weighing of many factors to create the most actionable evidence possible.

The methods choice factors I propose here are drawn primarily from my experien-ce conducting methodological reviews of the educational technology literature and

as a research methodologist and from my experience teaching and supervising edu-cational technology students.

Factors Influencing the Formulation of the Research Question

Of primary importance in methods choice is the formulation of the research ques-

(Greene et al., 2007). While the research question may be of primary importance in determining the right research methods, there are a variety of factors that are of primary importance in determining the right research question – (a) the research problem, (b) the research purpose and its associated tradition, and (c) the state of the previous research. So, by substitution, the factors that are of primary importan-ce in formulating the research question are the foundation on which methodologi-cal choices are made. (For the sake of simplicity, hereafter I use the term research question to refer to all of the following: scholarly research questions, evaluation questions, and development tasks.)

-nal technology research problems, it might be useful to make a distinction between the different kinds of research questions. One distinction I make is between kno-wledge-base questions and empirical research questions. The other distinction is between procedural research questions and structural research questions.

•

•

Methodological Factors in Educational Technology Research and Development


As the name suggests, knowledge-base questions are answered through an exa-mination of the knowledge base. The following are examples of knowledge-base questions:

What is known about best practices in user-centered design?

Across studies, what are the academic effects of tools that help students vi-sualize algorithms?

-terventions?

On the other hand, empirical research questions are meant to be answered through

research questions; however, they might also be knowledge-base questions if they had already been answered in the previous research:

What are the effects of a new technological intervention on the long-term and short-term memory retention of vocabulary words?

new intervention?

In what ways do teachers and students report that a new intervention can be improved?

The utility of answering each of these types of research questions varies depending on the circumstances of the research context. Answering knowledge-base questi-

are the current views about best practice. Also, answering knowledge-base ques-tions by conducting a literature review can help clarify what are the unanswered questions that are of import to the research community. On the other hand, ans-wering empirical questions can help add to the knowledge base when there is no existing knowledge or when the existing knowledge is inadequate or in doubt. As I explain later, the choice of an empirical question is often predicated by the answer to a knowledge-base question. In addition to the distinction between knowledge-base questions and empirical research questions, a distinction can also be made between structural and procedural research questions (Stake, 1995).

be broken down into its structural components. For example, the general research

issues, or structures, that are implied in the general research question:

What do teachers experience in terms of the phenomenon of sense of com-munity in online learning?

•

•

•

•

•

•

•

17

What do students experience in terms of the phenomenon of sense of com-munity in online learning?

-munity in online learning?

In others words, structural subquestions unpack the salient issues in a general re-search question.

In addition to unpacking research questions by issue, Stake (1995) suggested that research questions can also be unpacked by the research procedures to be used. For example, if one were using a phenomenological procedure to answer the general

in phenomenological research, might be asked:

community?

What meanings can be inferred from these statements?

What themes emerge from these meanings?

What are the contexts of and thoughts about the experiences of sense of com-munity?

In short, procedural subquestions unpack an overall research question by breaking it down into the research procedures that will be used.

By breaking down the overall research question into procedural or structural sub-questions, the research process and research reporting process become clearer and more manageable. For example, the researcher can rely on the procedural subques-tions to naturalistically guide the research process. Also, one often-used method of structuring the discussion section is to organize it by research subquestions. In each section of the discussion, the author answers a research subquestion by refer-

-

topic of the following section.

The Research Problem

-

was that a lack of research does not necessarily mean that there is a need for rese-arch; research needs to be rationalized by both a need for and a lack of research on that topic. It is the research problem that demonstrates the need.

•

•

•

•

•

•



In this section, I make a distinction among three types of research problems in educational technology: the scholarly research problem, the evaluation problem, and the development problem. These types of research problems correspond with the different purposes of educational technology research and their associated tra-ditions, which are discussed in the next section.

-

basic research problem, as explained below:

In applied research, the problem [is based on a] need, which may be based

shortcoming in educational or psychological services. The need is not, ho-wever, the problem. Any one need may be the basis for a number of different research problems, depending upon the research evidence that is available and judgments about how to best address the need. For example, the need to avoid the erroneous placement of bilingual minority students in special education classes might lead to research on the sensitivity of school person-

for the validity of the instruments used to classify bilingual students, or on

basic research, the assumed need is for adequate knowledge, and reference to public policy or needs data is usually not necessary. (p. 2)

The most frequently seen types of educational technology research problems (or the needs upon which they are based), which are implied by the major educational technology research questions that I discuss later, include:

a disconnect between how educational theory informs technologies for edu-cation, and vice versa;

a need for information about the best methods for educational technology research and development;

a need for information about the best methods to implement and improve the utility of technological innovations;

a need for information about the effectiveness of certain kinds of technolo-gical interventions; and

a need for information about what factors moderate the effectiveness of cer-tain kinds of technological interventions.

which I refer to here as evaluation problems, are local in scope. For example, an educational organization might have a need to respond to a local problem within their organization – perhaps there is a high degree of student attrition that needs to reduced, a need to determine if a certain distance education program should

•

•

•

•

•

19

be continued or abandoned, or a need to determine if a program had been imple-mented as promised. Evaluation problems are typically articulated by program sta-keholders.

Development problems, as the names suggests, concern the development of inter-ventions or a lack of knowledge about how to best develop those interventions. For

or adapting existing technological interventions to solve current educational prob-lems.

The purpose of educational technology research; whether it is scholarly, evaluative, or developmental; is to solve the types of problems mentioned above. In the next section, I discuss these different research purposes and the traditions with which they are usually associated.

The Purposes and Traditions of Educational Technology Research

Typically, research in educational technology is conducted for one or more of the following purposes:

to answer questions that are important for the development of an educatio-nal intervention;

to answer questions that are important to local stakeholders to improve, come to understand, or assign value to a program; or

the primary reason for conducting research is helpful in understanding methods choice. The research traditions that correspond primarily with the purposes of re-search listed above are (1) design-based research, (2) evaluation research, and (3) education research, respectively.

It is important to note that research traditions can easily overlap one another. For

might serve as a starting point for the development of an educational intervention. In the sections below, I go into more detail about each of these research traditions.

The design-based research tradition. A research tradition that has gained much credibility over the past few years and that works well for developing educational

-

1.

2.

3.



First, the central goals of designing learning environments and develo-ping theories or “prototheories” of learning are intertwined. Second, de-velopment and research take place through continuous cycles of design, enactment, analysis, and redesign.... Third, research on designs must lead to sharable theories that help communicate relevant implications to prac-titioners and other educational designers. Fourth, research must account for how designs function in authentic settings. It must not only document

-ding of the learning issues involved. Fifth, the development of such accounts relies on methods that can document and connect processes of enactment to outcomes of interest. (p. 5)

As shown above, design-based research has many characteristics, the most distin-

the traditional research framework, summative, generalizable, and rigorous stu-dies are valued; however, because those types of studies are long and resource in-tensive, they are not feasible for the initial development of an intervention. Instead,

to determine how to improve an intervention. After the intervention has been per-fected through many cycles of design and testing, only then does it makes sense to conduct a summative, large-scale, and resource-intensive study. What is more, design-based research is an exploratory sort of activity and, as such, can lead to

basic form of design-based research, no particular set of methods is prescribed; the appropriate method is the one that leads to the type of information that is needed

-

framework and how they compare with the phases of other design traditions, such

-

framework begins with an informed exploration phase that includes problem iden-

characterization. The next phase, enactment, includes researching the initial in-tervention design, creating a prototype, and then developing a fully detailed in-

For example, the results of an evaluation might indicate that the intervention needs to be redesigned. After another cycle of implementation and evaluation, it could be

-on, adoption, adaptation, and summative evaluation of the intervention.

-

and links to various other design-based research resources. Kelly (2003) edited a

21

Figure 2. Merging of design and research processes into the integrative learning design framework. From “The Role of Design in Research: The Integrative Learning Design Framework,” by B. Bannan-Ritland, 2003, Educational Re-searcher, 32(1), p. 22. Copyright 2003 by Sage Publications, Inc. Reprinted with permission.



special issue of Educational Researcher that contains a selection of articles that provide a thorough overview of the design-based research tradition.

The evaluation tradition. Three purposes are generally assigned to evaluation. Similar to design-based research, one purpose of evaluation research is to collect data that can be used to improve an intervention (formative evaluation). Another purpose is to collect data that can be used for decision-making or assigning va-lue to a program (summative evaluation). Yet another purpose is to make sense of

evaluation, evaluation research answers questions that are primarily of interest to local stakeholders.

There are a variety of evaluation traditions to choose from, but a standard method for conducting an evaluation consists of the following steps:

Develop a conceptual model of the program and identify key evalua-tion points,

Develop an evaluation design,

Collect data,

Analyze data, and

Provide information to interested audiences (Frechtling, Frierson, Hood, & Hughes; 2002, p. 15).

There are a many good resources for evaluation research. For example, the U.S. Na-tional Science Foundation has created a series of useful, free, and practitioner-ori-ented evaluation handbooks. The latest in the series is Frechtling, Frierson, Hood,

The User-Friendly Handbook for Program Evaluation. It pro-vides an overview of the types of evaluation, the steps involved in conducting an evaluation, an overview of quantitative and qualitative methods, and a section on strategies for culturally responsive evaluation. Other handbooks in this series in-clude The User-Friendly Handbook for Program Evaluation: Science, Mathema-tics, and Technology Education (Frechtling, Stevens, Lawrenz, & Sharp, 1993) and The User-Friendly Handbook for Mixed Methods Evaluation (Frechtling & Sharp, 1997). Seminal books in evaluation research include Herman (1987); Mark, Henry,

The education research tradition. -tion research tradition. While design-based research and evaluation research may indeed be types of research on education, I have chosen to use the term education research to refer to research that answers questions that are of interest to the edu-cation research community. Although design-based research and evaluation rese-

their primary function.

•

•

•

•

•

•

23

There is no shortage of high quality books and resources on the practice of educa-tion research. They are too numerous to describe here, but I do recommend Gall, Borg, and Gall (1996) as an introductory guide to the multifaceted literature on education research.

The State of the Previous Research

For many reasons, becoming familiar with the state of the previous knowledge on a

of a research question. First, conducting a literature review or needs analysis ma-

suggests that research can contribute to knowledge in the following ways:

It can contribute to an already established theory or line of empirical research,

It can help establish a new theory,

It can meet a practical need, or

It can make up for a lack of needed information about a problem or issue. (p. 34)

For example, the literature review should make it possible to determine whether there are established theories already and to what degree they have been substan-tiated. Or, from an empirical research point of view, a literature review can show

-ween those elements are understood, and whether the causal mechanisms under-

state of the previous research will have considerable impacts on the focus of the current research. In some sense, the literature review is the mother of the research question.

Second, the literature review provides a basis for comparing and contrasting cur-

--

-cate themselves in what I call a research family and get a clear picture of how their

research lineage. By research family, I mean the individual rese-archers or groups of researchers that investigate the same topic. By understanding

calls the tribes and territories research lineage, I mean the histo-rical line of research on a particular topic. By understanding the history of research

•

•

•

•



-tify what is needed at the present, and predict what will be needed in the future. For

research map – a visual representation of -

arch lineage. An example of an outstanding research map, which was created by a

in Figure 3. It shows the relationships among the previous research studies and the

Figure 3. Example of a research map. By F. Girardin, 2007. Copyright 2007 by F. Girardin. Reprinted with the permission of the author.

The Five Major Categories of Educational Technology Research Questions

In the sections above, I discussed, in general, the factors that go into choosing re-search questions. In the section below, I discuss what types of research questions

empirical approach to identify the major categories of research questions in educa-tional technology, between and within the design-based, evaluation, and education research traditions. With hope, identifying and describing the categories of ques-tions that are often seen in educational technology research will help add clarity to the debate about which methods are appropriate for answering these kinds of research questions.

25

Design-based research questions. 5) has given some suggestions for the categories of research and development ques-tions that are of critical importance. The list below summarizes the major research and development questions mentioned there.

-

authentic setting.

outcomes.

-ved.

through the steps in the informed exploration stage. Those steps are problem iden-, and audience cha-

racterization.

Evaluation questions.research is to answer questions that are important to program stakeholders. So, it is no surprise that questions in evaluation research come from people who are invol-ved in a program or intervention. Typically, evaluation questions are generated in two phases – a divergent question phase and a convergent question phase. In the di-vergent question phase the evaluator collects an unedited list of research questions from the people involved in the program – for example, from the administrators, practitioners, and clients. In the convergent phase, the evaluator and sometimes the stakeholders decide which of the questions from the divergent list need to be

Because there has been no review of the questions in educational technology evalu-ation reports, a lateral review of the questions in computer science education evalu-ations might provide some insight into the categories of questions that educational technology evaluators strive to answer. I make the assumption here that the body of computer science education research is more or less generalizable to the body of

th grade evaluation reports of computer science education programs that had been publis-hed before March 2005. I inferred the evaluation questions from those evaluation reports. For example, if an evaluator had examined student achievement as an out-come, then I assumed that at least one of the evaluation questions had to do with

1.

2.

3.

4.



the ability of the program to bring about student achievement. The factors that were examined are equally telling. For example, if gender had been examined as a factor, then I assumed that there was an evaluation question about whether the program had a differential effect for male or female participants. At any rate, Table 1 shows that the outcomes that the evaluation questions most often dealt with, in decreasing order of frequency, were attitudes, enrollment, and achievement in core courses and that the interaction factors that were examined most often were gender, ap-titude, and .

Table 1. The Question Topics in K-12 Computing Education Program Evaluations.

Question # Question topic Frequency

(%)

Outcome (out of 67 outcomes in 19 cases)

1 Stakeholder attitudes 17 (25.4)

2 Enrollment 13 (19.4)

3 Achievement in core subjects 14 (20.9)

4 Computer science achievement 9 (13.4)

6 Teaching practices 5 (7.5)

7 Intentions for future CS jobs/courses 3 (4.5)

8 Program implementation 2 (3.0)

9 Costs and benefits 2 (3.0)

10 Socialization 1 (1.5)

11 Computer use 1 (1.5)

Factors (from 19 cases)*

12 Gender 3 (15.8)

13 Aptitude 3 (15.8)

14 Race/ethnic origin 5 (26.3)

A Methodological Review of Program Evaluations in K-12 Computer Science Education

Education research questions. In this section, I present the categories of research questions that have been of import to the educational technology community over

help give more meaning to the research questions of the present.

27

In Tables 2 through Table 4, I summarize the results of three empirical reviews of the questions asked in educational technology research articles (Burghar & Turns, 1999a; Burghar & Turns, 1999b; and Burghar & Turns, 2000). In those reviews, Burghar and Turns used an emergent coding technique to create an initial set of research question categories from all the articles published over a two to four year time period from three major educational technology forums – the proceedings of Frontiers in Education (FIE), Educational Technology Research & Development

, and Human-Computer Interaction . Articles published between

Table 2. Major Categories of Research Questions from FIE (1997–1999).

Table 3. Major Categories of Research Questions from ETR&D (1997–1999).



Table 4. Major Categories of Research Questions from HCI (1995–1999).

Comparing and Contrasting Questions across Forums and Traditions

Several differences across the research questions between forums and research tra-ditions exist. First, evaluation questions tend to center more on program effective-ness and its moderators than the research questions in design-based research or in

-

-

While there are some differences in research questions across the traditions and forums, nonetheless, there is enough similarity that overall categories of research questions across traditions and forums clearly emerge. By synthesizing the ques-

-

1999b, and 2000), it appears that the questions in educational technology can be

major types of educational technology research questions and the sources of the sub questions on which they were based.

29

Table 5. The Five Major Categories of Educational Technology Research Questions.

Questions about theory and practice. These types of questions deal primarily with how educational and psychological theories can inform educational technology practice and how educational technology practice can inform those theories. These types of questions also include theoretical questions about the disciplinary identity of educational technology. Two hypothetical questions in this category are given below:

How has the theory of active student response been implemented in educa-tional technology interventions?

-se lead to increased academic performance, as the theory suggests?

•

•



Questions about research and development methods. These types of questions deal primarily with the conduct of educational technology research and development. They deal with the methods that can be used for conducting educational technology research and development and how those methods could be improved. Three hy-pothetical examples of research questions in this category are provided below:

What research methods do educational technology researchers tend to use?

Under what circumstances do they use those methods?

What are the strengths and weaknesses of using those methods under a va-riety of different research situations?

Questions about the implementation of technology. Two of the reviews presented here involved sub questions that deal with the implementation of technology. Some hypothetical examples in this question category are given below:

What factors help increase the likelihood that a teacher will adopt an educa-tional intervention?

What factors help increase the likelihood that a student will adopt an educa-tional intervention?

Questions about the effectiveness of a technological intervention. This group of questions includes formative questions about how to improve an existing techno-logy and summative questions about how well an existing technology works in ef-fecting a given outcome. Some hypothetical examples in this question category are given below:

achievement?

Questions about factors that moderate the effectiveness of a technological inter-vention. While the previous group of questions deals with the main effects of a technological interaction, this group of questions deals with the factors that mode-rate the effectiveness of an intervention. Some of the factors that are examined in these questions deal with group versus individual learning, the academic subjects involved, the type of technological intervention used, the setting of the instruction, the level of previous experience, gender, age, and so on. Some hypothetical examp-les in this question category are given below:

•

•

•

•

•

•

•

31

from using the educational technology intervention?

-tervention is used in a different setting?

Some Caveats

These categories of research questions come from articles that were written bet-

research questions will have changed. Some of the questions will have been answe-red and new questions will have replaced them. These categories of research ques-tions are only meant as a guide for situating and evaluating a set of current research questions by examining the research questions and traditions of the past.

Factors That Influence How a Research Question is Answered

question. Those factors included (a) the research problem, (b) the purposes and as-sociated traditions of the research, and (c) the state of the previous research. I also

research. In this section, I discuss the factors that are important to consider when choosing methods to answer a research question once it has been formulated. Tho-se factors include (a) the methods used in the previous research, (b) the research act implied in the questions, (c) and some salient dimensions in methods choice, such as the level of accuracy, utility, propriety, and feasibility of an investigation.

answer that question. For example, one might have to modify a research question if it is not feasible or if it can only be answered through an investigation that cau-ses excessive harm to participants. While it is true that the nature of the research question implies what type of research methods are appropriate, the factors that

questions that can be answered.

The Methods Used in the Previous Research

The research methods and procedures used in previous research can be an invalu-able guide to designing research. The previous research will show which methods have worked well in the past and which have not worked so well, which variables are important to examine and which can be left out, and what contextual and en-vironmental factors need to be taken into account. What is more, if it is important

that were used in the past so that it is easier to make comparisons across studies.

•

•



a topic using a method that has not been used before. Anyway, one has to be know-

Research Acts Implied in the Research Question

In order to be able to link research questions to research methods it might be help-ful to review the categories of research acts (i.e., the types of actions one takes while doing research) that are implied by the research question. Some authors call these the purposes of research, but I call them acts here to not confuse them with the research purposes mentioned earlier (i.e., developing an intervention, answering

-munity).

Several authors have put forward suggestions on what are the research acts in social science research. These include Gall, Borg, and Gall (1996), Jarvinen (2000), Mark,

Yin (2003). However, I have found it helpful to use the following categories of re-search acts to describe the kinds of activities that researchers do and the kinds of research questions they ask. Those categories are , description, com-parison, correlation, experimentation, and explanation.

their degree of ability to explain causal mechanisms. For example, one has to identi-fy a causal factor to be able to explain how it works in a causal model. However, that linearity does not mean that one type of research act necessarily needs to precede another type of research act. For example, one does not necessarily need to do expe-rimentation or correlation to make a causal explanation. And, it does not mean that one cannot switch back and forth repeatedly between research acts. For example, in

and explanation to arrive at a theory based on the data gathered.

-led exploration or orientation, deals with becoming aware of a phenomenon, its contexts, and its constructs. For example, in order to create a quantitative survey to

would have to identify the types of reactions that one wants to measure. Similarly,

often the purpose of quantitative correlational research approaches and in many qualitative research approaches.

Description. One might use quantitative or qualitative description to describe the

an important factor in some phenomenon, then a researcher might do a qualitati-ve study to provide a detailed description of the attributes of client satisfaction. It might turn out that client satisfaction has several sub factors, and the researcher

then, back to description to describe their attributes. In quantitative descriptive

33

research, a researcher might give out a survey to measure the degree of satisfaction

-

Comparison. The next type of research act, comparison, consists of two or more instances of description and an analysis of how those instances of description dif-

-ming choices of male students differ from the gaming choices of female students.

In comparison studies, researchers do not manipulate variables and do not assign participants to treatment or control groups. The point of contrast in a comparison study is usually on some normally nonmanipulable attribute, such as age, mother tongue, gender, or previous experience, and so on. In health research, comparison studies (also called causal-comparative studies or case-control studies) are fre-quently seen because often it is not ethical or possible to assign people to treatment

Correlationthe (co)relationships between variables. For example, an educational technology researcher might be interested in knowing whether the use of a certain feature in a technological intervention is related with an increase in academic achievement. One practical outcome of examining correlations is that, under instance of high correlation, predictions about the behavior of one variable can be made from the behavior of correlated variables. Some examples of correlational research questions

-

One important note is that correlation does not prove causation. Many occurrences are correlated but are not causally linked. Confounding factors can mask an ac-tual association or make it appear that an association exists when one really does not. For example, there is a positive correlation between the sale of cooling fans and drowning deaths, but obviously, one does not cause the other. The confounding factor is that the heat of the summer months is correlated with both an increase in the sale of cooling fans and with the number of people who go swimming (and subsequently drown).

While correlation does not prove causation, it can be an initial clue that a causal relationship exists. The type of research act discussed next, experimentation, can be used to help determine if a causal relationship does indeed exist.



Experimentation (causal description).

experiment, for example, by introducing a new version of a technological tool and comparing the academic results between the phases when the students used the newer version of the tool and the phases when the students used the older version. Experimentation might be thought of as a special case of comparison in which the researcher changes something about a situation and then makes a comparison. Or

-

2002, p. 9) are described. Some examples of experimental research questions fol-

While experimental research is prized for its ability for the causal description of phenomena, there are a few important caveats about experimental research and causal claims that need to be mentioned. First, while experimental research can ge-nerate information that can help support causal claims, it dos not guarantee causal certainty. Experimentation is a means, not an end, to arriving at sound causal

Experiments yield hypothetical and fallible knowledge that is often depen-dent on context and imbued with many unstated theoretical assumptions. Consequently, experimental results are partly relative to those assumptions and contexts and might well change with new assumptions or contexts….to the extent that experiments reveal nature to us, it is through a very clouded windowpane. (p. 29)

-

-serving the light going on and off, one could easily use causal descriptive reasoning

that -wing why or how

Causal explanation. As mentioned above, experimentation produces data that is useful for causal description. Unlike causal description though, which is used for determining that a certain cause leads to a certain effect, causal explanation can be used for explaining why or howexplanations often come about by examining a phenomenon in great detail.

for why turning on a light switch causes the light bulb to go on, a researcher using causal explanation would look into the walls and examine the wires, bulbs, swit-ches, fuses, circuit breakers, and so on. From that, the researcher could come up

35

from a bulb. By doing pattern matching between what elements theoretically are needed to make a light bulb work and what elements are actually in place, the rese-

There have been many useful descriptions of how causal explanation works. Scriven (1976) describes causal explanation as a research act that uses a modus operandiapproach – the same approach that a doctor uses to make a diagnosis or the same approach that a detective uses to catch a criminal. In short, in the modus operandi approach an observed pattern (e.g., a set of symptoms that a patient has) is matched with a known set of patterns (e.g., the set of symptoms associated with a particular illness). The often heard phrase in criminal investigation programs – this (pattern of evidence) is consistent with that (criminal phenomenon) – is evidence of the mo-dus operandi/pattern matching approach in action. Mohr (1999) describes causal explanation as a research act that uses physical causal reasoning – the same reaso-ning that lets physicists predict the movement of objects. By knowing the theories that underlie physical causes, physicists can make causal explanations of physical phenomena. However, the theories of human behavior are much different than the

explanation, not exclusively, as multiple cases of causal description. Whichever cha-racterization of causal explanation one adopts, the essence is that it allows one to explain why or how causal systems work.

Dimensions in Research Acts

correlation, experimentation, or explanation, it is also helpful to consider other di-mensions – including whether the research adheres to qualitative or quantitative traditions and the degree to which the research is generalizable, accurate, feasible, appropriate, and useful.

General vs. local. One key dimension in research is to what degree results are local or general – that is, the degree to which results are generalizable across units, treat-

– that is, conclusions that are meant to be generalized only to local participants, treatments, outcomes, or settings. For example, in a program evaluation, it is pro-

-ated because the funders of the evaluation are primarily interested in the results of their program and not necessarily interested in the results of other programs. But, stakeholders in similar programs would probably be interested. In most cases in traditional education research, conclusions have more worth if they are genera-lizable – that is, if the conclusions apply to other units, treatments, outcomes, or

-

Qualitative vs. quantitative. Traditionally, some of the research acts described above have been associated with either qualitative or quantitative traditions. For example, case study research has traditionally been associated with qualitative re-search; experimental research has been traditionally associated with quantitative



research. However, there is no reason that either quantitative or qualitative met-hods could not be used in any of the research acts. Theoretically, one could do an experiment in which only qualitative data were collected. Similarly, one could do a case study in which only quantitative data were collected, as Yin (2003) points out. There is growing support for combining qualitative and quantitative types of data

2006; Johnson & Onwuegbuzie, 2004).

. Another dimension of research is to what degree the

hypothesis. In some types of research, like grounded theory, the researcher refrains from making a research hypothesis until the data begins to accumulate. In that type of research, the researcher might have an idea or a topic to explore but does not try to gather evidence for or against any particular proposition. One could say that

often considered to be useful when there is little or no understanding of a phenome-non or when a line of research gets stuck and new hypotheses need to be generated

-ori hypothesis that is usually based on previous research or theory. In this type of research, one arrives at knowledge by positing a variety of hypotheses, testing the validity of those hypotheses, and eventually deciding on which hypothesis of many is the most likely. For example, a researcher might posit from theory or previo-us research that the method of instruction is more important than the medium of instruction in terms of student academic achievement. The researcher would then conduct an experiment in which evidence could be gathered that would either sup-port or discredit that hypothesis.

--

neralize across units, treatments, outcomes, or settings. While replication research

creates new information, replication is nevertheless a cornerstone of science and provides an excellent opportunity for beginning researchers to hone their craft.

phenomenon. Exploratory research that keeps ending up at the same conclusion can help build strong evidence for, or against, a claim.

Level of participation. This dimension involves two aspects: (a) how closely rese-archers become involved in the phenomenon and setting they are studying and (b) how closely the participants in a study become involved in the research process. In some types of research, like ethnographic research, the researcher becomes a

-her types of research there is a strict line between the researcher and participant. Also, in some types of research; like participatory action research or participatory evaluation; the participants collaborate with a researcher or the researcher acts as

37

Bradbury, 2001). In other types of research, the researcher is the only person who participates in the design, data collection, analysis, and reporting of research. The-re are many ongoing debates and discussions about the pros and cons of the diffe-rent degrees of researcher and stakeholder participation, but they are too numerous to go into here.

Accuracy. In some cases, it is necessary for research to have much accuracy; that is, it must produce sound information that is (a) comprehensive, (b) technically adequate, and (c) with judgments that are logically aligned with the data collected

makes sense to have much accuracy when lives and well-being are at stake or when policies or programs are involved that affect many people or require large amounts of resources. Also, in some cases accuracy is expected as a matter of fact, such as in dissertations or in articles in prominent journals. In other cases, however, less accuracy is acceptable. For example, it would certainly be impractical to conduct a randomized group experiment in every cycle of a design-based research study.

-ces on answering a large set of evaluation questions with much accuracy. Instead, it might be better to focus on answering the most important evaluation questions with more accuracy and answering the less important evaluation questions with less accuracy. Accuracy and feasibility are often tradeoffs.

One aspect of accuracy is reliability – the degree to which measurements are con-sistent over time, situations, or raters. Having high reliability is important in some types of research and less important in others. For example, high reliability of me-asurements might be important when creating an instrument to predict success in a graduate program in educational technology, but it would be less important in the early stages of a design-based research study when several informal investigations are being conducted to gain insights into how an intervention can be improved.

Utility

can vary across research traditions. For example, in formative evaluation the goal is to create information that will be used to improve educational programs or poli-cies. The stakeholders need to be able to easily understand and use the evaluation information. In other types of research, like basic research, the utility of the rese-arch is expected in the future. Although the utility of basic research is latent, basic research has been shown to be an essential factor in a large proportion of major

Propriety -

1994), is a critical dimension in all types of research. However, different types of research have different types of propriety issues. For example, meta-analytic rese-arch – research about research outcomes – generally does not involve propriety is-sues dealing with the treatment of human participants; however, it does involve ot-



the ethical treatment of human participants is a critical factor in the choice of a methodology. For example, deciding on whether to use a randomized experiment or some other research design can hinge on the ethical issues involved. For example, Boruch (2007) puts forth a set of questions, which follow, to determine if a rando-

Is the social problem serious? If the answer is yes, then consider a ran-domized trial to evaluate the purported solutions. Otherwise a trial is not worthwhile or ethical.

Are purported solutions to the problem debatable? If the answer is yes, then consider doing a randomized trial. If the answer is no, then adopt the purported solution.

Will randomized trials yield more defensible (less equivocal and unbi-ased) results than alternative approaches to estimating effects? If the answer is yes, consider mounting a randomized trial. If the answer is no, then rely on the alternative approach.

Will the results be used? If the answer is yes, then consider mounting a randomized trial. If not, forget about the trial, or redesign the rando-mized trial so that rights are protected. (pp. 56 – 57)

Feasibility. Feasibility, the degree to which research does not consume more time, money, or resources than necessary, is also an important consideration in research design. Some research traditions, like design-based research, are based on repea-ted, rapid cycles of investigation. And, therefore, it would be impractical, if not im-possible, to do a randomized trial each iteration. In short, one has to weigh the costs

Also, what may be feasible in one setting might not be feasible in another.

Conclusion

In summary, there is no simple answer for which method is most appropriate for a

the research problem, (b) the purposes of the research and their corresponding traditions, and (c) the state of the previous research. There are also factors that

questions, (e) the feasibility of the research, (f) safeguards for propriety, (g) the degree of utility needed, (h) the degree of generalizability needed, (i) the degree of stakeholder participation in the research, and (j) the degree of researcher partici-pation in the research setting, among other factors. For each research situation, the

research methods to use.

•

•

•

•

39

To aid in this process of considering and weighing the methodological factors men-tioned here, a list of key questions in educational technology methods choice can be found in the Appendix to this book. With hope, this list of questions will be useful for helping student researchers think through the considerations involved in met-hods choice and as an instructional aid for those who teach or supervise students of educational technology.



Questions to Consider

What are the similarities and differences between design-based research, evaluation research, and education research? In what ways can they be used to inform each other?

In this chapter I reported on the types of educational technology research questions that were often asked in the past. What types of educational technology research questions do you think will be asked in the future?

-arch acts implied in the questions?

What is your response to my claim that quantitative or qualitative data can be useful in any of the research acts mentioned above?

1.

2.

3.

4.

5.

41

Chapter 2

Types of Research Approaches in Educational Technology Research and Development

-cational technology research. In this chapter, I describe the major types of research approaches that are used in educational technology research and how they relate to the research acts mentioned earlier.

Although there are many ways to categorize research approaches, I have chosen to use an adaptation of the categories presented in Gall, Borg, and Gall (1996) for the

-proaches.

The quantitative approaches that I discuss are:

Survey research,

Experimental research.

Also in the section on experimental research I discuss the major threats to internal validity and the basic categories of experimental research designs.

Narrative research,

Ethnographic research,

•

•

•

•

•

•

•


Grounded theory research.

Although some of these research approaches are used less often in educational technology research than others, I, nonetheless, have included the rarely seen ap-proaches (e.g., narrative research and ethnographic research) here because of their high potential for answering important questions in educational technology rese-arch and development.

Survey Research

Survey research, whose related research act is attribute description, typically answers questions that deal with the frequencies or proportions in a population. An election poll is an example of survey research. Survey researchers might, for example, be interested in the percentage of people who will vote for a given candida-

-nal software. While questionnaires are common quantitative descriptive research instruments, any type of observational method – from standardized tests to direct observation – can be used to determine frequencies or proportions within a popu-lation.

In general, survey researchers either conduct surveys or censuses. In surveys, data are only collected from a representative sample and the results are inferred to the whole population. Surveys are conducted because it is often impractical or impos-sible to collect data about an entire population. For example, in an election poll, survey researchers might interview a representative set of individuals about how they are going to vote and, if the set is truly representative, the researchers can make a good prediction about how the population is going to vote. In censuses, survey researchers collect information about every individual in the population. For example, the U.S. government conducts a census in which they strive to gather information about every individual living in the United States. However, because of the time and expense involved they only conduct a census once every ten years.

To make a valid inference from a sample to a population, it is critical that the sample be representative of the population. One method that can help achieve the represen-tativeness of the sample is through random sampling – by choosing cases randomly

random selec-tion,does not ensure that a sample is representative. It is unlikely, but possible, that a random sample will not constitute a good representation of the population. The more cases that are randomly sampled, in a statistical sense, the more likely it is that the percentages in the sample are near the percentages in the population.

Often it is impractical or impossible to take a random sample so, instead, resear-chers take a purposive sample of typical cases – sampling in which the goal is to purposefully choose a set of cases that is representative of the population. Basi-cally, one can make an argument for the representativeness of a purposive sample, or any other type of sample, by assessing in what ways the sample is similar to the

•

•

43

example, if a researcher wanted to make an argument that the students in his or her

previous experience in computing, their socio-economic status, and so forth, and compare that information with information that is known about the population. If it turned out that the students in the class were all female Harvard students who went to a computer-science-oriented high school, then it might be hard to make an

-puter science students in general.

Besides purposefully sampling typical cases, a researcher might also use other ty-pes of purposive samples. For example, a researcher might be interested in extreme cases and purposefully sample the cases that exhibit extreme qualities.

Causal-Comparative Research

In causal-comparative research, which is also called case-control research, one ty-pically compares a group to one or more different groups or compares the same group at different times and does not manipulate a variable. For example, a rese-archer might examine whether male students differ from female students in how they experience community in online courses or the researcher might examine how perceptions of online learning have changed over the years.

-ful in situations in which an effect is known, but the cause is not known. For examp-le, causal-comparative research might be used to determine what caused students to drop out of an educational program by determining how those who dropped out

useful in those situations where the cause (the independent variable) cannot be ma-

gender or background. In these situations, comparative research can be used to gather evidence for, or against, a cause-effect relationship between variables; ho-

comparative research.

-ses other than the one being investigated. For example, imagine that a researcher

-tional classroom teaching and suppose that it turns out that students in traditional classrooms have better academic achievement than students in online classrooms.

-ning caused the better academic achievement because there are so many other factors that could have caused the increased achievement. For example, perhaps it is the case that online learners typically have full-time jobs and cannot devote as much time to study as students who can attend traditional classroom lectures. So, the real cause of the difference in academic achievement could actually be available study time. The crux of the matter is that in causal-comparative research it is extre-



that systematically rules out confounding causal factors.

In longitudinal research, which is appropriate for asking questions about develop-ments over time, there are four types of studies that are often used. In trend stu-dies a researcher chooses the same theoretical sample, but measures the theoretical sample at different times and with different participants. For example, in a trend

computer science students and repeat this every year – e.g., 30 students from all of the computer science students beginning in 2000, then 30 students from all the computer science students beginning in 2001, then 30 students from all the compu-ter science students beginning in 2002, and so on. In this way the researcher could

computer science students over time.

Another type of longitudinal study is the cohort study in which a researcher choo-ses a different sample at different points in time and the population remains cons-tant. For example, a researcher might choose a different set of 30 students, each year, from the cohort of students who began in 2000. To illustrate, in 2000 the researcher would sample thirty students from the 2000 cohort. In 2001, the rese-archer would choose another sample of students from the 2000 cohort and in 2002 the research would choose yet another sample of students from the 2000 cohort.

The third type of longitudinal study is a panel study in which a researcher chooses the same set of cases every year. For example, the same set of 30 students who originally had been chosen would be the same set of students who would be chosen every subsequent year.

obtained at only point in time, but the cases are chosen from different age groups. For example, in a cross-sectional study one might choose 30 students who started in 2001, 30 students who started 2002, and 30 students who started in 2003 to de-termine if there is a difference in the cohorts of incoming students over time.

Each of these types of longitudinal research has its advantages and disadvantages. While panel studies are more sensitive to small changes over time than are trend,

problems with attrition (i.e., problems with people dropping out of the study) and

other hand, trend cohort, and cross-sectional studies are easier to carry out and do not suffer from attrition or repeated measurement problems; however, they do not lead to data as rich or valid as panel studies.

Correlational Research

In correlational research, the researcher is interested in how one or more variab-les change in relation to how other variables change. For example, a correlational researcher might be interested in whether academic achievement increases, stays the same, or decreases as the amount of dialogue in an online course increases. Besides examining the relationships between two variables, correlational research

45

can be used for prediction or for examining the relationship between two or more

are plotted on two dimensions. Figures 4, 5, and 6 are examples of scatter plots on which the values on the horizontal axis (x) and the values on the vertical axis (y) are charted together. If two variables increase or decrease together they are said to be positively correlated (see Figure 4). If one variable increases while the other dec-reases, the variables are said to be negatively correlated (see Figure 5). And, if the variables tend to be independent, that is if an increase or decrease in one variable in not accompanied by an increase or decrease in another variable, then the variables are said to have no correlation (see Figure 6).

Figure 4. Scatter plot of a strong positive correlation, (r = .93).

Figure 5. Scatter plot of a strong negative correlation, (r = -.91).



Figure 6. Scatter plot when there is practically no correlation, (r = -.02).

educational technology, it might be important to note that the most frequently

denoted by an r, and its values range from 1.0 to -1.0. A positive value of r indicates that there is a positive correlation; a negative value indicates that there is a negative correlation, and a value of zero indicates that there is no correlation. Whether the correlation is positive or negative shows the direction of the correlation. The closer a correlation is to 1.0 or -1.0, the closer it is to being perfectly correlated. When the-re is a perfect correlation it is possible to know what the value of one variable will be by knowing the value of another. The nearness to 1.0 or -1.0 shows the degree of the correlation.

Figures 4, 5, and 6 have an r of .93, -.91, and -.02, respectively. So, the scatter plot in Figure 4 shows a very strong correlation in the positive direction, the scatter plot in Figure 5 shows a very strong correlation in the negative direction, and the scatterplot in Figure 6 shows that there is practically no correlation. Since there is practically no correlation in Figure 6, the direction is largely irrelevant.

One note about correlational research, which also applies to comparative research,

there might be a correlation between variable a and variable b because variable acauses a change in variable b; because variable b causes a change variable a; or that a third variable, say variable c, is correlated with and causes a change in both va-riable a and variable b. On this point, Gall, Borg, and Gall (1996) wrote,

(i.e., positive or negative) of the relationship between two or more variables

[other methods, like the experimental method]. (p. 414)

47

In the following section, I discuss the experimental method for arriving at causality

Experimental Research

Experimental research is a subset of comparative research in which the resear-cher manipulates a variable (called the independent variable) and determines how changing that variable effects one or more outcome variables (called depen-dent variables). For example, an experimental researcher might implement a new technological tool (the independent variable) in a classroom and then see how the implementation of that tool effects student academic achievement (the dependent variable). The crux of experimental research is that a researcher compares a factual condition (i.e., what happens when the variable is not manipulated – the control or baseline condition) with one or more counterfactual conditions (i.e., what happens when the variable is manipulated – the experimental condition) so that the unique

carefully chooses research designs to lessen the threat that something other than the variable being manipulated causes the outcome that is measured. Fortunate-ly, much research has been conducted on these threats that can lead to erroneous causal conclusions. These types of threats are called threats to internal validity and are the subject of the following section. Understanding these threats helps makes it possible to understand the rationale for the different types of experimental rese-arch designs.

Major Threats to Internal Validity

To give meaning to experimental research designs, it is necessary to understand the -

cularly suited to determining causal relationships, but threats to internal validity are factors that can make it appear that the independent variable is causing a result when in fact it is a different variable or set of variables that is causing the result. In other words, threats to internal validity in experimental research are hidden cau-sesmajor threats to internal validity: history, attrition, regression to the mean, matu-ration, instrumentation, testing, and selection.

History. Sometimes an unintended event outside of the experiment can affect the outcomes of an experiment. For example, suppose that there is loud construction work going on in a building where students are taking a test. It might turn out that the construction noise, and not a poorly designed intervention, could be the cause of low test performance.

Attrition. -metime the drop out is random and sometimes the drop out is non-random. When the drop out is connected with some factor that could affect the results, then attri-tion becomes a threat to internal validity. Imagine that there are 30 high-achieving students and 30 low-achieving students in an experiment. Suppose that 20 of the 30 low-achieving students dropped out of the experiment because the intervention



achieving students and the outcomes would appear better than they actually would have been if the low-achieving students had stayed in the experiment.

Regression to the mean. there is a tendency for an individual who has an extremely high or extremely low

measurement. This phenomenon is called regression to the mean. the mean becomes a problem when individuals are selected into an experiment on the basis of a selection test and when that selection test is also treated as a pretest. For example, if a researcher were interested in only investigating high-aptitude stu-dents, then he or she might give out an aptitude test (i.e., a selection test) to a diver-se set of students to determine which students have high-aptitude and, therefore,

if the researcher treated the results of the selection test as if they were the results of a pretest. In this case, for example, if some of the selected students had, by chance, gotten atypically high scores on the joint selection/pretest, their scores on a subse-quent measure probably would be nearer to their typical scores (i.e., scores nearer to the mean) just because of regression to the mean. This is because it is statistical-ly unlikely to get atypically high scores just by chance, two times in a row. In this case, regression to the mean would make the intervention look less effective then it actually might be. The way to avoid regression to the mean is to keep the selection and pretest measurements separate. A researcher should give a selection test, select students, then give a separate pretest and posttest.

Maturation. -gardless of an intervention. For example, it is a fact that as young students progress through puberty and onto adulthood their social skills typically develop. So in an experiment dealing with an intervention designed to increase the social skills of teenagers, then maturation rather than the intervention might be the cause of inc-reased social skills over time.

Instrumentation. When measurement instruments, including human raters, chan-ge over time, then the experimental outcome could be the result of the change of the instrument rather than the intervention. Say for example that a pretest turns out to be much harder than a posttest. In that case, a totally ineffective intervention

the two tests was what caused the apparent change between pretest and posttest measurements.

Testing. Sometimes simply the act of taking a test, or being measured, can affect the outcomes of that test or measurement. Organizations who train individuals to take tests capitalize on this fact by having their clients take a test repeatedly. The act of repeatedly taking a test will often cause the results of a test to improve with each repetition.

Selection. The threat of selection occurs when the participants in the experimental and control conditions are not equal to begin with. Say for example that a group of researchers conduct an experiment in which they introduce an educational inter-vention in one classroom, do not introduce the intervention in a different classroom,

49

and then measure academic achievement at the end of the year in both classrooms. It might be entirely possible, even probable, that one class had better performance at the beginning of the year and, therefore, that class would be expected to have better scores at the end of the year, regardless of the intervention.

One way to reduce the threat of selection is to randomly assign participants to eit-her control or experimental conditions. Strictly speaking, in experimental research in which there are two or more groups, if there is not random assignment then that type of investigation is called a quasi-experiment, not an experiment.

There has been an ongoing debate in the education research community about the

-

technology, it is a rare occasion indeed when a true experiment can be conducted. In education, participants often are entitled to self-select their interventions and it

-rol groups by chance, especially when the intervention is thought to be effective.

American Evaluation Association argue that carefully conducted quasi-experimen-tal research can lead to results as valid as the results in experimental research. See

One way to avoid the threat of selection is to use group experimental designs that include a pretest. Even if there are differences on the pretest, in some cases, sta-

adjust for the differences in pretest scores.

Group Experimental Designs

There are two basic dimensions to group experimental research designs. There is a within-subjects dimension and a between-subjects dimension. In the within-sub-jects dimension, the same participants (or participants who are matched) are me-asured at different points in time. For example, a group of participants might be gi-

data would be compared with his or her own posttest data to determine if the in-tervention had been effective. In the between-subjects dimension, different groups of participants are measured. For example, a group that received an intervention might be compared with another group who did not receive the intervention. The researcher would then compare the outcomes of one group to the outcomes of the other group to determine if the intervention had been effective. With these two dimensions in mind, it is then possible to make sense out of most of the basic ex-perimental research designs. As I will explain below, some experimental research designs are only within-subjects, some are only between-subjects, and some are a combination of both. The experimental designs that I deal with here are discussed

The one-group posttest-only design. As the name implies, in this research design there is only one group who is given a test after an intervention has been delivered.



This design is subject to almost all threats to internal validity, because many factors besides the intervention could have caused the outcome. The one-group posttest-only design is only recommended when there is considerable prior knowledge about the dependent variable, so that a comparison can be made between the outcomes of the experiment and the outcomes predicted by previous research or theory (Sha-

group posttest-only design is common in educational technology research.

One popular, but questionable, variation of this design is to use a retrospective posttest in which participants try to judge how much they learned as a result of the intervention. These types of retrospective reports are however known to be quite biased (Silka, 1989). These retrospective tests might also be thought of as survey

The posttest-only design with non-equivalent control groups. In this between-subjects design, the researcher compares the posttest results of an experimental

most threats to internal validity. For example, if a history threat were to happen it would probably happen to both groups equally and, in effect, would cancel itself out. Although this design lessens many threats, it is still vulnerable to the threat of selection. The groups may differ initially before the intervention and, therefore, would probably differ even after an ineffective intervention. Adding a pretest me-asurement is one way to improve on the posttest-only design with non-equivalent control groups.

The one-group pretest-posttest design. This is a within-subject design where one group is given a pretest measurement, an intervention is introduced, then a posttest measurement is given. In this design, selection is not a threat because there is not a control group; each participant acts as his or her own control. History, attriti-on, testing, regression to the mean, instrumentation, and maturation are the major threats in this design. It can be improved upon by adding a control group.

The untreated control group design with dependent pretest and posttest samples. This design has both within-subjects and between-subjects dimensions. There is a control group and experimental group and there are measures at two or more points in time. Within the groups, the pretest data for each participant is matched with his or her own posttest data. The major advantage of this design is that it combines fea-tures of within-subject and between-subject designs and, therefore, greatly reduces threats to internal validity. Also, in some cases it is possible to statistically control for differences between the experimental and control groups.

The list of experimental research designs mentioned above is by no means comp-lete; however, most of the other designs are variations on the ones above. Shadish,

-ning more about experimental research design.

Single-Participant Research

Single-participant research is a type of experimental research in which the empha-sis is on the performance of an individual over time rather than on group perfor-

51

mance. In single-participant research, the researcher takes several measurements of one case over time. Over those time periods the researcher will alternately imple-ment and withdraw an intervention (or variations of an intervention) and see how the outcome measurements change. For example, a researcher might measure a

-

and see if the results return to the baseline. This type of single-participant design is called an ABA design, where A is a time period when there is no intervention and where B is a time period where there is an intervention. Alternately, a researcher could have a baseline period, introduce the intervention, introduce a variation of the intervention, reintroduce the original intervention again, then withdraw the

time period where the variation of the intervention is introduced.

Typically, single-participant data are analyzed visually. Figure 7 is a graph of hypot--

rements in each phase. Suppose that the result being measured is percent of inter-vals observed in which a student was exhibiting off-task behavior (e.g., disrupting others). In the initial A phase, the intervals of off-task behavior were high. In the B phase when an intervention was implemented, the intervals of off-task behavior

intervals of off-task behavior decreased again. This high-low-high pattern provides evidence that the intervention worked in decreasing off-task behavior. Single-par-ticipant graphs are also useful because they intuitively illustrate variation between measurements and trends over time. Kazdin (1982) is a classic and highly recom-mended text on single-participant methods.

5060708090

100No intervention

(Phase A)Intervention (Phase B)

No intervention(Phase A)

010203040

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Figure 7. Example of a single-participant graph.



Single-participant research is well suited for design-based research. First, it does not require many participants, as group experimental research does, but it can ge-nerate strong evidence about causal relationships nonetheless. Second, single-par-

intervention to quickly identify which variations of an intervention work and which do not. The researcher can also make impromptu variations to the environment to see which environmental factors affect the results of the intervention and which do not. One downside though to single-participant research is that it only works in situations where testing is not a threat to validity, since many measurements need

-search, it is surprising that single-participant studies are not used more frequently in design-based educational technology research (as I show in the next chapter).

The Five Major Qualitative Approaches

for creating thick, rich description (Geertz, 1973). Also, unlike quantitative desc-riptive research; which involves answering who, what, where, how many, or how much questions; qualitative research often involves how or why questions.

Through thick, rich description qualitative methods can be used alone or paired

Yin, 2003). To give an example of pairing research methods, a qualitative study might be initially used to create a theory about a phenomenon and identify the im-portant variables, experimental research could be used to determine if that theo-ry holds up and is generalizable. Or alternately, qualitative research could follow

examined in detail.

narrative research, phenomenological research, eth-nographic research, case study research, and grounded theory research.

Narrative Research

Narrative research is suited for situations when it is meaningful to tell the life sto-ries of individuals. From these stories, insights about social phenomena and so-cial meanings can be drawn. A narrative researcher typically collects data about

-cational technology research one might tell the life story of a teacher and focus on

53

Phenomenological Research

The goal of phenomenological research is uncovering the essence of a lived experi-

interviews with individuals who have experienced a certain type of phenomenon. After bracketing (i.e., revealing) their own experiences about the phenomenon, the researcher picks out meaningful statements from the interview texts, gives meaning to those statements, and then goes on to provide a rich description of the essence of the phenomenon. In educational technology, for example, one might interview people who have participated in online courses and try to gain an understanding of how those people experience the phenomenon of community in online courses.

and van Manen (1990).

Ethnographic Research

In ethnographic research, the ethnographer strives to describe how a culture-sha--

technology, one might consider people who regularly participate in role-playing ga-mes as a culture-sharing group and, after collecting much detailed information, write a thick and rich description of how the culture-sharing group works and about

-

Wolcott (1994, 1999).

Case Study Research

contemporary phenomenon within its real life context, especially when the boun-

of a case study are to develop an in-depth understanding of a case, or multiple ca-ses, and to gain insight into the interaction between the phenomenon and the case. A case study researcher uses several sources of evidence; such as archival records, direct observation, interviews, and documents; and then analyzes that data though pattern-matching, explanation-building, and addressing rival explanations (Yin,

Stake (1995), and Yin (2003).

Grounded Theory Research

The goal of grounded theory research is to create a theory that is based on the data

questions in grounded theory research concentrate on how a phenomenon was ex-perienced and how the process unfolded. Typically a grounded theory researcher



conducts between 20 and 60 interviews, depending on how long it takes to reach a point of saturation – a point when no new information is gained from collec-

(open coding), assembles the data in new ways (axial coding), creates a story line that connects themes, or suggests new hypotheses. The ideal result of a grounded

55


How do each of the research approaches presented in this chapter relate to

Would you call a posttest questionnaire in which students are asked to ret-rospectively judge how much they learned from an educational intervention an example of survey research or an example of one-group posttest-only ex-perimental research?

-tional research?

Which of the research approaches above do you most often associate with educational technology research and development? Why?

-mized experiments are the gold standard of research? Or do you agree with others that research approaches other than randomized experimental rese-arch, such as qualitative research or quasi-experimental research, can lead to equally suitable evidence?

1.

2.

3.

4.

5.

6.


57

Chapter 3

A Meta-Synthesis of Methodological Reviews of Educational Technology Research

Thus far I have discussed the theoretical dimensions that can be used to chart educa-tional technology research. In this chapter, I describe the research methods that edu-cational technology researchers actually use in practice. I also compare those methods with the methods used in traditional education research.

have been used in the past to inform the research of the present and future. My in-tention is to identify the strengths and weaknesses of the past research so that the st-rengths can be built upon and the weaknesses can be remedied. The second is to yield information that can advance the dialogue on educational technology as a discipline.

This chapter is organized around the following questions:

What are the meta-categories that can be used to subsume the research catego-ries in other methodological reviews of educational technology research?

What are the proportions of quantitative, qualitative, and mixed methods rese-arch that educational technology researchers have tended to use?


research proper?

In what proportions do educational technology researchers choose (a) research approaches, (b) experimental research designs, and (c) measures?

How do educational technology researchers tend to report educational techno-logy studies?


1.

2.

3.

4.

5.

6.

7.


Since there have been many instances of research about research (meta-research) in educational technology already published, in this chapter I will simply concent-rate on synthesizing that research. To answer Questions 1, 2, and 3, I will report on

-dolph (2007a) and a methodological review of education research proper by Gorad and Taylor (2004). (By education research proper I refer to the education research literature published outside of forums typically associated with educational techno-logy.) Finally, to answer Questions 5, 6, and 7, I rely on a review of educational

2004 – one of the premier IEEE-sponsored conferences in educational technology.

To answer Questions 5, 6, and 7, I generalize the articles in the proceedings of the

What are the Meta-Categories that Can be Used to Subsume the Research Categories in other Methodological Reviews of Educational Technology Research?

Methodological reviews, literature reviews that concentrate on research practice,

-tify the complete set of those methodological reviews of educational technology.

It was a quantitative review (e.g., a content analysis) of research prac-tices, not a literature review in general or a meta-analysis, which fo-cuses on research outcomes.

education.

The review was written in English.

The candidate review’s categories were able to be subsumed under me-tacategories.

The review’s articles did not overlap with another review’s articles. (When reviews overlapped, only the most comprehensive review was taken.) (p. 21)

Thirteen methodological reviews of the educational technology literature were

•

•

•

•

•

•

59

-

Nodder, & Baker, 2001. Table 6 shows the research questions that were asked in

-rison, 2004) were ultimately excluded because they did not meet all of the criteria

the research questions that were addressed in each of the previous methodological reviews.

Table 6. Research Questions in Past Educational Technology Methodological Re-views.

Review Overview of research questions

Alexander & Hedberg, 1994

What, and in what proportions, are evaluation models used in evaluations of educational technology?

Caffarella, 1999 How have the themes and research methods of educational technology dissertations changed over the past 22 years?

Clark & Snow, 1975 What research designs are being reported in educational technology journals? In what proportions?

Dick & Dick, 1989 How do the demographics, first authors, and substance of articles in two certain educational technology journals differ?

Driscoll &Dick, 1999

What types of inquiry are being reported in educational technology journals? In what proportions?

Klein, 1997 What types of articles and what topics are being published in a certain educational technology journal? In what proportions?

Higgins et al., 1999 What do members of a certain educational technology journal want to read?

Phipps & Merisotis, 1999

What are the methodological characteristics of studies published in major educational technology forums?

Randolph, 2007c Are the same methodological deficiencies reported in Phipps & Merisotis (1999) still present in current research?

Randolph et al., 2005

What are the methodological properties of articles in the proceedings of ICALT 2004?

Ross & Morrison, 2004

What are proportions of experimental designs being used in educational technology research?

Reeves, 1995 What types of methodological orientations do published educational technology articles take? In what proportions?

Williamson et al., 2001

What types of research methods and pedagogical strategies are being reported in educational technology forums?

Note. From Computer Science Education Research at the Crossroads: A Methodological Review of the Computer Science Education Research: 2000-2005 (p. 24), by J. J. Randolph, 2007, Unpublished doctoral dissertation, Utah State University. Copyright 2007 by J. J. Randolph.



Table 7 shows the forums, the timeframe, and the number of articles reviewed in

reviews of over 905 educational articles, from eight reviews over the past 30 or so years. The forums that were covered in the previous reviews were AV Commu-nication Review Educational Communication and Technology Journal

Journal of Instructional Development Journal of Computer-Based Instruction Educational Technology Research & DevelopmentAmerican Journal of Distance Education Distance Education Jour-nal of Distance Education The Proceedings of the International Confe-rence on Advanced Learning Technologies mixed forum articles were all from one review (Williams et al, 2001). Of the 46 articles reviewed in William et al.

37 originate[d] from refereed journals or conference proceedings and the remainder from academic websites or Government departments.... In par-ticular we drew material from the conferences of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE) and from the

568)

Table 7. Characteristics of Educational Technology Reviews Included in the Quantita-tive Synthesis.

Review Forum Years covered Number of articles reviewed

Clark & Snow, 1975 AVCR 1970-1975 111

Dick & Dick, 1989 ECTJ 1982-1986 106JID 1982-1986 88

Higgins et al., 1989 ECTJ 1986-1988 40JID 1986-1988 50

Reeves, 1995 JCBI 1989-1994 123

Klein, 1997 TR&D 1989-1997 100

Williamson et al., 2001 Mixed 1996-2001 46

Randolph, 2007c AJDE 2002 12DE 2002 14JDE 2002-2003 40

Randolph et al., 2005 ICALT 2004 175a

Total 905Note. AVCR = Audio Visual Communication Review, ECTJ = Educational Communication and Technology Journal, JID = Journal of Instructional Development, JCBI = Journal of Computer-Based Instruction, ETRD = Educational Technology Research & Development, AJDE = American Journal of Distance Education, DE = Distance Education, JDE = Journal of Distance Education, ICALT = International Conference on Advanced Learning Technologies. a 175 investigations reported in 123 articles. From Computer Science Education Research at the Crossroads: A Methodological Review of the

(p. 25), by J. J. Randolph, 2007, Unpublished doctoral dissertation, Utah State University. Copyright 2007 by J. J. Randolph.

61

-search categories used in each review into three meta-categories. The three meta-categories that emerged were quantitative, qualitative, mixed-methods, and other. The other category consisted mostly of papers that did not deal with human parti-cipants, such as literature reviews, program descriptions, theoretical papers, and the like. Table 8 shows how each of the article categories was grouped into meta-categories.

Table 8. The Composition of Educational Technology Metacategories.

Qualitative Quantitative Mixed methods Other

Note. Computer Science Education Research at the Crossroads: A Methodological Review of the

What are the Proportions of Quantitative, Qualitative, and Mixed-Methods Research that Educational Technology Researchers Have Tended to Use?

In the previous section, three meta-categories for educational technology research emerged. Figure 8 shows the proportions for each of those categories.



Figure 8. Proportion of types of articles in educational technology journals. From Computer Science Education Research at the Crossroads: A Methodologi-cal Review of the Computer Science Education Research: 2000-2005 (p. 27), by J. J. Randolph, 2007, Unpublished doctoral dissertation, Utah State University. Copyright 2007 by J. J. Randolph.

How do those Proportions Differ over Time Periods and Publication Forums?

Figure 9 shows that there is considerable variability across forums in terms of the proportions of quantitative, qualitative, mixed methods, or other categories of ar-

Educational Technology Research and Development) appears to publish papers that do not report on investigations with human participants, while mixed forums tended to publish articles that do report on investigations with human participants.

Figure 9. Proportions of types of educational technology articles by forum. From Computer Science Education Research at the Crossroads: A Methodologi-cal Review of the Computer Science Education Research: 2000–2005 (p. 27), by J. J. Randolph, 2007, Unpublished doctoral dissertation, Utah State University. Copyright 2007 by J. J. Randolph.

Figure 10 indicates that there was a spike in qualitative educational technology re-search in the late 80s. However, the spike drops off rapidly and is replaced with

63

found that there was an increasing trend in the proportions of qualitative educatio-nal technology dissertations each year from 1977 until 1998.

There is one important caveat about Figure 9 and Figure 10 that should be noted: because of the nature of the methodological reviews included here, time period and forum are confounded (i.e., one cannot separate one from the other). For example, a particular forum may have been sampled more often in a particular time period

the time period, forum, or some other confound that was actually associated with the methodology category. However, because the time period trend of a decreasing amount of qualitative research over time (in journals) is consistent with trends in

pattern seems to match.

Figure 10. Proportions of types of educational technology articles by time period. From Computer Science Education Research at the Crossroads: A Methodologi-cal Review of the Computer Science Education Research: 2000-2005 (p. 28), by J. J. Randolph, 2007, Unpublished doctoral dissertation, Utah State University. Copyright 2007 by J. J. Randolph.

How do those proportions compare to the proportions in the field of education research proper?

-

educational technology research that has been discussed in the sections above. For

who conducted a review of 94 articles – 42 articles from the six issues published in 2001 in the British Educational Research Journalfour issues published in 2002 in the British Journal of Educational Psychology



Educational Ma-nagement and Administration. Although their sample was small, they provided additional evidence for the generalizability of their results by collecting data from other sources, such as:

-cluding researchers, practitioner representatives, policy makers and policy implementers;

a large-scale survey of the current methodological expertise and futu-re training needs of UK education researchers; [and a]

a detailed analysis and breakdown of the 2001 RAE [Research Assess-ment Exercise, 2001]. (p. 114)

-lt with investigations with human participants. It shows that education research proper had about a 30% higher proportion of articles dealing with human partici-pants than educational technology research.

Table 9. Comparison of the Proportion of Human Participants Articles in EducationalTechnology and Education Proper.

Field Human participants Yes No Total Percentage

yesAdjusted residual

Ed. tech 494 411 905 54.6 -5.5

Ed. proper 79 15 94 84.0 5.5

Total 573 426 999

Note. Ed. tech. = educational technology, Ed. proper = education proper. X2 (1, N = 999) = 30.21, p < .000. From Computer Science Education Research at the Crossroads: A Methodological Review of the

(p. 29), by J. J. Randolph, 2007, Unpublished doctoral dissertation, Utah State University. Copyright 2007 by J. J. Randolph.

Table 10 shows the cross-tabulations of the number of articles of each methodology

-

•

•

•

65

Table 10. Comparison of the Type of Methods Used in Educational Technology and Education Research Proper.

Note X N pComputer Science Education Research at the Crossroads: A Methodological Review of the

The conclusion I draw here is that educational technology researchers and educati-on researchers proper tend to do the same proportions of quantitative, qualitative, and mixed-methods research. Where they do differ is that educational technology researchers tend to write more articles that do not deal with human participants.

-gy have to do with theoretical and methodological considerations, it is no surprise then that educational technology research articles less often concern investigations with human participants than their counterparts in education research proper.

In What Proportions do Educational Technology Researchers Choose (a) Research Approaches, (b) Experimental Research Designs, and (d) Measures?

al. (2005), which is a methodological review of a census of articles published in -

-

technology investigations did not deal with human participants. This percentage

30.5% acceptance rate, which through personal experience is typical for educational technology conferences. I believe that these similarities give at least some prelimi-

of computer science education research, to other papers in educational technology.

In terms of the proportions of research approaches used, Table 11 shows the results



research, quantitative descriptive research (i.e., survey research), and qualitative research (in descending order) are the most frequently used research methods in educational technology research. The one article in the category in the original table was subsumed under the correlational category in the table pre-sented here.

Table 11. Research Approaches of ICALT Articles Dealing with Human Participants.

Research approach Frequency %

Experimental/quasi-experimental 21 41

Quantitative descriptive research 13 26

Qualitative 8 16

Correlational 6 12

Causal-comparative 3 6

Total 51 100

Note.

Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologiespermission.

12 shows which and in what proportions the variety of common experimental de-signs were used. The one-group posttest-only design, the weakest of experimental designs, was used most frequently.

Table 12. Experimental or Quasi-Experimental Designs in ICALT papers.

Design Frequency %

One-group posttest-only 9 43

Pretest-posttest, no controls 4 19

Repeated measures 4 19

Posttest-only, with controls 3 14

Pretest-posttest with controls 1 5

Single-participant 0 0

Total 21 100

Note.

Sutinen, 2005, in Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologieswith permission.

67

et al. (2005) that used experimental designs. The row totals do not sum to 100% because more than one measure could have been used in each article. As Table 13

most common measures used.

Table 13. Measures Used in ICALT Papers.

Measure Frequency (of 21)

%

Student questionnaire 19 91

Log files 6 29

Test (teacher/researcher made) 5 23

Interviews with users 5 23

Direct observation 4 19

Exercises 3 14

Teacher survey 2 10

Test (standardized) 2 10

Narrative analysis scheme 2 10

Number of resubmitted exercises 1 5

Time on task (electronic) 1 5

Focus groups 1 5

Pass rate 1 5

Note.

Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologiespermission.

How do Educational Technology Researchers Tend to Report Educational Technology Studies?

each article devoted to (a) reviewing the previous literature, (b) describing the program or intervention, and (c) describing the evaluation (research methods and results) of an intervention. They found that the mean proportions of literature re-view, program description, and evaluation were 18%, 47%, and 34% respectively; the implication being that educational technology researchers spend a good deal of



article space on describing their programs or interventions and spend little article space on reviewing the previous literature or on describing the evaluation of the

-dological review of the computer science education literature in which it was found that literature reviews and procedures were grossly underreported.

What Suggestions are Given for Improving Educational Technology Research?

In this section, I summarize and discuss the three major recommendations given in

-pants.

Avoid using the one-group posttest-only, attitudes questionnaire.

Review and report the previous research.

attention in educational technology research. The theoretical implication is that without a literature review, research questions are unlikely to contribute to infor-

developing a cumulative base of knowledge about educational technology and lear-ning design. Also, without literature reviews, developers of educational technology interventions are likely to develop interventions that have already been developed or develop interventions for which there is no need.

-ral, educational technology articles tend to emphasize a detailed description of the

-cedures of empirical investigations. For example, without an adequate description

science. Also, without an adequate description of settings and participants it is dif-

like computing science, is composed of many traditions (e.g., theoretical, enginee-ring, and empirical traditions). In the engineering tradition, the reporting conven-tion is to describe the intervention and its method of construction in great detail

less attention is given to the intervention description and more attention is given to the results and methods for evaluating the intervention. I hypothesize that the combining of these two traditions has led to hybrid papers that, unfortunately, are

•

•

•

69

-

validation of the intervention. Although this might border on piece-meal publica-

Avoid using the one-group posttest-only, attitudes questionnaire. The one-group posttest-design should be avoided because it is subject to almost all threats to vali-dity. For example, suppose that an educational technology researcher implemented an intervention in a course and gave out a questionnaire to students about whether they felt that their learning had increased during the time that they were in the course. First, there are many things that probably had an effect on reported lear-

-

of the course, among many other possible causal explanations.

While measuring attitudes (in this case, self-reports of learning) is an easy way to collect learning data, self-reports of learning are historically unreliable (Almstrum,

mono-operation bias and mono-method bias.

Questionnaires are ideal for collecting large amounts of data so that generalizations can be made from a sample to population. However, in the cases when the research priority is not to make generalizations, but rather to provide thick description of a phenomenon, measures other than questionnaires might be more appropriate. For example, in the often-seen case when an educational technology researcher imple-ments a new intervention in his or her own classroom and the research goal is to collect information that can be used to improve the intervention, it might be better

about how to improve the intervention.

Summary

In this section I summarize the results of some of the major methodological reviews of the educational technology research to answer the research questions listed be-low:

What are the meta-categories that can be used to subsume the research cate-gories in other methodological reviews of educational technology research?

What are the proportions of quantitative, qualitative, and mixed methods research that educational technology researchers have tended to use?


1.

2.

3.



-tion research proper?

In what proportions do educational technology researchers choose (a) rese-arch approaches, (b) experimental research designs, (c) outcomes, and (d) measures?

How do educational technology researchers tend to report educational technology studies?


A brief summary to each of those questions is given below:

One set of meta-categories that can be used to subsume the research cate-gories in other methodological reviews are quantitative, qualitative, mixed-methods and other. The other category consists of articles that do not deal with investigations with human participants, such as literature reviews, the-oretical papers, or program descriptions.

Articles that do not deal with human participants make up 46% of the ar-ticles published in educational technology research. Of the articles that do report on investigations involving human participants, the percentages of quantitative, qualitative, and mixed methods research are 56.7%, 35.2%, and 8.1%.

There is considerable variability in what type of methods the major journals -

pears that there was a spike of qualitative research in the late 1980s and a spike in quantitative research in the late 1990s and early 2000s. Note, howe-ver, that year and publication forum are confounded in these results.

While educational technology research tends to have more articles that do not deal with human participants than in education research proper, the proportions of quantitative, qualitative, and mixed-methods research is nearly the same.

Educational technology researchers tend to use (in descending frequency) experimental, quantitative descriptive (i.e., survey research), and qualitative research the most. When they do experimental research, they tend to use the one-group posttest-only design. In decreasing order of frequency, they tend

-asures and the outcomes of interest are typically student attitudes, academic achievement, and attendance.

Educational technology researchers tend to describe their interventions in great detail. However, they also tend to provide inadequate literature reviews

4.

5.

6.

7.

1.

2.

3.

4.

5.

6.

71

and inadequate detail in their reports of research done with human partici-pants.

-

and contextual research details, and avoiding the one-group posttest-only attitudes questionnaire.

7.




-ve and mixed-methods research in educational technology (Table 10), the proportions were 56.7%, 35.2%, and 8.1%, respectively. What conclusions

nearly 40% of articles dealing with human participants only presented anec-

educational technology?

Why do you think that educational technology research has a higher pro-portion of articles that do not involve human participants than in education research proper?

What is your opinion about the posttest-only attitudes questionnaire? What is its place in educational technology research?

1.

2.

3.

4.

73

Chapter 4

Data Collection in Educational Technology Research

In the previous chapter, I presented information about the types of data that educa-tional technology researchers tended to collect and what measurement instruments they tended to use. In this chapter, I will discuss in more detail the major methods of data collection and also point out how technologies can be used in the collection of research data.

To help frame this discussion, I will use the framework for valuing data collection

important in analyzing data collection methods: theoretical issues and practical issues.

rigor of the data generated, and (c) the philosophical issues underlying the data

value to a researcher depending on the research situation. For example, data that

or understand why an intervention works. On the other hand, that same data might indeed prove useful for a researcher trying to investigate whether students, in ge-

depth are preferred to data with breadth, or vice versa.

-red. For example, anecdotal observations have little rigor while single-participant observations, which involve carefully operationalizing variables and using multiple observers, have much rigor.

In terms of philosophical paradigms, some types of data collection lend themselves to some paradigms more than others. For example, if a researcher bases his or her research on the idea that there is not an objective reality and that reality is socially

measure objective states of reality.


-pes of data have different credibility for different audiences. Also, the variety of data collection methods differ in the amount of staff skills that are needed to do the data collection, the costs involved with the data collection, and the time nee-ded to collect the data. For example, questionnaires are usually easily and rapidly administered and cost relatively little. On the other hand, conducting ethnographic research takes much preparation on the part of the researcher and a commitment

Since it was found that questionnaires were the most frequently used instrument

four other most frequently used data collection methods in educational technology:

Questionnaires

Questionnaires are ideal for quantitative descriptive research when data need to be

They are ideal for research that is meant to have breadth, rather than depth. (I use the term questionnaire synonymously with survey.) For example, if the goal is to

might be appropriate. However, if the goal is thick, rich description, then other data collection methods, like interviews, might be more appropriate.

According to Frechtling (2002) the advantages of using questionnaires are that they are good for collecting basic descriptive data, they are inexpensive, and the quantitative data generated from them can be easily imported or transcribed into statistical software. On the other hand, questionnaires are most often based on self-reports, which are often biased, and their data usually lack depth and do not take context into account (Frechtling, 2002). Also, questionnaires are vulnerable to response bias – bias that occurs when the individuals who responded to a survey differ in important ways from those who did not respond. Typical problems are that people who drop out of an intervention tend not to get a postintervention satisfacti-

-

who did not respond to the survey differed from the people who did respond to the

about how to improve the response rate of surveys, including offering rewards, sen-ding follow-ups, and so on.

Questionnaires can have open and closed questions and can be administered in

responses. Many questionnaires use what is called a Likert scale in which respon-

75

dents select one response in a ranked series of responses. For example, respondents might be asked to select the degree to which they agree with a certain statement. The levels of agreement might be: strongly disagree, disagree, neither agree nor disagree, agree, or strongly agree. In open questions, respondents are allowed to give a free-form answer. For example, a respondent might be expected to write in what they liked or disliked about an intervention. The term questionnaire connotes that the questions be administered in writing; however, it is possible to use questi-onnaires in face-to-face interviews, in telephone interviews, or through computers. In fact, much research lately has gone into the differences in the quality of data generated from paper and pencil and online questionnaires. One consistent diffe-rence is that respondents tend to disclose more information and give less socially desirable answers in electronic surveys than in paper-based mail surveys (Kiesler

empirical research on the effects of different survey media on data quantity and quality.

-gure 11, is a computer-assisted self-interview tool, designed for administering ques-tionnaires or interviews to youths. The survey question appears on the screen in

-

data than when data are collected using a pencil-and-paper interview format. A

Nichols (1996), Saris (1989), or Saris (1991).

Figure 11. Virre: A computer-assisted self-interview tool. From “The Effects of a Com-puter-Assisted Interview Tool on Data Quality,” by J. J. Randolph, M. Virnes, I. Jormanainen, and P. J. Eronen, 2006, Educational Technology & Society, 9(3), p. 199. Copyright 2006 by the International Forum of Educational Technology & Society (IFETS). Reprinted with permission.



Questionnaires, whether computer-assisted or not, can vary substantially in their rigor. When accuracy and rigor are important, a researcher might need to go through the long process of doing a reliability or validity study of the questionnaire (Spector, 1992). In other cases, it might be enough just to do some pilot testing and then administer the study. At any rate, designing a questionnaire, especially a re-

questionnaire will need probably a few iterations of pilot testing and development. A reliability study and item analysis needs about 150 respondents per iteration (Spector, 1992). Therefore, before attempting to make a survey oneself, I suggest

have reliability and validity information provided for them. For example, the Texas -

course, on the Internet. The Educational Testing Service (n.d., n. p.) also maintains

makes information on standardized tests and research instruments available to re-

those brave enough to create one on their own, there are many good resources. Unfortunately, a detailed synthesis of survey construction and testing is beyond the scope of this text; however, there are many great resources – the most notable of

Survey Kit (2002).

Log Files

Educational technology researchers have a unique advantage over other types of researchers: educational technology interventions often have automated ways of recording user-generated data. That user-generated data is a boon for educational technology researchers because it is recorded unobtrusively and in a naturalistic setting. Also, because the process is automated, loads of data can be collected at almost no expense.

-cular intervention, what features they used, how often they used those features, or how often they used one feature in conjunction with another feature. Other examp-

student exercises (as in Laakso, Salakoski, Korhonen, & Malmi, 2004). In essence,

provide a framework and a review of the tools for observing users and gathering feedback about electronic environments. Some of the tools they mention are log

analytics), and user artifacts. Eye-tracking is yet another emerging technology for logging the behaviors of participants as they interact with learning environments

77

-nal technology researchers interested in unobtrusive ways of learning about their interventions.

the data collected are naturalistic. The participants are free to work as they normal-

not to say that the participants do not need to know that their activities are being recorded. To do ethical research, the participants must give their permission for their data to be recorded and to be used for research purposes. One disadvantage of

Tests

by a testing company, were the third most frequently used measure in educational technology research. Tests are useful when a researcher wants to document the sta-te of knowledge at one time or wants to measure how knowledge changes over time

-

score (unlike performance measurements), and are generally accepted by the public as an indicator of learning (Frechtling, 2002). On the other hand, there are also many disadvantages to testing. They might measure a construct other than the one

-asure knowledge unreliably, they might be biased towards certain populations, or they can be corrupted by teaching or coaching, among other disadvantages.

to administer, and not outside of the natural contexts of what students normally do in the classroom. However, teacher-made tests lack some of the desirable features that carefully-made (and usually expensive and unnatural) standardized tests do,

choice in general, there is always a trade-off in choosing what types of measures to use.

Like the creation of surveys, the creation of reliable and valid tests of knowledge is psychometrics, dedicated to

it. Therefore, it is often best to identify a pre-existing test if it is possible – again, check previous dissertations, academic databases, the Internet, and test collections. In research that requires much rigor, information about the reliability of a test is usually needed. A classic text on testing and test creation is Thorndike (2004).

For those displeased with traditional testing (see Wiggins, 1993), another opti-on is to concentrate on using performance measurements rather than traditional

demonstrations, student products, and problem-solving activities are more natu-ralistic, measure different types of knowledge than traditional tests do, and often



have more subjectivity in their scoring than traditional tests and that their scoring is not as straightforward, and, therefore, not as amenable to quantitative analysis, as traditional tests.

and to adapt to the level of skill of a test taker (computer-adaptive testing) (Wainer et al., 2000). See Alderson (2000) for a review of the future of technology in tes-ting.

Interviews

While surveys are ideal for collecting shallow knowledge over a broad area, inter-views are ideal for collecting deep knowledge over a limited area. In addition to generating rich data, they allow for face-to-face contact, enable a researcher to fol-low up immediately on unclear or ambiguous answers, gain access to information

administering the interview according to the needs of individual respondents or

questions that are appropriate to answer using interviews:

What does the program look and feel like to the participants? To other stakeholders?

What do stakeholders know about the project?

What thoughts do stakeholders knowledgeable about the program have concerning program operations, processes, and outcomes?

What are participants’ and stakeholders’ expectations?

What features of the project are most salient to the participants?

What changes do participants perceive in themselves as a result of their involvement in the project? (p. 51)

While interviews have many advantages, they are time-consuming to conduct and the interviewers need considerable training. Also, a great deal of time usually needs

-views, there is a possibility for inconsistencies between different interviewers. Be-cause of the face-to-face contact between interviewers and respondents, there is the possibility for interviewers to lead respondents. Finally, the volume of information that is generated can be very large and overwhelming.

Interviews can vary to the degree that they are planned a priori. In some cases, the interview protocol can be very strict and respondents are only allowed to choose

•

•

•

•

•

•

79

from a certain set of responses (i.e., a verbal questionnaire). In other cases the in-terview might be less structured and the interviewer has much freedom to explore

-viewer records the responses on the spot. In other types of interviews, the intervie-wer painstakingly transcribes audio or video recordings. In discourse analysis (see

elongations and emphases of syllables, overlaps between speakers, and other featu-res of spoken language. There are many good guides for collecting interview data.

Although not as often as interviews, focus groups are sometimes used in educatio-nal technology research. Essentially, a focus group is an interview with 8-12 people who share common characteristics. Originally used in market research, the focus group technique capitalizes on the group dynamic to generate insights that would not have been generated if the participants had been interviewed individually. Ac-

problems in project implementation; pretesting topics or ideas; identifying project strengths, weaknesses, and recommendations; assisting with interpretation of

-dations, which are too numerous to report here, for choosing between focus groups and interviews. A helpful resource for conducting focus groups is Stewart, Sham-

There are many technologies available to aid interviewing. Two of the major types -

1991). The latest generation of interview tools totally automates the personal inter-

Direct Observation

-pants, is another form of data collection used often in educational technology rese-arch. It comes in many forms. In one form, direct observation can be as simple as

a learning tool over time. In another form (as is the tradition in single-participant research), the researcher carefully operationalizes a behavior or set of behaviors and counts the number of times that the behavior occurs over a given period. For example, an educational technology researcher might observe how often students interact with each other when participating in a technologically oriented interven-

-liability in this type of research, additional researchers might also make observa-

behavior occurred (see Kazdin, 1982).

Another form of direct observation, in the ethnographic tradition, is to observe and

form of direct observation, albeit of a different character than the others, is verbal



talk aloud while interacting with the technology, thus revealing the cognitive pro-

observation can help explain the cognitive processes of users, its main shortcoming is that it is obtrusive. Having to explain what one is thinking while performing an

Observational techniques are advantageous because they allow the researcher to get

Also, they allow the researcher to come to understand the contexts important to the

important where it is not the event that is of interest, but rather how that event may -

servations are time-consuming and often require much training on the part of the observer. Also, the act of observation often changes that which is being observed.

-

recordings can be used as solid evidence. Second, recording an observation allows

insights that might have been missed had the recording not been reviewed. Howe-

the participants become accustomed to being recorded.

Mixing Data Collection Methods

There are many good reasons for using a variety of data collection methods when conducting educational technology research, the most of important of which is that combining methods, what Frechtling (2002) calls triangulation, increases the vali-

above have their strengths and weaknesses. By combining techniques, one can can-cel out the weakness of one technique by pairing it with a complementary techni-que. For example, the observation of behavior might be paired with interviews that expose the intentions behind those behaviors to gain a holistic picture of the cog-nitive and behavioral processes involved. Another reason is that different stages of research call for different methods of investigation. For example, Frechtling (2002) argues that one might begin an investigation with a qualitative focus group appro-ach to identify issues for further investigation, then conduct a survey to see to what degree that issue affects the population of interest, and conclude by conducting another qualitative inquiry to gain deeper insights into the issue. There are several

-Handbook of Mixed Methods

in Social & Behavioral Research, Designing and Conducting Mixed Methods Research, seminal article – Mixed methods research: A research paradigm whose time has come.

81


What data collection methods have you used in the past? What advice would you give to a colleague who intends to use one of those data collection met-

Of the data collection methods mentioned above, which has the most perso-nal appeal to you?

you anticipate some of the disadvantages? In particular, what happens when

I highlighted some of the technologies that are currently being used to aid in

are being used, in educational technology research?

1.

2.

3.

4.


83

Chapter 5

The Analysis of Educational Technology Research Data

As discussed in the previous section, the data generated from educational technolo-

interview transcripts, direct observations, talk-aloud transcripts, survey data, and many others. Accordingly, there is a large variety of approaches to analyzing the numerous forms of educational technology data.

I have found it helpful to group the approaches of data analysis into four catego-ries:

The quantitative analysis of quantitative data,

The quantitative analysis of qualitative data,

The qualitative analysis of qualitative data, and

The qualitative analysis of quantitative data.

The goal of this section on data analysis is simply to acquaint the reader with the different methods of analysis available and to refer readers to key resources for each method.

The Quantitative Analysis of Quantitative Data

--

ful for making sense of complex data, gaining insights into phenomena, making accurate predictions, making inferences from a population to a sample or from a

and testing theory, however, how to do statistics is a discussion far outside the scope of this text. While not every educational technology researcher may be inclined to do quantitative analyses, I believe that a well-rounded researcher at least needs to

1.

2.

3.

4.


2000).

The Quantitative Analysis of Qualitative Data

quantitative analysis of qualitative data might seem awk-ward, it is an increasingly seen method of analysis in educational technology re-search. One of its most popular manifestations is quantitative content analysis.

-tent analysis originated in media studies, but has been increasingly adopted by edu-cational technology researchers. While content analysis is often done with texts, it can be used actually with any sort of media – websites, movies, pictures, songs, and so forth.

In contrast to the qualitative analysis of qualitative data, according to Neuendorf -

hod and is closer to the positivistic paradigm, (b) emphasizes objectivity-intersub-jectivity (intersubjectivity is the standard that relates to how a group of people can agree that something is true), (c) uses an a priori design, (d) stresses reliability and validity, (e) creates generalizable results, (f) is replicable, and (g) is amenable to hypothesis testing.

quantitative content analysis to quantify the degree to which a cer-

can then be used for comparative, correlational, or even experimental research. In contrast, researchers use qualitative content analysis to specify what categories or events occur, why or how they occur, in what contexts they occur, or the meanings of their occurrences. In many ways, quantitative and qualitative content analysis are complementary. For example, a qualitative study could be used to identify the variables underlying a phenomenon. A quantitative content analysis study, using

to what degree those categories are present, how they quantitatively relate to each other, how they change over time, or how they relate to other variables.

Neuendorf (2002) suggests that there are nine steps in conducting a human-co-ded quantitative content analysis, which parallel the steps, more or less, of many

-

-lizations. One can use an a priori coding scheme (i.e., a coding scheme developed by someone else) or one can use an emergent coding scheme (i.e., a coding scheme based on categories that emerge along the way). This step also includes choosing a unit of analysis (e.g., a phrase, a sentence, or a speaking turn). With human coding, the fourth step is to create a coding book and coding form. In the coding book, one explains the variables in great detail and the procedures for coding the data. There

85

should be enough detail and so little ambiguity that two independent coders could consistently apply the same codes to the same units of analysis. The coding form is a form on which the coder inputs the data. (With computer coding, the fourth step

choose a sample. If the content to be examined is manageable, one might choose to do a census and code all of the content. If there is a large amount of content to code, one might choose a random sample (or some other type of sample) to code. The six-th step is to train coders in the coding book and coding procedure and, then, to pilot

The seventh step is to do the actual coding of the content and the eighth step is to

report the data. One might also do statistical analyses to identify relationships bet-ween content analysis variables, to identify changes in content analysis variables over time, or to identify relationships between content analysis variables and other, external variables. Neuendorf (2002) and Krippendorf (1980) are excellent resour-ces for those interested in quantitative content analysis. Osorio (1998) discusses the use of content analysis for analyzing transcripts in online courses.

The Qualitative Analysis of Qualitative Data

There are many popular methods for the qualitative analysis of qualitative data (e.g., Merriam, 2001; Miles & Huberman, 1994; Madison, 2005; Wolcott, 1994) all

data analysis spi-ral spiraland report writing are not distinct steps in the process – they are interrelated and

-

document, and organize their data. They segment their dataset by breaking it down into its units of analysis (e.g., sentences, paragraphs, a speaking turns, etc.). They can also document and organize their data, using the same logic that archaeologists use to document their artefacts, to create what Yin (2003) calls a case study data-base.

After setting up a system to manage the data, qualitative researchers do some form of reading and memoing. The researcher begins by sifting through the documents,

-archer might begin to write down memos in the margins about concepts that occur to the researcher (Merriam, 2001) or highlight meaningful statements in the texts (Moustakas, 1994).


87

Figure 12. A flowchart for the typical process of content analysis. From The Content Analysis Guidebook (pp. 50-51), by K. A. Neuendorf, 2002, Thousand Oaks, CA: Sage. Copyright 2002 by K. A. Neuendorf. Reprinted with permission.



Figure 13. The data analysis spiral. From Qualitative inquiry & research design: Choo-sing among five approaches (2nd ed.) (p. 151), by J. W. Creswell, 2007, Thousand Oaks, CA: Sage. Copyright 2007 by Sage Publications, Inc. Reprinted with permission.

interpretation. In this step, the researchers might begin by providing detailed desc-in situ, that is, within the

place to start in a qualitative study (after reading and managing data), and it plays a

to develop an initial set of categories for the data. Some researchers begin with a short list of general categories, and then in later iterations break the codes into in-

After the categories and subcategories for the data are established, the next step is -

ries about a phenomenon or identify relationships between persons, setting, and events.

-rices of data, hierarchical family trees, causal illustrations, social network maps, or

-ders to get their interpretations of the data and to increase the validity of the data through member checking.

I will discuss this step in more detail in the next chapter.

89

The prominent books on the analysis of qualitative data include Miles and Huber--

ral qualitative computer programs that can simplify the process of qualitative ana-

A thorough, but dated, review of 24 of these programs can be found in Weitzman

The Qualitative Analysis of Quantitative Data

Although the notion of qualitative analysis of quantitative data awkward, in fact, it is an everyday occurrence in quantitative research. (It might be more accurate to refer to this method of analysis as the visual analysis of quan-titative data.) For example, creating a graph of quantitative data is an example of qualitatively analyzing quantitative data.

Qualitative analysis of quantitative data is the main method of analysis of single--

through a visual analysis of the data it becomes clear that the scores during the intervention increased.

Mixed-Methods of Analysis

In reality, there is not a clear distinction between these methods of analysis. For example, some qualitative data analysts, such as Miles and Huberman (1994), ad-vocate counting the frequencies of categories, which is a technique of quantitative content analysis. Similarly, the process of creating categories in quantitative con-tent analysis is in fact a qualitative technique; statistical analysis often begins with a qualitative (visual) analysis of the data.

There is good reason for this mixing of methods of analysis. Earlier I reported on the arguments for using mixed-methods of data collection. The same arguments apply to using mixed-methods of data analysis: researchers can get a more holistic and valid view of a phenomenon by viewing and interpreting the phenomenon from

Johnson & Onwuegbuzie, 2004).




Of the four types of data analysis, what type are you most familiar and com-fortable with? What advice would you give to a colleague intending to do that

In your own research community, is there an implicit hierarchy of types of data analysis? Is one type given more merit than others?

-king of ourselves as scientists testing grand theories, and face the fact that we are technicians collecting and collating information, often in quantitative

-

1.

2.

3.

4.

91

Chapter 6

Reporting Educational Technology Research

There is a prevalent misconception that the research process ends after data are col-lected. However, the truth is that reporting is one the most important stages of the research process. No matter how carefully controlled a study may have been, no mat-ter the attention to detail in collecting and analyzing data, or no matter how large of a random sample was selected, a study is particularly vulnerable to bias at the reporting

and discussion of study outcomes is not congruent with what actually happened in the study. I have read countless educational technology studies in which the author care-

at the reporting stage include not reporting on relevant study complications, not pro-viding information about study operations that could have affected study outcomes,

Besides being a stage of the research process that is particularly vulnerable to the introduction of bias, much of the potential utility and acceptability of the research

-dy might have, poorly reported research is usually poorly regarded research. On this

-ript (such as truthfully reporting the impending end of the world or viable cold fusion) to rejection, yet even a trivial report has a reasonable chance of being published somewhere if the manuscript is well-written. Spend whatever time it takes to communicate economically and with probity of style. (2001, p. 342)

One way to help ensure that research is accepted by the research community is to fol-low the established conventions for reporting research. In this chapter I focus on those

method for using exemplars.


Reporting on Quantitative Investigations

The reporting of empirical quantitative investigations in education research is so-mewhat formulaic. It consists of a title, an abstract, an introduction (including a brief review of the literature), a methods section, a results section, a discussion section, and a conclusion section.

Title

The title of a report should be as concise as possible, but still give readers a good

Educational Technology & Society) limit the number of words in a title to ten.

Abstract

Abstracts in educational technology typically consist of a 120-300 word narrati-ve summary of the article. According to the Publication Manual of the American Psychological Association a narrative abstract of an empirical article should describe:

the problem under investigation, in one sentence if possible;

the participants or subjects, specifying pertinent characteristics, such as number, type, age, sex......;

the experimental method, including the apparatus, data-gathering procedures, complete test names......;

the conclusions and the implications or applications. (p. 14)

The abstracts for a review or theoretical article, methodological papers, or case stu-dies differ slightly, but the underlying logic is basically the same. See American

There has been a call lately for structured or claim-based abstracts in education research (Kelly & Yin, 2007; Mosteller, Nave, & Miech, 2004). For use both as a description and as an exemplar, I will include the structured abstract from Mostel-

Background: Approximately 1,100 education journals collectively publish more than 20,000 education research articles each year. Under current practice, no systematic way exists to move thestudies into the hands of the millions of education practitioners and policy-makers in the United States who might use them.

•

•

•

•

•

93

propose that education journals consider adopting a structured abstract, a structural innovation that focuses on the format of the article itself. The structured abstract would replace the paragraph-style narrative summary – typical-ly either an American Psychological Association-style abstract or “editor’s introduction” – now present at the beginning of many articles.

Intervention: A structured abstract is a formal and compact summary of

predictable structure that compresses information into a small space and can be read independently of the main body of the article. The structured abstract is longer and more detailed than the standard paragraph-style narrative summary. On the printed page, the structured abstract appearsbetween the title and the main body of the article. It includes basic items applying to all articles (i.e., background, purpose, research design, and conclusions) and several additional items that apply to some articles but not to others (i.e., setting,population, intervention, data collection and ana-lysis, and

: Analytic essay.

Conclusions: The structured abstract is a viable and useful innovation to help practitioners and policymakers systematically access, assess, and

Relative to current practice, the structured abstract provides a more robust vehicle for disse-minating research through traditional routes as well as through new chan-nels made possible by emerging technologies. (p. 29)

Introduction

-logical Association, 2001) suggests that an introduction should do three things: it should introduce the problem, it should review the previous research, and it should state the purpose and rationale for the study. The introduction should begin with a description of the problem and explain why the problem is important. It should also develop the background of a study by reviewing the literature. In an empirical ar-ticle, the literature review does not necessarily have to be comprehensive; it just has

review of literature how the current research will make a contribution to the body

for a list of ways that an article can make a contribution to the existing literature). Ideally, the literature review should serve as the basis for the purpose or rationale for the study.

-



(2001) are some examples of highly regarded books on conducting quantitatively oriented literature reviews.

In the problem introduction, the author shows that there is a problem. In the lite-rature review section, the author demonstrates what has been done to help solve that problem and what still needs to be done (or what needs to be known) for that problem to be solved. In the rationale and purpose section, one should state how the current research will make a unique contribution to solving the problem. Near the end of the introduction it is also appropriate to state research questions or hypot-heses and explain how those research questions or hypotheses are aligned with the previous research and aligned with the problem. The logic here is that answering the research questions will help make up for the lack of research (or development), which in turn, will help solve the problem.

Methods

The next section in an empirical paper in educational technology is the methods sec-tion. The methods section is customarily divided into subsections. If the research deals with human participants there should be a section on participants. That secti-on should report on how the participants were selected, the number of participants (in each group, if there were groups), the number of participants who quit the study,

-nicity and where possible and appropriate, characteristics such as socio-economic

In educational technology research, it would also be important to include informa-

years and type of schooling, academic aptitude, previous experience with the inter-vention, and so on. In educationally oriented research, it might also be important to include information about the educational setting where the research occurred. One might then label this section as Participants and Setting. It is better to err on the side of providing too much information in this section than too little.

There might also be a subsection on the instruments used in the study. In educatio--

ters or software used.

Another subsection, which is mandatory, is the description of the procedure used. -

riables were examined, what measures were used, how data were collected, who collected the data, how long the data were collected, how much data were collected, how data were analyzed, how many groups there were, what experimental research design was used, how participants were assigned to groups, how the research as-sistants were trained, among many other details. In essence, one should provide enough detail here that another researcher could more or less replicate the study solely by reading the description.

95

Results

In the results section, the author summarizes the results in enough detail to justify the conclusions. In traditional, quantitative, education research papers, the text of the results section is reserved for the presentation of information and for pointing out important pieces of information. If presenting statistical results, useful guide-

-arch Association (2006). The interpretations, conclusions, and implications should be reserved for the discussion section.

Discussion

The discussion section typically begins by answering the research questions, based on the information presented in the results section. If there were complications in the research (e.g., if something did not go as planned), it should be reported in the beginning of the discussion section. A mistake often seen in the discussion secti-on is that an author makes claims in the discussion section, but does not base the

It is also customary to examine the similarities and differences between the current

explain what are the implications of the research for solving the problem mentioned in the introduction. In essence, the discussion section uses the information in the results section to relate back to the introduction – to the research questions, to the previous research, and, of course, to the original problem.

Conclusion – The Take-Home Message

One of my academic writing mentors gave me some great advice about writing and presenting conclusions. He argued that the average reader or audience member is only going to remember two or three ideas from a research study or presentation, at most. He suggested, then, that an author should conclude by picking out the two

they get home. Basically, the take-home message is a statement, packaged in the -

that work particularly well in a take-home message are statements of (a) what had been known already, (b) what more is known now as a result of the study, and (c) why anybody should care.

In the paragraphs above I summarized the conventions for reporting quantitative, empirical investigations when human participants are involved. For conventional quantitative papers when human participants are not involved, such as literature



reviews, the sections might be slightly different, but the overall logic is the same: one should report on (a) the problem, (b) the previous research on the problem, (c) the approach to and rationale for solving the problem, (d) how the research was conducted, (e) the results that were found, and (f) the interpretation of the results

For educational technology researchers who do quantitative research with human participants the most important resource for learning how to report results is pro-

Common Defects in Quantitative Research Reporting

At least in quantitatively oriented social science articles, reviewers and editors tend

piecemeal publication, that is the separation of a single substantial re-port into a series of overlapping papers;

-on between two variables rarely has an interpretive value;

the reporting of negative results without attention to power analysis;

those related to the design and analysis) and the author’s interpreta-tion and discussion of the study’s outcomes (e.g., failure to report the statistical test at the level being claimed);

failure to report effect sizes;

failure to build in needed controls, often for a subtle but important as-pect of the study; and

exhaustion of a problem – there is a difference between ongoing rese-

the endless production of papers that report trivial changes in previo-us research. (p. 5)

•

•

•

•

•

•

•

97

Reporting on Qualitative Investigations-

titative research. Qualitative authors, in general, have much freedom to experiment with alternate forms of reporting and presenting research. Also, there is conside-rable variability between the reporting conventions (or lack of conventions) bet-

Reporting Narrative Research

Narrative research reports come in many shapes and forms. Some authors employ a narrative chronology – following the life of an individual over time. Other aut-

narratives that precede and follow the critical event. Other authors of narrative re-

methods, discussion, and conclusion). The narrative researcher is encouraged to employ the devices of biography and narration, such as transitions, metaphor, time-

write narrative reports.

Reporting Phenomenological Research

of phenomenology, is more like a conventional research report than a narrative re--

gestions for what sections are needed in a phenomenological report:

Chapter 1: Introduction and statement of topic and outline.

Chapter 2: Review of the relevant literature.

Chapter 3: Conceptual framework of the model.

Chapter 4: Methodology.

Chapter 5: Presentation of the data.

Chapter 6: Summary, implications, and outcomes. (p. 187)

Note that the sections listed above are more or less identical to the sections in a conventional quantitative research paper.

phenomenological research report. See Grigsby and Megel (1995) for a good examp-

•

•

•

•

•

•



Reporting Grounded Theory Research

-tions, grounded theory reports have a much broader set of reporting guidelines.

guidelines for writing up grounded theory reports:

Develop a clear analytic story. This is to be provided in the selective coding phase of the study.

Write on a conceptual level, with description kept secondary to con--

tion of the phenomenon being studied and more analytic theory at an abstract level.

Specify the relationships among categories. This is the theorizing part of the grounded theory found in axial coding when the researcher tells the story and advances propositions.

Specify the variations and relevant conditions, consequences, and so

variation and different conditions under which the theory holds. This means that the multiple perspectives or variations in each component of axial coding are developed fully. For example, the consequences in the theory are multiple and detailed. (p. 190)

(1986) chapter – is also a good resource for those writing up the grounded theory report.

Reporting Ethnographic Research

writes about four different types of ethnographic tales: realist tales in which the

of culture; confessional tales in which the author concentrates on his or her own

of realism and confessionalism.

Wolcott (1994) provides further advice on writing up an ethnographic report. Wol-cott suggests writing description, analysis, and interpretation sections. According

--

ressive focusing, a critical or key event, plots and characters, groups in interaction, -

•

•

•

•

99

case with a known case, evaluating the information, contextualizing the informa-tion within a broader theoretical framework, critiquing the research process, and

-archer can extend the analysis, make inferences from the information, do as di-rected or suggested by gatekeepers, turn to theory, refocus the interpretation itself, connect with personal experience, analyze or interpret the interpretive process, or

-

Reporting Case Study Research

Like many types of qualitative reports, case studies can take on a variety of forms. Yin (2003) has written an excellent chapter in which he discusses the different for-mats and structures for composing a case study report. The four formats are the

-

that only contains a cross-case analysis, and the question-and-answer format.

are listed below:

the linear-analytic structure, which follows a conventional reporting struc-ture;

the comparative structure, in which the same case is compared under diffe-rent conceptual models or alternative explanations;

the chronological structure, in which the events of a case are reported as they occurred over time;

the theory-building structure, in which evidence for a theory is presented in logical order;

the supporting evidence and details are given later; and

descriptive case studies.

Lincoln and Guba (1985), Merriam (1988), or Stake (1995).

•

•

•

•

•

•



Alternative Styles of Reporting

Above, I have presented the guidelines for writing up a conventional scholarly re-

-tegies designed to bolster the utility of research results. Those strategies include in-volving stakeholders in the reporting process; incorporating modern technologies such as chat rooms and Internet forums; and using techniques such as photography, cartoons, poetry, and drama. Angela Thody, a proponent of alternative styles of re-porting, adds to that list of alternative reporting strategies with examples such as:

readers’ theatre (where researchers [act] their respondents’ views);

speeches and then [invite] audience participation, assisted by mobile microphones);

debates (six researchers [have] exactly three minutes to put their ca-ses). (2006, p. 11)

Thody (2006) adds that the adoption of alternative styles is a postmodern reaction

reporting. It is also a reaction to what she calls the standardization of science repor-ting that results from publication manuals from organizations such as the Ameri-

On this point she writes,

Why, I mused, in the USA and Canada, so often depicted as lands of free-dom, is so little discretion allowed to, or taken by, highly intelligent acade-mics on how to present their work? Why have APA guidelines for writing up psychology experiments been adopted so wholeheartedly by other discipli-nes?.... Even when there are no strictures, such as when academics present their research orally, why do so many academics still elect to ‘read’ their papers and eschew the livelier arts of demonstration and teaching? (p. 10)

Using Exemplars

Whether one decides to use conventional or alternative means of reporting, the best

accomplish the kind of task you are trying to accomplish in an effective way

exemplars is reported below:

•

•

•

•

101

Step 1: Specify Your Intent.... Describe what you hope to accomplish in your writing project in at least three different ways. Settle on one distinct desc-ription. Seek advice about this objective from advisers and your writing community. (pp. 57 – 58)

Step 2: Choose Published Examples of Work with a Similar Objective.... -

Step 3: Examine the Structure and Tone of Successful Exemplars.... Outline -

voted to each topic. Also make notes about the “tone” used in your exemp-lar. Draw conclusions about the tacit rules for this kind of contribution to scholarly communication.... Identify the aspects of each exemplar that are particularly effective as well as any that are ineffective in communicating the author’s purpose. (pp. 60 – 61)

Step 4: Move... Beyond Your Exemplars... Use your analysis to identify whe-re you are making a unique contribution to the literature and think about the ways to usefully depart from your exemplars to more effectively deliver the primary message of your paper. (p. 62)

In Summary

In this chapter I have reported on the conventions for reporting both quantitati-ve and qualitative reports, discussed alternative styles of reporting, and presented

information to create high-quality, relevant, and publishable papers in educational technology.




If you have had experience submitting your articles to journals or conferen--

cess itself?

imperialism?

-rent state of education research:

The conception of our work that held that “studies” are the basic, fun-damental unit of a research program may be the single most counter-

a study culminates in the test of a hypothesis and that a hypothesis comes from a theory – this idea has done more to retard progress in educational research than any other single notion. Ask an educatio-nal researcher what he or she is up to, and they will reply that they are “doing a study,” or “designing a study,” or “writing up a study” for publication. Ask a physicist what’s up and you’ll never hear the word

-chive their work, one will seldom see the word “study.” Rather, physi-cists – the data gathering experimental ones – report data, all of it, that they have collected under conditions that they carefully described. They contrive interesting conditions that can be precisely described and then they report the resulting observations.) (2000, n.p)

-

Glass, and why?

4. At the end of this chapter, I suggested that the best way to learn how to do

exemplars in this way? If so, what exemplars have you used, for what purpo-se, and do you have any that are worth recommending to your colleagues or classmates?

1.

2.

3.

103

Chapter 7

Conclusions about Methods Choice in Educational Technology Research and Development


I have provided much information about methods choice in educational technology

taken into consideration when making methods choices and discussed the research approaches amenable to educational technology research. In the third chapter, I desc-ribed the methods choices that educational technology researchers tend to make. In

it might be useful to provide a summary of the process of making informed choices about methods choice in educational technology.

A key step in methods choice is arriving at appropriate research questions. However,

-search has been done to meet the purpose. As one reviews the research, one probably

-ment task) that advances the state of knowledge in some area. The review of literature is so important because one can only accidentally advance the state of knowledge in an area when one does not know what the state of the knowledge is.

After arriving at a question that can advance the state of the knowledge, the task is to take into consideration other factors that affect the choice of methods and make an initial decision about which approach to use. For example, it is important to consi-der whether the research question is best answered with a quantitative or qualitative approach (or both), the degree of generalizability that is needed, the degree of accu-racy that is needed, and the degree of researcher and stakeholder participation in the investigation. Two critical factors are propriety and feasibility; if a method cannot be found that is both appropriate and feasible, then the question might need to be

collection, data analysis, and reporting.

While the literature on a certain approach will provide much of the information on what methods of data collection, analysis, and reporting are possible, there are many variations within an approach that must be chosen and many details that should be worked out beforehand. These details might again affect the research questions.


So, in summary, there are many interrelated factors that come into play when choo-sing which methods to adopt: the purpose of the research, the state and type of previous research, the resources that are available, ethical restraints, among many other factors. The crux, then, of choosing the right methods is taking these factors into consideration and tailoring a balanced set of methods to the particular rese-arch situation. The art and science of methods choice begins with the case.

105

References

28(4), 593 – 603.

Alexander, S. & Hedberg, J. G. 1994. Evaluating technology-based learning: Which -

(233 – 244). Amsterdam: Elsevier.

-

887 – 890.

-

35(6), 33 – 40.

Association.

Becher, T. 1989. Academic tribes and territories: Intellectual enquiry and the cultu-


--

ments. New York: Springer.

-

-cher, 34(6), 3 – 15.

-

federal policies on evaluation methodology: Building the evidence base for

Evaluation, no. 113 (55 – 74). San Francisco: Jossey-Bass.

Brand, S. 1999. The clock of the long now: Time and responsibility. London:

Burghar, J. & Turns, J. 1999a. A survey of research questions in Educational -

April 11, 2007, from http://depts.washington.edu/pettt/papers/reviews/3-

-

Burghar, J. & Turns, J. 2000. A survey of research questions in Frontiers in

depts.washington.edu/pettt/papers/reviews/

-

--

409 – 429.

107

TESL-EJ, 2(3), A-1.

qualitative research. San Francisco: Jossey-Bass.

-

-

(2nd ed.). Mahwah, NJ: Lawrence Earlbaum.

science. Science, 192, 105 – 111.

the models and methods of usage-centered design. Boston: Addison Wesley

York: Sage.

-

Jossey-Bass.

-riers. Educational Technology & Society, 9(1), 276 – 288.

References


53 – 70). San Francisco: Jossey-Bass.

org/

Scott, Foresman/Little.

-

37(1), 81 – 87.

York: Wiley.

-

7 – 18.

from http://www.ets.org/testcoll/

Frechtling, J., Frierson, H., Hood, S. & Hughes, G. 2002. The user friendly hand-

May 10, 2005, from http://www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf

109

Frechtling, J. & Sharp, L. 1997. The user friendly handbook for mixed-method eva-

http://www.nsf.gov/pubs/1997/nsf97153/start.htm

Frechtling, J., Stevens, F., Lawrenz, F. & Sharp, L. 1993. The User Friendly hand-book for project evaluation: Science, mathematics and technology educati-

Gay, G. & Bennington, T. L. (Eds.) 1999. Information technologies in evaluation:

for Evaluation, no. 84. San Francisco: Jossey-Bass.

http://glass.ed.asu.edu/gene/papers/meta25.html

and standards-based curricula: The achievement cycle. Larchmont, NY: Eye on Education.

-

-

Methods choices are contextual, contingent, and political. In G. Julnes &

Building the evidence base for method choice in government sponsored -

co: Jossey-Bass.

Guba, E. G. 1990. The alternative paradigm dialog. In E. G. Guba (Ed.), The para-

References


Sage.

educational technology research and development. Educational Technology

-con.

Isaac, S. & Michael, W. B. 1990. Handbook in research and evaluation: A collection of principles, methods and strategies useful in the planning, design and

-

-forming federal policies on evaluation methodology: Building the evidence

-tions for Evaluation, no. 113 (1 – 12). San Francisco: Jossey-Bass.

methodology: Building the evidence base for method choice in government -

co: Jossey-Bass.

Kazdin, A. E. 1982. Single-case research designs: Methods for clinical and applied

Kelly, A. E. (Ed.) 2003. Theme issue: The role of design in educational research.

research: The need for claim-based structured abstracts. Educational rese-archer, 33, 133 – 138.

Opinion Quarterly, 50, 402 – 413.

111

-

Kvale, S. 1996. Interviews: An introduction to qualitative research interviewing.

Laakso, M-J., Salakoski, T., Korhonen, A. & Malmi, L. 2004. Automatic assess-ment of exercises for algorithms and data structures – a case study with

Helsinki University of Technology.

-

Mark, M. M., Henry, G. T. & Julnes, G. 2000. Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco: Jossey-Bass.

May, K. A. 1986. Writing and evaluating the grounded theory research report. In

Merriam, S. B. 2001. Qualitative research and case study applications in educati-on. San Francisco: Jossey-Bass.

Mosteller, F., Nave B. & Miech, E.J. 2004. Why we need a structured abstract in 33(1), 29 – 34.

Miles, M. B. & Huberman, A. M. 1994. Qualitative data analysis: An expanded

Mohr, L. B. 1999. The qualitative method of impact analysis. American Journal of Evaluation, 20(1), 69 – 84.

Sage.

References


Sage.

dissertation proposals. Utah State University, retrieved July 30, 2007, from

-

FL: WebNet.

contemporary research on the effectiveness of distance learning in higher

Halling (Eds.), Existential-phenomenological perspectives in psychology

-roads: A methodological review of computer science education research: 2000 – 2005. Unpublished doctoral dissertation. Utah State University.

dissertation

computer science education research. Manuscript submitted for publicati-on.

contemporary quantitative research methodology in distance learning. Informatics in Education, 6(1), 179-188.

-nen, E. 2005. A critical review of research methodologies reported in the

-

113

computer-assisted interview tool on data quality. Educational Technology & Society, 9(3), 195 – 205.

-

docs/dean/

for observing users and gathering feedback. In G. Gay and T. L. Bennington (Eds.), Information technologies in evaluation: Social, moral, epistemo-

(45 – 58). San Francisco: Jossey Bass.

Jonassen (Ed.) Handbook of research on educational communications and technology (2nd ed., 1021 – 1043.) Mahwah, NJ: Erlbaum.

Saris, W. E. 1989. A technological revolution in data collection. Quality and Quan-tity, 23, 333 – 350.

-tions for Evalution, no 58. San Francisco: Jossey-Bass.

-perimental designs for generalized causal inference. Boston: Houghton

References


-

-wah, NJ: Lawrence Earlbaum.

(Ed.), Transforming a Learning Society: The case of Finland. Exploratio-

-

-rieved June 14, 2007, from http://www.tcet.unt.edu/pubs/attcomp.htm

-

-

Association.

-municating and reporting: Enhancing learning in organizations (2nd ed.).

115

McGraw-Hill.

-

Earlbaum.

-

limits of testing. San Francisco: Jossey:Bass.

www.ascilite.org.au/conferences/melbourne01/pdf/papers/williamsona.pdf

References

117

Appendix

Key Questions in Educational Technology Methods Choice

-logy methods choice. I used those factors to create a list of key questions in methods choice.

This list can be used for several purposes. First it can be used to introduce students to the factors that are important to consider when planning educational technology research. In that capacity, it might serve as a classroom teaching aid or as an aid for supervising student researchers in the early stages of planning their theses or disser-tations.

research plans and come to understand how those research plans relate to other rese-

Third, the list of key questions in methods choice can be used to create a research

-



1. What is the research problem that you plan to resolve?

2. Is there a social problem related to your research problem? If so, what is it?

3. What is the primary purpose of your research?

a. To develop an intervention.

b. To answer questions important to local stakeholders.

4. What type of research contribution do you intend to make?

b. Help establish a new theory.

c. Meet a practical need.

d. Make up for a lack of needed information about a topic.

e. Other.

5. If you are investigating a phenomenon, what is the state of theoretical knowled-ge about the phenomenon?

a. There are no established theories.

b. There are theories, but they are not yet substantiated.

c. There are substantiated theories, but new theories need to be developed.

6. If you are investigating a phenomenon, what is the state of empirical knowled-ge about the phenomenon?

a. The important variables or the elements of a phenomenon have been

119

b. The associations between those elements or variables have been substanti-ated.

c. The causal mechanisms regarding the phenomenon are clear.

7. Which of the previous studies are related to your research and how are they rela-ted? (A good way to answer this question is to create a research map.)

9. What research methods were used in the previous research? (For example, you might answer this question by making a table in which you describe the following characteristics of the previous studies, as applicable: the research approaches, the methods of data collection, the methods of analysis, the variables examined, the settings involved, the participants involved, or other salient characteristics.)

10. What is your general research question and how does it relate to the research ques-tions asked in the previous research?

11. What are the sub-questions that unpack the general research questions? Are they procedural or structural or are they another type of research sub-question?

12. Which of your research questions are meant to be answered by examining the kno-wledge-base and which are meant to be examined empirically?

13. What category of educational technology research question does your general rese-arch question fall into?

a. Questions about theories and the practice of educational technology.

b. Questions about research and development methods.

c. Questions about technology implementation.

d. Questions about the effectiveness of an intervention.

e. Questions about the factors that moderate the effectiveness of an interven-tion.

f. Other.

14. Which of the following research acts are implied in your research question?



e. Experimentation (causal description).

15. What family of research approaches do you intend to use, and why?

a. The quantitative family.

b. The qualitative family.

c. Mixed-methods.

research approaches match up with the research acts in Question 12?

a. Survey research.

d. Experimental research.

e. Narrative research.

g. Ethnographic research.

i. Grounded theory research.

j. Other.

17. To what degree do you intend for your research to generalize across participants, interventions (or phenomena), outcomes, and settings?

19. To what degree will you involve stakeholders in the research process?

121

20. To what degree will you (the researcher) be involved in the research setting or involved in the phenomenon being investigated?

need? How many pages of transcripts do you intend to get?)

-

23. What safeguards are in place to ensure that your research is ethical?

24. What are the time and resources necessary to carry out your proposed rese-

research are worth the time and resources necessary to carry out the study?


multidisciplinary methods in educational technology research...

Documents