Research and Analysis for Public Policy and Public Management:

Principles and Practices from Active Learning

Dahlia K. Remler

School of Public Affairs, Baruch College, City University of New York

Department of Economics, The Graduate Center, City University of New York

National Bureau of Economic Research

Draft

Do not cite without the author’s permission

November 2, 2011

Acknowledgements: Gregg G. Van Ryzin and I developed many of these ideas and approaches jointly. All mistakes are my own.

1. Research and Analysis in Public Policy and Management

Most Masters of Public Administration (MPA) and Masters of Public Policy (MPP)

programs have some kind of statistics, quantitative data analysis or research requirement,

although the specific courses’ content and names vary (NASPAA, n.d.). Such requirements seem

essential, since public sector managers, non-profit managers and policymakers at all levels are

increasingly called upon to use evidence-based practice and data for both management and

accountability purposes. Yet students often find their research and analysis (R&A) courses

irrelevant—as well as difficult. In this paper, I describe principles and practices for MPA1 R&A

courses that improve student learning, interest and motivation.

Most of the principles and practices I describe are relevant for a wide variety of courses:

statistics, research methods, evaluation, data analysis, research in public administration and so

on. Although I emphasize quantitative methods, qualitative methods are also included and

many of the principles and practices apply equally to both. While the practices and principles

apply widely, they are particularly focused on some general learning objectives, which I expect

most MPA R&A courses share to some extent. Specifically, the objectives are that after the

course(s) students will be able to:

Critically consume researcho Spot weak or invalid conclusions in formal and “informal” research o Extract relevant and valid conclusions from research

Perform research in policy and practice capacities at a basic level Deal effectively with the quantitative aspects of public affairs

Since most MPA students do not become researchers or analysts, I recommend that primary goals

for all programs be that graduates are effective consumers of research and analysis and are

quantitatively literate. (Steen (2001, 2004) defines quantitative literacy, often also referred to as

quantitative reasoning, and which I discuss extensively below.)

1 The principles and practices described here are equally relevant to MPA and MPP students. Hereafter, in the interests of brevity, I will only refer to MPA students. The only relevant difference is that MPP students are more likely to consider research and analysis training of interest to them and therefore require less motivation for R&A courses (Infield and Adams, 2011).

1

I developed the practices and principles described in this article through two main

means: Trial and error in my own teaching2 and in collaboration with colleague Gregg Van Ryzin

as we wrote a research methods textbook (Remler and Van Ryzin, 2011); The teaching and

learning literatures on active learning and quantitative literacy.

The paper is organized as follows: In section 2, I describe those aspects of the diversity

and context of MPA students most relevant to R&A courses. In section 3, I describe the active

learning and quantitative literacy literatures, extracting the parts relevant for MPA R&A

courses, providing some examples and concluding with general principles. In section 4, I provide

descriptions of good practices for MPA R&A courses and many examples. In section 5, I briefly

conclude.

2. Diversity and context of MPA students

MPA students are diverse in many ways: age, ethnicity, other demographics, but one

form of diversity has a tremendous effect on R&A courses: prior quantitative training. I will

characterize (or caricature) that variation with three made-up composite bios, characteristics of

students I have taught. Alice hated math in high school, avoided quantitative courses in college,

majored in Ethnic Studies and is now a community organizer with a small not-for-profit. Brenda

was a math major in college who now, under the supervision of a senior researcher, does data

management and some analysis for a headquarters of a national religious organization. She

wishes to become an analyst in her own right. Carlos majored in history, is now a human

resources manager in a Federal government agency and wishes to move up as a manager in the

Federal government. Brenda is the only student with a strong quantitative background and an

interest in research per se, but Alice also is motivated to learn about research, because her

organization’s funders want evidence of its effectiveness. Carlos is only interested in

2For over seven years, I have taught the two-semester research and analysis sequence to MPA students at Baruch College of the City University of New York. My students included full-time, part-time and executive MPA students. For the five years prior to that, I taught research methods to Masters of Public Health students in health policy and management at Columbia University.

2

management and sees no relevance of the MPA course to his career, beyond meeting a

requirement. The ideal class would be useful, interesting, and challenging but not impossible

for all three students. As the sample composite bios illustrate, a second form of MPA diversity is

also important: the research and analysis requirements of students’ planned careers.

Which stages students are at in their careers are also important for designing and

implementing R&A courses. Students who have substantial career experience, particularly

“executive” students, have a wealth of examples and potential applications. Such students also

tend to be impatient with any material whose relevance they cannot see. Their math skills may

be weak since it may be many years since they last took a math or quantitative course in college

or even high school. In many public affairs careers, students well along in their careers have

probably seen the emphasis on data and evidence, giving them particular motivation for R&A

courses. With a few careers, however, students well advanced in their careers may have not

missed quantitative and analysis skills, and resent being forced to jump through a hurdle they

see as irrelevant. For many MPA students, relevance to policy and practice, and particularly

relevance to a job they have or would like to have, is essential for motivating and engaging

them.

Other students, however, have less career experience and some even start an MPA

program directly after an undergraduate degree. Yet other students may have experience in a

very different career and plan to make a career change. Therefore, when crafting assignments

and exercises, it is important to be aware of these students also. Nonetheless, since they

choose to do an MPA, they should at least have public affairs interests to draw upon.

Some MPA students attend full-time with little work outside school, while others attend

part-time and continue to work. Working while attending school can provide examples and

motivation in the same way that career experience does, but it also can make students only

interested in material they see as relevant. Working also takes time away from studies and

reduces the flexibility of students’ time, as can family obligations or other constraints.

Attending class at night after working makes it harder for students to pay attention, increasing

3

the need for methods for keeping their attention. The long, intense sessions common in

executive programs cause similar problems.

For all students, it is important to keep the learning goals relevant for MPA students. A

minority of MPA students, and even of MPP students, become researchers or analysts. We turn

next to principles that can help the achievement of R&A learning goals.

3. Active Learning and Quantitative Literacy: Principles for Research and Analysis Courses

Today most instructors have heard the idea that they should not just lecture but rather

employ active learning. But what exactly is active learning? Bonwell and Eison (1991) say that

active learning requires that “students must do more than just listen: They must read, write,

discuss or be engaged in solving problems. Most importantly… [they] must engage in such

higher-order thinking tasks as analysis, synthesis and evaluation.”

Of course, almost all courses require active learning tasks outside the classroom. What

is the evidence about the effectiveness of active learning in the classroom? Pascarella and

Terenzini (2005) summarize the experimental and quasi-experimental literature on active

learning in higher education, saying that, “studies either report better mastery of course

content when actively engaged in learning…or no significant learning differences or mixed

learning effects when comparing active to passive lecture instructional approaches” (p. 102).

Although not all studies provide enough data to determine an effect size, among those that do,

Pascarella and Terenzini estimated an average effect of .25 standard deviations. Studies using

observational data and control variables were consistent with the experimental and quasi-

experimental findings.

A critical feature of active learning approaches, as with any teaching strategy, is to

clearly define the learning objectives and then ensure that the approaches further those

objectives. Bonwell (1996) suggests several active learning techniques suitable for a

predominantly lecture class. Some that are suitable for R&A classes include: Short Writes, which

could be short interpretations or simple problem-solving; Think-Pair-Share, in which a short

problem or question is given and two students discuss it for two or three minutes, prior to a full

4

class discussion; Formative Quizzes, with a few short questions that allow students to test their

own understanding and practice applying the material. Bonwell and Eison (1991) describe many

lecture-substitute activities, including case studies, debates, and peer teaching.

Many active learning approaches described in the literature do not fit well with students

trying to learn essential quantitative skills that they could not learn on their own, particularly

when skills build on one another. For example, a debate using empirical evidence to support

positions would motivate students and is a wonderful way to synthesize and reinforce

application and critical analysis of evidence. But it is not a good method for first teaching

statistical significance to students who do not yet understand it. The literature on quantitative

literacy is therefore particularly useful, since it is focused on the right sort of learning for R&A

courses.

Steen (2001, p.8) describes several components of quantitative literacy, of which the

most relevant for MPA R&A courses are:

Confidence with Mathematics…Individuals who are quantitatively confident routinely use mental estimates to quantify, interpret, and check other information… Interpreting Data. Reasoning with data, reading graphs, drawing inferences, and recognizing sources of error… Logical Thinking. Analyzing evidence, reasoning carefully, understanding arguments, questioning assumptions, detecting fallacies, and evaluating risks… Making Decisions. Using mathematics to make decisions and solve problems… Number Sense. Having accurate intuition about the meaning of numbers, confidence in estimation, and common sense in employing numbers as a measure of things. Practical Skills. Knowing how to solve quantitative problems that a person is likely to encounter at home or at work…

I suspect that MPA programs all require some quantitative analysis or research course as

much to enhance their students’ general quantitative literacy as to teach statistics or any other

particular skills or knowledge. Despite the variation in MPAs’ career goals, some tasks should be

doable by all MPAs, both immediately upon graduating and long after finishing their degree.

Consider an example: An MPA’s organization has implemented a new policy and wants to see if

it “worked” to improve an outcome. The organization has individual level outcome measures

over time. (For example, an educational not-for-profit has added math majors as tutors and

5

wants to know if their centers have more students in the community coming for help with

math.) An MPA should be able to take the data in a spreadsheet or statistical package, graph

averages over time and interpret the results. More importantly, any MPA graduate should be

aware of the problems of statistical significance (that a change could be a fluke), even if they

cannot implement the correct statistical significance test, and be aware of the problems in

assuming that any change in outcomes was caused by the program change. Finally, he should

consider the outcome measure’s validity and reliability. Another example, which is both

qualitative and less formal, would be that when seeking feedback or thoughts on an issue, the

MPA graduate will always be attuned to the issue of the representativeness of those providing

feedback, even when time and resources constraints prevent obtaining a representative

sample.

Steen (2001, 2004) and references therein describe a variety of teaching practices to

increase quantitative literacy. I have selected the most important relevant recommendations

for MPA R&A courses from Collison et al’s (2008) adaption of Steen’s recommendations. They

are listed in Table 1.

Table 1: Strategies for Quantitative Literacy

Rule of Four: All applications and concepts presented as:

o Wordso Numberso Graphso Symbolso Translate from any one to the other

Practiceo Interpreting and writing about numberso Explaining equations in wordso Reading, interpreting and applying technical writing

Assignments and tests that require students to apply skills in applications that are meaningful to the students

o Examples involving familiar concepts are more effective than examples that require extra learning

6

o Examples which motivate and interest students are valuable A variety of different applications

o Increasing student role in framing the problem and in abstracting Spreadsheet and/or statistical software exercises integrated into course content

throughout the curriculum

The first strategy, the “Rule of Four,” means that concepts and applications should be

approached in four forms: equation, graph, numbers (data), and words, and that students

should be able to translate between any two forms. Although the Rule of Four was developed

for teaching calculus (Hughes-Hallett, Gleason and Flath, 2008), I have found it useful in all

quantitative subjects. While all four forms are useful, not all MPA students will become

comfortable with equations in some contexts, and therefore, to some extent, rationing

equations can be useful.

Consider a simple regression example, with data from a representative sample providing

individuals’ annual earnings and the number of years of education. The equation is

Earnings = const + B* Education.

(I recommend using the names of variables: Earnings, rather than “y”; Education, rather than

“x.”) A useful introduction or application of regression would include a scatterplot of the data,

with the best-fit ordinary least squares regression line, and a table (in software) showing the

numerical data. Students should also interpret the coefficient and constant in words, “For each

additional year of education a person has, we expect him to earn $B more annually” and “We

expect that a person with zero years of education would earn $const.”3 (They should use the

actual numbers, rather than B or const, of course.) Students should also describe other aspects

of the relationship in words, including tightness of fit. All four approaches together should be

used for introducing the concept and when students apply it, using statistical software to

estimate the constant and coefficient. It is relatively common to teach regression using the Rule

of Four, but the approach can be applied to hypothesis testing and many forms of data analysis.

3 The interpretation of the constant, of course, should be immediately followed with a discussion of how no one has anything close to zero years of education and the problems of out-of-sample predictions.

7

While all four parts of the Rule of Four are valuable, communicating in words about

quantitative content is particularly important. First, it is an extremely important skill in its own

right. MPA students, even those who become researchers or analysts, will work in the worlds of

policy or practice. Sentences like, “the logit results were an odds ratio for X1 of 1.6 with a p-

value of .007 and an odds ratio for X2 of 1.1 with a p-value of 0.26.” will not serve them well in

their careers even if they understand such sentences well. MPAs will need to be able to express

quantitative and technical information in the most accessible and meaningful manner possible.

Miller (2004, 2005) has written two books on writing about numbers and about multivariate

analysis, which include sections on writing for relatively broad audiences.

Second, describing numbers, equations and analytical concepts in words is an effective

way for many to learn. Students should practice interpreting entries in tables and statistical

package output in words. Being able to express the results in words both requires and aids

understanding. I recommend repeated exercises in which students interpret both individual

numbers within a table of results and the overall picture from the table. Sometimes non-native

English speaking students resent an emphasis on words in a quantitative class, having expected

that their weak English would matter little, if at all. For such students, suggest to them that they

start by practicing explaining in their native language and note that being required to explain

well in English will make them more effective at explaining well in their native language.

The third bullet in Table 1 refers to the importance of relevant applications. For MPA

students, who may already work or be well advanced in their careers, such applications are

extremely important, particularly for motivation. But applications are critical for all students

and they should be rich and compelling. The fourth bullet refers to the importance of many

different applications. Difficult concepts and methods take repeated efforts to learn. No matter

how motivating and useful a single example, students may not learn to generalize to other

situations unless they have practice generalizing. I return to this issue in later sections. Finally, it

is important for students to work with data in spreadsheets and/or statistical packages. Even

students who will do little data work of their own gain a far fuller understanding of the meaning

of statistics and data analysis by doing it themselves. I return to this issue later also. However, it

8

a class should not be dominated by either data cleaning or the details of a particular statistical

package, because most MPA students will be primarily consumers of research and analysis, not

producers.

The quantitative literacy literature also stresses that the practices and skills of

quantitative literacy must be reinforced across the curriculum. Ideally, other MPA classes in

subjects such as policy analysis, management, budgeting, communications, economics, politics,

and of course a capstone, would all reinforce these skills. While that is a good idea, from

hereon, I restrict myself to suggestions implementable in R&A courses alone.

In summary, I recommend the following four principles for MPA R&A courses:

1) Students should write and speak in words, interpreting the results or studies.2) Always teach using applications relevant to public policy and management. 3) Whenever possible, let students pick applications, particularly for longer assignments. 4) Use active learning.

In the next section, I flesh out these principles, particularly the active learning, with specific

practices and examples of their implementation.

4. Practices

In this section, I focus on those practices I have found effective in R&A, starting with

those that take the shortest amount of time and moving on to those who take the longest. In

every case, I include specific examples from my own teaching.4

4.1 One to Two Minute Individual Writing Exercise

Many of us ask our students questions and ask for volunteers to answer. While I confess

that this is my usual practice, it suffers from several problems. Students whose minds are

wandering or those who feel that they don’t understand will not actively try to answer the

question. Even students who do try may only have time to make an initial start before another

student answers the question, depriving the first student of a real opportunity to learn.

4 Many of these examples were developed working with Gregg Van Ryzin and he is the primary creator of some examples.

9

Therefore, I recommend making questions into very short writing exercises. Ask the question

and ask every student to answer it in writing. When most students are done, bring the class

together and ask for volunteers or call on students. In cases where there is a correct answer,

make sure that it is provided, clearly labeled as the correct answer and fully explained.

This approach can work both in the middle of a lecture and while discussing a study. This

approach has several advantages. It makes students realize if they do not understand or do not

know how to apply, potentially prompting students to ask useful questions. These exercises

make it much harder for students to tune out. When teaching skills that build on one another,

this approach can make sure the foundations are there before moving on. For example, if a

student cannot use the basic tools of path diagrams (circles for variables and arrows for causal

relationships), he will not understand subsequent material on mechanisms or common causes.

Example short writing exercises I have found effective include:

Identify independent and dependent variables in a particular application Describe the population sampled for a particular survey Interpret in words one number in a descriptive statistics table Determine if a particular result is statistically significant in a results table Calculate the margin of error for a poll State the null and alternative hypotheses Determine if a study is descriptive or causal; determine if a study is an experiment

This practice can also be used to help each student find relevance in the material for herself,

and thus become more motivated. Examples of such exercises include:

Describe a relationship between two categorical variables relevant to your work (or interests)

Think of a program and an outcome it is designed to improve in your work (or an area of interest). Do you have any estimates of the program’s effect on the outcome?

Think of a measure, such as a performance measure, used in your work. How valid do you think it is?

Using the first exercise will make cross-tabs seem much more relevant and interesting. Using

my fictional students to illustrate, Carlos, the government human resources management,

might be interested in the relationship between region and ethnicity of applicants. When

discussion of cross-tabs continues, Carlos’s example could continue to be used. Alice, the

10

community organizer, will likely have little trouble finding examples that would interest her

funders.

One problem with this practice is the large variation in students’ backgrounds and skills.

Illustrating again with my fictional students: Brenda, the math major and data analyst, will often

have the answer almost immediately, while Carlos and Alice will need more time. Either Brenda

sits around bored or Carlos and Alice get cut off and don’t have a chance to learn. To solve this

problem, prepare an “extra task or question” that you describe as only for students who have

time for it. For example, if students are being asked if a particular result is statistically

significant, the extra task could be assessing its practical (substantive) significance. If the main

task is interpreting a result, the extra task could an interpretation clear and accessible to a

journalist.

4.2 Think-pair-share

The think-paid-share approach is similar to the above individual writing exercise but asks

each student to work with another, usually after first pausing to think for herself. Tasks are

similar to the above but should take slightly more time. If students differ in ability, this

approach gives the stronger student the opportunity to teach, reinforcing that student’s

understanding and communication, while preventing the weaker student from floundering

alone. However, if the pairs have similar abilities, the pairing can exacerbate the problem due

to ability variation, with some finishing instantly and others floundering together. For that

reason, as well as the additional time for pairing up, I prefer the short individual exercise above

or longer group exercises described below. Nonetheless, think-pair-share is a standard in the

teaching and learning literature and it can be useful, particularly for less technical material. For

example, a pair could be asked to write a survey question asking respondents about how safe

they feel at school.

4.3 Longer Group In-class Exercises

For larger group exercises, students form groups of three to four students to do a

substantive task or solve a multi-part problem. I find this approach essential for tasks or

11

concepts that students find very difficult and which require multiple tries, with feedback, for

mastery. Examples include:

Finding alternative causal explanations for a correlation (e.g., reverse causation, common cause)

Creating a mechanism in a logic model of a particular program Interpreting regression coefficients and other statistical package output in terms useful

for policy or practice Predicting the direction of non-response bias for particular measures, and other biases,

in a particular survey Evaluating the generalizability and quality of causal evidence (internal validity) of a

particular quasi-experimental study

The first example is shown in Appendix A, in which I would expect all groups to get through part

(c). I often have the same kinds of exercises for out-of-class assignments, since students require

repetition. All of these tasks take significant time, sometimes a half hour or even more,

representing significant opportunity cost for class time. Therefore, I reserve such group

exercises for topics that students struggle with and that I consider very important.

The variation in student preparation and ability is again a problem. One approach is to

pre-select groups based on prior background (e.g., undergraduate major) to attempt to get

balance, although it can take time for students to physically form these groups. However, again,

the main solution is to have core tasks and extra tasks. For the logic model example, every

group should come up with at least one mechanism, but some groups may have time to find

several mechanisms or include moderating variables. If students are interpreting multiple

regression output from a statistical package, an extra task might be to discuss what other

control variables would be wanted and the bias from omitting them.

Circulating is essential for uncovering significant problems or misunderstandings. Make

sure to interact at least once with each group. If not, students may go off on the wrong track for

a long period or even the entire exercise. When I initially started to do small group exercises, I

made them much too difficult—the kind of problem appropriate for outside of class with much

more time. Not only did students not learn but they often confused one another and spread

bad approaches. The tasks should be meaty enough to warrant such a significant chunk of class

12

time, but almost all students should be able to complete the core tasks. The core tasks should

be doable from material taught immediately before or required as reading for the class.

4.4 Discussions of Studies

All MPAs should be consumers of research and analysis in many forms: peer-reviewed

academic journal articles, government and foundation reports, stray numbers or anecdotes

bandied about, advocacy, media accounts and more. They should be able to spot weak or

invalid conclusions and extract relevant, valid conclusions. MPAs will often work in settings

where time and resource constraints necessarily limit the quality of data and analysis and they

must know how to make the best of what they have. They will also need to go to the existing

literature and try to extract what is useful and relevant. Therefore, it is essential that they

practice reading and interpreting studies of many kinds: mainstream media presentations of

studies, mainstream media examples not of formal studies but ostensibly evidence,

government reports, advocacy reports, journal articles for which students know most methods,

journal articles which are too advanced for them but from which they can learn to extract

useful information. Management examples, whether from case studies or real problems, are

also useful.

Due to the scarcity of class time, readings should be done outside of class and discussed

in class. Since the discussion can stray far from the main learning goals, I suggest providing

questions to go with each reading. (See Appendix B for examples.) For long and/or difficult

articles, tell students which sections and/or issues to focus on.

One difficulty is ensuring that students do the readings before class. A possible approach

is a brief reading quiz that asks simple factual questions about the article (such as which city the

study took place in). Another problem is that only a few students may participate in the

discussion. The think-pair-share practice be useful for discussing questions on the readings,

before coming together as a full class. On-line discussion boards, with students required to give

a certain number of comments is another way to ensure broad participation. A final problem is

that even with specific questions, the conversation can go astray, and therefore attention must

be paid to keep on topic.

13

Reading studies is essential for research courses but cannot be easily used in a pure

statistics course. (That very fact reveals why an MPA program should offer more than simply a

pure statistics course.) However, in pure statistics classes, it is valuable to give students tables

of results for interpretation, with some context and set-up provided. Having students interpret

both individual entries and the large picture from tables of results will help with both

motivation and application skills.

4.5 Out-of-class Assignments, Including Data Analysis

An important purpose of out-of-class assignments is to reinforce the most difficult

and/or important topics, as well as to synthesize and apply more deeply. Out-of-class

assignments can provide opportunities for students to choose something relevant to them,

increasing their motivation and often their learning. Examples of out-of-class assignments that

can be chosen by students are a logic model and designing a survey design. (Appendix C

contains these two example out-of-class exercises.)

As already discussed, analyzing quantitative data is an important component to

developing quantitative literacy, as well as an important skill in its own right. Some instructors

use labs with statistical software to develop such skills. Labs are another form of active learning

in which students implement what they learn during class. Therefore similar strategies are

needed to cope with student variability. Whether or not there is a lab class or class time is used

for labs, I recommend out-of-class data analysis exercises, to allow for much deeper skills and

individual investigation.

For practical reasons, it is difficult for students to choose their own individual data sets

and therefore, in most classes all students will analyze the same data provided by the

instructor. However, in some contexts, it may be realistic for students to assemble their own

datasets from publicly available data (Aguado, 2009; Remler and Van Ryzin, 2011, p. 204). Even

if students all use the same datasets, they may be able to pick their own analysis (Hill 2003). For

example, students could select variables for descriptive statistics, choose their own control

variables or address different research questions.

14

The teaching and learning literature recommends that instructors use rubrics to

evaluate student assignments (Andrade, 2005). A rubric is “an assessment tool that lists the

criteria for a piece of work or what counts and articulates graduations of quality for each

criterion,” not only a ”checklist of criteria” but also “gradations in quality” (Andrade, 2005 p.

27). Appendix C includes two sample rubrics I have used to grade the accompanying two

sample out-of-class assignments.5 6 Students should receive the rubric with the assignment.

Andrade (2005) describes rubrics’ multiple strengths. They push instructors to clarify major

goals and expectations about those goals and to focus instruction, assignments and assessment

on those goals. They communicate priorities, desirable qualities and common pitfalls to

students. They provide informative feedback to students with less instructor time than is usual

for such individual and contentful feedback. While much of the peer-reviewed empirical

literature about rubrics focuses on writing instruction, there is recent work on assessing

quantitative reasoning in written assignments (Grawe, Lutsky and Tassava, 2010) and the

principles appear relevant to R&A courses.

4.6 The Case Against One Big Project

Many MPA research and analysis courses have students do one large research project as

their main or only outside-of-class assignment. There are many advantages to such an

approach. Students can select their topic, making it meaningful, increasing motivation and

learning. The work is very applied and forces students to deal with many real world

complexities. While it is great if students can do such projects in a capstone class, I argue

against them in standard core R&A courses.

First, MPA students need to learn quite a few distinct skills and concepts. To name just a

few, all MPA students need to aware of issues like: the representativeness of those who choose

5 I have only recently begun to use rubrics in my teaching and have no doubt that these could be improved substantially. These sample rubrics are revised versions of rubrics developed in coordination with my practitioner co-instructor, Gregorio Morales. 6 The Association of American Colleges and Universities (2010) has created an extensive rubric for undergraduate education, including quantitative reasoning. While it is far too general for a specific class, it is an excellent illustration of rubric design as well as principles of quantitative literacy.

15

to speak up about an issue, the idea of statistical significance (differences or trends possible

being a fluke), knowing when and why a proportion is needed, and not misinterpreting a

correlation as evidence of causation. Any given research project is likely to only touch on a few

of these skills. Moreover, students are quite likely to choose a qualitative research project,

which is a logical choice for many, since they are more likely to become producers of qualitative

evidence than producers of quantitative evidence. However, all MPA students should be able

consumers of at least basic quantitative evidence. They will not gain those skills by going a

qualitative research project. A further issue is that working with one example, even if it is rich,

makes students less likely to be able to generalize—to use their skills in a different context. If a

single project is to be used, it should be structured to ensure that a wide-range of skills are

included. (See Aguado (2009) for an example.) Of course, a big project has many advantages

and ideally students will also have the opportunity to complete one in a capstone course.

Instead of a single big research project, I recommend several out-of-class assignments,

along with shorter practice problems and at least one exam. This provides multiple chances to

learn with different applications. Otherwise, however, this recommendation is not based on the

teaching and learning literature, but on my own experience and my observations of others’

classes. In my experience, an exam focuses students’ minds on learning particular skills in a

manner that allows application to new problems. Of course, the short problems and exam

questions should still be applications and not just rote calculations or definitions. Appendix D

contains examples of sample problems and exam questions for research methods. Such

problems for a statistics course should include a great deal of interpretation in words of output

from statistical packages.

4.7 “Tell the story of the course”

In any course, but particularly in a technical course, students can lose sight of the big

picture: why they are doing this, what it is useful for, how it all fits together. To combat this

problem, periodically ask students to “tell the story of the course,” describing both

16

chronological and conceptual progression, and answering those three why, what and how

questions. They should be able to do this in a few sentences.7

For example, the story of the first semester statistics course that I teach would ideally go

something like, “We started by trying to describe one variable, such as gender or age,

graphically and with numerical summaries. We then moved on to the relationships between

variables, focusing in turn on the forms needed for different types of variables; for example,

cross-tabs describe the relationships between two categorical variables. We then turned to

inference, learning how precisely we estimate things from a sample and seeing if relationships

are statistically significant—or just a fluke.”

The story of the second semester research methods course that I teach would ideally go

something like, “We started with theories, which explain why variables vary, and with

qualitative research for initial explorations. We then turned to describing the world: how to

measure, how to sample, how to design surveys, how to find existing data. We moved next to

estimating and understanding causal effects: what happens to an outcome if we change

something—“what if” questions. Specifically, we covered the many causal explanations of a

correlation, trying to disentangle causal effects in observational data, randomized experiments,

and quasi and natural experiments.”

Obviously, these stories of the courses could be lengthened or shortened, include more

or less jargon, or include more or fewer examples. While students often struggle to do more

than list topics, it is useful for them to try and useful for the instructor to remind them of the

big picture.

4.8 Shortage of Time and Solutions to It

The literature on active learning lists several barriers to the use of active learning. As

already described, the biggest barrier for MPA R&A courses is not listed in the literature I have

read: the tremendous variation in student quantitative background and in interests in research.

7 This suggestion came from my colleague Neill Sullivan who asks students to tell the story of the class in Introduction to Public Affairs. It is a valuable tool in any class.

17

However, the literature does focus on the second biggest barrier: time. As Bonwell and Eison

(1991) state, with active learning “one cannot cover as much content in the time available” (p.

59). Of course, there is no reason that all the content must be covered orally in class. They note,

“as has been pointed out countless times, the lecture was outmoded by the invention of

printing and by cheap and easy access to printed words” (quoted in Bonwell and Eison, 1991, p.

60).

In today’s world, reading is far from the only means for delivering content outside of

class. Software to teach statistics exists and more is being developed. Many statistics textbooks

have accompanying packages with simulations and on-line exercises. In addition to such

statistics tools for sale, the Open Learning Initiative provides a free interactive statistics course

on-line (Open Learning Initiative, n.d.). Such software has the advantage that each student can

go at his own pace. Free videos teaching statistics abound on the web, although sorting through

them to figure out which ones are good takes considerable time.8 With technology today,

professors or departments can create their own videos—or audio recordings to accompany

visual slides. All of this allows content to be covered outside of class and class-time to be used

for those tasks where physical presence is most useful and important, such as discussions,

sample problems and the exercises described previously.

Of course, many instructors, including myself, will feel obligated to lecture on most of

the required content, because we know that all students will not learn all assigned material.

That does not mean that active learning should be given up for lecturing on all the required

content. The unfortunate truth may be that given the backgrounds of entering students, the

time available for class, and the time students have available outside class, it will not be

possible for many or even most students to learn all the required content.

One option is to teach less content—but more thoroughly and with greater student

mastery of the material. While that can be a good approach to some extent, two factors argue

against it. First, some students are able to learn all the content and they want and need it.

(Think of Brenda who is already doing some data analysis professionally and wishes to develop

8 I have not done this.

18

those skills further.) Reducing the content ill serves those students and makes the program less

attractive to them. Second, different students will want or be able to reach higher levels for

different topics. For example, Alice may not reach an advanced level with multiple regression,

but her background with non-profit funders may make her able and willing to reach advanced

levels with logic models and survey design. Reducing content in some areas would help Alice

but in other areas it would hurt her.

The best approach is to prioritize some content and reinforce that content through

active learning. Additional easy-to-learn material can be covered through readings and other

outside of class forms. Additional hard-to-learn material can be covered outside of class, letting

students know that it is the harder material and less of a priority. Harder additional material

can also be covered in the “extra” or more advanced tasks of in-class active learning.

4.9 Summary of Practices

In summary, there are a variety of active learning practices that can promote

quantitative literacy, skills and knowledge useful for MPA students. They range from short

individual writing exercises that interrupt lectures, to longer group in-class exercises, to

discussions of studies, to meaty applied out-of-class exercises, to on-line software, to short

applied out-of-class problems.

5. Conclusions

Seeing relevance is key to motivating MPA students. Providing relevant applications

ensures that they learn the skills relevant to their fields. When research and analysis courses

are based on relevant learning objectives and are taught well, students see the relevance and

are motivated. They understand that it is about the culture of evidence, data and performance

measurement they see and hear about so often. They should not think that research is only the

stuff in academic journals, although they should learn how relevant academic journals are to

them. They should see that research and analysis is about things that all MPAs will see in their

careers: client satisfaction surveys, figuring out if a new program is working, being wary of the

representativeness of the “squeaky wheel.”

19

The tools of active learning can aid all forms of education; MPA research and analysis

courses are no exception. However, the tools must be selected and adapted appropriately. On

the one hand, there is an existing teaching and learning literature we should make use of. On

the other hand, some of that literature is based on poor research, particularly poor causal

research. Moreover, very little of that literature is based on studies in MPA programs or on R&A

courses in MPA programs. Schools of public affairs and public policy are filled with people who

can do good research, particularly good causal research in applied settings. Following on calls to

increase the scholarship of teaching and learning (Boyer, 1990; Carnegie Foundation, n.d.),

perhaps some of us should work on good evidence about what works for MPA and MPP

students.

20

References

Aguado, N. Alexander. 2009. “Teaching Research Methods: Learning by Doing” Journal of Public Affairs Education 15(2): 251-260.

Andrade, Heidi Goodrich. 2005. “Teaching with Rubrics: the Good, the Bad and the Ugly” College Teaching 53(1): 27-30.

Association of American Colleges and Universities. 2010. Quantitative Literacy VALUE Rubric. Washington, DC: Association of American Colleges and Universities.

Ariguesta, Maria P. and Jeffrey Raffel. 2001. “Teaching Techniques of Analysis in the MPA Curriculum: Research Methods, Management Science and ‘The Third Path’” Journal of Public Affairs Education 7(3): 161-169.

Bonwell, Charles C. and James A. Eison. 1991. Active Learning: Creating Excitement in the Classrom. ASHE-ERIC Higher Education Report No. 1, Washington, DC: The George Washington University, School of Education and Human Services.

Bonwell, Charles C. “Enhancing the Lecture: Revitalizing a Traditional Format” in Using Active Learning in College Classes: A Range of Options for Faculty, Tracey E. Sutherland and Charles C. Bonwell, editors.

Boyer, Ernest. 1995. Scholarship Reconsidered: Priorities of the Professoriate San Francisco, CA: Jossey-Bass.

Carnegie Foundation for the Advancement of Teaching. n.d. Carnegie Academy for the Scholarship of teaching and Learning (CASTL). Retrieved October 22, 2011 from http://www.carnegiefoundation.org/scholarship-teaching-learning

Collison, Joe, Catherine Good, Sonali Hazarika, Matt Johnson, Jimmy Jung, Anita Mayo, Will Millhiser, Dahlia Remler, and Laurie Beck. 2008. “Report of the Provost’s Task Force on Quantitative Pedagogy.” Retrieved August 1, 2010 from http://www.baruch.cuny.edu/facultyhandbook/documents/TaskForceonQuantitativeSkills2008-09-05.doc

Grawe, Nathan D. and Carol A. Rutz. 2009. “Integration with Writing Programs: A Strategy for Quantitative Reasoning Program Development.” Numeracy 2(2): Article 2. Retrieved May 12, 2010 from http://services.bepress.com/numeracy/vol2/iss2/art2.

Hill, Carolyn J. 2003. “Can They Put it All Together? A Project for Reinforcing What Policy Students Learn in a First-Semester Quantitative Methods Course” Journal of Policy Analysis and Management 22(3): 473-481.

Hughes-Hallett, Deborah, Andrew M. Gleason and Daniel E. Flath. (2008) Calculus: Single and Multivariable. John Wiley & Sons. 5th edition.

21

Infield, Donna Lind and William C. Adams. 2011. “MPA and MPP Students: Twins, Siblings or Distant Cousins?” Journal of Public Affairs Education 17(2): 277-303.

Miller, Jane E. 2004. Chicago Guide to Writing about Numbers Chicago: University of Chicago Press.

Miller, Jane E. 2005. Chicago Guide to Writing about Multivariate Analysis. Chicago: University of Chicago Press.

NASPAA (National Association of Schools of Public Affairs and Administration) n.d. “The MPA/MPP Degrees” Retrieved October 22, 2011 from http://gopublicservice.org/degree.aspx.

Open Learning Initiative n.d. Retrieved October 22, 2011 from http://oli.web.cmu.edu/openlearning/forstudents/freecourses/statistics

Pascarella, Ernest T. and Patrick T. Terenzini. 2005. How College Affects Students: A Third Decade of Research. 2nd edition. San Francisco, CA: Jossey-Bass.

Remler, Dahlia K. and Gregg G. Van Ryzin. 2011. Research Methods in Practice: Strategies for Description and Causation. Thousand Oaks, CA: Sage.

Steen, Lynn Arthur. Editor. 2001. Mathematics and Democracy: The Case for Quantitative Literacy. National Council on Education and the Disciplines.

Steen, Lynn Arthur. (2004) Achieving Quantitative Literacy: an Urgent Challenge for Higher Education. Mathematical Association of America.

22

Appendix A: In-class group exercises

I. Correlation and causation

A research article reports that looking across schools, there is a correlation between mean test score and whether or not the school library has a qualified librarian. The Association of School Librarians picks up on the study and says that it shows that better librarians result in better student learning and test scores and therefore funding for qualified librarians should be increased.

(a) According to the librarians, what is the dependent variable and what is the independent variable? What is the unit of analysis in the study?

(b) Describe a theory that is consistent with the librarians’ view of what causes what. Use both words (a few sentences at most) and a path diagram. Make sure to include some intervening variables (i.e., a mechanism)—at least a start at convincing a foundation to give money to support qualified librarians.

(c) Describe an alternative theory that both contradicts the librarian’s position and explains the correlation in the study. Use a path diagram and words (a few sentences at most).

(d) What is the relevant counterfactual question?

(e) Explain in a few sentences what all of this has to do with the idea of endogeneity.

23

Appendix B: Questions for In-class Discussions of Readings

Reading Discussion Example I: Peer-reviewed research article

Reading is Cattaneo, Matias D., Sebastián Galiani, Paul J. Gertler, Sebastián Martinez, Rocio Titiunik. 2009. “Housing, Health and Happiness” American Economic Journal: Economic Policy 1(1): 75-105.

Reading instructions: You do not need to read section VI (robustness checks) and you can skim section III (data). Throughout, do not worry about the details but focus on the big picture of the basic idea of the study.

Questions on Catteneo et al “Housing, Health and Happiness”:

(1) What is (are) the research question(s)?

(2) What type of study is this?

(3) What are the outcome measures?

(4) How valid is the causal evidence (internal validity)? Justify it. What weaknesses are there?

(5) What were the findings? Focus on Table 5.

(6) Remembering the 3 main things to look at—statistical significance, practical significance and evidence of causation—how do you assess the findings?

(7) How generalizable are the results?

24

Reading Discussion Example II: Media articles

Read the stories on these web pages, in order

http://www.cnsnews.com/node/62812

http://www.nejmjobs.org/rpt/physician-survey-health-reform-impact.aspx

http://www.themedicusfirm.com/pages/medicus-media-survey-reveals-impact-health-reform

http://www.themedicusfirm.com/pages/survey

Questions on Medicus Survey of Physicians about Health Reform Readings

1) The New England Journal of Medicine (NEJM) is a peer-reviewed journal. Was the study peer-reviewed? Was it published in the NEJM? What does the Medicus Firm do?

2) What is the sampling frame or method of obtaining sampling units? What population does the sampling frame represent? What population does the survey interpreters claim that it represents? Are the two populations different in any way? Is there coverage bias in the “would try to leave medical practice” measure? Explain.

3) What is the size of the true sample? What is the size of the observed sample? (Explain the difference between the true and observed sample.) Which documents report the true sample?

4) Who is likely to respond and who is not? How might the propensity to respond produce bias in the “would try to leave medical practice” measure?

5) Discuss the relationship between stating in a survey that one “would try to leave medical practice” and actually leaving medical practice in fact. Are the persons surveyed competent to provide the desired information?

25

Appendix C: Examples of Out-of-class Assignments with Corresponding Rubrics

I. Logic model/mechanism assignment

Consider a policy or social program that actually exists, that you would like to propose or that someone else has proposed. Choose something that interests you and that you know something about. Prepare a description of the theory of the mechanism of how your program works.

Write this up as memo to a boss or collaborator who is working with you to develop the program. This is not someone you need to convince about the importance of the outcomes or the program.

Make sure that you including the following:

(1) What is (are) the outcome(s) (dependent variable(s)) the program is designed to affect? If there are many outcomes, restrict your analysis to one outcome or two closely related outcomes. (For example, your program program’s goal might be to raise high school graduation rates in urban areas and so the outcome is graduation rate.) Make sure that you state the outcome(s) explicitly.

(2) Describe your program—what it is. This should be as explicit as possible, not vague generalities. This section should be brief: a half double-spaced page at the most. Do not include marketing or promotion: your reader does not need to be convinced of the importance of the project. Write an objective and concrete statement of what the program literally does but do include implementation details.

(3) Using a path diagram and a narrative description, describe your theory of how the program is supposed to work. Both the path diagram and the narrative description should make clear the mechanism(s) through which the program will affect the outcome. So, if a link is not obvious, break it down into the steps along the way, illustrating the intervening variables.

This section should illustrate to your readers why they should believe that the program will work—will affect the outcome(s). It should also make clear what the weak linkages are.

This part is the main focus of the assignment.

Hints: The circles represent variables and the arrows represent causal effects. Make sure that you understand clearly the unit of analysis in your theory—the individuals to whom the variable applies: For example, is the program working on students, on schools, on cities?

There can be many mechanisms through which a program works. If so, pick only a couple and just note that there are other mechanisms. These should be more detailed than the logic

26

models you see in many grant proposals and papers. Each link should be spelled out and made believable.

(d) Qualitative research: Suppose that you want to make sure that one part (link or links) of your mechanism works how you think it does. Describe briefly (in two paragraphs or less) some qualitative research you could do once the program is running to see if this part of your theory is correct. What method (structured interviews, focus groups, observation, etc.) would you use? What would you expect to learn from this?

Notes:

Do not include introductions, motivations, background, marketing and so on.

Do not include inputs, resources, or (detailed) activities. This logic model is not an implementation-oriented one: It focuses on mechanism. Implementation is done more effectively after you understand clearly the mechanism.

Check that each separate causal link makes sense isolated. Check that you are not missing causal links between variables on the page.

27

Rubric for Logic Model Assignment

Component A level work B level work C level work F level work

Independent variable

Clearly defined indep var in narrative and path diagram

All effects on outcome (except contextual vars) lead ultimately from indep var

Clearly defined indep var in narrative and path diagram

Some effects on outcome lead ultimately from indep var

Some definition of indep var in either narrative or path diagram

No clearly defined indep var

Dependent variable

Clearly defined Dep var in narrative and path diagram

No other unspecified outcomes are de facto outcomes

Clearly defined dep var in narrative and path diagram

Almost no other unspecified outcomes are de facto outcomes

Some definition of dep var in either narrative or path diagram

Other unspecified outcomes are de facto outcomes

No clearly defined dep var

Intervening variable

Clearly defined Interv variables in narrative and path diagram

The bulk of preceding variables are logical causes and following variables are logical consequences

Does not confuse process and mechanism

Clearly defined Interv variables in narrative and path diagram

Most of preceding variables are logical causes and following variables are logical consequences

Does not confuse process and mechanism

Some definition of interv variables in either narrative of path diagram

Some of preceding variables are logical causes, and some following variables are logical consequences

Confuses process and mechanism

Intervening variables are not variables

Preceding variables are not logical causes and following variables are NOT logical consequences

28

Mechanics of Assg

Narrative portion is clear, concise and avoids marketing in favor of program description.

Clearly describes variables and mechanisms.

The bulk of path diagram has clearly drawn variables, explicit relationship arrows, and includes signs to show positive or inverse causal effects

Narrative portion is clear, concise and contains little marketing in favor of program description.

Describes variables and mechanisms,

Most of the path diagram has clearly drawn variables, explicit relationship arrows, and includes signs to show positive or inverse causal effects

Narrative portion is confusing and/or contains mostly program marketing language.

Path diagram is missing some variables and/or some arrows and signs

Narrative does not explain the program or variables.

Path diagram missing many arrows and signs.

Writing quality

Writing is very clear

Arguments are cogent and persuasive

Memo’s organization is sensible and clear

Language is correct and concise (with no repetition)

Writing is fairly clear

Arguments are fairly cogent and persuasive

Memo’s organization is mostly sensible and clear

Language is mostly correct

Some unnecessary repetition

Writing is unclear

Arguments are cogent and persuasive

Poor organization

Language has mistakes

A lot of unnecessary repetition

Writing is unclear

Arguments are cogent and persuasive

No organization

Language has significant mistakes

Much repetition

29

Rubric for Qualitative component

Component A level work B level work C level work F level work

Qualitative method

Appropriate qualitative method chosen

Qualitative method chosen

Non-qualitative method

No method

Description of research question

Clearly described purpose to qualitative research

Question clearly relates to link in mechanism

Research question described

Question sort of relates to link in mechanism

No research question described

No research question described

30

II. Survey Design Assignment

In this assignment, you will design a survey, including a questionnaire, to gather information on a population of interest. This population and the information that you gather about them should be relevant to some policy or practice question of interest.

Purpose

What is the purpose of this survey? What information are you seeking and how will it be used? Your purpose can be purely descriptive—the way the world is. Alternatively, your purpose can be to obtain variables (dependent, independent and/or control) to address a causal question. (Example: You survey adult immigrant English as a second language students and you want to know if having an immigrant instructor influences satisfaction with the instruction program.)

Explain, as specifically as possible, how the results will be used for policy or practice.

Population

Define the target population of interest. For example, it could be clients of a program you work on or a group, like parents of public school children.

Describe any differences between the ideal target population for your purpose and the actual study population. (Usually, it is not possible to study the ideal population…)

Measures

Describe the characteristics that you would like to know about your population—the measures or variables. Choose between four and eight substantive measures, plus any demographics (like age and gender) that are relevant. Write out the conceptual definition—the construct-- of each substantive variable. The construct is what you really want to capture.9

Identify which questionnaire item corresponds to each variable. Identify the level of measurement (nominal categorical, ordinal categorical, or quantitative).

9 For example, if you want to learn about income, do you want to include in that measure, unearned income from government benefits and income from investments? What is the deeper purpose or meaning of income in your study?

31

Discuss how well the measures from your questionnaire do at capturing the construct that you wish to measure. Describe any problems of answer bias that you anticipate and why.

Sampling Plan

Describe how you will sample and contact survey respondents. Will you survey everyone in the population (a census) or will you sample? If you will sample, will it be a simple random sample? How will you contact respondents?

Mode of data collection

Which mode of data collection will you use—in-person, telephone, self-administered questionnaire, Internet survey or some other approach? Briefly justify your choice, stating its pros and cons relative to alternative choices.

Questionnaire

Design and provide a questionnaire to measure your substantive variables and the needed demographic variables.

You may use a few standard questions for existing questionnaires, but you must credit them. Otherwise, the words of your questions must be your own.

Table shells and made up results

Illustrate the way that you will use the results of the survey by showing the shells of tables for results. Then make up examples of numerical results that you might conceivably find. Use the principles of good table construction described in chapter 15.

Critique survey, including predicting biases

What problems of coverage bias are likely? Describe what drives the differences between those who are in the sampling frame and those who are not. Predict the direction and extent of coverage bias for each of your variables. (For example, is the variable biased upwards? Is one category much less likely due to bias?)

32

What problems of non-response bias are likely? Describe what drives the differences between those who are more likely to respond and those who are less likely. Predict the direction and extent of non-response bias for each of your variables.

More generally, critique your survey. Explain its weaknesses but explain why it is still useful.

Note 1: The entire assignment should be at most eight double-spaced pages, not counting the questionnaire itself. This will require substantial editing to write all of the required information succinctly.

Note 2: Make sure that you read through the assignment carefully to make sure that you have answered each part.

Note 3: While you may get help from others both inside and outside this class, you need your own individual topic

Hint: Do not start this assignment by thinking about “what could I ask?” Instead, start by thinking about “what do I want to know?” What you can ask may fall short of what you want to know. But you want to know how they differ and you might be able to get closer.

33

Rubric for Survey Design Assignment

Component A Level Work B Level Work C Level Work F Level Work

Purpose

Survey explains its purpose clearly and concisely. It explains not only its intent but also how its findings will be used for policy or practice.

Survey explains its purpose clearly. It explains its intent and mentions how its findings will be used for policy or practice.

Survey explains its purpose in a confusing or incoherent manner. It fails to explain either its intent or how its findings will be used for policy or practice.

Survey does not explain its purpose or how its findings will be used for policy or practice.

Population

The target population is clearly defined and the choice is persuasively justified given the survey purpose.

The study population is juxtaposed with the ideal target population and any differences are clearly and concisely articulated.

The target population is clearly defined and the choice is somewhat justified given the survey purpose.

Makes mention of some differences between the study population and the ideal target population.

The target population is not clearly defined or its choice is illogical given the survey purpose.

Makes little mention of the differences between the study population and the ideal target population.

The target population is not defined.

Makes no mention of the differences between the study population and the ideal target population.

Measures Describes clearly and concisely 4 to 8 substantive variables as well as the constructs they are

Describes 4 to 8 substantive variables as well as the constructs they are intended to

Describes fewer than 4 substantive variables or describes them in a confusing

Fails to describe the variables used or makes no mention of the constructs intended to be

34

intended to measure.

Identifies questionnaire questions associated with each measure.

measure.

Identifies questionnaire questions associated with each measure.

manner and makes little mention of the constructs they are intended to measure.

Does not clearly identify questionnaire questions associated with each measure.

measured.

Does not identify questionnaire questions associated with each measure.

Sampling Plan

Clearly, concisely, and systematically explains how the sample will be constructed and how respondents will be contacted – including call-back strategies.

Makes appropriate use of technical terminology.

Clearly and concisely explains how the sample will be constructed and how respondents will be contacted.

Makes appropriate use of technical terminology.

Explains in a confusing manner how the sample will be constructed and how respondents will be contacted.

Does not make appropriate use of technical terminology.

Does not explain how the sample will be constructed and/or how respondents will be contacted.

Mode Clearly and concisely explains the mode of data collection used

Persuasively

Clearly explains the mode of data collection used

Justifies the choice of mode by mentioning its

Explains in a confusing manner the mode of data collection used

Cursorily justifies

Does not explain the mode of data collection used

Does not justify the choice of mode

35

justifies the choice of mode by clearly articulating its pros and cons relative to alternative modes

pros and cons relative to alternative modes

the choice of mode used and/or does not mention its pros or cons relative to other modes

Questionnaire

Has clearly worded questions that are suitable for the study population. Meets high standards of question wording.

Questions are suitable for the study population.

Most questions meet standards for question wording.

Questions are confusing but suitable for the study population. Many questions are not clear or well worded.

Questions are inappropriate for the study population.

Most questions are not clear or well worded.

Shell Tables

Creates clear and correct table shells

Table shells clearly address survey purpose

Puts reasonable and clear numbers in tables

Creates table shells that are not fully clear or not fully suitable

Table shells somewhat address survey purpose

Puts mostly reasonable numbers in tables

Creates table shells that are not clear or not suitable

Table shells do not address survey purpose

Puts unreasonable numbers in tables

No table shells

and/or no numbers in tables

36

Critique

Clearly, concisely, and thoughtfully explores the likely coverage and non-response biases: describing the differences between the respondents and the study population and prediction of the direction and extent of the biases

Clearly, concisely, thoughtfully, and realistically assesses the weaknesses and value of the survey

Explores the likely coverage and non-response biases: describes the differences between the respondents and the study population

But does not correctly predict the direction and extent of the biases

Assesses the weaknesses and value of the survey

Explores the likely coverage and non-response biases but does not describe the differences between the unreached population and those who responded and fails to give a prediction of the direction and extent of the biases

Cursorily assesses the weaknesses and value of the survey

Does not explore likely biases

Does not asses the weaknesses and/or value of the survey

37

Writing quality

Writing is very clear

Arguments are cogent and persuasive

Essay’s organization is sensible and clear

Language is correct and concise (with no repetition)

Writing is fairly clear

Arguments are fairly cogent and persuasive

Essay’s organization is mostly sensible and clear

Language is mostly correct

Some unnecessary repetition

Writing is unclear

Arguments are often not cogent

Poor organization

Language has mistakes

A lot of unnecessary repetition

Writing is unclear

Arguments are not cogent

No organization

Much repetition

38

Appendix D: Examples of short problems for practice and exams10

1) You want to conduct survey to learn how safe New York City public school students feel, on average, while at school.

(a) Describe in 1-2 sentences the sampling frame(s) you could use for this study.

(b) If your survey is limited to one question, write a good (closed-ended) survey question for the purpose described.

(c) Explain in 1-2 sentences why it would be better to have more questions to assess how safe students feel at school.

(d) If you wanted to do a qualitative study to explore students’ feelings of safety in school, which form of qualitative study would you choose? Explain why in 1-2 sentences.

2) A charter high school surveyed its alumni, from a list it maintains for fund-raising purposes, to find out how well they were doing in the job market. Nearly 50 percent responded, and the average (mean) salary calculated from the survey was $79,000.

(a) Predict the direction and extent of non-response bias of the mean salary, explaining your prediction.

(b) Recall that the sampling frame was a list maintained for fundraising purposes. Predict the direction of coverage bias of the mean salary, explaining your prediction.

3) In several observational studies, children who eat nutritionally balanced meals were found to get higher grades in school, on average. Some suggest that providing nutritionally balanced lunches to students in school will improve their learning.

(a) In the suggestion, what is the independent variable? What is the dependent variable?

(b) Create a path model showing one mechanism consistent with the suggestion above.

(c) Create a path model showing an alternative theory that is both consistent with the correlation described and contradicts the causation implicit in the suggestion. Briefly explain your path model in words.

10 Some of these problems are in Remler and Van Ryzin (2011).

39

(d) Design a quasi-experiment, using difference-in-differences, to determine the effect of providing nutritionally sound lunches on student learning. Imagine that you have an assessment that accurately describes students’ knowledge and skills. Describe briefly how you would design the quasi-experiment and analyze the data.

(e) In 1-2 sentences, discuss the generalizability of the results of the study you designed.

5) You are principal of a school and want to learn how well parents feel the school communicates with them. You send home a survey with students and then have each teacher follow up with a phone call to parents about how important the survey is. What kinds of non-response or coverage problems will you have? What bias (including direction of bias) do you think would result in your measure of school communication? Explain.

6) Suppose two manufacturing companies merge. Prior to the merger, company A had only one health plan option: a restrictive closed panel HMO while company B had several health plan options, ranging from the very restrictive to plans without any restrictions at all. After the merger, all employees have only the restrictive option that A had for all of this time. Luckily, the merger did not cause any lay-offs.

You have access to all of the health care utilization data for all employees from both A and B for several years before and after the merger. You want to estimate the effect of health plan restrictiveness on health care costs (spending on health care for an individual from all sources).

(a) What is the independent variable of interest? What is the relevant outcome (dependent variable)?

(b) Explain how this event can be seen as a natural experiment. Discuss the quality (or lack of quality) of the causal evidence. Explain.

(c) Describe how you would use this natural experiment and the available data in a differences-in-differences study to determine the effect of restrictive health plans on health care utilization and health care costs. Specifically, describe:

What data you would use How you would analyze that data (including a shell table) How you would interpret the results

40

7) Your very large home meals service serves mostly elderly people but also a few younger disabled people. Your boss wants to learn about the satisfaction of all your clients but is particularly worried about the younger disabled clients and wants to make sure that you have precise information about them. What kind of sampling should your satisfaction survey do? Explain.

8) Imagine that the subway line cleanliness measures are being revised. One proposal is to have raters count the number of pieces of trash in the center-most car of the train that arrives at the end of the line on Wednesday evenings at 8pm. Another proposal is to have raters ride the line several different times a week at different locations and say how dirty they think it is generally. Which measure is more reliable? Which measure is more valid? Explain.

9) A large randomized field experiment of treatment for alcoholics was conducted with 10,000 subjects. They were randomized to either an intensive short-term inpatient treatment program or a less intensive long-term outpatient treatment program with equal total costs. The main outcome measure is long-term (5-year) sobriety. Results showed that 35% of those in the inpatient program remained sober for 5 years while 55% of those in the outpatient program remained sober for 5 years.

(a) The p-value associated with a test of the difference between the programs was less than .001. What is the statistical significance of these results?

(b) What is the practical significance of these results?

(c) The study was performed according to standard ethical principles of informed consent: subjects were fully informed about what the study would entail and voluntarily chose to participate. What kinds of alcoholics would agree to participate in the experiment? What kinds would not?

(d) Someone reading this impressive study concludes that long-term sobriety rates of all alcoholics would be 20 percentage points higher if all alcoholics received long-term outpatient treatment than if all alcoholics received short-term inpatient treatment. Do you think their conclusion is valid? Explain.

(e) The study as described does not shed light on why the outpatient program had better results. Describe how the use of one qualitative method (e.g., open-ended interviews,

41

structured interviews, focus groups, observation) during the experiment could have shed light on this issue. Briefly describe which method you think would be best and why.

10) Excerpts from an Op-ed contribution from the New York Times Sunday City section, by E.S. Savas, Baruch School of Public Affairs Professor:

“TRANSIT officials are decentralizing the subway system to improve service, cleanliness and on-time performance by appointing individual managers for each of the 24 lines. …

The authority plans to start with the No. 7 and L lines and evaluate the pilot program by surveying riders after about three months. This implies only modest goals, as that time is too short for major improvements.

Moreover, more money and manpower are to be allocated to those lines, making it impossible to figure out whether any improvements result from better management or more spending. The plan seems loaded to elicit favorable comments in the short term from riders on those particular lines, which unlike the other 22 lines are isolated: they have separate tracks. …”

Evaluators of the present plan could use a differences-in-differences framework to evaluate the impact of their decentralization program. Savas notes three problems that undermine the ability of such an evaluation to allow generalizations to the long-term effects of creating individual managers for each of the 24 lines.

(a) Briefly describe the differences-in-differences framework that could be used to do an evaluation of the decentralization program.

(b) Explain the three problems Savas describes. Explain how they would undermine the desired generalization from your differences-in-differences study. (Make sure to briefly define generalization.)

42