snippet | nus social service research centre | issue 1 2020

SNIPPETCutting-edge Applied Research

Editor's Note

F E B R U A R Y 2 0 2 0 I S S U E 1

EDITOR'S NOTE

I N T H I S I S S U E

NUS SOCIAL SERVICE RESEARCH CENTRE

1© NUS SOCIAL SERVICE RESEARCH CENTRE ALL RIGHTS RESERVED

UPCOMING SSR EVENTS

WHY DO WE NEED REALISTEVALUATION?

WHAT ARE SYSTEMATICREVIEWS AND HOW ARE THEY

USEFUL?

Hi everyone,

Hope you had a great Chinese New Year break!

The Year of the Rat at our Snippet starts with two important articles

pertaining to research methodology not very much known in the local social

service community. Robyn Tan’s article on Realist Research is an emerging

field and is of particular relevance for practitioners and policy makers

looking for another perspective to evaluating and replicating service

programmmes.

The article by Timothy Teoh is a scholarly and concise description of what a

systematic review involves, how it is carried out and under what situations it

could be most useful. Both articles are a ‘must read’ for practitioners,

policymakers and graduate students looking for new directions in conducting

in-depth evidence-informed studies.

Happy reading!

by Dr Rosaleen Ow

P H O T O B Y M A R T I N R . S M I T H

The evaluation of a programme, a policy or an

intervention generally involves assessing one or more of

the following five domains: (1) the need for the

programme, (2) the programme design and its underlying

theory, (3) the implementation and service delivery, (4) the

impact, or outcomes, and (5) the cost and efficiency

(Rossi, Lipsey and Freeman, 2004).

However, in the local social service sector, programme

evaluation appears disproportionately focused on

assessing impact, probably driven by funding

requirements to address the central question, “Does the

programme work?” To attempt an answer, programme

evaluation approaches have held on to the experimental

(or quasi-experimental) research design as the mainstay

for effectiveness studies.

The experimental evaluation design is indeed the “gold

standard” except that it is applicable under rather limited

circumstances, such as, where the contexts in which the

programme is applied are the same or similar; the content

of the programme is clearly specified and standardised;

the process by which it is applied is fairly consistent; and

the outcomes are clearly defined and measureable (Walshe

2007). However, the experimental evaluation design does

not lend itself well when applied to complex and dynamic

systems.

F E B R U A R Y 2 0 2 0 I S S U E 1

Why do We NeedRealist Evaluation?by Robyn Tan

Research Fellow, Social Service Research Centre,

NUS


Social programmes are delivered in a world of existing

interventions and ever-changing contexts. The content

varies widely, for example, the programme may be

customised to meet the needs of an individual or a

community, or redesigned or changed over the course of

its duration. The process involves long implementation

chains with multiple stakeholders and interacting

programme components.

As these programmes require human volition to work,

the outcomes may vary and emergent effects are to be

expected. Attempts to “standardise” or “control” such

programmes to make them fit the experimental paradigm

would have completely missed the point. Furthermore,

there could be issues of insufficient sample size, the lack

of availability (and/or access) to a control or comparison

group, absence of a stable and singular intervention to

begin with and funder’s expectation to evaluate outcomes

even before the programme stabilises. Under such

circumstances, applying an experimental evaluation

design is akin to fitting a square peg into a round hole.

A different approach to evaluating effectiveness is much needed. Realist research is a form of theory-driven

evaluation that focuses on the theoretical basis of a policy, a programme or an intervention (why and how it works)

rather than only on its effectiveness (whether it works). The approach holds much promise in addressing a different

set of questions in evaluation – “what works, how it works, in what contexts does it work and for whom it works?”

Realist research, drawn from Ray Pawon and Nick Tilley’s seminal work, Realistic Evaluation (1997), is a collective

name for research underpinned by principles of realist philosophy. In a nutshell, realist evaluation collects and

analyses mainly primary data while realist review or synthesis, as its name suggests, is a synthesis of existing

evidence and uses mainly secondary data in its analysis (Pawson and Tilley 1997). In the past decade, realist research

has gained much traction and is increasingly used in health and social services in the UK and Europe, especially in

the evaluation of complex interventions, involving multiple stakeholders and interacting components.

Realist research is a form of theory-drivenevaluation that focuses on the theoretical basis

of a policy, a programme or an intervention(why and how it works) rather than only on itseffectiveness (whether it works). The approach

holds much promise in addressing a different setof questions in evaluation – 'what works, how it

works, in what contexts does it work and forwhom it works?'


For a start, we unpack realist research by first understanding its key tenets. First, programmes, policies and

interventions bring resources, ideas, opportunities that are intended to cause a change. In other words, a programme

“works” because it intends for participants to do something different, or behave differently, in order to effect change

in them or in their environment.

Second, programmes, policies and interventions require human volition to make them work. Participants are not

passive recipients but active agents of change. They interact with the resources, ideas and opportunities offered by the

programme. In their interaction, participants’ reasoning, their choices and decisions, in response to the programme,

are known as “mechanisms”. Participants’ response to the programme and the choices and decisions they make

depend on or are influenced by the “context” – these may include the individual’s capacities, interpersonal relations,

institutional and infrastructural system. Some contexts may enable particular mechanisms to operate or prevent them

from operating. The context influences how participants respond and make choices, resulting in “outcomes”.

Outcomes can be proximal, intermediate, or long term and include the successes or failures and the intended or

unintended consequences of the programme.

Third, programmes, policies and interventions are

“theories incarnate” (Pawson and Tilley 1997), that is,

they come with an underlying programme theory (or

theories) or simply put, a set of assumptions of what

the programme is expected to do (to address certain

problems) and how it is expected to work (to bring about

change). However, these theories often exist in the

minds of policymakers, programme designers or

practitioners.

Undertaking a realist research requires that these

theories be made explicit, by developing programme

theories about how, and for whom, to what extent, and

in what contexts a programme might work to bring

about change. The evaluation then tests and refines

these programme theories.

In realist terms, the programme theory is captured in

the form of context-mechanism-outcome (CMO) – an

analytical heuristic to explain how the programme

works (O) because of the action of some underlying

mechanism (M), which is activated under particular

contexts (C). Articulating, testing and refining CMO

configurations allows for an explanatory account of for

whom, in what circumstances and why the programme

works (Wong et al. 2016).

To better elucidate what contexts, mechanisms and

outcomes mean, we use the Family Independence

Initiative (FII) founded by Mauricio L. Miller as a case

example. In his book The Alternative – Most of what

you believe about poverty is wrong, Miller challenges our

assumptions of the poor as people who can’t manage

money, don’t take initiative and don’t help one another.

Undertaking a realistresearch requires that

theories underlyingprogrammes be madeexplicit, by developing

programme theoriesabout how, and for

whom, to what extent,and in what contexts a

programme might workto bring about change.

The evaluation then testsand refines these

programme theories.

Founded in 2001, the FII was set up as an alternative

approach “… to recognise the initiative of families

themselves, and to develop a way that makes resources,

connections, and money, available to them based on the

actions they take of their own volition”, with the aim of

helping participating families strengthen social

networks and support one another in achieving

mobility (Miller 2017, p. 130).

I read with great interest the radical approach

undertaken by the FII to trust and to invest directly in

low-income families instead of investing in run-of-the-

mills programmes or hiring professionals to run these

programmes for the poor. This piqued my curiosity,

followed by a host of questions – “What is it about FII

that works, how it works, in what contexts does it work

and for whom does it work?”


Equally important to the equation is the context. Mechanisms are activated only in particular contexts. Here, the

context matters and could influence the way in which families interpret a comparison via the journaling system.

For example, the group composition of participating families is a key consideration. Comparison with other people

most frequently occur between similar people or among people in similar life circumstances. However, the

participating families cannot be too homogenous as group homogeneity would mean fewer opportunities for

positive upward or downward comparison.

Focusing on the “journaling system” – one of the initiatives under FII, I adopt the realist lens to understand what

about the journaling system might work, for whom and under what conditions. The journaling system is a

technological platform that presents cloud-based, real-time feedback on the progress of participating families

(intervention). However, it isn’t about implementing the journaling system that works, but the way participating

families reason and respond to the ideas, resources and opportunities provided by the journaling system, and the

choices they make (mechanism) under a particular context that would bring about the change in them or in their

circumstances (outcome), whether intended or unintended.

Mechanisms should not be confused with programme activities. They are usually hidden and not directly

observable. One possible mechanism underlying this journaling system is “social comparison”. According to social

comparison theory originally proposed by Leon Festinger, people have a tendency to seek objective information to

evaluate their attributes, abilities and situations (Festinger 1954).

Comparison with others can influence people regarding how they feel or think about themselves and their

situations, and thus affecting the response and choices they make. Two main forms of social comparison –

“upward comparison” and “downward comparison” (Buunk et al. 1990), can be applied to our understanding of how

this journaling system might work. To elaborate, seeing other families do better (via the journaling system), may

invoke upward comparison where comparing with another’s situation that is perceived as better off and in turn

instil a sense of hope for improvement (enabling mechanism).

On the other hand, seeing other families do better may pose a threat to some who are not doing as well as they may

perceive themselves to be lesser in comparison (disenabling mechanism). Seeing other families do worse, may

invoke downward social comparison where they feel relieved about their own situation and in turn enhance

positive self-esteem and confidence (enabling mechanism). On the other hand, seeing other families that are worse

off could make them worry or fearful about their situation (disenabling mechanism).


However, it isn’t about implementing thejournaling system that works, but the way

participating families reason and respond to theideas, resources and opportunities provided by

the journaling system, and the choices they make(mechanism) under a particular context that

would bring about the change in them or in theircircumstances (outcome), whether intended or

unintended.

As such, there needs to be an injection of

heterogeneity, for example, in the form of peer role

models, akin to your everyday heroes, to showcase

what is reasonably possible to achieve for families in

similar circumstances. However, these are not your

exceptional, out-of-the-ordinary higher achievers,

whom families cannot relate to. In addition, the role

of a skillful facilitator is important to promote

positive interpretations among the families.

By adopting a realist evaluation, we can seek to test

and refine the initial programme theory – the

journaling system (intervention) might work to

strengthen social networks and support one another

(outcomes) because of social comparison among

participating families (mechanism), activated when

families in similar situations are brought together to

learn from one another in sessions facilitated by a

skillful practitioner (context).

In this example, social comparison is one possible

mechanism underlying the journaling system.

However, there are other mechanisms of change that

could motivate change in these families which the

study could elicit. In reality, there could be an infinite

number of contexts, mechanisms and outcomes out

there, which could possibly translate into an infinite

number of programme theories. A realist evaluation

may not be able to cover every conceivable

programme theory, hence, a decision has to be taken

to limit the scope of the study to examining the key

ones.

Lastly, there is no silver bullet that would solve all

social problems. Programmes, policies and

interventions, though well-intended, are not

universally successful and may work better in some

contexts than in others. The same programme could

invite different reasoning and response from different

participants. Since programmes work differently for

different people through different mechanisms of

change, in different contexts, programmes cannot

simply be replicated from one context to another and

be expected to achieve the same outcomes.

In other words, success cannot be repeated by a simple

matter of replicating programmes from one setting to

another. In reality, policymakers, programme

designers and practitioners must accept that

programmes work in different ways – they will be

adapted as they pass through the long implementation

chain and move into new contexts. The key is to

monitor these adaptations, and attempt to understand

why they occurred and how they may influence the

outcomes. Realist research can potentially offer a

theory-based understanding of “what works, for

whom, in what contexts, and why” that is transferable

to different settings, to allow for policies, programmes

and interventions a reasonable chance of success.

To learn more about how the realist approach can be

applied to evaluate social services, please look out for

the next run of SSR’s training workshop, titled,

“Introduction to Realist Evaluation for Social

Services”.


In reality, policymakers, programmedesigners and practitioners must accept thatprogrammes work in different ways – they

will be adapted as they pass through thelong implementation chain and move intonew contexts. The key is to monitor these

adaptations, and attempt to understand whythey occurred and how they may influence

the outcomes.

by Timothy Teoh

Research Executive, Social Service Research Centre,

NUS

One can also think ofsystematic reviews as a

review of literature whichis conducted in a

systematic manner andscoped to answer a

specific question.

What are Systematic Reviews and How are TheyUseful?

The systematic review process begins with the writing

of a proposal (known as a protocol), followed by

searching for studies, selecting studies, extracting data

from the studies, assessing the quality of the studies,

combining the findings, discussing and concluding

(see Figure 1). In many ways, traditional literature

reviews also require similar processes of searching,

selecting, synthesising and discussing. However, a

defining feature of systematic reviews is that these

processes are held to certain agreed upon standards

(Boaz, Ashby & Young, 2003).


Such standards, like PRISMA (Preferred Reporting

Items for Systematic Reviews and Meta-Analyses)

(Moher et al., 2009) and ENTREQ (Enhancing

Transparency in Reporting in the Synthesis of

Qualitative Research) (Tong et al., 2012), require

authors to spell out and clearly document each step in

the whole process. This clarity and transparency

allows for replication and serves to reduce the biases

often associated with ‘traditional’ literature reviews

(Littell, Corcoran & Pillai, 2008).

Introduction

As scholarship and technology continue to advance,

research abounds and is accessible like never before.

We can read of groundbreaking studies and other

relevant research across the globe. This works

wonderfully to our advantage, especially with shifts

towards evidence-based practice and policy. We want

to know and do ‘what works’ while being cost-efficient

and effective, and accountable to donors and funders.

But with so much research available which at times is

conflicting and contradicting, it begs the question: how

do we make sense of it all? Is there a way to

summarise all the relevant research on a topic and

reach a conclusion on the effectiveness of an

intervention or policy? One answer is through the use

of a methodology known as a systematic review. This

article will provide an overview of systematic reviews;

what they are and how they are useful.

What makes a systematic review“systematic”?

In simplest terms, systematic reviews refer to “a

method of making sense of large bodies of

information, and as a means of contributing to the

answers about what works and what does not”

(Petticrew & Roberts, 2006, p.2). One can also think of

systematic reviews as a review of literature which is

conducted in a systematic manner and scoped to

answer a specific question. Systematic refers to the

prescribed set of processes that employ certain

techniques to find, select and assess studies, and to

extract and synthesise the findings from them (Dixon-

Woods & Sutton, 2004).

Further differences between systematic reviews and traditional literature reviews can be seen within each of the

processes. These will be explained below in the framework of standards of a systematic review proposed by Boaz

and colleagues (2002, p.4-6).


Figure 1. Concept and Process of a Systematic Review (adapted from https://cccrg.cochrane.org/infographics)

Standard 1: Using protocols to guide the process

Similar to a research proposal, the protocol spells out the background of the review, the review question, search

strategies, inclusion and exclusion criteria for studies and plans for analysis. Systematic review organisations

like the Cochrane Collaboration, Campbell Collaboration and Joanna Briggs Institute (JBI) encourage or require

titles and protocols of systematic reviews to be registered with PROSPERO, the international prospective register

of systematic reviews, and even published, prior to the commencement of the review (e.g. Aromataris & Munn,

2017, Cumpston & Chandler, 2019). The registering of titles minimizes the duplication of reviews. The publishing

of protocols plays a key role in reducing bias as it encourages transparency and allows for the comparison

between what was done and what was planned (Centre for Reviews and Dissemination, n.d). Additionally,

differences between the protocol and review need to be explicated in the finished review (Cumpston & Chandler,

2019). This differs from a traditional literature review in which, while a proposal may be written, the degree of

transparency and accountability is minimal.

Standard 2: Focusing on answering a specific

question

Systematic reviews are typically designed to answer a

specific question, for example, how effective is an

intervention for a selected population under certain

circumstances in producing a particular outcome.

Various acronyms exist to help authors be explicit and

specific with the question of the review. These range

from “PICO”, which stands for Population,

Intervention, Comparison, Outcome; to CoCoPop:

Condition, Context, Population (Munn et al., 2018). The

acronym to use depends on the type of the review

being conducted.

The review question serves to establish the inclusion

and exclusion criteria which guide authors in deciding

which studies to include in the review. In contrast,

traditional literature reviews tend to be more general in

scope (Petticrew, 2001). For example, a broad topic like

school-based mindfulness interventions may be suited

for a traditional literature review. However, in the case

of a systematic review, the question would be more

focused, e.g. “What are the effects of mindfulness‐based interventions on cognition, academic

achievement, behavior, socio-emotional and

physiological outcomes?” (Maynard et al., 2017. p.23).

Standard 3: Seeking to identify as much of the

relevant research as possible

To identify as many relevant studies as possible,

systematic reviews undertake comprehensive searches

with carefully selected search terms, entered in

numerous databases. Preliminary searches help to

identify commonly used key terms. From there, with

consideration to the review question, authors craft

search strings—a combination of keywords and

Boolean operators (e.g. AND, OR and NOT)—to reach

an optimised search output. Aspects of the exclusion

criteria may also be applied here, e.g. if the review is

only looking for recent studies or studies conducted in

English, a date or language filter may be added.

Table 1 displays an example of the process from review

question to search terms. Beyond searching in

databases, systematic reviews also aim to include grey

(unpublished) literature, government reports, books and

the like (Boaz et al., 2002). This overall search strategy

is a necessary part of the protocol. Unlike traditional

literature reviews, this process is made explicit thereby

reducing the potential that relevant databases or studies

may be intentionally excluded.

9

© NUS SOCIAL SERVICE RESEARCH CENTRE ALL RIGHTS RESERVED

Table 1. Example of Research Question to Search String (adapted from Southall, 2018)

Standard 4: Appraising the quality of the research included in the review

Appraising the available evidence is arguably the centerpiece of a systematic review. Beyond synthesising and

summarising the evidence, systematic reviews must report on the quality of the included studies. This enables

readers to have confidence in the findings of a review. Various tools and checklists exist to guide and support

this process. These range from the Critical Appraisal Skills Programme (CASP) Checklists, to the JBI Critical

Appraisal Checklists and the Cochrane Risk of Bias Tool (CASP, 2018; Higgins et al., 2019: JBI, n.d.). The tools

require authors to take a detailed look at the methodology of each study. In doing so, a sense of the quality of the

study and its findings can be assessed, which leads to the extent to which they can be trusted. In traditional

literature reviews, such a process, if it occurs, is usually undocumented. This makes it hard to assess the quality

of the review.

Standard 5: Synthesising the research findings in the studies included

The appraisal of evidence also determines the weight each included study has. This has bearing on the

synthesised findings and the conclusions that are drawn. Various methods of synthesis exist, and the use of a

method depends on the type of systematic review being conducted and the studies available. Most systematic

reviews provide a narrative synthesis and summary table of included studies (Petticrew & Roberts, 2006). Where

available and appropriate, quantitative data from included studies can be combined through statistical methods

known as meta-analysis (Littell et al., 2008). Systematic reviews of qualitative evidence may draw on meta-

ethnographic or thematic analyses to synthesise the findings (Aromataris & Munn, 2017). Tools like GRADE

(Grading of Recommendations Assessment, Development, and Evaluation), and GRADE-CERQUAL (Confidence

in the Evidence from Reviews of Qualitative research) may be used to rate the quality of the synthesised findings

(Guyatt et al, 2011; Lewin et al., 2018). This differs from traditional literature reviews especially in the

consideration of the quality of the evidence in the process of synthesis, and the assessment of the strength of

synthesised findings.


Standard 6 & 7

Standard 6 and 7 are further considerations that occur within and beyond the systematic review process.

Standard 6 is “aiming to be as objective as possible about the research to remove bias”. This is ideally

incorporated at every stage of the review and further reinforced through the stipulated guidelines and as shown

in the preceding standards. Standard 7 is “updating in order to remain relevant” which is in line with the

continual production of research and the importance of keeping abreast of developments in the field. While not

all systematic reviews require updating (e.g. if commissioned for a specific purpose or policy need), reviews

should be updated where appropriate (Boaz et al., 2002).

As seen, a well-conducted systematic review would provide an overview of the relevant literature that is available.

Additionally, most systematic reviews also provide discussion on the effectiveness of an intervention or policy

(Petticrew & Roberts, 2006).

For such reasons, organisations may find systematic reviews particularly useful in the development of a new

programme. Finding and reading a systematic review on the programme could provide helpful pointers to guide

the development process and it would be more comprehensive than looking at a few studies on the topic. Some

organisations like the Campbell Collaboration, provide open access to their reviews, thus providing practitioners

with limited access to journals and databases, access to a summary of studies they would not be able to read

otherwise.

Policymakers may also find systematic reviews helpful in supporting decisions in policymaking as they provide a

base of evidence to refer to (Petticrew, 2001). Additionally, beyond answering questions on effectiveness,

systematic reviews can also be done on a variety of other questions like experiences, etiology and risk, etc.

(Aromataris & Munn, 2017), which may be of interest to policymakers as well.

And at the very least, systematic reviews provide a summary of the “robust studies in a particular field of work

which no policymaker or practitioner, however diligent, could possibly hope to read themselves.” (Petticrew &

Roberts, 2006, p.15).

Although systematic reviews can be useful in many ways, they may not always be appropriate to use. As with of

any methodology, ensuring that the use of a systematic review is appropriate for the task at hand is of utmost

importance. A key limitation of a systematic review is that it may not always be the “right tool for the job” as

required by stakeholders or funders (Petticrew & Roberts, 2006, p.28). As systematic reviews are designed to

answer specific questions, it would be good to determine the following before searching for or reading systematic

reviews.

11

© NUS SOCIAL SERVICE RESEARCH CENTRE ALL RIGHTS RESERVED

How are systematic reviews useful?

When are systematic reviews not useful?

What type of questions are you asking?

If your interest in a topic is broad and the questions you are asking are more general, for example, continuing on

mindfulness-based interventions, what are mindfulness-based interventions, what are they used for and how do

they work, etc.; reading a traditional literature review would be more appropriate. While it is commonplace for

systematic reviews to provide some background, it would not give a broad overview if that is what you are looking

for.

Does the review’s questions answer your questions?

However, if your questions are more specific (e.g. how effective are mindfulness-based interventions; do

mindfulness-based interventions help anxiety or depression, etc.), reading a systematic review could be more

helpful, provided one exists on your topic of interest and that the questions it answers addresses the questions you

are asking. This is important to ensure as systematic reviews are often specific to population and context. Reading

a systematic review on the effectiveness of school-based mindfulness interventions would not answer questions

about the effectiveness of mindfulness in adult populations, for example. Nor would reading a systematic review

about the effect of mindfulness-based interventions on anxiety answer questions about their effect on depression.


What should I consider when reading a systematic review?

Once you’ve determined that a systematic review can answer the more specific questions you are asking, here are

some considerations to keep in mind when reading a systematic review. Although the systematic review process,

with its procedures, standards and guidelines, aims to increase the quality of reviews and minimize bias, poorly

conducted and reported systematic reviews can still exist. Hence, it is still important for readers to appraise the

quality of reviews for themselves when reading them.

To this end, readers can ask the following questions to assess the overall quality of a systematic review (Oxman &

Guyatt, 1991; Oxman 1994 in Petticrew & Roberts, 2006, p.296):

1. Are the review’s inclusion and exclusion criteria described and appropriate?

2. Is the literature search likely to have uncovered all relevant studies?

3. Did the reviewers assess the quality of the included studies?

4. Did the reviewers take study quality into account in summarizing their results?

5. If there was a statistical summary (meta-analyses), was it appropriate (that is were the studies similar enough

to be statistically synthesized)?

6. Was study heterogeneity (differences between studies) addressed?

7. Were the reviewers’ conclusions supported by the results of the studies reviewed?

Another limitation of systematic reviews which should be kept in mind when reading a systematic review is the

“garbage in, garbage out” principle (Littell et al., 2008, p.16). While this typically applies to meta-analyses for which

the inclusion of biased or low-quality studies would skew the results, the principle stands for systematic reviews in

general as well. Where quality of studies included are low, useable findings or implications for practice or policy

are likely limited, with the main implication being for research and the need for higher quality work.

Conclusion

In conclusion, this article has discussed what systematic reviews are; what makes them systematic and distinct

from traditional literature reviews. This is not to say that traditional literature reviews are useless or unreliable.

Rather, systematic reviews can be useful in providing a summary of relevant research in ways which answer

specific questions and inform policy and practice. As seen, systematic reviews are not without limitations, nor are

they always appropriate for use. However, they do provide a means of making decisions in policy or practice more

evidence-based, or at the very least, more evidence-informed.

References

1. Robyn Tan, "Why do we need realist evaluation?"

Buunk, B. P., Collins, R. L., Taylor, S. E., Van Yperen, N. W. and Dakof, G. (1990) ‘The affective consequences of social

comparison: either direction has its ups and downs’, Journal of Personality and Social Psychology, 59(6): 1238 – 49.

Festinger, L. (1954) ‘A theory of social comparison processes’, Human Relations, 7(2): 117 – 40.

Miller, M. L. (2017) The Alternative: Most of what you believe about poverty is wrong, Lulu Publishing Services.

Pawson, R. and Tilley, N. (1997) Realistic evaluation, SAGE Publications Inc.

Rossi, P. H., Lipsey, M. W., and Freeman, H. E. (2004) Evaluation: a systematic approach, 7th Ed. California: Sage

Publications.

Walshe, K. (2007). Understanding what works – and why – in quality improvement: the need for theory-driven

evaluation, International Journal for Quality in Health Care, 19(2):57-59.

Wong, G., Westhorp, G., Manzano, A., Greenhalgh, J., Jagosh, J. and Greenhalgh, T. (2016), RAMESES II reporting

standards for realist evaluations, BMC Medicine, 14(96): 1-18.



2. Timothy Teoh, "What are Systematic Reviews and how are they useful?"

Aromataris, E. & Munn, Z. (Eds.) (2017). Joanna Briggs Institute Reviewers’ Manual. The Joanna Briggs Institute.

Available from: https://reviewersmanual.joannabriggs.org/

Boaz, A., Ashby, A. & Young, K. (2002). Systematic reviews: what have they got to offer evidence based policy and

practice? ESRC UK Centre of Evidence Based Policy and Practice. Centre for Reviews and Dissemination. (n.d.). About

PROSPERO. Available from: https://www.crd.york.ac.uk/prospero/#aboutpage

Cumpston, M. & Chandler, J. (2019). Chapter 2: Planning a Cochrane Review. In J. P. T. Higgins, J. Thomas, J. Chandler,

M. Cumpston, T. Li, M. J. Page, V. A. Welch (Eds), Cochrane Handbook for Systematic Review of Interventions, Version

6.0. Cochrane. Available from: www.training.cochrane.org/handbook

Critical Appraisal Skills Program. (2018). CASP Appraisal Checklists. Available from: https://casp-uk.net/casp-tools-

checklists

Dixon-Woods, M. & Sutton, A. (2004). Systematic review. In M. S. Lewis-Beck, A., Bryman & T. F. Liao (Eds.), The SAGE

Encyclopedia of Social Science Research Methods. Thousand Oaks: SAGE Publications.

Guyatt, G., Oxman A. D., Akl, E. A. et al. (2011). GRADE guidelines: 1. Introduction- GRADE evidence profiles and

summary of findings tables. Journal of Clinical Epidemiology, 64(4), 383-94.

Higgins, J. P. T, Savović, J., Page, M. J., Elbers, R. G. & Sterne, J. A. C. (2019). Chapter 8: Assessing risk of bias in a

randomized trial. In J. P. T. Higgins, J. Thomas, J. Cumpston, T. Li, M. J. Page, V. A. Welch (Eds), Cochrane Handbook for

Systematic Review of Interventions, Version 6.0. Cochrane. Available from: www.training.cochrane.org/handbook

Joanna Briggs Institute. (n.d). Critical Appraisal Tools. Available from:

https://joannabriggs.org/ebp/critical_appraisal_tools

Lewin, S., Booth, A., Glenton, C. et al. (2018). Applying GRADE-CERQual to qualitative evidence synthesis findings:

introduction to the series. Implementation Science 3(2).

Maynard, B. R., Solis, M. R., Miller, V. L & Brendel, K. E. (2017). Mindfulness-based interventions for improving

cognition, academic achievement, behavior, and socioemotional functioning of primary and secondary school students.

Campbell Systematic Reviews (5).

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G. & The PRISMA Group. (2009). Preferred Reporting Items for Systematic

Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7).

Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct?

A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Medical Research

Methodology, 18(1), 5.

Oxman, A. (1994). Checklists for review articles. British Medical Journal, 309, 648–51.

Oxman, A. D. & Guyatt, G. H. (1991). Validation of an index of the quality of review articles. Journal of Clinical

Epidemiology, 44, 1271–8.

Petticrew, M. (2001). Systematic reviews from astronomy to zoology: myths and misconceptions. British Medical Journal,

322. 98-101.

Petticrew, M. & Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing.

Tong, A., Flemming, K., McInnes, E., Oliver, S. & Craig, J. (2012). Enhancing transparency in reporting the synthesis of

qualitative research: ENTREQ. BMC Medical Research Methodology, 12(181).

Southall, J. (2018). Bodleian Library Workshops - Creating and Using an Advanced Search String. Available from:

https://libguides.bodleian.ox.ac.uk/workshops/handouts

Article Contributors Dr Robyn Tan, Research Fellow, NUS Social Service

Research Centre

Timothy Teoh, Research Executive, NUS Social

Service Research Centre

The Snippet TeamEditorial Team Review Editor: Dr Rosaleen Ow

Executive Editor: Dr Neo Yu Wei

Editorial Assistant: Nurul Fadiah Johari

Contact Us NUS Social Service Research Centre

Faculty of Arts and Social Science

National University of Singapore

The Shaw Foundation Building, Block AS7,

Level 3, 5 Arts Link,

Singapore 117570

Email: [email protected]

Phone: 6601-5019

Website: http://www.fas.nus.edu.sg/ssr/index.html

Upcoming SSR Events


SSR Training Programmes 2020

Click below to register:

Participatory Research for Social Services

(Tuesday, 3 Mar 2020)

https://mysurvey.nus.edu.sg/EFM/se/543BE5C23273AEA1

https://mysurvey.nus.edu.sg/EFM/se/543BE5C22AC4D39C

snippet | nus social service research centre | issue 1 2020

Documents