PROTO

COL DEV

ELOPM

ENT AND

SEARCHING FO

R QUALITATI

VE

EVIDENCE

S Y S T E MA T I C R E V I E W

WO R K S H O P

K U L E U V E N 4 - 6 J U N E 2 0 1 2

Janet Harris - Updated material from Angela Harden, Three-day systematic review workshop, K U Leuven, 6th to 8th May 2011

WHY DEVELOP A PROTOCOL?Helps to plan the review and anticipate

problemsEncourages comment and reviewAids transparency and auditCan build in flexibilityRequired by review organisations like

Cochrane and Campbell

ELEMENTS OF A REVIEW PROTOCOLTitleBackgroundReview question/objectivesInclusion and exclusion criteriaSearch strategyScreeningQuality assessmentCoding and data extractionSynthesis methodsConflicts of interest

EXAMPLE OF A REVIEW PROTOCOL THAT INCLUDES QUALITATIVE RESEARCH

http://onlinelibrary.wiley.com/o/cochrane/clsysrev/articles/CD009105/frame.html

Leiknes KA, Berg RC, Smedslund G, Jarosch-von Schweder L, Øverland S, Hammerstrøm KT, Høie B. Electroconvulsive therapy for depression (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5. Art. No.: CD009105. DOI: 10.1002/14651858.CD009105.

PICO• Population• Intervention• Comparison• Outcome

SPICE• Setting (where? in what

context?• Perspective (who?)• Intervention (or

phenomenon of interest)

• Comparison (what else?)• Evaluation (how well –

what result?

REVIEW QUESTIONS

SEARCHING SYSTEMATICALLYCommon principles and challengesApproaches to searchingSearch sourcesBibliographic databasesFilters for electronic searchingSummary of key points

BUT first……..

SEARCHING IS A SKILLED AND TIME CONSUMING ACTIVITY

Tap into the skills of information, subject and systematic review specialists, to:

Identify sources to searchDeal with technicalities (e.g. of databases)

Plan time for developing and testing searches

COMMON PRINCIPLES Searching is the act of seeking studies that

might be relevant for the review’s question The search strategy is guided by the review

question(s) Practically constrained (by database

limitations, restricted time and resources available to reviewers)

As in other stages, reviewers: Have a rationale for their searching methods: Draft, test and implement a structured search plan

(‘search strategy’); Report fully on their methods and findings.

COMMON CHALLENGES • Profusion of published and unpublished

material• Much hidden: only 50% abstracts presented at

conferences are later published in full• Different databases use different terms to

classify studies• Different databases cover different, largely

discrete areas of literature• As individuals we are constrained by our own

disciplinary and policy area knowledge.

APPROACHES TO SEARCHING Comprehensive searching

Considers ideal as having access to all studies that answer the review question. An unbiased sample is next best.

Purposive searching E.g. to identify main themes in the literature Searching plans may develop as evidence

comes to light.

SOURCES OF QUALITATIVE RESEARCH (1)Qualitative research may be published as: Journal articlesBooks and book chaptersConference papersProject reportsDissertationsAny more formats?

SOURCES OF QUALITATIVE RESEARCH (2)Qualitative research may be found through: Bibliographic databasesSpecialist research registersLibrary cataloguesConference proceedingsPersonal contactsHandsearching journalsWebsitesSearch engines (Google scholar)Reference lists of relevant studies

BIBLIOGRAPHIC DATABASESSubject specificE.g. Medline, Embase, CINAHL, ERIC

Multiple disciplinesE.g. Social Science Citation Index, ASSIA

Format specificDissertation abstracts, Index to theses

SEARCHING BIBLIOGRAPHIC DATABASESTwo main approaches:Topic only searchTopic plus qualitative filter

Two types of search termsControlled vocabularyFree text

A NOTE ON TERMINOLOGY• No agreed definition of qualitative

research• Houses a broad range of methodologies

and methods• Indexing on databases is inconsistent• Titles and abstracts may not employ the

word ‘qualitative’ but may use more specific terms

E.g. focus groups, interviews, grounded theory, discourse analysis, phenomenology, Glaser and Strauss, Nvivo, Nudist, views, perspectives, experiences, stories, narrative, themes, findings

DEFINITIONS RANGE FROM RELATIVELY SIMPLE…

“Research that derives data from observation, interviews or verbal interactions and focuses on the meanings and interpretations of

participants”Holloway and Wheeler (1995)

Holloway W, Wheeler S (1995) Ethical issues in qualitative nursing research, Nursing Ethics, 2:223-232

….TO ELABORATE AND COMPLEX…Qualitative research is an umbrella term for an array of attitudes

toward and strategies for conducting inquiry that are aimed at discerning how human beings understand, experience, interpret

and produce the social world…it encompasses richly detailed descriptions and in-depth, particularised interpretations of

persons and the social, linguistic, material and other practices and events that shape their lives and are shaped by them.

Qualitative research typically includes, but is not limited to, discerning the perspectives of….the actors…point of view.

Although both philosophically and methodologically a highly diverse entity, qualitative research is marked by certain defining imperatives that include a case orientation…to

analysis, sensitivity to cultural and historical context, and reflexive accounting practices to optimise validity…..

Sandelowski M (2003) Qualitative research. In M Lewis-Beck, AE Bryman & TF Liao (Eds) The Sage Encyclopedia of Social Science Research Methods. Thousand Oaks, Ca: Sage.

OPTIMAL SEARCH FILTERS• Aim to balance sensitivity and

specificity

• Some databases may have built in filters

• Information specialists have developed and tested filters for others

• Lots of room for further development and evaluation - build this into your reviews

OPTIMAL QUALITATIVE FILTER FOR CINAHLVarious combinations of the following

tested: interviews.sh; interview.tw; attitude.sh; qualitative

studies.sh; qualitative stu$.mp; thematic analysis.sh; audiorecording.sh; grounded theory.sh; study design.sh

Optimal combination: Interview.tw OR audiorecording.sh OR qualitative

stu$.mpsh = subject heading tw= textword - word or phrase in title or

abstractmp = multiple posting – word or phrase in

title, abstract of subject headingWilczynski et al. (2007) Search strategies for identifying qualitative studies in CINAHL, Qualitative Health Research 17:705-710.

EVALUATION OF THREE STRATEGIES ON SIX* DATABASES

Strategy No. of records

identified

No. relevant to review

Controlled vocabulary e.g. in Medline ‘Qualitative Research’, ‘Nursing Methodology Research’

3537 191

Free-text terms40 plus commonly used terms e.g. ‘ethnograph$’, ‘lived experience$’, ‘grounded theory’

3451 172

Broad-based terms 3 free text terms ‘qualitative’, ‘findings’, ‘interview$’, Controlled vocab term ‘Interviews’

3912 187

TOTAL 7420 262

Shaw et al. (2004) Finding qualitative research: an evaluation of search strategies, BMC Medical Research Methodology 4:5

*MEDLINE, EMBASE, CINAHL, British Nursing Index, ASSIA and SSCI

KEY POINTS FOR DEVELOPING YOUR SEARCH STRATEGY

• Seek the input of an information specialist asap.• Undertake some initial scoping and searching.• Identify relevant sources.• Use a methods filter if your review only focuses

on qualitative research.• Document each part of your search and its

results.• Invest in good information management

systems and people.

FURTHER READINGSandelowski M, Barroso J (2007) Handbook for

synthesising qualitative research. New York: Springer (see chp 3)

Flemming K, Briggs M (2006) Electronic searching to locate qualitative research: evaluation of three strategies. Journal of Advanced Nursing, 57: 95-100.

Stansfield C et al. (2010) Search wide and dig deep: Identifying ‘views’ research for systematic reviews. Paper presented at the Joint Colloquium of the Cochrane and Campbell Collaboration, Keystone, Colorado, USA, 18th to 22nd October.

Stansfield C et al. (forthcoming, 2012) Finding relevant studies. In Gough D, Oliver S, Thomas J (Eds) Systematic Reviews in the Social Sciences. London: Sage.