An International Project on the Effectiveness of

Psychotherapy and Psychotherapy Training

(IPEPPT): Research Framework and

Protocols

Robert ElliottUniversity of Toledo

Research-Practice Gap in Era of Evidence-Based Mental Health

Numerous contemporary attempts to link research & practice in psychotherapy

Top-down solutions: Empirically-supported treatments Evidence-Based Practice

Based on: Randomized Clinical Trials research model Therapist-as-research-consumer model

Results have been mixed

Research-Practice Integration as a Two-way, Dialectic Process Success is more likely if we add a more

integrative, bottom-up strategy Building on Mental Health Services/

Therapy Effectiveness paradigm Existing RCT research makes space for

grass-roots-based research in real world practice and training settings

=Practice-based Evidence

Example: Practitioner Research Networks (PRNs) USA: Pennsylvania (Ragusea, Borkovec,

Castonguay) UK: National Health Service CORE

research team (Barkham, Evans et al.) Latest trend: Practice-based research in

training clinics and centers (e.g., Castonguay et al.)

Practice-Based Therapy Research in Training Sites

Training site research movement: USA, Europe Research on psychotherapy process/outcome is

essential for understanding and improving psychotherapy practice in all orientations

Being able to use and carry out research is an important aspect of therapist competence

Best way to learn therapy research methods: Do research during basic therapy training Primary professional socialization process Create habits that carry over into later practice

Principles for Practice-based Research

1) Make research relevant to actual practice of therapy

2) Use methods that support therapy rather than interfere with it

3) Actively and continuously involve therapists in selection of research questions and methods

4) Include inexpensive and easy-to-use instruments of key elements (therapeutic alliance, client problem severity)

5) Encourage variety of research methods (qualitative & quantitative; group & single-case)

6) Create research networks of training sites using similar, pan-theoretical instruments

International Project on the Effectiveness of Psychotherapy and Psychotherapy Training (IPEPPT) Formally initiated, June 2004, by:

Italian Coordinamento Nazionale Scuole di Psicoterapia (CNSP; >5,000 therapists)

Italian Federation of Psychotherapy Associations (FIAP; 21 psychotherapy associations: >10,000 therapists)

General Goal: To improve psychotherapy and psychotherapy training in a broad range of theoretical approaches by encouraging systematic research in therapy training institutes and university-based training clinics.

IPEPPT General Scientific Steering Committee Robert Elliott, Scientific Director (University of Toledo-

USA) Alberto Zucconi, Coordinator (University of Siena-Italy) David Orlinsky (University of Chicago-USA) Franz Caspar (University of Freiburg) Louis Castonguay (Pennsylvania State University-USA) Glenys Parry (University of Sheffield-UK) Bernhard Strauss (Friedrich Schiller University Jena-

Germany)

IPEPPT: Current Status Still in formation stage Not a single study The “Project” = Promoting practice-based

research in Europe, North American and elsewhere

Finding partners Creating/finding tools

E.g., conceptual/organizing concepts

IPEPPT Specific Objectives 1. To construct a list of agreed-upon general

pantheoretical recommendations for evaluating: Key aspects of therapy, especially in training centers Key aspects of therapy training outcome Not a “Core Battery”

2. To facilitate the development of specific treatment and training outcome protocols for particular: Therapy approaches (e.g., Systemic therapy) Client populations (e.g., people living with schizophrenia) Linguistic/national groups (e.g., Italy)

3. To facilitate national/international collaborations

IPEPPT Draft Research Framework Such a project requires a guiding

conceptual framework for determining what to measure and how to measure it

Work-in-progress 8 measurement domains:

4 Research themes 2 Levels (Star design)

Research Theme A. General/ Pantheoretical

B. Treatment- Specific

I. Therapy Outcome

e.g., general problem severity

e.g., theory-based dysfunctional processes

II. Therapy Process

e.g., therapeutic alliance

e.g., therapist techniques

III. Client/Ther-apist Background

e.g., demographics e.g., preference for type of therapy

IV. Training Outcome

e.g., productive vs. unproductive practice pattern

e.g., therapist skill development

Framework: Eight Therapy Measurement Domains, with examples of key concepts

Structure: (1) “Star” Design Main body of the star = General outcome/

process protocol Shared by all orientations (General/

Pantheoretical) Provides common metric

Star rays = Specialized protocols for different therapy approaches and different countries (Treatment/Population/Language Specific)

“Star” Design for Sample Concepts within Therapy Outcome Domain for Studies of Four

Different Therapies

Qualitative perceptions of

change

General problem severity

Interpersonal/ relational issues

Dysfunctional Attitudes

Target Problems

Implicit Cognitive Biases

Self-Ideal Discrepancy

Experiential Access

Self-Esteem

Level of Object Relations

Maturity of Defenses

CCRT Change

Relational Satisfaction

Family Environment

Interpersonal Empathy

CBTCBT Experi-Experi-entialential

Psycho-Psycho-dynamicdynamic

Family/Family/CouplesCouples

Structure: (2) Nested Priority Lists

Not a single “core battery” Allow flexibility while encouraging

consistency within & across approaches Three Levels of Priorities:

1) Measurement domains are prioritized2) Within each measurement domain, key concepts

are ranked by approximate importance3) For each concept, available instruments are also

described (researchers prioritize)

Research Theme A. General/ Pantheoretical

B. Treatment- Specific

I. Therapy Outcome

e.g., general problem severity

e.g., theory-based dysfunctional processes

II. Therapy Process

e.g., general problem severity

e.g., therapist techniques

III. Client/Ther-apist Background

e.g., demographics e.g., preference for type of therapy

IV. Training Outcome

e.g., productive vs. unproductive practice pattern

e.g., therapist skill development

Framework: Eight Therapy Measurement Domains, with examples of key concepts

Example: General Therapy Outcome Domain

Key concepts in a possible recommended priority order: (“Star”) (1) General problem severity (quantitative)

Give every 2 sessions to reduce data loss from drop-out (2) Interpersonal/relational functioning (3) Qualitative perceptions of change (4) Individualized problems/goals (5) Health care utilization/costs (6) Quality of life/life satisfaction/well-being

Instrument (length) Reference ShorterForms(length)

Scale Basis,Points &Time Frame

Cost Non-EnglishTrans-lations

Symptom Checklist-90-Revised (SCL-90-R) (90 items)

Derogatiset al., 1976

BriefSymptomInventory (53items)

Distress5 pointsPast week

$2 each(scoringextra)

10 (incl.Italian)

Clinical Outcomes inRoutine EvaluationOutcome Measure(CORE-OM; 34items)

Evans etal., 2002

CORE-SF(18 items)

Frequency5 pointsPast week

Free Italian,Slovak,Norwe-gian

OutcomeQuestionnaire (OQ-45) (45 items)

Lambert etal., 1996

OQ-30 (30items); OQ-10 (10 items)

Frequency5 pointsPast week

One-timelicensingfee (e.g.,$30 for astudent)

German,Dutch,Spanish,Italian

Treatment OutcomePackage (TOP) AdultClinical Scales v4.0(58 items)

Kraus etal., 2005

TOP 37 (37items)

Frequency6 pointsPast 2 weeks

Free Spanish

Common General Symptom Severity Instruments

Research Theme A. General/ Pantheoretical

B. Treatment- Specific

I. Therapy Outcome

e.g., general problem severity

e.g., theory-based dysfunctional processes

II. Therapy Process

e.g., therapeutic alliance

e.g., therapist techniques

III. Client/Ther-apist Background

e.g., demographics e.g., preference for type of therapy

IV. Training Outcome

e.g., productive vs. unproductive practice pattern

e.g., therapist skill development

Framework: Eight Therapy Measurement Domains, with examples of key concepts

Example: General Therapy Process Domain

Key concepts in possible recommended priority order: (1) Therapeutic alliance (2) Therapist and client response modes (3) Perceived helpful aspects of therapy (4) Perceived session effectiveness

Common Therapeutic Alliance Instruments

Instrument(items)

Reference Scale Basis,Points &Time Frame

Subscales Informant Non-EnglishTranslations

WorkingAllianceInventory(WAI) (36items; 12 itemshort form)

Horvath &Greenberg,1989;Hatcher,2005a

Frequency7 points(revised shortform: 5points)

BondTaskagreementGoalagreement

ClientTherapistObserver

DutchDanishFrenchItalian

CaliforniaPsychotherapyAlliance Scale(CALPAS; 24items)

Gaston &Marmar,1994

Agreement-Disagreement6 points

PatientWorkingCapacityPatientCommitmentTherapistUnderstandingandInvolvementWorkingStrategyConsensus

ClientTherapist

FrenchPortugueseItalian

Penn HelpingAllianceQuestionnaire-II(Haq-II; 19items)

Luborskyet al., 1996

Agreement7 points

-- ClientTherapist

GermanFrenchNorwegianDutch(earlierversion)

Different Levels of Research Protocol are Possible

I. Minimum Protocol II. Systematic Case Study Protocol III. Maximum Protocol

Other Protocols: IV. General Training Protocols V. Specific Research Protocols

I. A Recommended Minimum Protocol: Applications

Easy to use: Limited to one measure from each of the first three research domains

Can use with own clients Provides basic treatment monitoring for

individuals & agencies Other versions are possible (e.g., different

outcome or process measures)

I. A Recommended Minimum Protocol: Elements

(1) General therapy outcome instrument Client problem severity Give at odd-numbered sessions (short form)

(2) General therapy process Therapeutic alliance (use short from)

(3) Client/therapist background measure Standard practice: Client/ therapist demographics Client diagnosis, presenting problems Type of therapy

II. Systematic Case Study Protocol: Applications

Use for student case study requirements Meets emerging standards for systematic single

case research New online journal: Pragmatic Case Studies in

Psychotherapy (Rutgers University, Editor: Fishman)

II. Systematic Case Study Protocol: Elements

A. Therapy Outcome: (1) Weekly/biweekly outcome measure+ (2) At least one other quantitative outcome measure+ (3) Qualitative outcome assessment (e.g., post-therapy interview)

B. Therapy Process (1) Therapeutic alliance+ (2) Detailed record of therapy (process notes and/or recordings)+ (3) Qualitative perception of helpful aspects (post-session and/or post-

therapy) C. Client/therapist background

Client/ therapist demographics; client diagnosis, presenting problem; type of therapy

II. Systematic Case Study Protocol: Research Questions

(1) Did the client change substantially over the course of therapy?

(2) If the client changed, did therapy make a substantial contribution?

(3) What brought about the client’s changes?

II. Systematic Case Study Protocol: Emerging Evidence Standards (1) Rich case record, including both quantitative

& qualitative data (2) Replication/convergence across methods (3) Critical examination of alternative views

(e.g., Hermeneutic Single Case Efficacy Design, Elliott, 2002): Non-change explanations (e.g., measurement error) Non-therapy explanations (e.g., extra-therapy events)

(4) Narrative coherence Narrative model of predisposing and process factors Use for generalizing to other cases

Include measures of at least one concept in each of the eight domains

Appropriate for research centers (e.g., Center for the Study of Experiential Psychotherapy)

Also consortia of cooperating centers: Each center measure some variables

III. Maximum Protocol

IV. General Training Research Protocols: Issues Outcomes of therapy training not well understood Difficulties:

Lack of agreed-upon measures of therapist functioning and skill

Must measure therapist change longitudinally over several years of training

Possible applications: Use research to improve training Meet requirements of accrediting and funding agencies

IV. General Training Research Protocols: Promising Concepts

General therapist facilitative interpersonal skills (e.g., coping with common difficulties)

Quality of therapist professional involvement and growth (e.g., Orlinsky; Collaborative Research Network [CRN])

Qualitative perceptions of effects and important aspects of training (e.g., qualitative interviews)

Change in therapist self concept (e.g., Scilligo, SASB Introject scales)

V. Specific Protocols

= Star rays Applications: For specific theoretical approaches,

client populations, or language groups Requires working committee for each group

Identify relevant therapy outcomes, processes, background variables (or training outcomes)

Do protocol and measure development research Establish virtual communities for exchanging ideas

V. Specific Protocol Example:

Person-Centered and Experiential Psychotherapy International Research Group (PCEP-IRG)

Current core members: University of Toledo (Elliott & team) Ohio University (Anderson & team) Katholieke Universiteit Leuven (Leijssen & team) Universities of Strathclyde & Abertay, Scotland

(McLeod, Cooper)

V. PCEP-IRG Outcome Protocol: Promising Developments

Center for the Study of Experiential Therapy Research Protocol (CSEP- 2):

Self--determined problems/goals: Personal Questionnaire (PQ-10)

Self-concept (content & coherence) Qualitative Self-Description interview Tennessee Self-Concept Scale 2 (long, short forms)

Experiential processing: Toronto Alexithymia Scale (TAS-20)

Need positive mental health measures, self-coherence, etc.

V. Specific Training Research Protocols

Some Possible Types of Specific Training Outcomes: Treatment-specific intervention skill Case formulation skill Therapist personal development (e.g.,

maturity, identification with orientation, values)

Promising New Therapy Research Methods Make this Work Possible

Systematic qualitative research methods Interpretive single case designs (Fishman,

Elliott) Using early outcome to identify & repair

problems (Lambert: Signal alarm methods) New, powerful psychometric methods

(Rasch analysis/Item Response Theory) Virtual communities (Community Zero)

Invitation to Dialogue - 1

1) Provide comments and suggestions on the framework & concepts presented here: Robert.Elliott@utoledo.edu

2) Form or join online discussion groups or virtual communities

Closed sites; must apply for membership General info: www.communityzero.com/ipeppt Example: www.communityzero.com/pcepirp

3) Begin implementing the minimum protocol design with your own clients and in your own training setting.

Invitation to Dialogue - 2

4) Convert traditional case presentation training requirements into systematic case study exercises

5) Help with translations of key research instruments

6) Contribute to psychometric research: Improve existing instruments Equate different instruments for same concepts

7) Collaborate with groups with similar interests to generate data for pooling.