A new group technique to discover in evaluationA new group technique

to discover in evaluationMarie Gervais, Ph.D.

marie.gervais@saaq.gouv.qc.ca

Geneviève Pépin, M.Sc., O.T. (c)Quebec, Canada

5th Biennal Conference of the European Evaluation SocietySevilla, Spain / October 2002

AbstractAbstractTRIAGE, or TRIAGE, or TTechnique for the echnique for the RResearch of esearch of IInformation nformation

by by AAnimation of a nimation of a GGroup of roup of EExperts, is an inductive and xperts, is an inductive and structured method for collecting information, which aims to structured method for collecting information, which aims to obtain a group consensus. The goal of this technique is to obtain a group consensus. The goal of this technique is to quickly and efficiently provide quality informative material to quickly and efficiently provide quality informative material to enable decisionenable decision--makingmaking. . TRIAGE complements and TRIAGE complements and distinguishes itself from the main group techniques used in distinguishes itself from the main group techniques used in evaluation up until nowevaluation up until now ((Delphi technique, Nominal Group Delphi technique, Nominal Group TechniqueTechnique, , Focus GroupFocus Group). ).

The definition, the context for use as well as the The definition, the context for use as well as the stages in the usual process of the TRIAGE technique will be stages in the usual process of the TRIAGE technique will be presentedpresented. . Many examples of the use ofMany examples of the use of TRIAGE being TRIAGE being applied in different evaluation contexts be presented. These applied in different evaluation contexts be presented. These examples will illustrate the richness, the flexibility and the examples will illustrate the richness, the flexibility and the potential of this technique. The strengths and shortcomings potential of this technique. The strengths and shortcomings of this technique will also be discussed.of this technique will also be discussed.

History of TRIAGEHistory of TRIAGE

TT echniqueechnique forfor RR esearchesearch ofof IInformationnformation by by AA nimationnimation of aof a GG rouproup

ofof EE xperts xperts ((PlantePlante & & CôtéCôté, 1993), 1993)

Developed in the field of educationDeveloped in the field of education(Laval (Laval UniversityUniversity, , QuebecQuebec, Canada), Canada)

Transfer appliedTransfer applied tto the health field (1996)o the health field (1996)FamilyFamily of Group Techniquesof Group Techniques

Alternative for Alternative for FocusFocus Group, Nominal Group Group, Nominal Group Technique (NGT), Technique (NGT), DelphiDelphi TechniqueTechnique

TRIAGE vs FocusGroup

TRIAGE vs FocusGroup

Individual phaseQuestionnaire

Collective phaseFurthers interactionsbetween participantsUngoing data analysis and validationRole of animatorGroup and task oriented

Individual Individual phasephaseQuestionnaireQuestionnaire

Collective phaseCollective phaseFurthersFurthers iinteractionnteractionssbetween participantsbetween participantsUngoingUngoing data data analysis analysis and and validationvalidation

RoleRole of of animatoranimatorGroup Group and task orientedand task oriented

No individual phase

Collective phase

Furthersinteractionsbetween participantsStrenuous data analysis andvalidation processRole of animator

No No individualindividual phasephase

Collective phaseCollective phase

FurthersFurthersiinteractionnteractionssbetween between participantsparticipantsStrenuousStrenuous data data analysis analysis andandvalidation validation processprocessRoleRole of of animatoranimator

TRIAGE vs NGTTRIAGE vs NGTIndividual phaseQuestionnaire

Collective phase

Furthersinteractionsbetween participants

Ungoing data analysis andvalidation

Role of animator

Individual Individual phasephaseQuestionnaireQuestionnaire

Collective phaseCollective phase

FurthersFurthersiinteractionnteractionssbetween between participantsparticipants

UngoingUngoing data data analysis analysis andandvalidationvalidation

RoleRole of of animatoranimator

Individual phaseQuestionnaire

Collective phaseInteraction limited to group leader and participantsUngoing data analysis andvalidationRole of animatorTask-oriented

Individual Individual phasephaseQuestionnaireQuestionnaire

Collective phaseCollective phaseInteractionInteraction limited limited to to group leader group leader and and participantsparticipantsUngoingUngoing data data analysis analysis andandvalidationvalidationRoleRole of of animatoranimatorTaskTask--orientedoriented

TRIAGE vs DelphiTRIAGE vs DelphiIndividual phaseQuestionnaireCollective phaseFurthers interactionsbetween participantsUngoing data analysis and validationRole of animatorGroup and task-oriented

Rapid and efficient process

Individual Individual phasephaseQuestionnaireCollective phaseCollective phaseFurthersFurthers iinteractionnteractionssbetween participantsbetween participantsUngoingUngoing data data analysis analysis andand validationvalidationRoleRole of of animatoranimatorGroup Group and taskand task--orientedoriented

RapidRapid and and efficient efficient processprocess

Individual phaseQuestionnairesNo interactionNo collective phase

Strenuous data analysis andvalidation process

No group animation

Time consumingprocess

IndividualIndividual phasephaseQuestionnairesQuestionnairesNo No iinteractionnteraction

No No collective phasecollective phase

StrenuousStrenuous data data analysis analysis andandvalidation validation processprocess

No group No group animationanimation

Time Time consumingconsumingprocessprocess

Characteristics of TRIAGECharacteristics of TRIAGE

InductiveInductive techniquetechnique

InteractiveInteractive techniquetechnique

ContinuousContinuous visual support visual support

ConsensusConsensus--focusedfocused technique technique StructuredStructured technique for the gathering technique for the gathering of of informatioinformation n made up of 3 successive made up of 3 successive phasesphases

TRIAGE... Preparation phaseTRIAGE... Preparation phase

DefinitionDefinition & validation & validation of themes or of themes or questionquestions of s of researchresearch//evaluationevaluation

ElaborationElaboration of data collection of data collection tools tools

Formation of expert groupFormation of expert groupKKeyey informersinformers ((usually usually 6 to 12)6 to 12)

TRIAGE... Production phaseTRIAGE... Production phase

Individual Individual pproductionroduction phasephasePostal questionnairePostal questionnaire5 5 answers answers to to each each question question givengiven by by participantparticipantss ((indicatorsindicators, issues, , issues, pperceptionserceptions, suggestions, suggestions, , needsneeds, , strengthsstrengths, , weaknessesweaknesses, …), …)Transcription of information on Transcription of information on numbered numbered cardscardsNon Non treated treated informationinformation

Collective pCollective productionroduction phasephaseWork Work session of session of thethe expert group (2expert group (2--3 3 hourshours))F lF l ii t

Collective Phase… DescriptionCollective Phase… Description

MemoryStoring of answers

GroupingsInformation processing

SelectionFinal choice of indicators / Can be priorized

Refrigerator Ambiguity, difficulty: Indicator kept to be discussed later

VetoFor diminishing tension / Will be treated later by external experts

GarbageIndicators not relevant

MemoryMemoryStoringStoring of of answersanswers

GroupingsGroupingsInformation Information processingprocessing

SelectionSelectionFinal Final choicechoice of of indicatorsindicators / Can / Can be priorizedbe priorized

Refrigerator Refrigerator AmbiguityAmbiguity, , difficultydifficulty: : Indicator keptIndicator kept to to be discussed be discussed laterlater

VetoVetoFor For diminishingdiminishing tension / tension / Will be treated laterWill be treated later by by externalexternal expertsexperts

GarbageGarbageIndicatorsIndicators not relevant not relevant

11

9988

7766

5544

3322

1010

Groupings

Refrigerator Veto Garbage

SelectionMemory

Visual support

66331010

11

9988

77

5544

22

Participants can’t

agree on 3. It is put in

the refrigerator for a while and will be discussed

later.

Concensus is not possible for 10. It will be treated by

externalexperts later.

6 is not relevant. It

is put in the

garbage can.

66331010

99

88

77

55

4422

1111

111 and 8 share similar ideas under different

names. A new indicator that

includes these ideas is then created.

2 and 4 share a similar

meaning, onlythe most accurate

indicator is kept.

44 2222

5 contains more than

one idea. It is divided in 2

new indicators.

551313

7 and 9 are kept as they

are.

7799

1212

66331010

991111

22

1212

771313

2, 9, 11 and 12 are the final

selection. They can be put in

order of priority.

Applications in evaluationApplications in evaluationProjecProject:t: Evaluation of the effectiveness of Evaluation of the effectiveness of rehabilitation rehabilitation programsprograms

Participative Participative and pluralistic approachand pluralistic approach53 participants 53 participants

Decision makers Decision makers / / Caregivers Caregivers / / UsersUsers of of the the programsprograms

800 propositions of 800 propositions of indicatorsindicatorsSelection Selection by consensus of by consensus of thethe 15 «15 « bestbest » » indicators according indicators according to 5 dimensions of to 5 dimensions of effectiveness effectiveness ElaborationElaboration andand validation of validation of thethe model of model of Dimensions of Dimensions of programprogram (Gervais, 1996)(Gervais, 1996)Model Model and selected indicators later usedand selected indicators later used for for formalformal evaluationevaluation of multipleof multiple programsprograms

Applications in evaluationApplications in evaluationProjeProjecct:t: Identification of operational Identification of operational components of a mental health program components of a mental health program in a general hospitalin a general hospital -- Formative Formative evaluationevaluation

36 participants / 6 groups36 participants / 6 groups((doctorsdoctors, nursing staff, , nursing staff, healthhealth professionalsprofessionals, ,

patients)patients)

343 indicators grouped by theme 343 indicators grouped by theme (communication, team (communication, team workwork, management, …), management, …)

Identification of the strengths and Identification of the strengths and weaknesses of the program weaknesses of the program Identification and succeeding Identification and succeeding implementation of a solution planimplementation of a solution plan

Applications in evaluationApplications in evaluationPProjectroject:: Development of a satisfaction tool Development of a satisfaction tool for the for the traumatic brain injury traumatic brain injury clienteleclientele

10 10 Quebec specialists Quebec specialists in in the fieldthe field of of user’s user’s satisfactionsatisfactionValidation of Validation of thethe 5 d5 dimensionsimensions of of care care and their and their correspondingcorresponding standardsstandardsPhase of item Phase of item generation generation 8888--item prototype item prototype questionnairequestionnaire (5(5--point rating point rating scalescale) ) Satisfaction questionnaire Satisfaction questionnaire actually under actually under experimentation experimentation in in the Quebecthe Quebec traumatology traumatology networknetwork

Applications in evaluationApplications in evaluationProject:Project: Improving the validity of a Improving the validity of a measurement tool for the social measurement tool for the social participation of a clientele with physical participation of a clientele with physical disabilitiesdisabilities

EvaluationEvaluation of of thethe relevance relevance and degree of and degree of difficultydifficulty of of the the 58 58 test test itemsitems29 participants / 3 groups 29 participants / 3 groups fromfrom 3 3 rehab rehab centers centers in in thethe province of province of QuebecQuebecInterInter--group group validity validity Confirmation of results from Rash analysisConfirmation of results from Rash analysisRecommandations made to Recommandations made to the authorsthe authors of of thethe test to test to developdevelop a version a version especially especially adaptedadapted to to the targeted clientele the targeted clientele (spinal (spinal

Applications in evaluationApplications in evaluationProjeProjecct:t: Evaluation Evaluation of a of a community community resource associatedresource associated with the Quebec with the Quebec health health networknetwork

Goal of Goal of the resourcethe resource: To : To enhanceenhance thethe social social integration integration of people living of people living with disabilitieswith disabilities77 participants 77 participants ((usersusers, , family membersfamily members, ex, ex--usersusers, , intervenersinterveners, network , network actorsactors))Identification of the strengths and Identification of the strengths and weaknesses of the program according to weaknesses of the program according to the model of the Dimensions of the program the model of the Dimensions of the program (Gervais, 1996)(Gervais, 1996)Implementation of improvement strategiesImplementation of improvement strategies

Applications in evaluationApplications in evaluationStrengths of the community resourceStrengthsStrengths of theof the community community resourceresource

1. Centered clients objectives

2. Participative management

1. Centered clients objectives

2. Participative management

1. Devoted humanresources / bilinguism

2. Structure of the community resource

1. Devoted humanresources / bilinguism

2. Structure of the community resource

1. Occupational and recreational programmation

2. Involvement and attitude from the employees

1. Occupational and recreational programmation

2. Involvement and attitude from the employees

1. Occupational alternative to work

2. Satisfaction for the close relatives

1. Occupational alternative to work

2. Satisfaction for the close relatives

1. Psychological benefits (well-being, normalization, confidence, self-esteem, sense of belonging)

2. Social contacts (experimentation of valued social roles, decreasedisolation)

1. Psychological benefits (well-being, normalization, confidence, self-esteem, sense of belonging)

2. Social contacts (experimentation of valued social roles, decreasedisolation)

1. Lack of thattype of resource in the actualnetwork

2. Clients: severe physical and cognitive disability

**strategicstrategic

**systemicsystemic

*structural*structural

**operationaloperational

**specificspecific

Applications in evaluationApplications in evaluationImprovement goals for the community resourceImprovementImprovement goals for thegoals for the community community resourceresource

1. Marketing of the communityresource in nearby community organisation and partners

2. Long term vision of the communityresource(financial survival)

1. Marketing of the communityresource in nearby community organisation and partners

2. Long term vision of the communityresource(financial survival)

1. Accessibility (area and tansportation)

2. Uncertain and restricted financing

1. Accessibility (area and tansportation)

2. Uncertain and restricted financing

1. Programming: amount and variety2. Human resources: stability

1. Programming: amount and variety2. Human resources: stability

2. Few partners3. Lack of

openmindedness

2. Few partners3. Lack of

openmindedness

1. Social integration objective to pursuit

2. Empowerment and assuming responsabilities are to be developed

1. Social integration objective to pursuit

2. Empowerment and assuming responsabilities are to be developed

1. Younger clients

2. Be carefulnot to encourage dependency

**strategicstrategic **systemicsystemic

*structural*structural

**operationaloperational

**specificspecific

Conclusion : Strengths of TRIAGEConclusion : Strengths of TRIAGE

Can be used on a small, medium or large scaleCan be used on a small, medium or large scaleCan be adapted to different contextsCan be adapted to different contextsAllowsAllows the simultaneousthe simultaneous exploration of exploration of several several facets facets of of the phenomenon the phenomenon to to be studiedbe studiedIs Is flexible, flexible, easy to use and inexpensiveeasy to use and inexpensiveRapidly produces valid informationRapidly produces valid information -- No No additionaladditional data data analysis requiredanalysis requiredInteractions pInteractions promoteromote an early and continuous an early and continuous implication on behalf of participants in the implication on behalf of participants in the evaluation processevaluation processStimulates a reflexive practice Stimulates a reflexive practice for for thetheparticipantsparticipantsVisual Visual support support promotes empowermentpromotes empowerment

Conclusion : Pitfalls of TRIAGEConclusion : Pitfalls of TRIAGEScope of theme to be studiedScope of theme to be studiedChallenges with animationChallenges with animation

Groups Groups dynamics skills dynamics skills of of the animatorthe animatorManagement of Management of problematic behavioursproblematic behaviours

Choice of participantsChoice of participantsContent expertContent expertBe Be confortable confortable within within a group situation a group situation

(communication (communication skillsskills, , level level of of concentration)concentration)

Comparability of results to be Comparability of results to be demonstrateddemonstrated