objective assessments of quality of life: how much do they agree with each other?

Journal of Community & Applied Social Psychology, VoE. 5 , l -19 (1995)

Objective Assessments of Quality of Life: How Much do they Agree with Each Other?

JONATHAN PERRY AND DAVID FELCE Welsh Centre for Learning Disabilities Applied Research Unit, University of Wales College of Medicine, 55 Park Place, Cardiff, CFI 3AT, U.K.

ABSTRACT

Information was collected on 14 objective measures of service quality in 15 staffed houses for people with learning disabilities. Measures or dimensions of the measures were grouped under six headings: (1) quality of housing; (2) social and community integration; (3) social interactions; (4) development; ( 5 ) activity; and (6 ) autonomy and choice. Rank order correlation coefficients were calculated to explore the extent to which different quality assessments within each category agreed. Overall, there was a reassuring level of agreement. Some of the lack of agreement found could be attributed to the fact that, even though addressing the same general area, measures were focused on subtly different facets of the phenomenon under study. Other disagreement seemed to stem from the interaction between the nature of the information and the process for obtaining it. Comparisons between measures which required staff to give an opinion about how the setting was organized, and those which either required staff to comment on resident activity or employed independent observation, tended to disagree. The study illustrates that the basis for translating a multidimensional definition of quality into measurable terms is developing. Further research to examine the interrelation- ship between measurement approaches is warranted.

Key words: Learning disability, mental handicap, quality of life, quality monitoring, evaluation.

The quality of life of service users has been increasingly proposed as the ultimate criterion for the assessment of the effectiveness of social care delivery in the field of learning disabilities (Landesman, 1986; Brown, 1988; Heal and Chadsey-Rusch, 1985). However, a continuing problem for research in this area is that a definitive conceptualization of quality of life has still not been established. It follows that while much evaluation has accompanied service reform, measurement has not fully reflected the breadth of the concept. Development of a better understanding of how to define and measure quality of life has, therefore, been called for as an urgent priority (Landesman, 1986), a call which has prompted but also reflected a considerable emphasis being given to the issue (Brown, Bayer, and MacFarlane, 1989; Schalock, 1990; Borthwick-Duffy, 1992).

This research was supported by a grant from the Department of Health and Welsh Office.

CCC 1063-3995/95/0 10001-1 9 0 1995 by John Wiley & Sons, Ltd.

2 J. Perry and D. Felce

Building on conceptual and empirical work on the general population and a range of disability groups (Andrews and Withey, 1976; Campbell, Converse, and Rodgers, 1976; Baker and Intagliata, 1982), many commentators now recognize that quality of life encompasses both the objective conditions of a person’s life and their subjective satisfaction with life (Emerson, 1985; Parmenter, 1988; Brown, Bayer, and MacFar- lane, 1989; Schalock, Keith, and Hoffman, 1990; Cummins, 1992; Felce and Perry, in press). Many authors also agree that personal aspirations or values play an important part in how a person may assess their life (Andrews and Withey, 1976; Campbell, Converse, and Rodgers, 1976; Emerson, 1985; Cummins, 1992; Felce and Perry, in press). Thus, quality of life might best be thought of as an interaction between the circumstances or mode of a person’s life, their satisfaction with its various facets, and their personal goals and values.

Within each of these dimensions, researchers have tended to operationalize sub- domains, such as homelife, work, material standards, health, social life, development, activity, choice, and emotional well-being as separate facets of life which contribute to overall quality (Felce and Perry, in press). This breadth of concern has been reflected to some extent in the variety of variables studied in a number of comparative service evaluations (Conroy and Bradley, 1985; Felce, 1989; Lowe and de Paiva, 199 1; Raynes, Wright, Shiell and Pettipher, 1994). Evaluation methodologies aimed at encapsulating some aspect of service quality have proliferated in the last quarter of a century (see Raynes, 1988). However, little is known about their relative proper- ties and, specifically, whether measures with essentially the same purpose agree; that is, give a similar indication of the respective qualities of a number of services. The aim of this study was, therefore, to investigate agreement between different measurement methods. It relates to a review of the quality of life construct which we have conducted (Felce and Perry, in press), which proposes a multi-element frame- work for the concept, which includes the objective and subjective assessment of various life domain concerns. Of the former, a number of objective indicators concerned with material well-being, social well-being, personal development, activity, and choice have been studied here. They were applied to a sample of 15 staffed houses for people with learning disabilities and, to that extent, are restricted to those quality of life aspects based in the home or conducted from the home. Correla- tions between measures were investigated. Data were collected twice over a 2-year period in order to explore the consistency of any significant correlations found.

THE RESEARCH MEASURES

The measures investigated were a number of instruments commonly used in the assessment of quality in residential services for people with learning disabilities. Each is described in greater detail in the Methods section.

Material Well-being An important aspect of material well-being is housing quality, for which four indicators have been used: (1) the Characteristics of the Physical Environment Scale (Rotegard, Bruininks, and Hill, 1981); (2) the Physical Quality Scale (Conroy and Bradley, 1985); (3) the original facility subscore from the Programme Analysis of

Quality of life measurement 3

Service Systems 3 (PASS 3) (Wolfensberger and Glenn, 1975); and (4) the revised physical facility appearance subscore derived from the same source (Wolfensberger, 1983).

Social Well-being Social well-being is reflected by the range and frequency of social contacts, the ex- tensiveness of community activities and by the quality of interpersonal interactions within the household. Social contacts and community activities were measured in this study by direct recording of the activities which people undertook (see Firth and Short, 1987; de Kock, Saxby, Thomas, and Felce, 1988), the use of the Index of Community Involvement-Form 2 (Raynes, Sumpton, and Pettipher, 1989a) and by the social integration ratings within PASS 3. Interpersonal relationships were measured by the relationship dimension of the Community Oriented Programmes Environment Scale (COPES) (Moos, 1974), the social distance items of the Group Home Management Schedule (GHMS) (Pratt, Luszcz, and Brown, 1980), the interactions rating within PASS 3, and by direct observation of social activity patterns.

Personal Development Personal development was the yardstick used by early researchers to assess alterna- tives to institutional care (Tizard, 1964). Although often not the sole focus, the measurement of skill acquisition or developmental progress has been a consistent part of service evaluation since that time (Close, 1977; Schroeder and Henes, 1978; Hemming, Lavender, and Pill, 1981; Felce, de Kock, Thomas, and Saxby, 1986; Lowe, de Paiva, and Felce, 1993). The Adaptive Behavior Scale (Nihira, Foster, Shellhaas, and Leland, 1974) has been one of the most commonly used measures in this area. It was used in this study to reflect personal development together with the personal growth dimension of the COPES and the developmental growth orientation ratings within PASS 3.

Use of skills in the form of engagement in everyday activity is closely related to personal development. Direct observation of the extent of engagement has been a common feature of service evaluation (see Hatton and Emerson, in press). Such direct observation in this study has followed the method set out by Beasley, Hewson, and Mansell (1989). Other indicators of activity or service support for activity included are the activity domain of the Characteristics of the Treatment Environment Scale (CTE) (McLain, Silverstein, Hubbell, and Brownlee, 1977), the involvement subscale of the COPES, the age and culture-appropriate activity, routines and rhythms ratings within PASS 3 and the Index of Participation in Domestic Life (Raynes, Sumpton, and Pettipher, 1989b).

Autonomy and Choice Autonomy and choice have been addressed by use of the autonomy subscale within the COPES, the autonomy domain of the CTE, the age-appropriate autonomy and rights and individualization ratings within PASS 3, the depersonalization, block treatment and rigidity of routine items within the GHMS, the Choice Making Scale (Conroy and Feinstein, 1986), and the Index of Adult Autonomy (Raynes and Sump- ton, 1986).

4 J. Perry and D. Felce

METHOD

Settings and Residents Fifteen houses from four counties in South Wales participated in the study, 14 throughout the 2-year data collection period and one for just the first year. All were small, staffed community residences (ranging in size from between one and seven places), and all but one had been in operation for a t least a year. The services were recruited by approaching service managers in the four counties who were asked to nominate services that would provide a range of qualities. As well as ensuring that the sample of houses selected reflected such a variety in anticipated quality, it was also structured to represent a good range of dependency levels. Eleven of the 15 houses were managed by local authorities, two by voluntary agencies, one by a health authority and one was privately operated. Four houses were located in cities or large towns, ten were in or on the edge of small towns and one was on the edge of a village.

At the outset, the houses catered for a total of 58 residents, of whom 31 were male and 27 were female. Ages ranged from 19 years to 67 years with the mean age being 36 years. Residents in three houses had raw ABS scores which were only achieved by the upper quartile of people in the sample from which the reference norms were derived. Residents in four houses had raw ABS scores which were in the lowest quartile of the reference sample. The raw scores of the other houses were more or less evenly spread across the second and third quartiles. During the course of the study, apart from the three residents lost through the withdrawal of one of the houses, one resident died and three residents moved to alternative accom- modation. Staffing levels in the 15 houses were, on the whole, related to the ability of the residents and varied from staff: resident ratios of 1 : 1 or greater throughout the day to 1 : 6 or less.

Measurement Details of the chosen measures are itemized under six headings: (1) quality of housing; (2) social and community integration; (3) social interactions; (4) development; (5) activity; and (6) autonomykhoice.

Quality ofhousing. The Characteristics of the Physical Environment Scale (CPE) (Rotegard, Bruininks, and Hill, 1981) comprises five Likert scales, one for each functional area of the home: (1) living room; (2) dining room; (3) bedrooms; (4) bathroom; and (5) garden. The researcher asks for a tour of the facility and completes the scales en route. Each scale ranges from 1 to 6 and represents a continuum of homeliness. Examples of what constitutes each pole of the scale are given, and provision is made for comments. Finally, the researcher describes the immediate vicinity of the facility by ticking one or more items from a list provided. Administration took about 10 min per house. High levels of inter-item and full scale reliability were found in the authors’ study of 236 residences.

The Physical Quality Scale (PQS) (Conroy and Bradley, 1985) is a measure of the physical quality of the environment similarly derived from a rater touring the facility. Each item is rated from zero to three. Three items refer to the building’s exterior and the neighbourhood. Two items rate bedrooms in terms of personalization and individualization and the remaining items are concerned with cleanliness, amount


of light, odours, orderliness, and state of furniture in other rooms. The PQS took about 10 min to complete in each house.

Programme Analysis of Service Systems 3 (PASS 3) (Wolfensberger and Glenn, 1975) derives from Wolfensberger’s (1 972) elaboration of the normalization principle. A team of trained raters evaluate the service on 50 ratings, each containing three to six levels. Each rating is accompanied by an explanation of its rationale to provide scoring guidelines. Levels are weighted to reflect the importance of the aspect of service being evaluated. Levels on each rating are determined by the team through discussion and conciliation after each rater has determined their individual assessment. The total score represents the quality of the project: 73% of the total score represents the extent of adherence to the normalization principle; 13% reflects the presence of other ideologically based service practices and 14% reflects administrative efficiency. Quality of the facility in PASS 3 was initially reflected by combining ten ratings concerned with the location, environs, building design, fittings, appearance, comfort and decor of the setting to form a facility subscore. A variant was also explored in this study by considering the facility score minus the proximity ratings, which are concerned with the location of the service. The revised subscale, the physical facility appearance subscore, was also used. This was introduced in 1979 and combines six of the ratings used previously (concerned with building design, fittings, appearance, comfort and decor) with a seventh also concerned with building design, which was not previously included in the facility subscore.

Internal consistency reliability coefficients of 0.92 were obtained by Flynn (1980). Inter-rater reliability of between 0.70 and 0.93 have been obtained with varying numbers of team raters (Flynn and Heal, 1980). In this study, teams of three raters were assembled for each house to be assessed from the body of individuals who have presented or provided team leadership within PASS 3 workshops held in the UK. Teams visited services for a full day.

In the analysis, the subcomponents of the CPE, PQS, and PASS 3 subscores that related solely to the internal standard of the settings were also calculated and compared.

Social and Community Integration. The Index of Community Involvement- Form 2 (ICI) (Raynes, Sumpton, and Pettipher, 1989a) is a short scale designed to measure the extent of resident involvement in social activities and their use of facilities based in the local community. There are two versions of the ICI. The more recent was used in the current study, an adaption of the scale originally used by Raynes, Pratt, and Roses (1979). Respondents are asked whether or not each resident has participated in 14 listed activities in the previous 4 weeks. They are also asked if the person has been on holiday in the last year. No training is said to be needed, and the ICI is simple and brief to administer and score. It can be scored for individuals or groups. Inter-rater reliability of 92% and internal reliability coefficients of 0.85 (group) and 0.77 (individual) have been obtained (Raynes, 1988). In this study, staff members acted as respondents.

The frequency and nature of social contacts and community activities were also directly recorded over two 4-week periods to provide a complete tally of events residents engaged in. Staff were asked to complete recording forms and return them to the researchers after the month was completed. This resulted in low compliance in the first year, with only six houses providing a complete record. In the second

6

year, services were asked to return completed forms weekly and were given telephone reminders to fill in the records. Data were obtained for 12 of the remaining 14 collaborating services. The results on this indicator are only presented for the second year.

Two rating clusters within PASS 3 are concerned with physical and social integration. Various combinations were explored: the combined scores of all physical and social integration ratings, the combined scores of just the social integration ratings, the scores on the single rating concerned with socially integrative social activities and the latter two quantities in combination with four of the physical integration ratings: access, physical resources, program-neighborhood harmony and congregation and assimilation potential.

Finally, the ICI and social and community activities event data were divided into their respective social and community activities components and compared separ- ately.

J. Perry and D. Felce

Sociul interactions. The Community Oriented Programmes Environment Scale (COPES; Moos, 1974) is one of a series of social climate scales that conceptualize three major dimensions of social environments: (1) the relationship; (2) the personal growth; and ( 3 ) the maintenancekhange dimensions. Within these dimensions there are ten subscales: (1) involvement; (2) support; (3) spontaneity; (4) autonomy; (5) practical orientation; (6) personal problem orientation; (7) anger and aggression; (8) order and organization; (9) programme clarity; and (10) staff control. The relationship dimension relevant here comprises the involvement, support, and spontaneity subscales and assesses how involved individuals are in the service, the support stair provide and the mutual support between service users. Each subscale comprises ten questions. Respondents mark an answer sheet to indicate whether each of the items is true or false for their setting. The COPES has been widely used in community settings with staff and residents. Internal consistency of the subscales was found to range from 0.62 to 0.89 (Moos, 1988). Correlations of test-retest reliability ranged from 0.60 to 0.98 (Moos, 1988). In the current study, as many staff as possible in each setting were convened to complete the COPES and it took them 15 to 20 min to do so. Their responses were scored individually and averaged to give a setting score.

The Group Home Management Schedule (GHMS) (Pratt, Luszcz, and Brown, 1980) was developed for use in residential facilities for adults with learning disabilities in Canada and is a modified version of the Revised Child Management Scale (King, Raynes, and Tizard, 1971). It measures the extent to which management practices are institutionally or individually oriented as operationalized through the extent of block treatment, depersonalization, social distance and rigidity in routine. Inter- actions are reflected in the social distance items of the scale. The scale contains 37 items, each scored on a 3-point rating scale. Higher scores signify institutionally oriented practices. Administration of the GHMS is by structured interview with staff members. This took about 30 min per house. Construct reliability has been established (see Raynes, 1988).

The interactions rating from PASS 3 was included in this section. Interactions between staff and residents, and the social initiatives or responses

of residents were also observed using a 20-s momentary time sample encoding observations into a handheld portable computer (a Psion organizer). The observer is


prompted by the computer to observe the behaviour of a particular resident and any contact received from staff every 20 s. The observer then presses precoded keys to input data. The observational definitions used followed those given by Beasley, Hewson, and Mansell (1 989): residents’ social activity comprised clear social activity, unclear social activity, and no social activity; social interaction from staff comprised positive, negative, and neutral contact, contact in the form of assistance and contact from other residents (this code was used in conjunction with one or more of the other four codes to distinguish contact between residents from contact from staff). The data are saved automatically onto datapacks, which can be uploaded to an IBM-compatible computer, analysed and printed at a later date. Provision is made in the software for groups to be observed by observing individuals in rotation. In this study residents were observed in random rotation for blocks of 10 min at a time.

Each house was observed for 11 h a day for 3 days over a 2-year period. Individuals were not observed in personal situations or when out of the house. The number of observations made therefore varied according to when people got up, when they retired, and the extent of their activity outside of the house. A second observer collected data simultaneously with the first for 5% of the time, to enable inter-observer reliability to be checked. Percentage occurrence agreement on each code was calculated by dividing the total number of agreements of occurrence by the total agreements plus disagreements for that code and multiplying by 100. Percentage non-occurrence agreement on each code was calculated by dividing the total number of agreements on non-occurrence by the total number of agreements and disagreements for that code and multiplying by 100. The reliability data for the social interaction data and for the observations relating to resident activity (see ‘Activity’ below) are given in Table 1.

Table 1. Percentage occurrence and non-occurrence reliabilities for the observational codes

Code

No activity Leisure Personal/self-care Domestic/practical chores No social activity Clear/unclear social activity No contact from others Total contact from others

Occurrence reliability 1%) 91.7 76.9 84.9 82.3 97.8 66.3 95.8 64.1

Non-occurrence reliability (%)

81.3 91.0 97.9 99.0 73.3 96.5 75.8 98.5

Development. The Adaptive Behavior Scale (Part One) (ABS) (Nihira, Foster, Shellhaas, and Leland, 1974) consists of 66 items spanning ten domains of adaptive behaviour: independent functioning, physical development, economic activity, language development, numbers and time, domestic activity, vocational activity, self- direction, responsibility, and socialization. Items are structured so that the respondent either has to select one of several possible responses, or select all statements which apply. The ABS was administered by interview with a member of staff who knew the person well and generally took about 30 min to complete. A total ABS raw score was calculated by combining the domain scores with the exception of vocational

8

activity. This was omitted because the items relate to work outside the home, and this is not relevant to the quality of the residential service. Percentile rank scores were calculated by comparing the raw scores for each domain with the domain profile for the appropriate age group in the ABS manual. An overall percentile rank score was calculated by averaging domain rank scores. Inter-respondent reliability was checked by obtaining independent assessments from two members of staff for a sample (1 3%) of residents. Reliability on each domain was calculated by dividing the number of times staff agreed on an item within the domain by the total number of domain items and multiplying by 100. Overall reliability was calculated by a weighted average of domain reliabilities. The reliability data are given in Table 2.

J. Perry und D. Felce

Table 2.

Domain YO agreement Domain YO agreement

Percentage inter-respondent item by item agreement on the ABS

Physical development Language development Numbers and time Economic activity Indevendent functioning

88.5 Domestic activity 87.5 86.7 Self-direction 63.1 87.5 Responsibility 58.3 66.7 Socialization 73.8 82.2 Overall 80.7

Scores from two of the scales were also included: the personal growth dimension of the COPES and the developmental growth orientation rating cluster from PASS 3. The former comprises four subscales: autonomy, practical orientation, personal problem orientation, and anger and aggression. Correlations were investigated using the total dimension score, the total dimension score minus anger and aggression and the practical orientation subscale alone.

Activity. Two indicators of activity were taken from the COPES and PASS 3. The involvement subscale from the COPES was used, as was a combination of the age and culture-appropriate activities, routines, and rhythms ratings from PASS 3. In addition three other measures were used: (1) resident activity; (2) the Index of Participation in Domestic Life; and ( 3 ) the Characteristics of the Treatment Environment Scale.

Resident activity was measured by direct observation at the same time as staff: resident interactions were observed using a 20-s momentary time sample, handheld Psion computers, and the observational definitions put forward by Beasley, Hewson, and Mansell (1 989) as described above. Residents’ non-social activity comprised leisure, personal/seIf-care, domestic/practical tasks, education, and no activity. Residents’ social activity was as set out above. Other observational details are also as described earlier. Correlations were explored using the total extent of engagement in activity, and the extents of subcategories: domestic/practical and leisure.

The Index of Participation in Domestic Life (IPDL) (Raynes, Sumpton, and Petti- pher, 1989b) assesses the extent to which a facility provides opportunities for individual residents to be involved in domestic tasks. It is a measure of opportunity rather than of resident skill level or competence per se. The measure consists of a list of 13 domestic tasks and respondents indicate whether each task is done independently, done with help from staff, or not undertaken at all. Scoring takes account of the number of tasks done and the level of independence with which they are


done. No training is said to be required and the IPDL is simple and brief to administer and score. The authors report inter-rater reliabilities ranging from 95 to 97% and an internal consistency coefficient of 0.89. In this study, staff acted as respondents.

The Characteristics of the Treatment Environment Scale (MR/DD version) (CTE; McLain, Silverstein, Hubbell, and Brownlee, 1977) is a scale to assess the developmental potential of a community setting for people with learning disabilities. Developed from Jackson’s (1964) measure of the therapeutic value of the environment, the CTE comprises two factors: (1) autonomy, the degree to which residents are encour- aged to function independently; and (2) activity, the variety and frequency of social and recreational activities made available to residents. The current version has 48 items and is a self-administered survey which can be carried out on an individual or a group basis. Staff are instructed to read each statement and to mark a Likert scale to indicate its truth or falsity for their setting. The CTE was administered to as many staff as were available and took about 30 min to complete. The activity factor was used in this section.

Autonomy and choice. Aspects of four of the scales already described were used. The CTE autonomy factor described in the last paragraph was included, as were the autonomy subscale from the COPES and a combination of the age-appropriate autonomy and rights and individualization ratings from PASS 3. A combination of the depersonalization, block treatment, and rigidity of routine items from the GHMS was also included in this section. Two further short scales were used.

The Choice Making Scale (CMS) (Conroy and Feinstein, 1986) was developed to estimate the extent to which staff encourage residents to make choices. It is a self-completion questionnaire with six sections containing a total of 24 items. The sections refer to choice over food, house/room, clothes, sleeping/waking, recreation, and other issues. Each item is rated on a 4-point scale ranging from ‘no opportunities for making choices’, to ‘yes, opportunities for making choices all of the time’. Total scores range from 24 to 96, with higher scores representing more opportunities for making choices. The scale was completed by a single member of staff in each house and took 5 to 10 min. Using the scale in a recent study, Raynes, Wright, Shiell and Pettipher (1 994) obtained a reliability coefficient of 0.96 using Cronbach’s alpha.

The Index of Adult Autonomy (IAA; Raynes and Sumpton, 1986) was designed to assess the extent to which adults are given opportunities to make decisions about various aspects of their daily lives. It contains 11 items, each of which is scored on a 3-point scale. Scores are summed to provide an index score. The higher the index score, the greater the opportunity to participate in decision making. As with the CMS, the index was completed by a single member of staff in each house and took 5 to 10 min. Raynes, Wright, Shiell and Pettipher (1994) obtained a Cronbach’s alpha coefficient of 0.82.

Data Collection Each setting received a monthly visit from research staff. The general design of the study was to complete all measures within a year, and repeat them during a second year. The PASS 3 assessment was conducted once because of its difficult logistics and it was sequenced to occur at the end of the first year so that it could straddle the 2 years of the study. Observations of staff and resident behaviour were conducted three times in order to be able to examine data stability. Averages for

,AJ94CA300 00 000009698 0590 00273

10

each year’s data are used. The ABS was carried out at the start of the study, at the end of the first year and at the end of the second year so that change during the first and second years could be calculated.


Analysis All measures with the exception of the social and community activities event data and the ABS either produced percentage scores or were converted to percentages of the total possible score. The frequency data on social and community activities were entered into the analysis as average frequencies per resident per month. Change in ABS was considered in two forms: (1) change in raw scores; and (2) change in average percentile ranks. When measures yielded individual data, the scores for individuals in each setting were averaged. These data transformations resulted in a series of average setting scores for each measure. Measures were grouped according to which quality of life domain they appeared to tap as described above.

Correlation was investigated using Spearman’s rank-order statistic, a non parametric method (see Siegel, 1956). This is a less powerful procedure than Pearson’s product moment coefficient, for which parametric assumptions need to be made. The range of obtained scores for the various measures is relevant to the interpretation of correlations. These are presented in Table 3. Data on the CPE, PASS 3, ICI,

Table 3.

Measure Score range (Yn)

Score ranges and correlations between t l and t2 data

Correlation tl t2 tl - t2

CPE 28-80 40-92 0.53* PQS 49-7 1 49-7 1 0.38 PASS Integration 23-85 NA PASS Facility 29-82 N A PASS DGOT 0-34 N A PASS Other 0-90 NA ICI 12-72 28-77 0.94*** GHMS 1 0 4 7 11-39 0.93*** COPES Relations 61-100 51-95 0.69** COPES Growth 41-78 38-76 0.70** COPES Autonomy 22-88 23-80 0.73** CTE Activity 63-89 58-90 0.62** CTE Autonomy 56-86 51-87 0.89*** ABS (YO rank) 14-89 15-87 0.97*** Engagement 18-80 13-89 0.97*** IPDL e l 0 0 14100 0.77*** IAA 52-100 NA CMS 53-99 NA

t DGO, developmental growth orientation rating cluster; NA, not available as measure only administered once. * p < 0.05; * * p < 0.01; ***p < 0.001,


ABS, observed engagement, and IPDL reflected the possible scale range, or a substan- tial part of it, meaning that the correlations to be reported apply across the full extent of the measures investigated. Data on the PQS, COPES, CTE, GHMS, IAA, and CMS were more restricted tending towards the more positive half of the score spectrum (positive GHMS scores being low). Correlations for these measures should not be taken to imply necessary agreement in the remainder of their possible ranges. However, in general, the score ranges show an absence of enforced clustering due to floor or ceiling effects.

Table 3 also shows correlations between the data obtained at time 1 (tl) and at time 2 (t2) for the various measures. Reliability data were collected only for the direct observational data and the ABS, in keeping with common practice to accept the established reliability of the other scales reported. However, correlation across the data points can give an indication of data stability, and a conservative assessment of test-retest reliability given the possibility of real change in the interven- ing year. Correlations could not be calculated for the PASS 3, IAA, and CMS assessments, as these measures were only done once. All of the other measures showed significant tl - t2 correlation, with the exception of the PQS. Positive relationships were found on both measures of housing quality but the lower significance level of one and non-significance of the other was in keeping with the level of change that had occurred in some settings between assessment points.

RESULTS

Quality of Housing Among the total scores, significant correlations were found between the PQS, CPE, and PASS physical facility appearance score but not the original PASS 3 facility score at tl . However, no significant correlations were found among the t2 assessments (Table 4). Significant and consistent correlations were found between the CPE and

Table 4. Quality of housing: correlations between the CPE, PQS, PASS 3 facility, and PASS 3 physical facility appearance subscores both in total and for the subset of items relating to the internal area of the home (n = 14)

CPE PASS 3 PASS 3 Facility Physical subscore facility

appearance subscore

Total scores 11 0.4F 0.28 0.49"

(0.02) PQS

CPE tl 0.16 0.58* (t2) (0.17) (0.41)

t l 0.4 1 0.35 0.41* PQS

CPE tl 0.59* 0.64* (t2) (0.54*) (0.57*)

(t2) (0.09) (-0.24)

Internal scores

(t2) (0.20) (-0.16) (0.15)

* p < 0.05.

12

Table 5. of PASS 3, and the frequency of social and community events undertaken (n = 14)

f. Ferry and D. Fetce

Social and community integration: correlations between the ICI, relevant ratings

Frequency of PASS 3 PASS 3 PASS 3 event? Social/ Social Socially

physical integration integrative

ratings activities integration ratings social

ICI t l - -0.02 0.23 0.31

Frequency of events (t2) (0.59*) (0.53*) (0.57*)

t Frequency of event data are for t2 only and based on an n = 12. * p < 0.05.

(t2) (0.01) (-0.05) (0.03) (0.26)

PASS 3 scores relating to the internal quality of the housing. The PQS, although significantly correlating at tl with one of the PASS subscores, appeared to report a slightly different aspect of quality than the other two measures.

Social and Community Integration Significant correlations were found between the event data and the various indicators drawn from PASS 3 (Table 5). No significant correlations were found between the ICI and any other measures. This remained true when the ICI total score was separ- ated into social contact and community activity components and each subscore compared with the separate frequencies of social and community activities in the event data. The various formulations from PASS 3 were highly intercorrelated and similar to each other in the extent of correlation with the other measures.

Social Interactions The COPES relationship dimension correlated negatively with the GHMS social distance dimension at t l and positively with observed total social contact each resident received (Table 6). Significant correlations were also found between the GHMS social distance dimension and the PASS 3 interactions rating at t l and at t2. Moderate

Table 6. Social interactions: correlations between the COPES relationships dimension, the GHMS social distance items, the PASS 3 interactions rating, and observed staff: resident interaction@ = 14)

~ ~ ~ ~ ~

GHMS PASS 3 Observed Social distance Interactions Staff: resident

interaction

COPES Relationships tl -0.55* 0.44 0.50" (t2) (-0.39) (0.42) (0.13)

GHMS Social distance t l -0.65** -0.49*

PASS 3 Interactions t l -0.02 (t2) (0.16)

(t2) (-0.49*) (-0.58*)

* p < 0.05; * * p <0.01


but non-significant correlations were found between the COPES relationship dimension and the PASS 3 interactions rating. Observed total contact correlated significantly with the COPES relationship dimension at t l but not at t2. Observed total contact also correlated negatively and significantly with the GHMS social distance at t l and t2. The PASS 3 interactions rating and observed staff:resident contact were not found to correlate.

Development The developmental growth orientation dimension of PASS 3 correlated significantly with the COPES personal growth dimension (Table 7). The relationship was

Table 7. mental growth orientation ratings, and the COPES personal growth dimension (n = 14)

Development: correlations between change in ABS raw score, the PASS 3 develop-

PASS 3 COPES Developmental Personal growth

growth orientation

ABS raw score t l -0.17 ( t2) (-0.07)

PASS 3 developmental t l growth orientation (a

0.19

0.61* (0.46")

(-0.28)

* p < 0.05.

unchanged in terms of significance when the anger and aggression element of personal growth was removed. Change in raw ABS score did not correlate with either of the scales and the same was true for change in rank ABS score.

Activity Table 8 shows consistent and highly significant relationships between the total IPDL score; the age and culture-appropriate activities, routines, and rhythms dimension

Table 8. Activity: correlations between the COPES involvement scale, CTE activity domain, PASS 3 age and culture appropriate activity, routines and rhythms ratings, IPDL and observed resident engagement in activity (n = 14)

COPES CTE PASS3 Total Engagement Engagement Involvement Activity A&C engagement in leisure in domestic

ARRt in activity activity activity

IPDL t1 0.09 -0.16

COPES involvement 11 0.75*** ( t2 ) (0.63**) (0.1 1)

( t2) (0.41) CTE activity t l

( t2) PASS 3 A&C ARR tl

( t 2 )

0.73** 0.91*** (0.78***) (0.91***)

-0.08 0.21 (0.40) (0.45)

-0.07 0.06 (0.18) (-0.02)

0.73** (0.72**)

0.68** (0.83***) 0.30

(0.67**) 0.20

(-0.03) 0.63**

(0.69**)

0.84*** (0.82***)

(0.31) -0.09

- 0.04 (0.1) 0.74**

(0.85***)

t A&C ARR, age and culture-appropriate activity routines and rhythms. * * p < 0.01; ***p < 0.001.

of PASS 3; and total engagement in activity. There were equally high correlations with two subcomponents of engagement-leisure and domestic activity-which were themselves highly intercorrelated with each other and with total engagement. The

14

COPES involvement and CTE activity dimensions were significantly correlated (p < 0.001) at tl and almost correlated at the p < 0.05 level of significance at t2. However, neither scale correlated significantly with the other activity measures, except for the relationship between the COPES involvement subscale and observed engagement in leisure activity at t2.


Autonomy and Choice There were either significant or near significant correlations between all the measures at tl and t2 (Table 9). The COPES autonomy subscale correlated significantly with all other measures, and the remaining measures correlated significantly with at least three or four of the other measures,

Table 9. Autonomyichoice: correlations between the CTE autonomy domain, COPES autonomy scale, PASS 3 age-appropriate autonomy and rights and individualization ratings, GHMS depersonalization, block treatment, and rigidity of routine items, IAA and CMS (n = 14)

COPES PASS 3 GHMS IAA CMS Autonomy Aut & Indt Dep, BT & RRS

CTE Autonomy tl 0.69** (t2) (0.77**)

COPES Autonomy t l

PASS 3 Put & Ind tl

GHMS Dep, BT & RR 11

IAA t l

( t2)

(t2)

(t2)

(t2)

0.40 -0.72** ~ ~

(0.34) (-0.50*) (0.65**) (0.60*) 0.54* -0.71** ~ -

(0.36) (-0.63**) (0.61*) (0.62*) -0.59* ~

(-0.64**) (0.34) (0.52*) - -

(-0.43) (-0.44)

(0.51*)

t Aut & Ind, age-appropriate autonomy and rights and individualization; t Dep, BT & RR, depersonalization, block treatment, and rigidity of routine. * p < 0.05; * * p < 0.01; ***p < 0.001.

DISCUSSION

This study has grouped and compared various indicators of service quality in terms of the domain of experience they appear to tap. Clearly, in such an exercise there is a danger that particular emphases are ignored in forcing what is essentially a pragmatic clustering of supposedly like measures. For example, PASS 3 was not developed as a scale of quality of life but as a way of quantifying the extent to which a service conformed to the principle of normalization. Other indicators have been developed as scales of quality but they rely on different underlying constructs and, therefore, could be viewed as differing in purpose. For example, the GHMS is derived from the initial work of King, Raynes, and Tizard (1971), which stemmed from Goffman’s (1961) sociological analysis of total institutions, whereas the COPES derives its theoretical basis from social psychology. Thus, although each of these three measures may appear to be concerned in a part of their content with a similar issue such as the social milieu of the setting, their different perspectives imply that they need not agree. However, research such as has been conducted here is required if measurement indicators are not simply to proliferate. The common ground between

Quality of l f e measurement 15

them needs to be studied. In this respect, the extent of correlation found between the indicators as grouped is reassuring.

The results on measures of housing quality showed close agreement between the CPE and the physical facility appearance subscore of PASS 3 . The PQS, although overlapping with the CPE in particular, evidently measures a different aspect of housing quality. Scrutiny of the measures’ contents reveals some difference in orientation of the items, with the CPE items being largely concerned with the homeliness of the environment and PQS items having more to do with space, cleanliness, and physical standards of decoration and furnishings. Areas of overlap between the PQS and CPE relate to the degree of personalization and homeliness of bedrooms. The lack of agreement of the original PASS 3 facility subscore with the other indicators in total, and the lower correlations found for the scores relating to internal housing quality, suggest that the revised physical facility appearance subscore introduced in 1979 is to be preferred.

Although based on very different content, the social and community integration indicators from PASS 3 correlated significantly with the data on the frequency of social and community events. The PASS 3 ratings concentrate on the resource and image context for integration as well as on the assessment of the frequency of integrative events. Several rating combinations from PASS 3 were, therefore, tried to reflect these three elements. However, each was highly correlated with the other and all correlated with the detailed event data collected. The same could not be said for the ICI. There was a separation between the ICI and the other measures of social and community integration. This may be explained by a difference in the focus of the ICI. It is structured to measure whether individuals have done a variety of social or community activities in the last month. It therefore has a limited sensitivity to the frequency with which events are undertaken, not discriminating other than between occurrence and non-occurrence. It is, however, sensitive to the range of events undertaken in a way that the frequency of event data may not be. The absence of correlation between the ICI and data emphasizing frequency of occurrence is understandable. This study would suggest that the ICI might best be used in conjunction with measures more sensitive to frequency or adapted to bring such sensitivity within its own specification.

Measures of social interaction were generally found to correlate at levels at, or approaching, significance. The GHMS social distance dimension seemed to reflect, albeit inversely, the other three measures, which with the exception of the direct observations were concerned with the quality and quantity of interaction. The observed staff: resident interaction was purely a measure of the frequency of contact and this correlated moderately strongly with the COPES relationship and GHMS social distance dimensions. However, it and the PASS 3 interactions rating showed little agreement. The latter judges whether interactions are warm and positive and free from devaluation. Assessment may be concerned more with the quality of interactions than their quantity. The measurement of quality and quantity clearly comple- ment each other and both have their uses and are to be recommended.

The significant correlations between the PASS 3 developmental growth orientation rating and the COPES personal growth dimension set them apart from the change in ABS raw score as measures of development. The distinction may be explained by the fact that PASS 3 and COPES are concerned with structures and processes, which indicate an environment as conducive to personal development, whereas

16

change in raw ABS score measures personal development per se. The information gained by the COPES is by interview with staff. It may be apparent to staff that it relates mainly to the quality of environmental arrangement for which they are responsible. PASS 3 has a greater element of direct observation but much of the information for the major rating of relevance here, the intensity of relevant program- ming, results from what staff say they do. Although staff are also the informants for the ABS, the measure is specifically concerned with what residents can do-their skill levels-as opposed to any aspect of environmental milieu. The findings of this study suggest that a staff account of the developmental quality of a setting does not agree with a longitudinal assessment of behavioural development. Although the latter has methodological difficulties and is time consuming, its replacement by cross-sectional proxies cannot be recommended on the basis of these results.

Measures concerned with resident activity also fell into two relatively distinct groups. The IPDL, the PASS 3 age and culture-appropriate activities, routines and rhythms ratings and the total level of engagement and its subcomponents--engagement in domestic activity and engagement in leisure activity-were highly intercorrelated. The COPES involvement and CTE activity dimensions were also highly intercorrelated but independent of the first set of measures. Again, a possible explanation may lie in the difference in focus of the measures and the interaction of focus with informant. The IPDL, PASS 3 ratings, and observed activity focus on the activity of residents. Even though the IPDL is a staff informant measure, the data gained correlated highly with independent observation. The COPES and CTE are staff informant measures which are much more directed at staff organization of the setting. This line of inquiry does not seem to produce a similar picture to that which gauges the impact on residents. This is true even when closely related data sets are compared. For example, the CTE activity domain is concerned with the availability of leisure activities. A near zero correlation with observed engagement in leisure activities was found at both time periods.

Finally, the finding that correlations between the measures of autonomy and choice were almost universally significant and consistent across t l and t2, suggests that the various measures assess similar quality concerns. Information on all measures was gained by staff report except for the element of direct observation inherent in the PASS 3 evaluation. A further development of measures of similar kind appears unnecessary, Development of assessment measures which are independent of staff report would seem more useful.

Overall, the results tend to confirm that progress has been made in rendering a multidimensional structure of the quality of life measurable. Broad agreement between measures reflecting similar aspects of life was found, which will bring greater confidence to the choice of measurement instruments in the future. However, certain differences were also found. Even within quality of life domains, more than one measure may be needed to represent quality concerns fully. For example, there seem to be several aspects to what is meant by housing quality, range and frequency of social, community, and other activities are both relevant and the quantity and quality of interactions and activity need to be addressed. The findings also indicate a tendency for disagreement between measures that gained information from staff about the organization of the setting and those that gained information either from staff about residents or independently by direct observation. One could ascribe this result to the traditional distinction between the quality of care (or process) and


Quality of life measurement 1 I

the quality of life (or outcome). However, particularly in recent times when quality of life has become the dominant objective, it is difficult to define quality of care or process other than that which leads to quality of life. This study would suggest that process perspectives need to be complemented by data on outcome.

Although involving considerable data gathering, this study was still limited to a sample of 15 houses and a range of available measures. Further research is required both to replicate and extend it. One must treat the findings with some caution until similar relationships between measures are demonstrated by further research. Although many of the correlations were consistent across the two data collection periods within this study, some were not. Such inconsistency adds to the need for caution. However, ultimately, the validation of short, easy to use and non-intrusive quality scales, such as the IPDL, as consistently and highly correlated with the more time-consuming and complex direct observation of activity patterns may be of considerable practical value in promoting the routine evaluation of the quality of housing services for people with learning disabilities.

REFERENCES

Andrews, F.M. and Withey, S.B. (1976). SocialIndicators of Well-being: Americans’Perceptions of Life Quality, Plenum Press, New York.

Baker, F. and Intagliata, J. (1982). ‘Quality of life in the evaluation of community support systems’, Evaluation and Program Planning, 5,69-79.

Beasley, F., Hewson, S. and Mansell, J. (1989). MTS: Handbook for Observers. Centre for the Applied Psychology of Social Care, University of Kent at Canterbury.

Borthwick-Duffy, S.A. (1992) ‘Quality of life and quality of care in mental retardation’, in L. Rowitz (ed) Mental retardation in the year 2000, Springer-Verlag, Berlin.

Brown, R.I. (1988). Quality of life for handicappedpeople, Croom Helm, London. Brown, R.I., Bayer, M.B. and MacFarlane, C.M. (1989). Rehabilitation Programmes: The

Performance and Quality of Life of Adults with Developmental Handicaps, Lugus Productions Ltd, Toronto.

Campbell, A., Converse, P.E. and Rodgers, W.L. (1976). The Quality of American Life: Percep- tions, Evaluation and Satisfactions, Russell Sage Foundation, New York.

Close, D.W. (1977). ‘Community living for severely and profoundly retarded adults: a group home study’, Education and Training ofthe Mentally Retarded, 12,256-262.

Conroy, J.W. and Bradley, V.J. (1985). The Pennhurst Longitudinal Study: A Report of Five Years Research and Analysis, Temple University Developmental Disabilities Center, Phila- delphia.

Conroy, J.W. and Feinstein, C. (1986). The Choice Making Scale, Conroy and Feinstein Associates, Philadelphia.

Cummins, R.A. (1992). Comprehensive Quality of Life Scale-Intellectual Disability (3rd edn), Psychology Research Centre, Melbourne.

de Kock, U., Saxby, H., Thomas, M. and Felce, D. (1988). ‘Community and family contact: an evaluation of small community homes for adults’, Mental Handicap Research, 1,127-140.

Emerson, E.B. (1985). ‘Evaluating the impact of deinstitutionalisation on the lives of mentally retarded people’, American Journal of Mental Deficiency, 90,277-288.

Felce, D. (1989). Stafed Housing for Adults with Severe and Profound Mental Handicaps: The Andover Project, BIMH Publications, Kidderminster.

Felce, D. and Perry, J. (in press). ‘Quality of life: Its definition and measurement’, Research in Developmental Disabilities.

Felce, D., de Kock, U., Thomas, M. and Saxby, H. (1986). ‘Change in adaptive behaviour of severely and profoundly mentally handicapped adults in different residential settings’, British Journal of Psychology, 77,489-501.

18

Firth, H. and Short, D. (1987). ‘A move from hospital to community: evaluation of community contacts’, Child: care, health and development, 13, 341-354.

Flynn, R.J. (1980). ‘Normalisation, PASS and service quality assessment. How normalking are current human services?’ in R.J. Flynn and K.E. Nitsch (eds) Normalisation, Social Integration & Community Services, University Park Press, Baltimore.

Flynn, R.J. and Heal, L.W. (1980). ‘A short form of PASS 3 for assessing normalisation: Structure, interrater reliability, and validity’, in R.J. Flynn and K.E. Nitsch (eds) Normalisa- tion, Social Integration & Community Services, University Park Press, Baltimore.

J . Perry and D. Felce

Goffman, E. (1961). Asylums, Doubleday, New York. Hatton, C. and Emerson, E. (in press). ‘Deinstitutionalisation in the UK: impact on service

users’, in J. Mansell and K. Ericsson (eds) Proceedings of the International Conference on the Dissolution of Institutions and the Development of Community Services, April, 1993. Chapman and Hall, London.

Heal, L.W. and Chadsey-Rusch, J. (1985). ‘The Lifestyle Satisfaction Scale (LSS): assessing individuals’ satisfaction with residence, community setting and associated services’, Applied Research in Mental Retardation, 6,475490.

Hemming, H., Lavender, T. and Pill, R. (1981). ‘Quality of life of mentally retarded adults transferred from large institutions to new small units’, American Journal of Mental Dej- ciency, 86, 157-169.

Jackson, J. (1 964). Toward the comparative study of mental hospitals: Characteristics of the treatment environment. In A. F. Wesson (ed) The psychiatric hospital as a social system Charles C. Thomas, Springfield, IL.

King, R., Raynes, N. and Tizard, J. (1971). Patterns ofresidential care. Routledge and Kegan- Paul, London.

Landesman, S. ( I 986). ‘Quality of life and personal life satisfaction: definition and measurement issues’, Mental Retardation, 24,141-143.

Lowe, K. and de Paiva, S. (1991). NIMROD: An overview. HMSO, London. Lowe, K., de Paiva, S. and Felce, D. (1993). ‘Effects of a community-based service on adaptive

and maladaptive behaviours: a longitudinal study’, Journal of Intellectual Disability Research, 37, 3-22.

McLain, R.E., Silverstein, A.B., Hubbell, M. and Brownlee, L. (1977). ‘Comparison of the residential environment of a state hospital for retarded clients with those of various types

es’, Journal of Community Psychology, 5,282-289. Moos, R. (1974). The Social Climate Scales: An Overview. Consulting Psychologists Press,

Palo Alto. Moos, R. (1988). Community-Oriented Programs Environment Scale Manual (2nd edn), Con-

sulting Psychologists Press, Palo Alto. Nihira, K., Foster, R., Shellhaas, M. and Leland, H. (1974). A A M D Adaptive Behavior Scale,

American Association on Mental Deficiency, Washington D.C. Parmenter, T.R. (1988). An analysis of the dimensions of quality of life for people with

physical disabilities. In R.I. Brown (ed) Quality of life for handicappedpeople, Croom Helm, London.

Pratt, M.W., Luszcz, M.A. and Brown, M.E. (1980). ‘Measuring the dimensions of the quality of care in small community residences’, American Journal of Mental Deficiency, 85,188-194.

Raynes, N.V. (1988). Annotated Directory of Measures of Environmental Quality for use in Residential Services for People with a Mental Handicap, The University Department of Social Policy and Social Work, Manchester.

Raynes, N.V. and Sumpton, R.C. (1986). ‘The index of adult autonomy’, in Raynes, N.V., Wright, K., Shiell, A. and Pettipher, C. (1994). The Cost and Quality of Community Residen- tial Care, David Fulton Publishers Ltd, London.

Raynes, N.V.. Pratt, M.W. and Roses, S. (1979). Organisational Structure and the Care of the Mentally Retarded, Croom Helm, London.

Raynes, N.V., Sumpton, R.C. and Pettipher, C. (1989a). The Index of Community Involvement, The University Department of Social Policy and Social Work, Manchester.

Raynes, N.V., Sumpton, R.C. and Pettipher, C. (1989b). The Index ofParticipation in Domestic Life, The University Department of Social Policy and Social Work, Manchester.


Raynes, N. V., Wright, K. , Shiell, A. and Pettipher, C. (1994). The Cost and Quality of Com- munity Residential Care, David Fulton Publishers Ltd, London.

Rotegard, L.L., Bruininks, R.H. and Hill, B.K. (1981). Environmental Characteristics of Resi- dential Facilities for Mentally Retarded People, University of Minnesota, Minneapolis, MN.

Schalock, R.L. (1990). Quality of Life: Perspectives and Issues, American Association on Mental Retardation, Washington D.C.

Schalock, R.L.,Keith, K.D. and Hoffman, K. (1990). Quality of Life Questionnaire; Standardi- zation Manual. Mid-Nebraska Mental Retardation Services, Hastings, NE.

Schroeder, S. and Henes, C. (1978). ‘Assessment of progress of institutionalized and deinstitu- tionalized retarded adults: A matched-control comparison’. Mental Retardation, 16, 147- 148.

Siegal, S. (1956). Non Parametric Statistics: For the Behavioural Sciences, McGraw-Hill Koga- kusha, Tokyo.

Tizard, J. (1964). Community services for the mentally handicapped, Oxford University Press, London.

Wolfensberger, W. (1972). Normalisation: The Principle of Normalisation in Human Services, National Institute of Mental Retardation, Toronto.

Wolfensberger, W. and Glenn, L. (1975). Program analysis of service systems: Handbook and Manual (3rd edn), National Institute on Mental Retardation, Toronto.

Wolfensberger, W. (1983). ‘New PASS subscores’, in Guidelines for Evaluators During a PASS, PASSING, or Similar Assessment of Human Service Quality, National Institute on Mental Retardation, Toronto.

objective assessments of quality of life: how much do they agree with each other?

Documents