development to measure symptoms depression general...

Journal ofEpidemiology and Community Health 1997;51:549-557 549

Development of a scale to measure symptoms ofanxiety and depression in the general UKpopulation: the psychiatric symptom frequencyscale

Malin Lindelow, Rebecca Hardy, Bryan Rodgers

AbstractObjectives-The psychiatric symptomfrequency (PSF) scale was developed toassess symptoms of anxiety and depres-sion (ie affective symptoms) experiencedover the past year in the general popula-tion This study aimed to examine the dis-tribution of PSF scores, internalconsistency, and factor structure and toinvestigate relationships between totalscores for this scale and other indicatorsofpoor mental health.Participants-The Medical ResearchCouncil national survey of health anddevelopment, a class stratified cohortstudy ofmen and women followed up frombirth in 1946, with the most recentinterview at age 43 when the PSF scale wasadministered.Main results-The PSF scale showed highinternal consistency between the 18 items(Cronbach's a=0.88). Ratings on items ofthe scale reflected one predominant fac-tor, incorporating both depression andanxiety, and two additional factors of lessstatistical importance, one reflecting sleepproblems and the other panic and situ-ational anxiety. Total scores were calcu-lated by adding 18 items of the scale, andhigh total scores were found to be stronglyassociated with reports of contact with adoctor or other health professional anduse ofprescribed medication for "nervousor emotional trouble or depression," andwith suicidal ideas.Conclusions-The PSF is a useful andvalid scale for evaluating affective symp-toms in the general population. It isappropriate for administration by layinterviewers with minimal training, isrelatively brief, and generates few missingdata. The total score is a flexible measurewhich can be used in continuous or binaryform to suit the purposes of individualinvestigations, and provides discrimina-tion at lower as well as upper levels ofsymptom severity.

(J Epidemiol Community Health 1997;51:549-557)

Depressive and anxiety disorders together con-stitute a considerable proportion of all psychi-atric disorders in adults. The United Statesnational comorbidity survey estimated the 12

month prevalence of depressive disorders at1 1.3% and that of anxiety disorders at 17.2 %.1There is substantial comorbidity of these twogroups of disorders in clinical and generalpopulation samples2" and, particularly for lesssevere disorders, it appears arbitrary to attemptto draw a distinction. The terms "commonmental disorders"5 and "neurotic disorders"6have been applied to this broader category, ashas "affective disorders",7 although usage ofthis last term in the United States typicallyexcludes anxiety disorders.8 The above preva-lence estimates do not include the manyindividuals who experience symptoms that fallshort of criteria for formal diagnoses. In theOPCS surveys of psychiatric morbidity inGreat Britain,6 in which the clinical interviewschedule was used, a category of mixed anxietyand depressive disorder was defined. This cov-ered symptoms of anxiety and depressionwhich did not meet criteria for psychosis,depressive disorder, generalised anxiety disor-der, phobias, obsessive-compulsive disorder, orpanic disorder. The one week prevalence forthis mixed category was 77 per thousand out ofa total of 160 per thousand for all neurotic dis-orders. Overall, patients with affective symp-toms account for considerable use of services,especially primary care,9 and a substantial pro-portion of time off work,'0 not to mention thecontribution these symptoms make to socialrole handicaps and personal distress.

This subject is therefore of major impor-tance in health services and epidemiologicalresearch. However, investigating affective dis-orders, as for psychiatric disorders in general, ishampered by difficulties in measurement.Clinical diagnostic evaluation is time consum-ing and costly, and impractical for large scalecommunity studies. This has led to thedevelopment of alternative approaches, includ-ing self completion inventories, such as theCenter for Epidemiologic Studies depressionscale (CES-D)" and the general health ques-tionnaire (GHQ),"2 13 and more extensivestructured interviews such as the present stateexamination (PSE)'4 15 and the compositeinternational diagnostic interview (CIDI). '6There are now many instruments, but selectingone for a particular investigation is notnecessarily a simple process. They vary in manyrespects, including the time and resourcesneeded for training interviewers, the time andcost of administration, the necessity for main-

MRC National Surveyof Health andDevelopment,University CollegeLondon MedicalSchool, Department ofEpidemiology andPublic Health, 1-19Torrington Place,London WC1E 6BT.M LindelowR Hardy

NH&MRC SocialPsychiatry ResearchUnit, The AustralianNational University,B Rodgers

Correspondence to:Dr R Hardy.

Accepted for publicationJanuary 1997

on 23 April 2018 by guest. P

rotected by copyright.http://jech.bm

j.com/

J Epidem

iol Com

munity H

ealth: first published as 10.1136/jech.51.5.549 on 1 October 1997. D

ownloaded from

http://jech.bmj.com/

Lindelow, Hardy, Rodgers

taining standards during fieldwork, the de-mands placed on respondents, the range ofsymptoms included, the time frame of assess-ment, the dimensions of severity encompassed,the provision or not of subscales or diagnosticsubgrouping, the distribution of total scores(for continuous measures), and the thresholdfor identifying disorders (for discrete meas-ures). The advantages and disadvantages ofindividual methods of evaluation must be con-sidered in the context of any given study'sresources, scope, and objectives

In planning the 43 years of age (1989) followup of the Medical Research Council's nationalsurvey of health and development (NSHD), forwhich assessment of mental health comprisedonly a part of the full interview, such issues hadto be addressed in choosing a method for theevaluation of affective disorders. The NSHD isa prospective longitudinal study of 5362 menand women born in England, Scotland, andWales during the week 3-9 March 1946.17 Alllegitimate, single births from non-manual andagricultural backgrounds, and one quarter ofchildren from manual class backgrounds wereselected for inclusion in the study. Data havebeen collected at regular intervals, at least everytwo years during childhood and less frequentlyduring adulthood, with the previous collectionhaving been in 1982 when the survey memberswere 36 years of age.The short form of the PSE had been used in

the 36 year follow up, but concerns regardingits suitability had arisen. From a practical pointof view, the considerable time required, bothfor training interviewers and administrationduring fieldwork, made the PSE expensive,while also limiting the time available for assess-ing other aspects of psychological functioning.There was a tendency for interviewers' ratingsto 'drift' over the course of the data collection'8and although audiotaping a proportion ofinterviews helped to maintain standards, thiswas both time consuming and more intrusivethan wished. Furthermore, the very skeweddistribution of total PSE scores, with manysubjects scoring zero, was a particular disad-vantage in a longitudinal study such as theNSHD where differences in symptom levels inthe subclinical range may be important forpredicting future symptoms of greater severityand where the study of well being (in contrastto 'ill being') may be of interest. Finally, thediagnostic classification given by the PSE didnot prove useful in that few variables (past orcontemporary) were found to discriminatebetween diagnostic groups.'9 20These concerns led to the specification of

particular requirements for the instrument tobe used at the 1989 data collection as follows.* In addition to the obvious necessity for

internal reliability, face validity, and predic-tive validity with respect to external criteria,it was necessary that items should first coverthe symptoms indicative of both depressiveand anxiety disorders.

0 The time frame for assessment should besufficiently long to avoid high severity ratings

arising from brief recent episodes, but not so

long as to introduce problems of recollec-tion.

* The format of the schedule should besuitable either for administration by layinterviewers or for self completion.

* The need for training of interviewers or forinstruction of subjects should be kept to a

minimum.

* The time for administration/completionshould be short, preferably less than 10 min-utes.

* Resources for monitoring the quality ofassessment during fieldwork should be mini-mal, and any methods of quality controlshould not interfere with assessment.

* The schedule should provide a scale whichgives discrimination at the lower, as well as

the upper, end of its range. Its distributionshould be suitable for the application of dif-ferent cut off points identifying high scoringindividuals, and for studying psychologicalwell being as a continuum.When the data collection was planned in

1988, there was no single instrument that metall these criteria, so a new scale was developed(by BR) to incorporate all seven desirablecharacteristics listed above. Of the existinginstruments, the psychiatric epidemiology re-

search interview (PERI)2' perhaps came theclosest to the specifications. However, the itemsincluded in the PERI did not have a close cor-

respondence to symptoms contained in stand-ard classifications of the relevant psychiatricdisorders. There were additional difficultiesarising from the scales being developed for theUnited States population which meant thatmany items would have required rewording.Other possible candidates included the CES-Dscale," but this typically assesses symptomsover just one week. The various forms of thegeneral health questionnaire22 were consideredinappropriate because of the unspecified pe-riod of assessment and because the responseframe calls for comparison with how individu-als usually are or usually feel. A number ofsymptom check lists, such as the Malaisescale,23 were problematic because items scoredonly as zero or one (ie absent or present), whenthey have high thresholds and are substantiallyintercorrelated, will generate a large proportionof zero values for total scores and yield poordiscrimination at lower severity levels.

KEY POINTS* The psychiatric symptom frequency

(PSF) scale is reliable and valid for evalu-ating symptoms in the general popula-tion.

* It is suitable for use by lay interviewerstakes little time, and generates fewmissing values.

* It provides discrimination at the lower, aswell as the higher, end of the scale.

* ROC analysis is a valuable means ofquantifying the validity of symptomscales.

550



j.com/

J Epidem

iol Com

munity H


ownloaded from


The psychiatric symptom frequency scale

This paper describes the development of thenew instrument in the context of the NSHD,highlighting the characteristics required. Theaim of the analysis is to examine the distribu-tion, internal consistency and factor structureof the PSF scale, as well as its relationships withsuicidal ideation, and with use of prescribedmedication and contact with a doctor for'nervous or emotional trouble or depression'.

MethodsINSTRUMENT DEVELOPMENTPossible items were selected from the shortform of the PSE, used previously in the study,'8and from other instruments covering symp-toms of unipolar depressive disorders, agora-phobia, social phobia, panic disorder, and gen-eralised anxiety disorder. Items from theNSHD 36 year study that had contributed leastto the discrimination of those seeking treat-ment included worrying, tension pains, freefloating anxiety, specific phobias, anxietyavoidance, social withdrawal, self depreciation,lack of self confidence, ideas of reference, guiltyideas of reference, pathological guilt, expansivemood, ideomotor pressure, grandioseideas/actions, obsessional checking/repeating,obsessional cleanliness/rituals, obsessionalideas/rumination, derealisation, and deperson-alisation. A number of these symptoms arecomparatively rare in the general population.'8The least discriminating items were omitted.Somatic items were excluded because of fearsthat they could contribute to age differences ifused in future follow ups as the inclusion ofsomatic items has been the subject of concernfor other instruments.22 24 25 Crying (or tearfulness) was also excluded because of the particu-larly strong gender difference for such an itemin adult samples.Wording of items followed the principles for

the study's interview as a whole, being as sim-ple as possible. In some instances, PSE itemswere suitable or required only minor modifica-tions, eg "Have you felt on edge or keyed up ormentally tense?" When PSE wording was inap-propriate, items from other instruments wereadapted. In order to maximise the informationobtained from relatively few questions, it wasdecided to utilise multiple response categoriesor a rating scale for each item. A number ofdimensions of severity were considered, includ-ing: frequency of onset, duration, subjectiveintensity and degree of associated disability.Frequency and duration appeared the mostappropriate, and permitted the use of acommon response frame for all items.A period of assessment of at least a few

months was thought necessary in order tominimise the impact of brief recent episodes. Along time period would also reduce theproportion of subjects who had never experi-enced a particular symptom, and thereforeminimise the number scoring zero on the fullscale. Although accurate recollection over oneyear may be problematic, the choice of 12months had the advantage of giving a simplereference point for the start of the period, whilealso limiting possible seasonal variations ininterviews conducted at different points in the

year. This one year time frame was alsoconsistent with other sections of the question-naire requiring retrospective information.

PILOT STUDYA pilot study for the 1989 interview was carriedout using a sample close in age to the NSHDcohort, identified from GP registers in twogeographical areas of England. Interviews wereconducted by NSHD scientific staff in therespondents' homes. A trial version of the PSFscale was included in the study, with subjectsasked 'how often' they had felt or experiencedeach symptom over the last year and given ananswer card to indicate responses on a six pointscale. These responses and their respective rat-ings were never (0), occasionally (1), some-times (2), quite often (3), very often (4), andalways (5). Problems with the questionnairewere identified not only from routine adminis-tration, but also by asking subjects at the end ofeach interview about their reaction to andinterpretation of specific questions, and bydouble checking responses given during theinterview and requesting elaboration on these.One difficulty with the symptom ratings wasidentified very early in the pilot phases.Specifically, there were large individual differ-ences in the meaning attached to the terms onthe answer card used to indicate frequency.The same objective frequency of symptomscould span as many as three points on the sub-jective scale and, consequently, the instrumentwas modified so that open-ended responseswere elicited without use of an answer card.These responses were collated and used toconstruct a rating scale for interviewers as fol-lows:* 0 = never in the last year.* 1 = up to 10 days in total, less than once a

month.* 2 = a spell up to one month, once or twice a

month, 'a months worth'.* 3 = a spell up to four months, once or twice

a week, three to ten times a month.* 4 = a spell of over four months, three or more

times a week, 11 or more times a month.* 5 = every day in the last year.This scale reflected the tendency for spontane-ous replies to be given either in the form of anepisode or a rate (typically weekly or monthly),but did not distinguish between these twomodes of responding.The pilot interview also led to some modifi-

cations to the wording of items. One item cov-ering difficulties in making decisions wasdeleted because it was often construed by sub-jects as a personality characteristic independ-ent of psychological distress or disorder. Thisleft 18 items plus one set of questions relatingto suicidal ideation and acts that were reformu-lated into a composite Guttman type scale,ranging from never feeling that life was hardlyworth living in the last year through toattempting to take one's own life, still yieldinga 0 to 5 rating, but forsaking responding interms offrequency of experience. The resultantquestions, shown in the Appendix, were used inthe interviews conducted with the NSHD

551 on 23 A

pril 2018 by guest. Protected by copyright.

http://jech.bmj.com

/J E

pidemiol C

omm

unity Health: first published as 10.1136/jech.51.5.549 on 1 O

ctober 1997. Dow

nloaded from



sample, with interviewers rating open-endedresponses on symptom frequency.

MAIN STUDY: SAMPLE AND DATA COLLECTIONThe interviews at which the PSF was adminis-tered were conducted by trained nurses during1989-1990 when survey members were 43years of age. At that time it was known that 365(6.8%) of the original cohort of surveymembers had died, and 618 (11.5%) had leftthe country. Of those thought to be still livingin Britain 3262 (74.5%) were successfullyinterviewed. Of those not interviewed, 646(14.8%) had refused to participate further inthe study, 276 (6.3%) were untraced, and 195(4.5%) were not willing to be interviewed onthis particular occasion. However, the popula-tion interviewed at the age of 43 years were, inmost respects, representative of the native bornpopulation of that age.26 Certain small groupsof individuals are, however, under-represented,specifically those with poor literacy skills,learning disabilities, and serious psychiatricdisorders.The total score on the PSF scale was calcu-

lated by adding the scores of the first 18 itemsof the instrument, excluding the questions onsuicidal ideation. Most subjects interviewed(3196/3262=98%) provided a response forevery item, thus suggesting that the PSF has agood completion rate. The group of individualswith at least one missing item were no differentto those with no missing items in terms of gen-der, socioeconomic status, and educationalachievement. For subjects missing only one ortwo of the contributing items, total PSF scoreswere estimated using the technique of equiva-lent centile points. As a consequence, only 20subjects, with more than two missing items,had to be excluded from the analysis due tomissing data on the PSF.The item referring to suicidal ideation and

acts was dichotomised to contrast individualswho reported neither (at most stating that atsome point over the past year they had felt thatlife was hardly worth living) with those whoadmitted having thought that they would bebetter off dead, thought of taking their own life,made plans to take their own life, or attemptedto take their own life. Only a small proportionof the respondents (4.1%) fell into this lattergroup.

In a prior part of the interview, survey mem-bers were asked about their service contact andtreatment histories regarding a wide range ofmedical complaints experienced over the pastyear. Within this context they were asked iftheyhad seen a doctor or other health professionalfor 'nervous or emotional trouble or depres-sion'. Survey members who responded af-firmatively to this were contrasted with thosewho denied such contact, independently of thenumber of consultations that had occurred.Similarly, information regarding the use of pre-scribed medication for 'nervous or emotionaltrouble or depression' was requested. Again,those reporting such use were contrasted withthose giving negative answers.

In the NSHD population, 5.8% reportedcontact with a doctor or other health profes-

sional, and 4.3% reported that they were takingprescribed medication. As expected, almost allof those taking such medication also reportedhaving seen a doctor or other health profes-sional. The rate for reported contacts with ahealth professional is, as expected, lower thanreported prevalence rates for affective disor-ders. However, the rates for service use foundin the NSHD correspond well with those foundin the OPCS Morbidity Statistics from GeneralPractice,9 where using their findings for annualprevalence of 'neurotic disorders' diagnosed invisits to general practices, a rate of 4.3% in apopulation the age of the NSHD would beexpected.

Exclusion from all analyses of those surveymembers with missing values for either theseadditional variables or the PSF scale resulted ina final sample size of 3058 (94% of those inter-viewed).

STATISTICAL METHODSStatistical analysis was undertaken in twosteps. First the internal consistency and factorstructure of the scale were examined usingCronbach's a and factor analysis. Second thecriterion-related validity of the scale was exam-ined by reference to contact with a doctor orother health professional, use of prescribedmedication, and suicidal ideation. Here, bothbinary and continuous versions of the PSFwere investigated, using X2 tests and receiveroperating characteristic (ROC) curves27 re-spectively.An ROC curve may be used to examine

whether any proposed diagnostic score, meas-ured on a continuous numerical scale, can dis-criminate between a "diseased" group and a"normal" group. Essentially, it examines thedegree of overlap of the distributions of thediagnostic score for groups defined by anyspecified criterion variable. In the presentstudy, no formal clinical diagnoses of "affectivedisorders" were made, so that a strict definitionof "diseased" and "normal" was not available.Such clinical diagnoses may, in any case, beconsidered too stringent for many epidemio-logical purposes and are themselves of ques-tionable validity. Hence, other criteria indica-tive of poor mental health were used. Theseindicators were whether an individual had beentaking prescribed medication for "nervous oremotional trouble or depression", had been incontact with a doctor or other health profes-sional for the same reason, or had reported sui-cidal ideation.For each point, or cut off, on the PSF scale,

subjects may be classified as scoring high orlow, ie scoring above or below that cut off.Individuals classified as high and who are alsopositive on a given criterion variable, for exam-ple those taking medication, are termed truepositives (correctly ascertained by the PSF),while those classified as hig,h and who arenegative on the criterion variable, that is thosenot taking medication, are termed false posi-tives (incorrectly ascertained by the PSF). TheROC curve itself is a plot of the true positiverate against the false positive rate for everypoint of the PSF scale, that is sensitivity plotted

552 on 23 A


http://jech.bmj.com

/J E

pidemiol C

omm


ctober 1997. Dow

nloaded from



1500rU,

100

50

0-0 5 10 15 20 25 30 35

PSF score

Figure 1 Frequency distribution of the lowest 95% of the psychiatric symptom frequency(PSF) score.

against one minus specificity. The degree ofoverlap, in terms of their PSF scores, betweenthe two groups defined by the criterionvariable, is measured by calculating the areaunder the curve (AUC). The AUC and itsassociated standard error were calculated usingthe non-parametric method of Hanley andMcNeil."8ROC curves can also be used to identify cut

off points for dichotomising continuous scales.Clearly there can be no unequivocal solution,as any dichotomy is arbitrary, and considera-tion must be given to costs and benefits of mis-classification in the specific clinical or researchsituations in which the dichotomous variable isused. A cut off derived from the ROC curvewas compared with a cut offbetween the scoresof 22 and 23, identifying 12.9% of the sampleas scoring above the threshold, that has beenused in previous analyses.29 30

ResultsDISTRIBUTION OF TOTAL PSF SCOREThe distribution of the total PSF score,excluding the highest 5%, is shown in figure 1.Fewer than 10% of survey members scored 0,with the remaining scores ranging up to 90.95% of scores were equal to or less than 35,thus giving the distribution a long right-handtail.

INTERNAL CONSISTENCYThe internal consistency of the scale, asindicated by Cronbach"s a was relatively high(0.88). Furthermore, values for a with eachitem deleted in turn showed that eliminatingany one of the 18 items from the scale resultedin little change in the value of a. This suggeststhat none of the items detracted from the reli-ability of the overall scale.

FACTOR ANALYSISExploratory factor analysis of the first 18 itemswas undertaken, using principal componentanalysis. Three factors with eigenvalues largerthan 1.0 were extracted, and a varimax rotation

performed. This gave one predominant factor(eigenvalue 6.12), accounting for 34% of thevariance, and two subsidiary factors whichaccounted for less than 7% of the varianceeach. All except four items were strongly asso-ciated with the first rotated factor, making it anoverall factor incorporating symptoms of bothdepression and anxiety. The other four itemswere linked to the second and third factors.One of these factors was concerned with twoitems reflecting sleep disturbance and the otherwith two items covering panic and situationalanxiety, and both arose from the relatively highcorrelations between each pair of items (0.56and 0.54 respectively). When analysis wasrestricted to the first (unrotated) factor, all 18items showed factor loadings in the range 0.43to 0.71.

ROC ANALYSES PREDICTING SERVICE USE AND

SUICIDAL IDEATIONThe ROC curve for the PSF score with use ofmedication as the criterion variable is shown infigure 2. The diagonal line represents thehypothetical case of no separation between thetwo distributions, and can be referred to as theline of no information. Hence, the further theROC curve lies from this line of no informa-tion, the better the performance of the scale indiscriminating between the criterion groups.Hence, in the case considered here, there iscertainly some degree of discrimination by thePSF scale (fig 2).The AUC for a random ROC curve is 0.5,

whereas a perfect scale, completely separatingthe two groups, gives an AUC of 1.0. Hence,the relatively large AUCs for all three criterionvariables considered here (table 1) indicate thatthe PSF score distinguishes significantly be-tween the two groups of interest. Subjects withhigh scores on the PSF were more likely to havesought professional help in relation to emo-tional problems, to be prescribed medicationand to have had suicidal feelings. ROC curveswere then produced and AUCs calculated formen and women separately for each of thethree criterion variables. The null hypothesisthat the AUC for men was equal to the AUC

0.8- | B (low cut off)

> 0.6 ----A (high cut off) ,

CDCl) 0.4

0 0.2 0.4 0.6 0.81 - Specificity

Figure 2 Receiver operating characteristic (ROC) curvefor the PSF score where the diagnostic criterion is use ofprescribed medication.

553 on 23 A


http://jech.bmj.com

/J E

pidemiol C

omm


ctober 1997. Dow

nloaded from



Table 1 Results from receiver operating characteristic (ROC) analysis

Cut off maximising specificityCriterion variable Area under curve (SE) and sensitivity

Contact with a doctor 0.84 (0.016) 13/14Use of prescribed medication 0.86 (0.014) 13/14Suicidal ideation 0.92 (0.012) 19/20

Table 2 Percentage of those with low and high psychiatric symptom frequency (PSF)scores reporting contact with a doctor, use ofprescribed medication, and suicidal ideation(PSF cut off between 22 and 23)

Low PSF score High PSF scoreCriterion variable (n=2694) (n=364) p value (Z2 test)

Contact with a doctor 2.7% (n=74) 28.0% (n=102) 0.000Use of prescribed

medication 2.1% (n=56) 21.2% (n=77) 0.000Suicidal ideation 1.0% (n=27) 25.5% (n=93) 0.000

for women was tested in each case. Resultsindicate that for both service use criterion vari-ables, the PSF scale performs equally well foreach gender. For suicidal ideation, however, theresult of the test is significant at the 5% levelwith the AUC for men (AUC=0.95,SE-0.013) being greater than that for women(AUC=0.90, SE=0.018).The cut off used in previous analyses, that is

between 22 and 23, was applied to the 18-itemscale, shown as point A in figure 2. Although inthe past this cut off had been applied to a 19item scale with suicidal ideation and actsincluded, the low prevalence of positive re-sponses to this item meant that its omissioncaused minimal alteration to the distribution oftotal scores. In respect of identifying subjectswho have taken prescribed medication, this hadhigh specificity (91%), but relatively low sensi-tivity (58%). Alternatively, the point on theROC curve nearest to the top left-hand corner(point B) is the one giving the best balance ofspecificity and sensitivity. In table 1, this isreferred to as the cut off maximising specificityand sensitivity and is between scores of 13 and14. The same cut off (13/14) was found forcontact with a doctor or other health profes-sional as for use of prescribed medication, andthis classified 28.5% of the total sample as highscorers. In subsequent binary analyses, the twocut offs of 13/14 and 22/23 were compared.

BINARY ANALYSES USING SERVICE CONTACT AND

SUICIDAL IDEATION AS CRITERIAThe associations between binary versions ofthe PSF and respondents" reports of servicecontact for "nervous or emotional trouble ordepression" and suicidal ideation were investi-gated, applying X' tests of significance. Thisconfirmed that reports of health servicecontact, use of prescribed medication, and sui-cidal ideation were all strongly related to highPSF scores. For the higher cut off, typically a

Table 3 Percentage of those with low and high psychiatric symptom frequency (PSF) scoresreporting contact with a doctor, use ofprescribed medicine, and suicidal ideation (PSF iscut off between 13 and 14)

Low PSF score High PSF scoreCriterion variable (n=2187) (n=871) p value(/2 test)

Contact with a doctor 1.4% (n=31) 16.6% (n=145) 0.000Use of prescribed medication 1.1% (n=25) 12.4% (n=108) 0.000Suicidal ideation 0.6% (n=12) 12.4% (n=108) 0.000

quarter of those with high PSF scores reportedeach of having had contact with a doctor orother health professional, having used pre-scribed medication, or having experienced sui-cidal ideation over the past year, compared tounder 3% of those with low scores (table 2).For the lower cut off, only about 1% of thosewith low scores reported affirmatively on crite-rion variables, compared with 12-17% of thosewith high scores (table 3). In all cases thesecomparisons were highly significant.

DiscussionThe advantages and disadvantages of the PSFscale for epidemiological research are dis-cussed here in relation to three aspects of theinstrument:* The content and format of the scale,* Its psychometric properties, including inter-

correlations of items and associations withexternal criterion variables and

* The utility of resulting measures, both indiscrete and continuous forms.

This will act as a guide to the suitability of thescale for particular research purposes and tothe feasibility of incorporating components ofthe general approach into the development ofnew instruments.

CONTENT AND FORMATThe items of the scale cover many of the symp-toms of the target anxiety and depressivedisorders.8 " The most prominent exclusion, ofsomatic symptoms which could arise fromphysical illnesses, should be advantageous forsamples that include the physically ill, the eld-erly or those experiencing physical stresses.32The inclusion of items covering sleep problemscould introduce difficulties with physically ill orelderly subjects. Sleep problems do appear, infact, to hold up as indicators of depression inelderly subjects,33 but their prevalence is alsoknown to rise with age.9 In some circumstancesit may be advisable to omit these items from thescoring of the instrument or to contrastfindings with them included or not. The omis-sion of items on self worth may be a weaknessin certain contexts, although one that could beaddressed if necessary. Since the developmentof the PSF, a report from the epidemiologiccatchment area (ECA) programme has identi-fied "worthlessness or guilt" and "trouble con-centrating or thinking" as the two symptomgroups most predictive of the onset of majordepression.34 However, these analyses wereconducted for subjects who had no lifetimehistory of major affective disorder at the initialinterview when they reported these symptoms,and their implications for the prediction ofcontemporary disorder are not clear. Else-where, data from the same study have indicatedthat "feelings of worthlessness" are relativelyuncommon amongst subjects who currentlymeet criteria for major depressive disorder,35indicating poor sensitivity. It is notable, too,that the symptoms of anxiety and depression(SAD) scale, derived by the selection of itemsdiscriminating patient and normal groups,36contains no items relating to self worth. Thereare some additional minor peculiarities reflect-

554 on 23 A


http://jech.bmj.com

/J E

pidemiol C

omm


ctober 1997. Dow

nloaded from



ing the roots of the PSF scale. For example,enclosed space is mentioned in the item cover-ing situational anxiety even though fear ofenclosed spaces is usually classified with simplephobias.The instrument satisfied the practical and

administrative requirements laid down in theintroduction. The format of the scale, requiringrating of open-ended responses is straightfor-ward to carry out, although inappropriate forself-completion, and requires minimal trainingand monitoring of interviewers. Interviewersneed to be able to cope with queries or condi-tional responses, for example not to includesleep problems resulting from noisy neigh-bours, and also to elicit and aggregate re-

sponses when subjects have experienced bothacute episodes and periodic symptomatology.In general, the approach of rating the fre-quency of symptoms, even over a 12 monthperiod, appears to be satisfactory. The use ofthe 12 month assessment period is well suitedto studies of chronic adversity, risk associatedwith more distant past events and circum-stances (eg childhood adversity), and changesin symptomatology associated with long-termtransitions, as identified by demographic or

socioeconomic factors. It would be inappropri-ate for studies wishing to investigate short termchanges in symptomatology, such as those fol-lowing life events. For the latter type of study, itwould be possible to adapt the PSF to inquireof symptoms over a shorter period and toamend the rating scale for responses. The fullscale, including questions on suicidal ideationand acts, is relatively brief, giving an averageadministration time of about seven minutes.This is particularly useful for studies withbroad interests, where limited time is availablefor assessing mental health.

PSYCHOMETRIC PROPERTIES

The PSF scale has high internal consistency (a= 0.88), thus indicating good reliability. Thiscompares well with other similar scales, such as

the GHQ-30 which was found to have an a

value of 0.9,37 the CES-D with a value of 0.85,"and the Malaise with a value of 0.80.38 Such a

level of internal consistency, as well as indicat-ing good reliability, can also be interpreted as

showing considerable redundancy of items.This is not, however, attributable to the inclu-sion of synonymous items, as care was taken toavoid such duplication. The implication of thisredundancy is that even briefer scales could bedeveloped with little loss of reliability and,hopefully, validity. It is inevitable, however, thatreducing the number of items will increase theproportion of individuals scoring zero on thescale, and this would have some disadvantages.Two prominent specific examples of redun-dancy involved sleep problems and the associa-tion between situational anxiety and auto-nomic symptoms of panic. It is likely that thecorrelations between the relevant pairs ofitems, resulting in the second and third factorsof the principle component analysis, arise fromreports referring to the same occasions, that issubjective anxiety and panic occurred togetherfor many individuals, as did trouble getting off

to sleep and waking up early. Althoughdifficulty getting to sleep and early waking havebeen considered to have some diagnosticspecificity, it may be acceptable in the contextof measuring general affective symptomatologyto incorporate both symptoms into a singleitem. The failure to identify two separatefactors representing depression and anxiety isconsistent with other studies.2 3 Even whenseparate constructs have been indicated bymore sophisticated item-analysis techniques,the resultant subscales are highly correlatedand do not map perfectly onto the symptomconstellations contained in classificationsystems.39The ROC and contingency table analyses

using external criterion variables gave theunexceptional findings that PSF scores werestrongly associated with seeking help fromhealth professionals, being prescribed medica-tion and suicidal ideation and acts. Indeed, ascale professing to measure "affective disor-ders" would be expected to produce such asso-ciations and hence the results are encouragingin this respect. The scale was most successful indiscriminating between subjects with andwithout suicidal ideation, as indicated by thegreater area under the curve. Whereas this maybe a genuine reflection of reluctance to seekhelp by many individuals suffering fromsignificant psychopathology,40 41 it is also possi-ble that it is an effect of the methods wherebydifferent criterion variables were collected.That is, information about suicidal ideationwas obtained in conjunction with the PSFscale, whereas information about health servicecontact and use of prescribed medication fornervous or emotional trouble or depression wascollected at an earlier part of the interview,during which subjects were also asked abouttheir physical health. These circumstances maywell have influenced reporting.

In the absence of gold standards fordiagnoses in psychiatry, ROC analysis providesa valuable means of quantifying the validity ofsymptom scales. The incorporation of similarcriteria in other studies, or in investigationsusing more than one method of assessment,would permit direct comparisons of differentinstruments. The traditional difficulty of estab-lishing predictive validity in psychiatric epide-miology has led to an over-reliance on reliabil-ity and construct validity, and ROC analysisoffers an alternative approach. It is advisable toobtain several criterion variables in suchstudies, as any single outcome (such ashelp-seeking) may reflect unwanted biases, egpropensity to consult a doctor independent ofseverity of symptoms. In addition to the criteriaused in the present study, sickness absencefrom work and interference with social roleswould be suitable candidates.

UTILITY OF THE TOTAL PSF SCOREA very large proportion of subjects had nomore than two missing items on the scale(99.4%) and could therefore be ascribed a totalPSF score. The long upper tail of the distribu-tion of this total score is a desirable attribute,permitting the application of alternative or

555 on 23 A


http://jech.bmj.com

/J E

pidemiol C

omm


ctober 1997. Dow

nloaded from



multiple cut points to suit the aims of specificinvestigations. The results for binary forms ofthe PSF, using two different cut off points,illustrate how both may be useful in epidemio-logical research, but for different purposes.When using the higher cut off, a relatively largeproportion of subjects with high scores reportpositively on the criterion variables, although acertain number of those with health servicecontact, use of prescribed medication, and sui-cidal ideation still fall into the group with lowscores. This may therefore be appropriate forresearch where interest is in defining a popula-tion at high risk of clinical disorder. The lowercut off was successful in putting a greaternumber of those with affirmative answers onthe criterion variable in the high scoring group,and is therefore more appropriate as ascreening measure.The discrimination at the lower end of the

PSF scale, less than 10% of subjects scoredzero compared to almost half on the corre-sponding total PSE score obtained in 1982,18 isa prerequisite for the study of well being and isof particular use in longitudinal studies wheresymptoms at a subclinical level may be predic-tive of later disorder. The overall distribution oftotal scores remains very skewed, however, butthe relatively fine gradation of the scale allowstransformation to a more normal distribution ifappropriate for necessary analyses. This wouldstill leave truncation at the lower extremity, butthe fact that a significant number of individualsdo not report experiencing any of 19 symptomsat any time during a 12 month period suggeststhat further attempts to achieve greater dis-crimination could be futile. Either there is agroup of very resilient (or protected) individu-als in the general population or there is a reluc-tance to admit to affective symptoms.The PSF performed well in this nationally

representative sample of 43 year olds. Theresults of the ROC analyses were, overall, simi-lar for men and women. The differencebetween the AUCs for suicidal ideation,although being statistically significant, wassmall in terms of any practical implications asfor both sexes the discrimination of the PSFscale was high. However, since the NSHD is acohort of individuals who are all of the sameage, no conclusions can be drawn about howthe PSF operates across different age ranges.Furthermore, since there are very few individu-als in this sample with serious psychiatricdisorders, it is not known whether the scale issuitable for such a group and hence, studieslooking at the properties of the scale in clinicpopulations would be informative. In addition,there are other aspects of the performance ofthe scale, such as the test-retest and inter-raterreliability, which have not been assessed in thepresent study, but which could be consideredin future work.

In conclusion, the PSF scale is a useful addi-tion to the set of instruments available forevaluating affective symptoms in adults. It isparticularly suited to studies of general popula-tion samples where data collection is by layinterviewers. The PSF requires little time foradministration and yields a very small propor-

tion of missing values. The total score repre-sents one predominant dimension, consistentwith the construct of "common mental disor-ders" or "neurotic disorders", incorporatingsymptoms of anxiety and depression. Thisscore is extremely flexible and appropriate foruse as a continuous measure of affectivedistress or as a categorical outcome measure,using a variety of cut points, where high scoringindividuals are likely to have disorders of clini-cal significance. Its discrimination in the lowerrange is particularly valuable for longitudinalstudies of onset of disorder. The quantificationof discriminability by reference to severalexternal criterion variables provides the oppor-tunity for comparison with other possibleinstruments, and this general approach isrecommended as a means of assessing validityin preference to the traditional reliance onindices of reliability.

The authors wish to thank Professor Mike Wadsworth for help-ful comments on previous drafts of this paper.

Funding: the research was funded by the Medical ResearchCouncil.Conflicts of interest: none.

1 Kessler RC, McGonagle KA, Zhao S, et al. Lifetime and12-month prevalence of DSM-III-R psychiatric disordersin the United States. Arch Gen Psychiatry 1994;51:8-19.

2 Eaton WW, Ritter C. Distinguishing anxiety and depressionwith field survey data. Psychol Med 1988;18:155-66.

3 Feldman LA. Distinguishing depression and anxiety in self-report: Evidence from confirmatory factor analysis on non-clinical and clinical samples. Jf Consult Clin Psychol1993;61:631-38.

4 Hunt C, Andrews G. Comorbidity in the anxiety disorders:the use of a life-chart approach. J Psychiatric Res 1995;29:467-80.

5 Goldberg DP, Huxley P. Common mental disorders: a bio-socialmodel. London: Routledge, 1992.

6 Meltzer H, Gill B, Petticrew M, Hinds K. The prevalence ofpsychiatric morbidity among adults living in private households.OPCS surveys of psychiatric morbidity in Great BritainReport 1. London: OPCS Social Survey Division, 1995.

7 Paykel ES, ed. Handbook of affective disorders. 2nd ed. NewYork: Guilford Press, 1992.

8 American Psychiatric Association. Diagnostic and statisticalmanual of mental disorders. 4th ed, rev. Washington, DC:American Psychiatric Association, 1994.

9 McCormick A, Fleming D, Charlton J. Morbidity statisticsfrom general practice: fourth national study 1991-92. SeriesMB5 No 3. London: HMSO, 1995

10 Department of Health and Social Security. Digest of statisticsanalyzing certificates of incapacity June 1969-May 1970.London: HMSO, 1973.

11 Radloff LS. The CES-D scale: a self-report depression scalefor research in the general population. Applied PsychologicalMeasurement 1977; 1: 385-401.

12 Goldberg DP. The detection of psychiatric illness by question-naire. London: Oxford University Press, 1972.

13 Goldberg DP, Hillier VF. A scaled version of the generalhealth questionnaire. Psychol Med 1979; 9: 139-45.

14 Wing JK. Use and misuse of the PSE. BrJ Psychiatry 1983;143: 111-17.

15 Wing JK, Cooper JE, Sartorius N. Present State examination.London: Cambridge University Press, 1974.

16 World Health Organization. Composite International diagnos-tic interview. Geneva: WHO, 1995.

17 Wadsworth MEJ. The imprint of time: childhood, history andadult life. Oxford: Clarendon Press, 1991.

18 Rodgers B, Mann SA. The reliability and validity of PSEassessment by lay interviewers: a national populationsurvey. Psychol Med 1986; 16: 689-700.

19 Rodgers B. Behaviour and personality in childhood aspredictors of adult psychiatric disorder.Y Child Psychol Psy-chiatry 1990; 31: 393-414.

20 Rodgers B. Socio-economic status, employment and neuro-sis. Soc Psychiatry Psychiatr Epidemiol 199 1;26: 104-14.

21 Dohrenwend BP, Shrout PE, Egri G, Mendelsohn FS. Non-specific psychological distress and other dimensions of psy-chopathology. Arch Gen Psychiatry 1980; 37: 1229-36.

22 Goldberg DP, Williams P. A user"s guide to the general healthquestionnaire. Windsor: NFER-Nelson, 1988.

23 Rutter M, Tizard J, Whitmore K. Education, health andbehaviour. London: Longman, 1970.

24 Hirst MA. Evaluating the Malaise inventory: an item analy-sis. Soc Psychiatry 1983; 18: 181-84.

25 McGee R, Williams S, Silva PA. An evaluation of theMalaise inventory. _ Psychosom Res 1986; 10: 147-52.

556



j.com/

J Epidem

iol Com

munity H


ownloaded from


The psychiatric symptom frequenicy scale

26 Wadsworth MEJ, Mann SL, Rodgers B, Kuh DL, HilderWS, Yusuf EJ. Loss and representativeness in a 43 yearfollow-up of a national cohort. J Epidenmiol ConmmtutnityHealth 1992; 46: 300-4.

27 Hsiao JK, Bartko JJ, Potter WZ. Diagnosing diagnoses:receiver operating characteristic methods and psychiatry.Arch Gen Psychiatry 1989; 46: 664-67.

28 Hanley JA, McNeil BJ. The meaning and use of the areaunder a receiver operating characteristic (ROC) curve.Radiology 1982; 143: 29-36.

29 Rodgers B. Pathways between parental divorce and adultdepression. 7 Child Psy,chol Psychiatry 1994; 35: 1289-308.

30 Rodgers B. Reported parental behaviour and adult affectivesymptoms. 1. Associations and moderating factors. PsycholMed 1996;26:51-61.

31 World Health Organization. The ICD-1O classification ofnmental and behaviouiral disorders: diagnostic criteria forresearch. Geneva: World Health Organization, 1993.

32 Grant G, Nolan M, Ellis N. A reappraisal of the Malaiseinventory. Soc Psychiatry Psychiat Epideniol 1990;25:170-78.

33 Mackinnon A, Christensen H, Jorm AF, Henderson AS,Scott R, Korten AE. A latent trait analysis of an inventorydesigned to detect symptoms of anxiety and depressionusing an elderly community sample. Psychol Med 1994;24:977-86.

34 Dryman A, Eaton WW. Affective symptoms associated withthe onset of major depression in the community: findingsfrom the US National Institute of Mental Healthepidemiologic catchment area program. Acta PsychiatricaScandintavica 1991 ;84: 1-5.

35 Eaton WW, Dryman A, Sorenson A, McCutcheon A.DSM-III major depressive disorder in the community: Alatent class analysis of data from the NIMH Epidemiologiccatchment area programme. Br ] Psychiatry 1989;155:48-54.

36 Bedford A, Foulds GA, Sheffield BF. A new personal distur-bance scale (DSSI/SAD). Br J Soc Clin Psychol 1976;15:387-94.

37 Goodchild ME, Duncan-Jones P. Chronicity and thegeneral health questionnaire. Brit _7 Psychiat 1985;146:55-61.

38 Rodgers B, Power C, Hope S. Parental divorce and adultpsychological distress: evidence from a national birthcohort. In press.

39 Goldberg DP, Bridges K, Duncan-Jones P, Grayson D.Dimensions of neuroses seen in primary-care settings. Psy-cholMed 1987;17:461-70.

40 Henderson JG, Pollard CA, Jacobi KA, Merkel WT.Help-seeking patterns of community residents with depres-sive symptoms. Affect Disorders 1992; 26: 157-62.

41 Maier W, Lichterman D, Oehrlein A, Fickinger M. Depres-sion in the community: A comparison of treated and non-treated cases in two non-referred samples. Psychopharmacol1992;106 (Suppl):79-81.

AppendixTHE PSYCHIATRIC SYMPTOM FREQUENCY (PSF)SCALEI would like to get some idea about how you havebeen feeling about things over the last year. Howoften:-(1) have you felt on edge or keyed up or

mentally tense?(2) have you been in low spirits or felt miser-

able?

(3) have you felt particularly low or de-pressed first thing in the morning?

(4) have you had the feeling that somethingterrible might happen?

(5) have you had days when your thoughtswere muddled or slow?

(6) have you had no appetite, not countingperiods of physical illness?

(7) have you been in situations, such as in acrowd or an enclosed space or meetingpeople, when you have become undulyanxious?

(8) have you been in situations when you feltshaky or sweaty or your heart pounded oryou could not get your breath?

(9) have you had trouble getting off to sleep?(10) have you had trouble with waking up and

not being able to get back to sleep?(11) have you been frightened or worried

about becoming ill or dying?(12) have you felt fidgety or restless?(13) have you found it hard to concentrate on

things or found your thoughts drifting offto other things?

(14) have there been days when you tired outvery easily?

(15) have there been days when you found itdifficult to get things done or had troublegetting started on things?

(16) have you had the feeling that the futuredoes not hold much for you?

(17) have you been so caught up in your ownthoughts that you neglected things?

(18) have you seemed to lose interest inthings?

Additional item concerned with suicidal ideationand actsIn the last year have you ever:-(i) felt that life is hardly worth living? No... 0(ii) thought that you really would be better off

dead? No... 1(iii) thought about taking your own life? No ... 2(iv) made plans to take your own life? No...3(v) attempted to take your own life? No...4

/Yes. .5

557



j.com/

J Epidem

iol Com

munity H


ownloaded from


development to measure symptoms depression general...

Documents