British Journal of Addiction (1991) 86, 1211-1220

THE COLLABORATIVE ALCOHOL-RELATED LONGITUDINALPROJECT

An integrated approach to meta-analysis inalcohol studies^

BRYAN M. JOHNSTONEf, E. VICTOR LEINO,MICHELLE M. MOTOYOSHI, MARK T. TEMPLE,KAYE MIDDLETON FILLMORE & ELIZABETH HARTKA

Institute for Health & Aging, University of California, San Francisco, 201 Filbert Street^Suite 500, San Francisco, California 94133-3203, USA

With the fcUowlog collaborators: Saime Ablstrom (Finland), Peter Allebeck (Sweden), Arvid Amundsen (Norway), Jules AngstSwitzerland), Geliisse Bagnall (UK-Scotland), Ann Brunswick (USA), Remi Cadoret (USA), Sally Casswell (New Zealand), Lorraine deLabry (USA), Norman Giesbrecht (Canada), Bridget Grant (USA), Thomas Greenfield (USA), Joel Grube (USA), Bernd Guether(Federal Republic of Germany), Thomas Harford (USA), Ludek Kubicka (Czechoslovakia), Steve Manske (Canada), Mark Morgan(Ireland), Harold Mulford (USA), Leif Ojes|o (Sweden), David Peck (Scodand), Martin Plant (Scotland), Cbris Power (UnitedKingdom), Lee Robins (USA), Anders Romelsjo (Sweden), David Rosen (USA & Shetland Islands), Ronald Schlega! (Canada), MartinSieber (Switzerland), Soren Sigvardsson (Sweden), Rainer Silbereisen (Federal Republic of Germany), Ronald Stall (USA), MeirTeichman (Israel), Richard Wilsnack (USA) and Sharon Wilsnack (USA).

AbstractThe research design and methods utilised by the Collaborative Alcohol-Related Longitudinal Project aredescribed. The project design addresses the critical need to develop procedures to assess the replicability ofresearch results in alcohol studies. Key features of the research plan include: re-analysis of original data frommultiple longitudinal studies of drinking behavior in the general population; centralization of all dataanalyses, developed and implemented by an interdisciplinary core staff; development of the research plan andinterpretation of results in co-operation with original investigators of studies included in the project; and use ofmodeling procedures from meta-analysis to quantify the relative contribution of factors infiuencing thedistribution of effect estimates across studies, including both methodological differences and aggregate levelvariables. The final section describes statistical methods for meta-analysis used by the project, includingprocedures for the calculation and combination of estimates of effect magnitude, categorical and continuousmodeling procedures for use with effect sizes, and random effects models.

•This work was supported by a National Institute on Alcohol Abuse and Alcoholism (NIAAA) grant (#R01 AA07034) and by a NIAAAResearch Scientist Development Award (#K01 AAOOO73) to the fifth author. The Collaborative Alcohol-Related Longitudinal Project isincluded in the plan of work of NIAAA as a World Health Organization Collaborating Center on Research and Training in Alcohol-Related Problems and is also affiliated with the WHO Global Program on Prevention and Control of Alcohol and Drug Abuse. Order ofauthorship in the project is designated by the following criteria: (a) the fist author has taken principal responsibility for organizing andwriting the research paper; (b) persons making substantial contributions follow the first author in alphabetical order; (c) collaborators,having reviewed the paper and its findings in accordance with accuracy and representation of their data and project goals. Portions of thisreport were presented at the Second Meeting of the Collaborative Alcohol-Related Longitudinal Project, Mark Hopkins Hotel, SanFrancisco, California, July 31-August 4, 1989,

tPreseni address: Department of Behavioral Science, University of Kentucky College of Medicine, Lexington, Kentucky, USA40536-0086.

1211

1212 B. M. Johnstone et al.

IntroductionThis paper describes the research design andmethodological procedures utilized by the Collabo-rative Alcohol-Related Longitudinal Project. Sec-tion 1 presents ari overview of the project's researchactivities. Participants in tbe project have developedan integrated approach to quantitative researchsynthesis that both minimizes the possibility of errorin large-scale collaborative research of this kind, andmaximizes the contributions of original investigatorsand multiple disciplinary pterspectives. Section 2describes the methodological basis for analysesreported in the accompanying research papers fromthe project (Fillmore et ai, 1991a; Hartka et al,1991; Temple et ai, 1991). It is hoped that thecollective effort described below will contribute tothe critical need for the development of standardsand procedures for the conduct of research syn-theses in alcohol studies.

The analytic basis for the Collaborative Project isprovided by meta-analysis, or the combination andanalysis of data from multiple studies (Hedges andOlkin, 1985; Chalmers et al, 1987a,b). As Hedges(1987a) has emphasized, this approach has severalfundamental advantages over traditional efforts tosynthesize and interpret the results of multiplestudies on a common research question. Twocapabilites of the technique are particularly salientin this context. First, meta-analysis makes use offormal procedures to weight resuls of varyingprecision across studies, quantify the consistency ofcombined effects, and judge the appropriateness ofthat combination. Secondly, categorical and contin-uous modeling approaches available in meta-anaty-sis can identify and evaluate hypothesized explana-tions for systematic variation in results acrossstudies. This makes it possible to use a singleanalytic framework to assess the effect of verydifferent types of factors, e.g. design effects andcultural or historical variation, on study differences.

Despite these advantages, it has become clear thatthe design and conduct of effective quantitativeresearch syntheses can be a very difficult enterprise.Wachter (1988) and others have panicularly notedconcerns that the retrospective application of forma!mathematical procedures to combine the results ofstudies, in the absence of controlled conditions,measurement scales designed to be homogeneous, orrigorously established statistical independence, maylimit the validity of conclusions in research syn-theses. Moreover, there is a danger that the assess-ment of prior studies may be reduced to a routinizedtask in which details of interpretation, and the

contributions of original investigators are lost in anindiscriminate effort to pool results and determine'overall' effects.

Research designThe design of the Collaborative Project addressesthese and other significant challenges for theconduct of effective meta-analyses. Althoughresearch syntheses have become increasinglyimportant in social and health-related research,techniques and applications explicitly designed tocombine results or data from epidemiological orother non-experimental studies are still relativelynew (Louis, Fineberg & Mosteller, 1985; Dyer,1986; Greenland, 1987; Jenicek, 1989), andguidelines are especially lacking for projects whichanalyse primary data from original studies ratherthan published reports, or which attempt tocombine results from studies that cross national,cultural and historical boundaries.

Our approach builds on prior experience in thefield (Hedges and Olkin, 1985; Dyer, 1986), ac-knowledgement of the need to adjust for theadditional potential for bias in the combination ofresults from non-experimental studies (Ozmin-kowski, Wortman & Roloff, 1988), and a commit-ment to the development of a fully interdiscipli-nary approach to the explanation of drinkingbehavior that permits the testing of competinghypotheses using a common study population anda coordinated analytic framework (Fillmore,1988). The need to develop effective means forresearch synthesis is perhaps especially importantin alcohol studies, because of the great variety ofresearch designs, analytic methods, and explana-tory paradigms which have been applied to drink-ing behavior. Meta-analysis, with its capacity toidentify and measure the effects of factors thatgive rise to differences in results from studies, isan especially important tool in this process.

Figure 1 summarizes major components of theresearch plan of the project. The basic design callsfor a repetitive sequence of complementary tasks,completed in turn by an interdisciplinary core staffresponsible for co-ordinating all research activites,data management and analysis, and secondly, a bodyof collaborating investigators responsible for assess-ing research plans and interpreting results. Datafrom studies included in the project are centralizedat the University of California, and all analysis isconducted by the core staff using common analytic

Meta-analysis in alcohol studies 1213

CORE STAFF

Data documentationDevelopment of comparative constructs

Preliminary analysis pian

COLLABORATING INVESTIGATORS

Specification of theory and hypothesis testingAssessment of comparative constructs

Final analysis plan

CORE STAFF

Data standardization and assessmentData collection: aggregate level factors

Analysis and meta-analysis

ICOLLABORATING INVESTIGATORS

Assessment of analysis resultsRevised hypothesis testing and analysis plan

Satellite analysis projects

ICORE STAFF

Assessment of methodological and aggregate level factorsFinal analysis and meta-analysis

Figure 1. Research design of the Collaborative Alcohol-Related Longitudinal Project.

models for parallel analyses of individual studiesand meta-analysis. This approach significantlyenhances the fiexibility of the project, reducesthe potential for error, and balances the participa-tion of original investigators in the interpretationof results with the intensive efforts of a core staffwho design and conduct all primary analyses. Thedevelopment of project data documentationsystems, comparative constructs, data standardiza-tion procedures, and assessment of aggregate leveleffects is briefly described below. Specifics ofprocedures for meta-analysis are discussed in theflnal section.

Data documentation procedures and development ofconstructsAs many authors have emphasized, the validity ofresearch syntheses is crucially dependent uponcareful assessment of the equivalence of measures tobe compared across studies (Rosenthal, 1984;Hedges, 1986; Wolf, 1986). In the present project,access to original data has enabled careful appraisalof the comparability of measures. However, thisdesign feature also posed significant data manage-ment problems. The system that has emerged servesboth to organize the formidable amounts of infor-mation contained in multiple longitudinal data sets,

1214 B. M. Johnstone el al.

and contributes to the systematic development ofcomparative constructs for use in the meta-analysis.

In the formative stages of the project's develop-ment, collaborating investigators from each studyincluded in the meta-analysis completed a basicquestionnaire on available measures in his or herproject. From these preliminary documents, mem-bers of the core staff developed a comprehensive listof measures significantly represented across studies.Collaborators also provided complete question-naires, codebooks, system file listings, proceduresfor variable construction and weighting, and detailsof research design and sampling procedures em-ployed in the study. After the arrival of thisinformation, core staff members first surveyed thedocumentation, identifying matches with the gen-eral variable list. Secondly, essential documentationon each variable selected was entered into a study-specific data base. Thus the resulting system candisplay variation in measures across studies quicklyat any level (e.g. at the level of item wording or thatof response category). It also provides a fiexiblemeans to incorporate additional data sets into thegeneral data base, document data transformations,and permit others to reconstruct and evaluateproject procedures.

Core staff members have also had to developappropriate theoretical constructs for each antece-dent and outcome measure used in analyses, andstandard operationalizations of these constructs.This essentially qualitative step—judgment of thecomparability of measures across studies—necessar-ily precedes the quantitative aggregation of results,and has been one of the most challenging aspects ofresearch syntheses (Hedges, 1986). This phase ofthe study was designed as an integral extension ofthe documentation process, relying as closely aspossible on the information available from the database of measures as the foundation for operationali-zations. After entry for each study-specific data basewas complete, it was integrated into a cross-studydata base containing variable records from all datasets included in the meta-analysis. At this point,cross-study information on variables included in themeta-analysis could be rapidly extracted and inte-grated using capabilities of the data base manage-ment system; these cross-classified materials be-came the basis for the development of 'standard'cross-study codes for each of the variables includedin the meta-analysis. This process was necessarilyimperfect. However, it does represent perhaps themost systematic attempt to evaluate the comparabil-ity of measures across studies to date in the field.

Specification of theory and hypothesis testingAfter development of principal comparative con-structs was complete, members of the core staffcreated a preliminary research and analysis plan,and submitted all materials to collaborating investi-gators at the first collective meeting of the project(Johnstone, 1987). Participants were asked to judgethe appropriateness of comparative constructs, de-bate the theoretical basis for meta-analysis inparticular research domains of alcohol studies andagree upon a final analysis plan. At the secondmeeting of the project, collaborators evaluated theresults of primary analyses and meta-analyses,discussed interpretations and developed revisedanalysis plans (Fillmore, 1989). Satellite analyses,conducted by subsets of project participants, werealso initiated for the intensive analysis of particularresearch domains. This general interactive process isrepeated, on a smaller scale and through the mail,with each new analysis conducted by the project.Although difficult to organize, this procedure estab-lishes a crucially important system of checks andbalances for the design and interpretation of large-scale research syntheses.

Data standardization and assessmentPrior to analysis, original data from each of theindividual studies is converted to a common codingscheme. The decision to 'standardize' measuresaccording to a common operationalization for eachconstruct was taken for a number of reasons:

(1) to simplify the application of parallel analy-ses across studies;

(2) to enhance the comparability and interpreta-bility of results;

(3) to enable creation of a final cross-study database with all measures organized according toa common format.

(This procedure is followed for all variables, withthe exception of complex scale scores, which aresimply recoded to refiect a common directionality).

The majority of constructs represent behavioralphenomena commonly measured in social surveys,e.g. alternative dimensions of drinking behavior.However, the development of uniform operationali-zations for even such ostensibly 'simple' measures isa formidable challenge, and necessitates systematicevaluation of the effects of transforming originalmeasures. Quantitative assessment of the contribu-tion of such methodological factors to differences inresults across studies is a basic objective of the

Meta-analysis in alcohol studies 1215

project. The documentation system described abovemakes it possible to test the effects of suchdifferences across studies at the level of individualmeasures. As described above, details of eachoriginal measure are recorded in the general database. This information is then used to code andassess methodological differences; because full do-cumentation on the original item is present, thissystem can enable identification of significant dif-ferences between measures at many levels, forexample, item wording or original response code.

techniques. Although adaptation of such proceduresfor use in meta-analyses is very new, this task isessential, if research syntheses are to evaluaterigorously the effects of cultural and historicalvariation on differences between study results(Blalock, 1984; Nesselroade & Von Eye, 1985;Raudenbush & Bryk, 1985; Bryk & Raudenbush,1987). Details of project procedures and results forthis specialized segment of the analysis are pre-sented elsewhere (Johnstone & Sawyer, 1989; Fill-more et al, 1990, Leino et al, 1990).

Assessment of aggregate level factorsIn addition to methodological differences betweenstudies, significant cross-national and cross-histori-cal variation is present in studies included in themeta-analysis. This unusual design feature of theproject has made it necessary to develop means toassess the potential contribution of such aggregatelevel factors to differences in results across studies.A three-pronged approach has been developed tomeet this difficult objective, including

(1) careful integration of the knowledge baseprovided by the participation of originalinvestigators;

(2) systematic efforts by the core staff to collectand standardize relevant aggregate level data;

(3) adaptations of contextual effects models andassociated aggregate level data analytic ap-proaches for use with meta-analysis.

The first component of this approach capitalizeson collaborators' specific expertise with respect tothe studies they represent. Toward this end, anextensive questonnaire was developed for responseby individual investigators, designed to tap theirknowledge of the cultural and historical circum-stances under which their longitudinal research wasconducted. Effectively, this method uses partici-pants as ethnographic informants, to enhance thevalidity of hypothesis testing in collaborative re-search. Information gathered from collaboratinginvestigators is supplemented by the collection of anindependent body of alcohol-related aggregate data.Members of the core staff have collected availablesocial statistics and data on public policy from therelevant countries, regions and states representedfor the period from 1880 to the present (reflectingthe range of birth cohorts represented in data setsincluded in the project).

The final component of this approach consists ofthe application of aggregate level data analysis

Meta-analysis proceduresThe analytic approach adopted by the projectpermits examination of the combined data set frommultiple perspectives. Eight basic steps are con-ducted for each research question:

(1) disaggregation of individual studies by prin-cipal classification variables;

(2) calculation of effect estimates within categ-ories for each relationship under evaluationin each study;

(3) calculation of combined results across studieswithin categories for each research question;

(4) tests of the homogeneity or consistency ofcross-study results;

(5) use of categorical models to test the consis-tency of effect estimates across discretegroups of studies;

(6) modeling of cross-study variation in effectestimates, using an analogue to multiplelinear regression, to quantify the relativecontribution of alternative study level char-acteristics to differences in results;

(7) use of random effects models, if additionalvariation remains to be explained;

(8) re-aggregation of pooled results calculated inthe third step to obtain an overall effectestimate, using appropriate weights for eachsignificant cross-classified category.

Calculation of effect estimatesStep 1 consists of disaggregation within data setsaccording to principal demographic variables, ini-tially by sex, age and race/ethnicity with subsequentdivisions dependent on the results of primary testsof homogeneity. This preliminary task is especiallyimportant in research syntheses of general popula-tion studies, because of the diverse demographiccomposition of the samples included. Particularemphasis is placed on the disaggregation of samples

1216 B. M. Johnstone et al.

by age, because basic theoretical objectives of theproject focus on the hypothesized invariance of age-related variation in alcohol consumption and prob-lems (Fillmore et ai, 1988).

After disaggregation is complete, estimates ofeffect magnitude (the principal unit of analysis inmeta-analysis) are calculated within studies using acommon analytic model for each research question.Effect estimates are calculated independently withinstudies for each research question, and the resultscombined across studies, rather than by pooling datafrom alternative studies and conducting a singleanalysis. As a number of investigators have noted,use of pooled aggregates to derive effect estimatesmay be misleading, because the results may reflectthe sample composition as much as underlyingeffects present within relevant strata (Bickel, Ham-mel & O'Connell, 1975; Hedges, 1987a). Thisapproach is also similar to procedures adopted bythe International Collaborative Group to combineresuls from multiple prospective epidemiologicstudies (Dyer, 1986).

The project has made use of a variety of teststatistics derived from primary analyses as effectestimates in hypothesis testing. Procedures for themeta-analysis of effect sizes (i.e. standardized meandifferences) are presented below for illustrativepurposes. Alternative indices of effect magnitudeutilized in the project have also included propor-tions or means (for estimates of prevalance, inci-dence and other event rates), and unstandardizedregression coefficients derived from either multiplelinear or logistic regression analysis. Details of theprocedures for meta-analysis of effect sizes arepresented in Hedges & Olkin (1985), and Hedges &Becker (1986). Analogous procedures for the meta-analysis of results of regression analyses are dis-cussed in Greenland (1987).

The effect size (g) is the most common unit ofanalysis in meta-analyses. Defined by Glass,McGaw & Smith (1981) as the standardized differ-ence between experimental and control groupmeans, this value is calculated as

(1)

where s consists of the pooled within-group stan-dard deviation, approximated by

as the standardized mean difference between firstand final measurement point for outcome measures.Hedges & Olkin (1985), however, have demon-strated that f is a slightly biased estimator of thepopulation effect size, tending to overestimate thisvalue in small samples. They define an unbiasedestimator d, obtained by multiplying ^ by a constantdependent upon the sample size of the study. That

IS ,

For within-subjects designs, the value ofapproximated by

(3)

(4)

where n refers to the total number of subjects in thestudy. This unbiased estimator constitutes the finalestimate of effect magnitude.

Combining effect estimatesSteps 3-8 employ a variety of procedures tocombine effect estimates across studies for eachresearch question and evaluate the distribution ofresults. In step 3, a preliminary estimate of theoverall or pooled measure of effect across studies isprovided by the calculation of a weighted meaneffect estimate (d.). Derivation of this value andsubsequent statistical tests for use with effect sizesare dependent upon calculation of the systematicand non-systematic variance of the unbiased estima-tor from each observation. Hedges & Olkin (1985)demonstrate that the sampling distribution of d isapproximately normal, with variance given by

nE+nc-2

In the present context, effect sizes are calculated

(5)

That is, the variance of d is completely determinedby the samples sizes and the value aid. As Hedges &Becker (1986, p. 30) note, the capability to deter-mine the non-systematic variance of d from a singleobservation (or study) is the key to meta-analysis.Effectively, this capability permits the researcherboth to make use of all the degrees of freedomamong different values of d to estimate systematiceffects, and also to derive the non-systematicvariance necessary for the construction of statisticaltests.

After derivation of the variance, weighted averageeffects across studies are calculated for each rela-tionship under evaluation. This value, denoted by

Meta-analysis in alcohol studies 1217

d., is computed as the weighted sum of the results,divided by the sum of the study weights, namely.

d.

I ̂ A

1(6)

where

(V)

A standard error for the weighted mean effect iscomputed as the inverse of the square root of thesum of the weights, or

(8)

Confidence intervals for the weighted average effectcan be approximated as follows,

(9)

(10)

A z transformation, defined as

d.

assesses whether the overall estimate of effect acrossstudies is non-zero. Analogous procedures areavailable for combination of other test statistics,including proportions and regression coefficients.

The appropriateness of the resulting weightedmean or summary effect is dependent on theassumption that studies are estimates of the sameeffect, that is, that differences between estimates aredue to non-systematic error. After calculation ofsummary effects within each category for eachresearch question, this assumption is evaluated instep 4, using a test of homogeneity of results acrossmultiple studies (2 ) , which has a chi-square distri-bution (cf. Hedges & Olkin, 1985, Ch. 6), and isapproximated by

i -1

Multiple assessments of homogeneity of effects maybe conducted for particular research questions, withre-computation of pooled effects after partitioningstudies along characteristics potentially associatedwith significant heterogeneity in results. Tests of

homogeneity quantify the consistency of effectestimates across studies; however, they do notprovide estimates of the effects of particular factorson patterning in effect estimates.

Categorical and continuous modeling procedures foreffect estimatesIf effect sizes are heterogeneous across studies,categorical modeling procedures can be used to testthe effect of discrete study level characteristics, e.g.methodological differences between studies, onvariation in the distribution of effect sizes. Hedges& Olkin (1985, C. 7) derive an analogue to theanalysis of variance for effect sizes that partitionsthe general homogeneity statistic 2 , '"to compo-nents refiecting between-group ( 2 B ) ^^^ within-group (2«) homogeneity. These values, analogousto the partitioning of sums of squares in standardanalysis of variance, can be used to test whethervariation between and within groups of effect sizesis significant, and whether the resulting model iscorrectly specified. The between-group homogene-ity statistic consists of a weighted sum of squares ofgroup mean effect size estimates about the generalweighted mean. This value is approximated by

Q^ = lw,id^~d.y (12)

where d. is calculated as in (6) above, d, is theweighted average of the effect estimates in the ;thgroup, and w, is defined as the reciprocal of thevariance of d, (Hedges & Becker, 1986, p. 36).

The within-group homogeneity statistic Q^ repre-sents the sum of the homogeneity statistics calcu-lated independently for each of the groups, that is,

2* = 2^.,+ ••• + 2 . (13)

Both statistics have approximate chi-square distri-butions for use in significance testing. If more thantwo groups are present, both apnon and ^posterioricomparisons between groups, analogous to contrastsin standard analysis of variance, can be calculated.

The full range of regression procedures can alsobe applied in meta-analysis, if continuous studylevel predictors are hypothesized to affect thedistribution of effect sizes across studies. Hedges &Olkin (1985) have defined an analogue to conven-tional multiple regression for use in meta-analysis,using weighted least squares regression, with effectsizes weighted as in (7) above. This model tests thehypothesis that the population effect sizes Sj...S^are a linear function of the predictors jc,... x , thatis, S,^P^+fitXi+ ... +PpXp, where ^^, /?„.. .fi are

1218 B. M. Johnstone et al.

unknown regression coefficients (Hedges & Becker,1986, p. 43).

Results of the analysis provide direct estimates ofthe regression coefficients for each predictor, butstandard errors must be adjusted using the squareroot of the residual mean square of the regression,that is

(14)

where p^ is the regression coefficient, SE0^ is theunadjusted standard error, and MS^ is the error orresidual mean square (Hedges & Becker, 1986, p.44). A test of significance of the regression coeffici-ent is also calculated, using the standard normaldistribution, and based on the transformation

. = 4 . (15)

Finally, a test for overall model specification,fitssentially a test of whether significant systematicvariation in effect sizes is unexplained by the model,is provided by the weighted sum of squares aboutthe regression line. This statistic also has a chi-square distribution (Hedges & Olkin, 1985, p. 172).Thus, a comprehensive set of procedures can beemployed to evaluate hypothesized explanations forvariation in effect sizes across studies. Greenland(1987) provides the model and notation for bothcategorical and continuous modeling procedures formeta-analysis using regression coefficients derivedfrom logistic or ordinary least squares regressionanalysis.

Random effects models and derivation of a summaryeffectProcedures in prior steps are based on fixed-effectsassumptions. However, if large amounts of unex-plained variance in the distribution of effect esti-mates remain after regression modeling and otherdiagnostics, random effects models can be substi-tuted. This formulation assumes that the populationvalues of the effect estimates are samples from adistribution of the effect size parameters (or, in thecase of Bayesian estimation, that the parametershave a prior distribution). Variability between effectsizes is, therefore, due in part to true variability inthe population parameters, in addition to samplingerror about the parameter value. A distribution oftrue or population effects is therefore assumed. Fullnotation of the model is presented in Hedges &

Olkin (1985, pp. 189-203), and Raudenbush & Bryk(1985) have extended the approach to includemixed linear models with both fixed and randomeffects. The advantage of this approach is thatestimates of precision may more accurately reflectpreviously unaccounted-for sources of variation incross-study results. However, as Greenland (1987,p. 26) notes, use of random-effects assumptions innon-experimental research must be applied onlyafter careful consideration of the justification fortheir application.

The final step calculates a summary estimate ofthe overall effect for each research question underassessment, by recalculating combined results, usingappropriate weights for significant predictors de-rived from modeling procedures. The completesequence of analyses for each research questionprovides a comprehensive examination of the over-all effect for the relationship under evaluation, andsystematic assessment of the methodological andother factors which may influence variation in theresults across studies.

ConclusionTaken together, research procedures adopted by theCollaborative Alcohol-Related Longitudinal Projectcomprise a cohesive research plan for large-scalemeta-analyses in alcohol studies. Key design fea-tures of the project include:

(1) re-analysis of original data from multiplelongitudinal studies of the general popula-tion;

(2) centralization of all data analyses, designedand conducted by an interdisciplinary corestaff, in a single location;

(3) comprehensive inclusion of the expertise oforiginal investigators in the development ofthe research plan and the interpretation ofresults;

(4) systematic assessment of variables hypothe-sized to influence the distribution of effectestimates across studies, including bothmethodological and aggregate level factors;

(5) utilization of the multilevel analysis frame-work for research synthesis provided bymeta-analysis.

Can meta-analysis of general population studiescontribute significantly to the explanation of drink-ing behavior and problems across the life course?Can elements of this project serve as a template forresearch synthesis efforts in other domains of

Meta-analysis in alcohol studies 1219

alcohol studies? It is, of course, difficult at presentto answer these questions with any certainty,because the project is still very much in process; theresults presented in the papers accompanying thisrepon are prebminary, and significant heterogeneityin combined results remains to be explained.Important theoretical and methodological chal-lenges also remain to be fully addressed by theproject. However, the very process of developing aproject of this nature has been exciting, leadingparticipants into a number of new speculationsabout future directions for research (Fillmore etal.,1989; Grant & Johnstone, 1990-91).

We can also be encouraged by the emergingresults of meta-analyses in other research domains.In an unusual study, Hedges (1987b) has comparedthe results of meta-analyses from experimentalstudies in social scientific research with the resultsof quantitative synthesis in the physical sciences; theresults suggested that social research can achievecomparable levels of consistency to those in the'hard sciences'. This study suggests the broadpotential significance of research syntheses in socialscience, and especially their importance as a guageof (and contributor to) the 'cumulativeness' ofresearch. Short of conducting a massive multi-sitecomparative prospective study of life course varia-tion in drinking behavior and problems, quantitativeresearch synthesis appears to be the most promisingdirection for the urgently needed effort to evaluatesystematically the replicability of research results instudies of human drinking behavior.

ReferencesBiCKEL, P. J., HAMMEL, E. A. & O'CONNELL,J. W. (1975)

Sex bias in graduate admissions: data from Berkeley,Science, 187, pp. 398-404.

BLALOCK, H. M . (1984) Contextual-effects models: theo-retical and methodological issues. Annual Review ofSociology, 10, pp. 353-372.

BRYX, A. S. & RAUDENBUSH, S. W. (1987) Application ofhierarchical linear models to assessing change. Psycholo-gical Bulletin, 101(1), pp. 147-158.

CHALMERS, T . C., LEVIN, H . , SACKS, H . S., REITMAN, D. ,BERRIER, J. & NAGALINGAM, R. (1987a) Meta-analysisof clinical trials as a scientific discipline. I: control ofbias and comparison with large co-operative trials.Statistics in Medicine, 6, pp. 315-325.

CHALMERS, T . C , BERRIER, J., SACKS, H . S., LEVIN, H. ,REITMAN, D . & NAGALINGAM, R. (1987b) Meta-analysis of clinical trials as a scientific discipline. II:replicate variability and comparison of studies that agreeand disagree. Statistics in Medicine, 6, pp. 733-744.

DYER, A. R. (1986) A method for combining results from

several prospective epidemiologic studies. Statistics inMedicine, 5, pp. 303-317.

FILLMORE, K. M . (1988) Alcohol Use Across the LifeCourse: Seventy Years of International LongitudinalResearch (Toronto, Addiction Research Foundation).

FILLMORE, K. M . (Ed.) (1989) Final Report of the SecondMeeting of the Collaborative Alcohol-Related Longitudi-nal Project (San Francisco: Institute for Health & Aging,University of California, San Francisco).

FILLMORE, K. M . , GRANT, M . , HARTKA, E., JOHNSTONE,

B. M., SAWYERS, S., SPEIGLMAN, R. & TEMPLE, M . T .

(1988) Collaborative longitudinal research on alcoholpToblem^, British Joumal of Addiction, 83, pp. 441-444.

FILLMORE, K. M . , HARTKA, E., JOHNSTONE, B. M . ,

LEINO, E. V., MOTOYOSHI, M . & TEMPLE, M . T . (1989)

Research developments in alcohol studies: the contribu-tions of social and epidemiological research, paperprepared for Report of the National Institute on AlcoholAbuse and Alcoholism, 'Ten Year Plan'.

FILLMORE, K. M . , HARTKA, E., JOHNSTONE, B. M . ,

LEINO, E. V., MOTOYOSHI, M . & TEMPLE, M . T . (1991)

A meta-analysis of life course variation in drinking,British Journal of Addiction, 86, pp. 1221-1268.

FILLMORE, K. M . , JOHNSTONE, B. M . , LEINO, E . V. &

AGER, C. R. (1990b) A cross-study contextual analysisof effects of in dividual-level drinking and group-leveldrinking factors, paper presented at the 16th AnnualAlcohol Epidemiology Symposium Budapest, Hungary,June 3-8.

GLASS, G . V,, MCGAW, B. & SMFTH, M . L . (1981) Meta-

Analysis in Social Research (Beverly Hills, Sage).GRANT, M . & JOHNSTONE, B. M . (1990-91) Research

priorities for drug and alcohol studies: the next 25 years.International Joumal of the Addictions, 25(2A), pp,201-219.

GREENLAND, S. (1987) Quantitative methods in the reviewof epidemiologic literature, Epidemiologic Reviews, 9,pp. 1-30.

HARTKA, E. , JOHNSTONE, B. M . , LEINO, E. V.,

MOTOYOSHI, M . , TEMPLE, M . T . & FILLMORE, K. M .

(1991) A meta-analysis of depressive symptomatologyand alcohol consumption over time, British Joumal ofAddiction, 86, pp. 1283-1298.

HEDGES, L . V. (1986) Issues in meta-anaiysis, in:E. Z. RoTHKOPF (Ed.) Review of Research in Education(New York, Erlbaum), pp. 353-399.

HEDGES, L. V. (1987a) Meta-analysis. Presented at theFirst Meeting of the Collaborative Alcohol-RelatedLongitudinal Project, Berkeley, California, July 28.

HEDGES, L. V. (1987b) How hard is hard science, how softis soft science? The empirical cumulativeness of re-search, v^merican Psychologist, 42(2), pp. 443-455.

HEDGES, L . V. & BECKER, B. J. Statistical methods in themeta-analysis of research on gender differences, in:J. S. HYDE & M. C. LINN (Eds) The Psychology of

Gender: Progress Through Meta-Analysis (Baltimore,The Johns Hopkins University Press), pp. 14-50.

HEDGES, L. V. & OLKIN, 1. (1985) Statistical Methods forMeta-Analysis (New York, Academic Press).

JENICEK, M. (1989) Meta-analysis in medicine. Where weare and where we want to go, Joumal of ClinicalEpidemiology, 42(1), pp. 35-44.

JOHNSTONE, B. M . (Ed.) (1987) Repon: First Meeting ofthe Collaborative Alcohol-Related Longitudinal Project

1220 B. M. Johnstone et al.

(Berkeley, California: Alcohol Research Group, MedicalResearch Itistitute of San Francisco).

JOHNSTONE, B. M . & SAWYER, S. (1989) Aggregate levelexplanations for change and stability in individualdrinking behavior: conceptual and methodologicalissues, presented at The Natural Course of Drinking—Implications for Public Health: A National Confer-ence, San Diego, California, January.

LEINO, E. V., AGER, C. R., FILLMORE, K. M . &

JOHNSTONE, B. M . A cross-study contextual analysis ofgroup influences and individual infiuences of changingmarital status on individual drinking behavior over time,paper presented at the 16th Annual Alcohol Epidemiol-ogy Symposium, Budapest, Hungary, June 3-8, 1990.

LOUIS, T . A., FINEBERG, H. V. & MOSTELLER, F . (1985)

Findings from public health from meta-analyses, An-nual Review of Public Health, 6, pp. 1-20.

NESSELROADE, J. R. & VON EYE, A. (Eds) (1985)Individual Development and Social Change: Explana-tory Analysis (New York, Academic Press).

OZMINKOWSKI, R. J., WORTMAN, P. M. & ROLOFF, D. W.(1988) Inborn/outborn status and neonatal survival: ameta-analysis of non-randomized studies. Statistics inMedicine, 7, pp. 1207-1221.

RAUDENBUSH, S. W. & BRYK, A. S. (1985) Empirical Bayesmeta-analysis. Journal of Educational Statistics, 10,(2),pp. 75-98.

RosENTHAL, R. (1984) Meta-Analytic Procedures forSocial Research (Beverly Hills, Sage).

TEMPLE, M . T . , FILLMORE, K. M . , HARTKA, E.,

JOHNSTONE, B. M . , LEINO, E. V. & MOTOYOSHI, M .

(1991) A meta-analysis of change in marital andemployment status as predictors of alcohol consumptionon a typical occasion, British Journal of Addiction, 86,pp. 1269-1281.

WACHTER, K. W. (1988) Disturbed by meta-analysis?Science, m , pp. 1407-1408.

WOLF, F . (\.9&6) Meta- Analysis: Quantitative Methods forResearch Synthesis (Beverly Hills, Sage).