Advances in Survey Methods and Measurement Techniques

Methodological advances of the past twenty years have been sparked by the develop- ment of opinion research centers, by the expansion of government-sponsored research during World War 11, by the professional associations, and not least by the polling debacle of 1948. These advances have been reflected in a series of outstanding, and in- creasingly sophisticated, studies. Research methodology has been adapted to a wider field of subject matter, rapid strides have been made in attitude measurement tech-niques, and principal sampling problems have been partially solved.

The author, Professor of Social Statistics at Princeton and President of the Ameri- can Association for Public Opinion Research for 1957-58, has been one of the foremost practitioners of the application of statistical technique to social problems ever since 1931 when he became Director of the Bureau of Social Research, Pittsburgh

T o SEE CLEARLY the advances in survey and attitude research methods which have taken place during the past two decades, we should start a bit farther back. Much of the present stage of technological advance in opinion research is foreshadowed by Rice in the papers he published during the Twenties and assembled in 1928 in his Quantitative Methods in Politics.' They are also reflected by Lundberg in his Social Research? published in 1929. These men were not writing prophetically, as was Jules Verne. They reported and synthesized work that was then being done by Floyd Allport, Harold Gosnell, L. L. Thurstone and many others, much of it of a pioneering naturee3 They worked primarily in the fields of social psychology, sociology, and political science.

As we moved into the Thirties, the time of the Depression and the New Deal, there were other developments. Claude Robinson made an extensive survey of the primitive opinion surveys conducted by newspapers and maga- zines up to that time, and his report, entitled Straw Votes, was published in 1932.4 Henry Link and others, using the "quota method," conducted surveys largely in the field of market research, although some parts of their work could properly be classified as opinion research? The Literary Digest, which

1 Stuart A. Rice, Quantitative Methods in Politics, New York, Alfred A. Knopf, 1928. 2 George A. Lundberg, Social Research, New York, Longmans Green and Company, 1929.

Stuart A. Rice, "Statistical Studies of Social Attitudes and Public Opinion," Chapter XI in Statistics in Social Studier, edited by Stuart A. Rice, Philadelphia, University of Pennsylvania Press, 1930.

4 Claude E. Robinson, Straw Votes, New York, Columbia University Press, 1932. 6 Henry C. Link, "Some Milestones in Public Opinion Research," International /ournu2 of

Opinion arjd Attitude Research, 1 (1947) 36-47.

80 PUBLIC OPINION QUARTERLY

had conducted several large polls in the Twenties on "prohibition" and other issues,' forecast closely the percentage of the vote won by Roosevelt in 1932.7 In the same year, Margaret Hogg8 pointed out the weaknesses of unemploy- ment surveys and made a strong case for "probability sampling." Many large surveys9 were undertaken by agencies of the Federal Government under the New Deal Administration, !greatly enriching the experience and "know-how" of the people who had begun to specialize as survey personnel. Market re- search was growing apace. It stimulates interest in polling and contributed much to the methods that were used in the field and in the central office.

In 1936, the Literary Digest's cumbersome straw vote went badly askew, pointing to an overwhelming victory for Landon.'' The relatively good showing of the newly launched Gallup Poll1' and the close estimate from the Fortune PolP2 stirred public interest in opinion surveys. The lesson d 1936 seemed to be that there was no "safety in numbers" in straw votes, but one could have confidence in the results of relatively small surveys that were pre- pared carefully and conducted skillfully. An adequate and true "cross-section" could be obtained with only 2000 persons or so, selected by "quota sampling," and most opinions could be elicited adequately by interviewers who followed a carefully pretested list of questions.13 This kind of polling gained great popularity. Many state and local polls were organized. For several years it overshadowed other types of opinion research and included many surveys conducted by or for private business concerns that were never published, such as the subscriber surveys started by the American Telephone and Telegraph Company in 1929.14 Interest in opinion research was scattered, however, and the methods still lagged behind those developed in some similar fields of research. This was the situation when the Public Opinion Quarterly first appeared.

6 Walter F. Willcox, "An Attempt to Measure Public Opinion About Repealing the Eighteenth Amendment," lournal of the American Statistical Association, 26 (1931) 243-262.

7 T h e Literary Digest, November 5 , 1932. 8 Margaret H. Hogg, "Sources of Incomparability and Error in Employment-Unemployment

Surveys," Journal of the American Statistical Association, 25 (1930) 284-294. Q Frederick F. Stephan, "History of the Uses of Modern Sampling Procedures," Journal of

the American Statistical Association, 43 (1948) 12-39. 10 T h e Literary Digest, October 31, 1936. IlGeorge Gallup and Claude Robinson, "American Institute of Public Opinion Surveys,

1935-38," Public Opinion Quarterly, 2 (1938) 373-399. 12 "Fortune Quarterly Survey," Fortune, July 1935 and succeeding issues. 13 See the critiques of the polls and their methods in a group of papers assembled under the

general theme, "The Public Opinion Polls: Dr. Jekyll or Mr. Hyde?" Public Opinion Quarterly, 4 (1940) 212-285. Daniel Katz and Hadley Cantril, "Public Opinion Polls," Sociometry, 1 (1937) 155-179 and Daniel Katz, "The Public Opinion Polls and the 1940 Election," Public Opinion Quarterly, 5 (1941) 52-78.

14 Arthur H. Richardson and C. Theodore Smith, "Finding Out What People Think of Us," Bell Telephone Magazine, 25 (1946) 1-15.

81 ADVANCES IN METHODS AND TECHNIQUES

TIlE FIRST DECADE

There have been few dull moments in either foreign or domestic politics during the two decades since the Qumterly was established. The United States went from neutrality and isolationism to extensive involvement in world affairs, first in the Second World War and then in the critical post-war period punctuated by the Berlin blockade, Korea, and Suez. Opinion abroad, as well as commu~~ication and opinion at home, became an increasing con- cern of public officials and leaders in the business world. Simultaneously, in- terest in opinion research grew in the academic world.

Not long after the Quarterly first appeared, the Office of Radio Research was established at Princeton with Cantril, Lazarsfeld and Stanton as direc- tors. It subsequently was moved to Columbia University, where it became the Bureau of Applied Social Research. In 1941, the National Opinion Research Center was founded at the University of Denver. These centers, and others formed later, were active in developing better methods and training research personnel, as well as in conducting research projects that were not likely to be undertaken by commercial agencies. In this they were aided by grants from private foundations and government research agencies.

World War I1 called forth a great expansion of opinion research, notably in the Office of War Information, the Department of Agriculture, the War Production Board, and the office of the Director of the Army Information and Education Division. Along with this expanded activity came experi- mentation in methods and the development of improved techniques. The outstanding example is the work that culminated as the volume on Meamre-ment and Prediction by Stouffer and his associates.15 The work in the De- partment of Agriculture under Rensis Likert and Angus Campbell was con- tinued after the War by many of the same researchers, who established the Survey Research Center at the University of Michigan. Sometime after Harry Field's untimely death, the National Opinion Research Center, under the direction of Clyde Hart, moved to the University of Chicago. Other centers developed, with or without a corporate identity. Thus, opinion research became one of the primary interests of a number of well-staffed centers, and the emergency work of wartime settled into a more regular, though still somewhat hectic, peacetime operation.

The opinion polling activities that received such a stimulus in 1936 ex-panded at home and abroad. A central file of poll results, as well as of the Gallup survey punch cards, was maintained by Cantril in the Office of Public Opinion Research at Princeton University. A compilation of poll results was published by Cantril in a 1200 page quarto volume, entitled Public Opinion

1 5 Samuel A. Stouffer, Louis Gutunan, Edward A. Suchman, Paul F. Lazarsfeld, Shirley A. Star, John A. Clausen, Measurement and Prediction, Princeton, Princeton University Press, 1950.

82 PUBLIC OPINION QUARTERLY

1935-1946, that assembled work done during only the first half of the Quar-terly's existence!"

During all this growth of opinion polling, there were serious discussions and debates about methods, especially those of interviewing and sampling. In March 1944, a series of meetings on techniques used in the measurement of opinion was held in Washington by the Washington chapters of the American Statistical Association and the Institute of Mathematical Statistics. Gallup's methods were discussed in testimony before a Congressional Com- mittee in December of that year?7 Quinn McNemar published a critical re- view of methods in his "Opinion-Attitude Methodology," in 1946.''

In 1947, the American Association for Public Opinion Research (AAPOR) was organized. Its annual meetings have assembled people who engage in opinion research, or use its results, and have brought out in public discus- sion many of its methodological problems. The Quarterly serves as the official publication of AAPOR and reflects its discussions in the papers that are published and in the summary proceedings of the annual meetings.

The methods that are used in opinion surveys and attitude measurement are also used quite widely in other types of research. Consequently, it is not surprising that a great deal of methodological discussion is found at the meet- ings and in the publications of the American Psychological Association, American Political Science Association, American Sociological Society, Amer- ican Marketing Association and American Statistical Association.

The elections of 1948 proved to be almost as severe a crisis for the polls as the election of 1936 had been for the Literary Digest. Earlier that year Louis Bean had written on "How to Predict election^,"'^ mixing information about past elections and analysis of trends with a critique of polling methods and some words of caution. In reviewing Bean's book for the Quarterly?" the author of this article suggested that the question "How?" could be an- swered in one word, "Don't." (That answer was no prediction of what was about to happen; otherwise it would have been self contradictory.)

Gallup issued a release a few days before the election pointing out that his

16 Hadley Canuil, Public Opinion, 1935-1946, Princeton, Princeton University Press, 1951. For a view of what has been done in the second decade in only one segment of public opinion research, compare the extensive bibliography by Bruce L. Smith and Chitra M. Smith, Inter-national Communication and Political Opinion, Princeton university Press, 1956.

1 7 78th Congress, Second Session on H.R. 551, Hearings Before the Committee to Investigate Campaign Expenditures; December 28, 1944, Washington, Government Printing Office, 1945.

18 Quinn McNemar, Opinion Attitude Methodology, Psychological Bulletin, 43 (1946) 289- 374.

19 Louis H. Bean, How to Predict Elections, New York, Alfred A. Knopf, 1948. 20Frederick F. Stephan, Review of "How to Predict Elections," Public Opinion Quarterly,

12 (1948) 498-499.

83 ADVANCES IN METHODS AND TECHNIQUES

polls had been in error by 4 percentage points on the average in past elections, but even he neglected to include this warning when his final press release was written and published.

The outcome of the election is well-known. Truman won by a clear mar- gin, and a sharp reaction against polling set in. It appeared to threaten other kinds of re~earch.~' In editorials and elsewhere there was exultation to the effect that "The People Know Be~t,"~%nd that no modern poll or scientific device can tell them how they think. Some of this blast of critical reaction was well deserved, some sheer demogogery. Over-onfidence in polling was due for deflation. Further improvements in method were clearly needed. None- theless, opinion research had not been proven completely wrong or worthless by the particular failures of the 1948 polls.

A substantial report on the polls was published by the Social Science Re- search Council after a "crash" investigation by a special committee.23 While it turned up no single cause for the failure, it explained rather fully the opera- tions of the polls and analyzed their principal weaknesses.

The debate over the meaning of the 1948 "fiasco" continued at the meet- ings of various professional societies, at a special conference in Iowa and in the journals, particularly the International Journal of Opinion and At-titude Research and the Quarterly. It subsided without reaching any sharp conclusion. The major polling organizations continued their operations mak- ing some improvements in their methods and exercizing greater caution, es- pecially at the election of 1952. The public has not again fallen under the spell of the polls, but it is more tolerant toward them than it was at the end of 1948. Other types of opinion research are distinguished from polling some- what more clearly. There has been a gain in methodological sophistication, although few of the fundamental questions of method have been solved.

In the past ten years there has been a further growth of research, marked more by its extension into new subject fields than by mere increase in the rate of activity in old fields. It has gained ground in analysis as well. Berel- son, Lazarsfeld and McPhee exhibit this increased scope and refinement of research in V ~ t i n g ? ~a profound study of the 1948 election in Elmira, New York, that goes well beyond the earlier study of Sandusky, Ohio, in T h e

21Robert K. Merton and Paul K. Hatt, "Election Polling Forecasts and Public Images of Social Service," Public Opinion Quarterly, 13 (1949) 185-223.

22 Morris L. Ernst and David Loth, The People Know Best, Washington, D. C., Public Affairs Press, 1949.

23Frederick Mosteller, Herbert Hyman, Philip J. McCarthy, Eli S. Marks, and David B. Truman, The Pre-election Polls of 1948, New York, Social Science Research Council, 1949.

24Norman C. Meier, and Harold W. Saunders (editors), T h e Polls md Public Opinion, New York, Henry Holt and Company, 1949.

25 Bernard R. Berelson, Paul F. Lazarsfeld, and William N. McPhee, Voting, Chicago, Uni- versity of Chicago Press, 1954.

84 PUBLIC OPINION QUARTERLY

People's Choice.26 Campbell, Gurin and Miller, in their impressive nation- wide study of the 1952 election, T h e Voter D e ~ i d e s ? ~ use considerably more sophisticated techniques than in the earlier Michigan study, T h e People Elect a President." A great step has been taken by Katz and Lazarsfeld toward un-derstanding individual opinion in relation to group opinion in Personal In- fluence?' a study of people who tend to shape the opinions of others, and also by Stouffer in a comparative study of the attitudes of civic leaders and the population at large in Communism, Conformity, and Civil Libertie~.~' These and many other studies that exemplify advances in research might be listed, and discussed with emphasis on the relation between the substance and the method in each, but there is no room for that here. It has been done very notably in the "Researcher Meets the Critic" sessions at the annual meetings of AAPOR.

One of the most clear-cut evidences of the advance in methods and tech- niques is exhibited in the progression of books and articles principally de- voted to methodology. Cantril's Gauging Public Opinion?' Zeisel's Say It Wi th Payne's T h e Art of Asking Questionsa3 Parten's Surveys, Polls, and Sample~ ,3~ and many handbooks and manuals not generally avail- able, reveal the accumulation of experience, coupled with critical reexamina- tion and testing, that spell progress. A remarkable literature on interviewing and sampling technique has been published in the Quarterly and elsewhere. Only recently has it begun to be incorporated into textbooks and other com- ~rehensive volumes. The volumes by Jahoda, Deutsch and and by Festinger and K a t ~ ; ~ ~ the appendices in Katona's Psychological Analysis of Economic B e h a ~ i o r ; ~ ~ and especially the volumes by Hyman on Interviewing

2ePaul F. Lazarsfeld, Bernard, E. Berelson, and Helen Gaudte, The People's Choice, New York, Duell, Sloan & Pearce, 1944.

'27Angus Campbell, Gerald Gurin, and Warren E. Miller, The Voter Decides, Evanston Illinois, Row, Peterson and Company, 1954.

28 Angus Campbell and Robert L. Kahn, The People Elect a President, Ann Arbor, Survey Research Center, University of Michigan, 1952.

29 Elihu Katz and Paul F. Lazarsfeld, Personal Infltrence, Glencoe, Illinois, The Free Press, 1955.

30 Samuel A. Stouffer, Communism, Conformity, and Civil Liberties, Garden City, New York, Doubleday and Company, Inc., 1955.

31Hadley Cantril, Gauging Public Opinion, Princeton, Princeton University Press, 1944. 32 Hans Zeisel, Say It With Figures, New York, Harper and Brothers, 1947. 33Stanley L. Payne, The Art of Asking Questions, Princeton, Princeton University Press,

1951. 34 Miidren Parten, Surveys, Polls, and Samples, New York, Harper and Brothers, 1950. 35 Marie Jahoda, Morton Deutsch and Stuart W. Cook, Research Methods in Social Relations,

New York, The Dryden Press, 1951 (two volumes) 36Leon Festinger and Daniel Katz (editors), Research Methods in the Behavioral Sciences,

New York, The Dryden Press, 1953. 37 George Katona, Psychological Analysis of Economic Behavior, New York, McGraw-Hill

Book Company, 1951.

85 ADVANCES IN METHODS AND TECHNIQUES

in Social Resear~h,3~ and Survey Design and Analysis,a9 are all significant landmarks by which our present stage of progress will be identified when the QuarterZy marks its Fiftieth Anniversary.

It may be suitable to add three brief sections to complete this rapid survey of what happened in opinion research, and particularly in research methods, since the Quarterly was founded. The first will be on survey operations and interviewing, the second on attitude measurement, and the third on sampling.

SURVEY OPERATIONS

One of the great advances of the past quarter century has been the demon- stration that it is both feasible and worth while to collect a great variety of social data by sampling surveys. The monumental study entitled Recent So- cial Trends?' commissioned by President Hoover just before the Great De- pression, foreshadowed these advances in many ways, but came too soon to participate in them and benefit from them. The Depression stimulated sur- veys of income and expenditures, unemployment, and other relatively sensi- tive subjects. Then the Second World War showed that soldiers and even enemy civilians could be interviewed successfully. Postwar studies of preju- dice, and Kinsey's surveys of sexual behavior, added evidence that highly personal information and beliefs could be obtained from the general public by personal interview.

These demonstrations were not without their flaws, and they rested on some very important provisos. It was necessary that the approach to the re- spondents be made in an appropriate manner and skillfully, and that the interviewing be done equally well. Interviewers had to be selected and trained carefully. Their work had to inspected and checked. The necessary techniques of conducting interviews in each case had to be discovered by an analysis of experience and by clever experimentation.

Some of the results of research on interviewing were startling. Reports submitted by interviewers were often found to be incomplete, even to the extent of omitting half or more of the significant material in the interview, and sometimes they were quite incorrect. Interviewers did not do precisely what they were instructed and trained to do.41 Earlier, when interviewers were less carefully selected and trained, some of them were found to have falsified their reports. The problem of "cheating" came up for lively discus- sion in the Quarterly, and "cheater traps" were added to many question-

88 Herbert H. Hyman, Interviewing in Social Research, Chicago, University of Chicago Press, 1954.

39 Herbert Hyman, Survey Design and Analysis, Glencoe, Illinois, The Free Press, 1955. 40 The President's Research Committee on Social Trends, Recent Social Trends in the United

States, McGraw-Hill Book Company, 1933. 4 1 Stanley L. Payne, "Interviewer Memory Faults," Public Opinion Quarterly, 13, (1949)

684-685.

86 PUBLIC OPINION QUARTERLY

naire~.~' The interviewers were probably blamed too much; the fault was partly that there was a degree of naivete and neglect in the management of interviewing staffs. Studies by Hyman and many others, and wisdom de- rived from the struggle to improve the quality of interviewing, have gone far to bring this important part of the survey process to an adequate level of excellence.

Problems of coding43 and analysis have likewise been brought to light and studied. Important advances have been made toward their solution. It became apparent that the methodological problems of operating a survey are interconnected. A survey of opinion is usually a complex system of research activity involving many methods and techniques. T o be effective as a source of information, to be useful as a guide to action and policy the total survey must be managed well. This requires a fine blend of preparation, pretesting, skill in the use of techniques, control of operations, and coopera- tion between the producers and users of opinion survey resultsP5 The past two decades have seen increasingly distinguished performance in the manage- ment of research operations.

MEASUREMENT OF ATTITUDES

The technique of measuring attitudes and opinion has come a long way since 1928, when Thurstone declared that "attitudes can be m e a ~ u r e d . " ~ ~ The state of measurement at the founding of the Quarterly is reflected in Guil- ford's Psyclzometric Methods47 and in the review by the Murphys and New- time, progress has been made in the use of the older methods. Guttman's new comb in the last chapter of their Experimental Social P ~ y ~ h o l o g y . ~ ~ Since that development of "scale analysis"49 has been tested and applied widely. His theory, running on ahead of technique, has reached several advanced positions

42 Leo Crespi, "The Cheater Problem in Polling," Public Opinion Quarterly, 9 (1945) 431- 445. Herbert H. Hyman, Interviewing in Social Research, Chicago, University of Chicago Press, 1954.

43 Julian L. Woodward and Raymond Franzen, "A Study of Coding Reliability," Public Opinion Quarterly, 12 (1948) 253-258. J. Durbin and A. Stuart, "An Experimental Companion Between Coders," Journal of Marketing, 19 (1954) 54-67.

44Rensis Liiert, "The Sample Interview Survey as a Tool of Research and Policy Formation" in T h e Policy Sciences, edited by Daniel Lerner and Harold D. Lasswell, Stanford, Stanford University Press, 195 1.

45 Russell L. Ackoff and Leon Pritzger, "The Methodology of Survey Research," Intmational Journal of Opinion and Attitude Research, 5 (1951) 313-335.

4'3L. L. Thurstone, "Attitudes Can Be Measured," American Journal of Sociology, 33 (1928) 529-554.

47 J. P. Guilford, Psychometric Methods, New York, McGraw-Hill Book Company, 1936. 48 Gardner Murphy, Lois Barclay Murphy, and Theodore M. Newcomb, Experimental Social

Psychology (revised edition), New York, Harper and Brothers, 1937. 49Louis Guttman, "An Outline of Some New Methodology for Social Research," Public

Opinion Quartwly, 18 (1954) 395-405.

87 ADVANCES IN METHODS AND TECHNIQUES

in "facet analysis" and a variety of scaling models. "Latent structure analysis" was developed by Lazarsfeldso in the analysis of the attitude studies in T h e American Soldier and has been carried toward finished technique. The Riley's have extended the theory and application of scale analysis in sociometric studies.61 Coombs has developed another type of scaling and measurementF2 Finally, mathematical models for attitude measurement and change have been perfected, notably by Anderson and others, in Mathematical Thinking in T h e Socia2 SciencesF3

The Quarterly published two featured review articles on T h e American Soldier and Measurement and Prediction that are classics for the library of opinion research.54 It has also published a number of outstanding studies of the validity of interview data. While these do not involve the more modern measurement techniques, they illustrate the type of research which will be necessary if we are going to validate these newer methods. The Denver study by Parry and Crossleys5 was as startling in its way as some of the studies of interviewing bias and error. Hyman's "Do They Tell the Truth?"56 was an- other and earlier classic study of validity. One phase of the maturing of opin- ion research is an increasing concern with the accuracy of the results of meas- urement, whatever the technique by which these results are achieved. Testing validity is the greatest problem that confronts opinion research today.

SAMPLING THEORY AND PRACTICE

The double decade since the Quarterly's first issue has been a period of remarkable advance in sampling theory and practice. With some exceptions, notably the surveys of the Survey Research Center and of several commercial and government research agencies, opinion research has lagged behind the statistical agencies in adopting the more modern sampling methods. It has been reluctant to change its original procedures, and has made improvements in them to avoid more radical modifications. Nevertheless, the trend in the

50 Paul F. Lazarsfeld, "A Conceptual Approach to Latent Structure Analysis," in Mathe-maticd Thinking in the Social Sciences, edited by Paul F. Lazarsfeld, Glencoe, Illinois, The Free Press, 1954.

51Matilda White Riley, John W. Riley, Jr., and Jackson Toby, Sociological Studies in Scale Analysis, New Brunswick, Rutgers University Press, 1954.

52C. H. Coombs, "Mathematical Models and Measurement Theory" in Decision Processes, edited by R. M. Thrall, C. H. Coombs, and R. L. Davis, New York, John Wiley and Sons, 1954.

53 T. W. Anderson, "Probability Models for Analyzing Time Changes in Attitudes," Chapter 1 in Footnote 50.

64 Paul F. Lazarsfeld, "The American Soldier-An Expository Review," Public Opinion Quarterly, 13 (1949) 377-405. M. A. Girshick and Daniel Lerner, "Model Construction in the Social Sciences," Public Opinion Quarterly, 14 (1950) 710-729.

65Hugh J. Parry and Helen M. Crossley, "Validity of Responses to Survey Questions," Public Opinion Quarterly, 14 (1950) 61-80.

5'3Herbert Hyman, "Do They Tell the Truth?" Public Opinion Quarterly, 8 (1944) 557-559.

- -

88 PUBLIC OPINION QUARTERLY -United StatesF7 like that noted by Moser in Great BritainFS is toward random or probability methods.

Sampling has been badly misunderstood throughout the period in spite of the appearance of excellent treatises by Yate~,5~ Deming,Go Cochran,G1 and Hansen and his associatesP2 Many textbooks now include chapters on modern sampling methods, too. The outlook is for better comprehension of sampling problems and more effective sampling practice. The Quarterly may, therefore, carry more discussion of sampling problems in opinion research in the future, although the problems discussed may be quite different from those that have been debated so heatedly in the past.

Several basic problems were prominent in the past:

(a) What effect does the sampling really have on the final results? (b) How much control and what kind of control must be exercised over the selection

of persons to be interviewed? How serious is the loss of part of the sample that cannot be found or will not cooperate?

(c) How di5cult and how expensive are the probability methods? (d) How important is it to have accurate measures of sampling error?

These problems may have been formulated in different terms, such as "What kind of sample do I have to get?," but fundamentally there was within the field of opinion research a deep-seated difference of opinion about sampling, based on differences in both background and current operating position. Similar differences were evident in other fields. Everywhere they appear to. be on the way to being resolved by accumulating experience and serious study of sampling method.

One reason the first problem persisted was that opportunities were seldom presented or taken for testing sampling methods and dissecting out the ef- fects of the sampling from the effects of interviewing and other factors. How- ever, Field and Hart,'63 Politz,'j4 Hochstim and Smith,G6 the Division of Re-

67 Frederick F. Stephan, "Sampling in Studies of Opinion, Attitudes, and Consumer Wants," Proceedings of the American Philosophical Society, 92 (1948) 387-398.

58 C. A. Moser, "Recent Developments in the Sampling of Human Populations in Great Brit- ain," ]ournal of the American Statistical Association, 50 (1955) 1195-1235.

59 Frank Yates, Sampling Methods for Censuses and Surueys, London, Charles Griffin and Company, 1949.

60 W. Edwards Deming, Some Theory of Sampling, New York, John Wiley and Sons, 1950. 61 W. G. Cochran, Sampling Techniques, New York, John Wiley and Sons, 1953. 82 Morris H. Hansen, William N. Hunvitz, and William G. Madow, Sample Survey Meth-

ods and Theory, New York, John Wiley and Sons, 1953. 82111 an experimental study conducted in Denver and a nationwide test of a quota sample

survey that included questions on labor force status for comparison with the Census Bureau's Monthly Survey of the Labor Force.

64Alfred Politz, "BMB Evaluation Study," Report submitted to the Broadcast Measurement Bureau, 1947.

85 Joseph R. Hochstim and Dilman M. K. Smith, "Area Sampling or Quota Control-Three Sampling Experiments," Public Opinion Quarterly, 12 (1948) 73-8 1.

89 ADVANCES IN METHODS AND TECHNIQUES

search Techniques at the London School of Economics,B6 and others did per- form such tests in opinion surveys.

The second problem-that of control over respondent selection-was studied by Hilgard and Payne, whose article "Those not at Home: Riddle for pollster^"^^ is another classic piece of research published by the Quarterly. Stantonag Lazarsfeld:' Franzena9 Baur," Ford,", 72 Zeisel," andC l a ~ s e n , ~ ~ many others studied the problem as it occurred in the case of mail question- naires. The general lesson of these studies has been clear: Insufficiently con- trolled sampling leads to unrepresentative samples, these lead in turn to dis- torted inferences, which presumably then lead to undesirable consequences. Much sampling in the past has been insufficiently controlled.

The third problem-the difficulty and expense of probability sampling- has been discussed by Manheimer and Hyman,73 William~,'~ Wat~on,'~ Cor- dell," G ~ o d m a n , ~ ~Canne11Y7 KingY8 and others. It is inherently complex but real progress has been made both toward finding answers for it and to- ward inventing more feasible and less expensive methods. There have been serious miscalculations of cost and a general lack of good cost data. There have been difficulties and added costs in changing over from one type of sam- pling to another. There are capital costs that can be spread over many surveys but otherwise would be a heavy burden on a single survey. Progress in sam- pling continues as these problems are overcome.

66C. A. Maser and A. Stuart "An Experimental Study of Quota Sampling," Iournal o f the Royal Statistical Society, Series A, 116 (1953) 349-405.

67 Ernest H. Hilgard and Stanley F. Payne, "Those Not at Home: Riddle for Pollsters, "Pub-lic Opinion Quarterly, 8 (1944) 254-261.

6s Frank Stanton, "Notes on the Validity of Mail Questionnaire Returns," lournal o f Applied Psychology, 23 (1939) 95-104.

89 R. Franzen and P. F. Lazarsfeld, "Mail Questionnaire as a Research Problem," Iournal o f Psychology, 20 (1945) 293-310.

70 E. Jackson Baur, "Response Bias in a Mail Survey," Public Opinion Quarterly, 11 (1947) 594-600.

71Robert N. Ford and Hans Zeisel, "Bias in Mail Surveys Cannot be Controlled in One Mailing," Public Opinion Quarterly, 13 (1949) 495-502.

72 J. A. Clausen and R. N. Ford, "Controlling Bias in Mail Questionnaires," lournal of the American Statistical Association, 42 (1947) 497-511.

73Dean Manheimer and Herbert Hyman, "Interviewer Performance in Area Sampling," Public Opinion Quarterly, 13 (1949) 83-93.

74 Robert Williams, "Probability Sampling in the Field," Public Opinion Quarterly, 14 (1950) 316-330.

75 Alfred N. Watson, "Note on 'Probability Sampling in the Field: A Case Study'," Public Opinion Quarterly, 14 (1950) 610-61 1.

78 W. N. Cordell, "Blueprints Don't Fly," a paper presented to the American Marketing As- sociation and American Statistical Association, December 27, 1948.

77 R. Goodman and C. Cannell, "Sampling Errors and Components of Interview Costs in Relation to Sample Design," (mimeographed) Survey Research Center, 1949.

7s Arnold J. King, "Sampling in Marketing Research," a paper presented to the American Marketing Association and the American Statistical Association, December 27, 1948.

90 PUBLIC OPINION QUARTERLY

The fourth problem-that of measuring sampling error-is common to all statistical analysis. Most uses of opinion research appear to be on the border of quantitative research, and can be satisfied by quite approximate results except when a close decision is involved. In the future there will be a growing need for accurate estimates of sampling error in conjunction with meas-urement error and various biases.

The discussion of such problems was virtually absent in the days of straw votes. Since then there have been bitter lessons and warnings. Many agencies have broken away from their commitment to rule-of-thumb and quota meth- ods. There is increasing acceptance of probability methods and a growing demand for them. Various forms of "modified probability sampling" have been developed. While they are improvements, their errors and biases must be determined before a firm judgment of their merits can be made. Purpo- sive methods have been found useful in certain surveys.70 The advances made in the past appear to be heading toward a situation where sampling methods will be selected from a large collection of possibilities, and will be designed or adjusted to fit the survey in which they will be used. All this will be guided not by precedent or prejudice but by well worked out theory and by technical knowledge of how the various methods work.

There are still great gaps in the theory of sampling method. Almost noth- ing constructive has been done on the theory of purposive and quota sam- pling. The theory of bias is meager, based largely on a few studies. The theory of sampling items in a universe of attitude content is yet to be formulated adequately. There is much to be done.

So much for methods. They are merely tools but without them there would be no craftsmanship, no art. They help us know more about public opinion and understand it better. How far they will progress is the story only the future will tell, but we can get glimpses from those who have looked ahead calmly and written for the Quarterly what they saw. Julian Wood- ward's presidential addresss0 is still about the best look ahead we have.

The Public Opinion Quarterly has been a great forum for the discussion of opinion research and research methods for a generation. It will be just that for generations to come.

79 Charles Metzner, "Purposive Sampling in a Study of Community Differences in Reaction to Aromic Energy," a paper presented to the American Statistical Association.

80 Julian L. Woodward, "Public Opinion Research 1951-1970: A Not-Too-Reverent History," Ptiblic Opinion Quauterly, 15 (1951) 405-421.