The Committee on NationalStatistics (CNSTAT)

36 Years of Care and Feeding of the Federal Statistical System

Presentation to APDU, 9/24/2008

Constance F. CitroDirector, CNSTAT

Themes of My Talk• CNSTAT and the National Academies:

• What, Why, and How?• Value for/Perspective on the Statistical System

• Range of Studies/Workshops/Seminars:• Concepts—Cost of Living, Poverty, Food Insecurity• Reviews of Surveys/Reviews of Agencies/Programs• Reviews of Censuses Past and Present—and the ACS• Cross-Cutting Issues—Confidentiality and Data Access,

Survey Automation• “P&P”—Principles and Practices for a Federal

Statistical Agency• How APDU Can Build on Work of CNSTAT and Others

The National Academies Structure

National Research Council created as operational arm in 1916

National Academy of Sciences given congressional charter in 1863National Academy of Engineering, Institute of Medicine added in 1964, 1970

The National Academies Structure

NRC divided into major operational units for studies/workshops/seminarsCommittee on National Statistics is standing board under the social sciences

Committee on National Statistics (CNSTAT)

Created in 1972 on recommendation of President’s Commission on Federal Statistics: “to provide an outside review of federal statistical activities”

Unique among other Academies units, foreign science academies for its focus on the federal statistical system

Helps to bring together the agencies in a highly decentralized system

Key Features of Academies/ CNSTAT Consensus Panels

Volunteers: Members of study panels serve pro bono (reimbursed only for expenses)

Independence from Sponsor Control: Sponsors do not select panel members and have no access to report until final release

Rigorous External Review: Overseen by Academies’ Report Review Committee; reviewers chosen for expertise and balance

Wide Dissemination of Reports: Sponsors do not control release

Value for Agencies

Breadth and Depth of Expertise: CNSTAT panels engage leading academics and other experts on agencies’ scientific/technical challenges

Cross-cutting Issues: CNSTAT tackles system-wide challenges (e.g., data access and confidentiality in the Internet age)

Credibility: Reports often enable support from Congress/executive for innovation and improvements in agency programs

Downsides for Agencies• Slow:

• Projects, particularly panel studies, can take 1-2+ years• Events may overtake the study

• Pricey: • Volunteers are free, but staff, conference facility, travel, management, etc. are not

• Time-Intensive:• To get most value from study, agency staff must work hard to respond to panel (there is an offsetting benefit to agency from interacting with outside experts)

• Independent:• Two-edged sword

CNSTAT Studies

CNSTAT Studies—Concepts

Observations> Concepts need to keep up with a changing world• Consumer Price Index has had several revisions, in data sources, measurement, and the meaning of major components, such as housing• BEA is moving forward on satellite accounts to capture important components of economic activity that are not easy to include in the main accounts

> So, too, the poverty measure needs to be revised (NYC has led the way; bill being drafted)

NYC Poverty Rates by Borough

2006 ACS Official NYC*Bronx 26.6% 27.9%Brooklyn 21.5 27.0Manhattan 16.8 20.4Queens 11.7 19.6Staten Island 8.4 13.1

New York City 18.9 23.0

*Using NAS method and auxiliary data sources

CNSTAT Survey Reviews

Observations, Suggestions• Major ongoing surveys need regular review, updating, and, sometimes, complete redesign from the ground up• The ACS should be viewed as in a developmental stage at least for the next 10 years; users should be vocal in support and in suggestions for improvement• APDU should press producers for full documentation and funders for resources for “methods panel”-type R&D• APDU dissemination of Federal Register notices re new and revised surveys for comment is great; perhaps could also disseminate advisory committee meeting minutes, other information

CNSTAT Agency/ProgramReviews 1985: Immigration Statistics

1986: NCES

1997: BTS

2007: State and Local Government Statistics (Census)

2007, 2009: BJS

2008: Vital Statistics Workshop

Observations, Suggestions>Agencies sometimes need a major overhaul—

• All too often today, agencies lack the resources to carry out their entire portfolio well, but no one wants to give up long-standing data series• Sometimes agencies have lost—or never sufficiently acquired—a sense of what it means to be a statistical agency

>Change is possible—it takes leadership and inside/outside support; APDU can help

2000 Census Evaluation>Panel to Review the 2000 Census conducted a comprehensive assessment, issuing 9 reports from 1999-2004

>Major problems—wholly imputed households; duplicate enumerations; high long-form sample nonresponse rates; poor quality group quarters data; inadequate evaluations for planning the next census

Bottom Line:>Major successes—halted decline in mail response through advertising, outreach, design of mailing package; hired more than enough enumerators; data capture via OCR

2010 Census PlanningPanel on Research on Future Census Methods reviewed emerging 2010 census plan, including replacing long-form sample with ACS

As early as 2001 letter report—and with particular strength in 2004 final report—panel presaged Bureau’s current problems with handheld computers

Recommended “system architect,” attention to logical architecture

Warned of failure to focus on requirements

American Community Survey2007 Report—Comprehensive Review of the ACS and Differences/Similarities with the Long-Form Sample

Major challenges for users:

• Period estimates (1-, 3-, 5-year)

• Multiple estimates for larger areas

•Larger sampling error than long-form sample

•Inadequacies of population controls

•Collision of ACS/controls/ census in 2010

Current Census/ACS Work

Current panels charged to advise on:

2010 census coverage measurement

2010 CPEX program leading to testing program for 2020

Recent panel examined a custom addition to the American Community Survey—a question on field of study in college, used by NSF

May study group quarters for the ACS and census

Cross-Cutting Issues>Very important function of CNSTAT

>Often have to cobble together funding

>Confidentiality/data access issues always with us—APDU contributed helpful letter; need to be vigilant that the balance doesn’t tip too much against access

CNSTAT White Paper

First edition, 1992 Second edition, 2001 Third edition, 2005 Fourth edition, 2009

(forthcoming)

PRINCIPLES/GOALS>Relevance, Credibility with Users, Trustworthiness with Providers

PRACTICES (selected)>Strong Position of Independence>Openness about Sources and Limitations of Data>Wide Dissemination of Data>Commitment to Quality/Active Research Program>Professional Advancement of Staff

UNDER THREAT TODAY>Not only resources, but also independence, research capability, and adequacy of technical staff pipeline

2005 P&P

Cooperation/CoordinationStatistical Agencies

OMB Statistical and Science Policy OfficeAPDU – CNSTAT – COPAFS – COSSA et al.

Statistical AgenciesOMB Statistical and Science Policy Office

APDU – CNSTAT – COPAFS – COSSA et al.

• The agencies need more help than ever• Cooperation and coordination are essential to make the best use of limited agency resources and to leverage the work of outside groups• As a long-time APDU member, I’m thrilled to see it revitalized so that it can play its part in ensuring a vital federal statistical system for the future.

Questions?/Feedback

http://www.nationalacademies.org/cnstat

Reports: http://www.nap.edu

Connie Citro: ccitro@nas.edu; (202) 334-3009