The Integrity of Data Management

Responsible Conduct of Research Workshop

Howard University

November 2, 2010

Peggy Carr, Ph.D.

Associate Commissioner

National Center for Education Statistics

1

Overview of Statistical and Research Standards

2

OMB Guidelines for Federal Statistical Agencies

16 Federal Statistical Agencies adhere to Office of Management and Budget Guidelines to maintain high standards of performance Development of concepts and methods

Planning and design of surveys and other means of collecting data

Collection of data

Processing and editing of data

OMB, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, February 22, 2002, (67 FR 8752-8460)

3

Best Practices for the Conduct of Research

4

Best Practices for the Conduct of Research

1. Conceptualize Research Problem

2. Operationalize Study Design

3. Engage in Peer Review and Networking

Activities

4. Publish and Publish Early/Often

5. Be Ethical

5

Oprah Video Clip

1. Conceptualize Research Problem

7

Literature ReviewLiterature Review

Statement of ProblemStatement of Problem

HypothesesHypotheses Research QuestionResearch Question

Definition of Terms, Concepts, and

Constructs

Definition of Terms, Concepts, and

Constructs

Conceptual

Operational

2a. Operationalize Study Design

The Concept of Study Design

Hypothesis 1Hypothesis 1

Explained VarianceExplained Variance

Unexplained VarianceUnexplained Variance

Sampling Error

Sampling Error

Measurement Error

Measurement Error

Validity

Reliability

2b. Operationalize Study Design

Validation of Constructs Face – Content validation across judges Criterion – Predictive and concurrent Construct – Methodological approaches to validation

9

Jealousy Envy

2c. Operationalize Study Design

Reliability of Measures

Test and Retest – stability over time

Equivalency – consistency across forms

Internal – internal consistency

10UNRELIABLE RELIABLE

2c. Operationalize Study Design

Research Design

Internal Validity – refuting rival plausible hypotheses

External Validity – generalizability of findings

Choice of Designs – experimental, quasi, survey,

correlational

11

2c. Operationalize Study Design

Quality Assurances of Data Data Checks (e.g., out of range variance, criterion reference

checks, replication)

Thoughtful and Appropriate Data Analysis The nature of your data Limitations of statistical technique

Discuss Findings and Conclusions with Colleagues Relate back to literature review Implications for future research Flaws and limitations

12

NEXT

2c. Operationalize Study Design

Trend in fourth-grade NAEP mathematics average scores

13

2c. Operationalize Study Design

14

3. Engage in Peer Review and Networking Activities

Find a specialty and stick with it

Attend professional meetings and conferences – share your research

Keep an open mind and learn from others

15

4. Publish and Publish Early/Often

Start your publication record early in peer reviewed journals As first, second, or third author

Publish and/or present often Regardless of professional focus

Annually

16

5. Be Ethical

Be committed and follow through

Adhere to Subjects’ Rights Institutional Review Board Confidentiality and Privacy Rights

17

Questions?

18