maintaining quality control standards using … quality control standards using telephone...
TRANSCRIPT
![Page 1: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/1.jpg)
RTI International is a trade name of Research Triangle Institutewww.rti.org
Maintaining Quality Control Standards Using Telephone Interviewing and Data Analysis
Presented byTiffiny Fambro, Tamara Terry, and Jaki Brown
RTI International
Presented atInternational Field Directors & Technologies Conference 2010
May 18, 2010 ? Chicago, IL
3040 Cornwallis Road ¦ P.O. Box 12194 ¦ Research Triangle Park, NC 27709
![Page 2: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/2.jpg)
5/20/2010
2
www.rti.org
Objectives
• To demonstrate the effectiveness of integrating a multi-modal approach to confirm field interviewer data quality on a large-scale national survey, the National Survey on Drug Use and Health (NSDUH)
• To provide additional knowledge on procedures for maintaining data quality standards for field studies
![Page 3: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/3.jpg)
5/20/2010
3
www.rti.org
Acknowledgement
National Survey on Drug Use and Health (NSDUH):• This project is funded by the Substance Abuse and Mental
Health Services Administration, Office of Applied Studies, underContract no. 283-2004-00022 and Project no. 0209009
• The views expressed in this presentation do not necessarily reflect the official policies of the Department of Health and Human Services; nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. Government
• This presentation is sponsored by RTI International’s Survey Research Division
![Page 4: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/4.jpg)
5/20/2010
4
www.rti.org
Study Background
• The National Survey on Drug Use and Health (NSDUH) is a general population survey
• Conducted by RTI under contract with SAMHSA since 1988
• The nation’s largest ongoing source of information on substance use behaviors
• Data collected in all 50 states and the District of Columbia
![Page 5: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/5.jpg)
5/20/2010
5
www.rti.org
Study Background (cont’d.)
• Data collected on a quarterly basis each year
• Approximately 700 Field Interviewers (FIs) staffed
• Approximately 180,000 household screenings and 67,500 interviews completed annually
![Page 6: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/6.jpg)
5/20/2010
6
www.rti.org
Overview
• This large-scale field survey uses combinations of data quality assurance methods to ensure a high level of data quality
• NSDUH uses a telephone verification team, a 2nd
follow up telephone verification team, mail verification, data quality reports, and field verifications
![Page 7: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/7.jpg)
5/20/2010
7
www.rti.org
NSDUH Process
The NSDUH is comprised of non-interview and interview cases
• Randomly selected addresses
• Household screenings on hand-held computer
• 0, 1, or 2 persons selected for the interview from the household
• Interview conducted on a laptop computer
• Telephone verification– 15% of completed interviews
– 5% of non-interview cases (no one was selected)
![Page 8: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/8.jpg)
5/20/2010
8
www.rti.org
Verification Information
• Non-interview cases– Phone and first name collected in hand-held computer
• Interview cases– Information gathered on Quality Control (QC) form
• R’s phone number• Address• Date (of interview – entered by FI)• FI name• Case ID• Parent or legal guardian who granted permission for minor R
![Page 9: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/9.jpg)
5/20/2010
9
www.rti.org
Telephone Verification
• Data Quality Coordinators (DQCs)
- RTI staff that handle all verification data quality issues with the field staff and the telephone interviewing staff
• Telephone Interviewers (TIs) training
- CATI (Computer Assisted Telephone Interviewing)
- History of NSDUH
- FI materials
- Mock cases
- Frequently Asked Questions
![Page 10: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/10.jpg)
5/20/2010
10
www.rti.org
Telephone Verification
• CATI system used by TIs
• Conduct a brief 2 minute interview
• Verify quality of FIs work and their adherence to project procedures and protocol
• Responses are keyed into the verification system database
![Page 11: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/11.jpg)
5/20/2010
11
www.rti.org
Telephone Verification: Call Back Team
• The Call Back Team is a small group of telephone interviewers who conduct follow up calls to flagged or problem cases
• Problems and/or protocol violations are flagged in CATI
• Some flagged or problem cases are sent to the Call Back Team’s database
![Page 12: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/12.jpg)
5/20/2010
12
www.rti.org
Telephone Verification: Call Back Team
• Call Back Team works cases located on the CMS
– Flagged cases are sent to a NSDUH web application in the form of problem sheets in the Case Management System (CMS)
– Initial problem is identified and investigated
• Questionnaire not read verbatim
• Addresses problem found by TI on initial call
![Page 13: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/13.jpg)
5/20/2010
13
www.rti.org
Telephone Verification: Call Back Team
After Call Back Team completes verification, the DQCs:
• Review cases worked by Call Back Team
• Resolve cases using same web application
• Read problem sheets and are able to determine the reason for the discrepancy
![Page 14: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/14.jpg)
5/20/2010
14
www.rti.org
Telephone Verification: Call Back Team
DQCs use Problem Sheets to:
• Determine if the case is considered:
– unresolvable
– an error
– not a problem
– unable to contact
– invalid
![Page 15: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/15.jpg)
5/20/2010
15
www.rti.org
Mail Verification
• Mail verification form sent to all Rs without contact information
• Standardized form with 7 questions to assess whether FI followed project protocols during the interview
• Returned using pre addressed envelope
• Responses keyed into the verification system
![Page 16: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/16.jpg)
5/20/2010
16
www.rti.org
Telephone Verification Production Report
NSDUH Telephone Verification Production Report
• Used to monitor the verification response rates
• The report is updated daily and shows the progress of TIs in verification calls for the quarter
• The report uses information from the Call Center Services Verification Status Report that is run everyday
![Page 17: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/17.jpg)
5/20/2010
17
www.rti.org
Data Quality Reports
• Information captured from the initial telephone verification call, the follow-up call by the Call Back Team, and the mail-in verification responses are illustrated on Data Quality Reports
• The main 4 reports are: Increased Verification Report, Data Quality Summary Report, Interview Lengths Report, and the Verification Form Error Report
![Page 18: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/18.jpg)
5/20/2010
18
www.rti.org
Data Quality: Increased Verification Report
• Increased Verification Report - An increase in the percentage of cases to be called and verified by TIs
• Interviewers are selected for increased verification based on:
– Not following project protocol
– Data quality concerns
– New interviewers that may have struggled through training
![Page 19: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/19.jpg)
5/20/2010
19
www.rti.org
Data Quality: Summary Report
• The Data Quality Summary Report, provides a summary of various reports and tracks:
– FI production – Number / percent of households reporting no phone – Refusals for verification information – Missing quality control forms – The Call Back Team– Short interviews
![Page 20: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/20.jpg)
5/20/2010
20
www.rti.org
Data Quality: Interview Lengths Report
• The Interview Length Report details all interviews that were completed in less than 30 minutes or more than 3 hours
– Short interviews can be an indicator that the FI did not follow proper procedures
– Long interviews may show that the FI needs re-training
![Page 21: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/21.jpg)
5/20/2010
21
www.rti.org
Data Quality: Verification Form Error Report
• This Report captures errors reported on verification quality control forms such as:
– Missing Date or Time
– Missing or Incorrect FI ID
– Missing or Incomplete Case ID
– Also captures refusals by respondents to fill out form
![Page 22: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/22.jpg)
5/20/2010
22
www.rti.org
Data Quality Reports
• Main reports capture information from the hand-held computer and the laptop.
• These reports are used by the DQCs to help analyze the Data Quality Summary, Interview Lengths, and Verification Form Error Reports.
– Short FI Level Report
– Team Status - FS Level Report
– Missing Screening Data Report
– Record of Call Time Discrepancy Report
![Page 23: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/23.jpg)
5/20/2010
23
www.rti.org
Field Verifications
• Reasons for Field Verification (FV)
- A large % of the FIs work that can not be contacted by the TIs
- Potential falsification is suspected
• The DQC will train an interviewer to re-contact the completed cases of the original FI
• If falsification is found the individual is released from the project
![Page 24: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/24.jpg)
5/20/2010
24
www.rti.org
Conclusions
• The NSDUH project uses a combination of various methods, to maximize the reliability of the field interview process
• We hope that the information provided in this presentation illustrated the benefits of a multi-modal verification process and can help other developing projects to construct the most effective data quality monitoring system
![Page 25: Maintaining Quality Control Standards Using … Quality Control Standards Using Telephone Interviewing and Data Analysis Presented by Tiffiny Fambro, Tamara Terry, and Jaki Brown RTI](https://reader034.vdocument.in/reader034/viewer/2022042620/5ace9ecd7f8b9ad24f8b8325/html5/thumbnails/25.jpg)
5/20/2010
25
www.rti.org
Thank You!
• For more information:
• Tiffiny Fambro, RTI International, [email protected]
• Tamara Terry, RTI International, [email protected]
• Jaki Brown, RTI International, [email protected]