introduction to encounter data validation presenter: thomas miller, ma executive director, research...

Introduction to Encounter Data Validation

Presenter:

Thomas Miller, MA

Executive Director, Research and Analysis Team

1

Welcome

About me Rules for engagement Presentation overview

• The importance of encounter data • Trends in Federal policy• CMS protocols• Florida EDV study

2

Objectives

1. Learn why Encounter Data Validation studies are important.

2. Identify the core evaluation components outlined in CMS’ protocols for validating the quality of encounter data.

3. Understand the proposed scope of work for Florida Medicaid’s SFY 2013-2014 encounter data validation study.

3

Importance of Encounter Data

Accurate and complete data are critical to success of managed care programs

Essential for overall management and oversight of Florida’s Medicaid program– Ability to monitor and improve quality

of care– Establish performance measures– Generate accurate and reliable reports– Obtain utilization and cost information

5


6

http://blogs.informatica.com/perspectives/wp-content/uploads/2011/02/jman72h1.jpg


Used by MCOs and the State for many purposes– Performance measure development and calculation– Performance improvement measurement– Focused studies/quality activities– Rate-setting– Compliance monitoring– Provider practice patterns

7

Key Trends

Importance of Federal and State monitoring– Development of core measurement sets

• Medicare versus Medicaid• Health care reform• Holding health care accountable

Data, not anecdotes

8

Key Trends in the News

9

Key Trends

Findings from a recent article in Medicare and Medicaid Research Review, Assessing the Usability of MAX 2008 Encounter Data for Comprehensive Managed Care– Objective: Assess availability, completeness,

quality, and usability of encounter data– Results: High rates for reporting by key encounter

data types– Conclusions: Completeness and quality of

encounter data were high

10

Objectives




1111

Objectives




13

EQR Protocol Developed and refined with the maturation of the

External Quality Review program

14

EQR Protocol

Specific guideline for External Quality Review Organizations (EQRO) to use when assessing completeness and accuracy of encounter data.

Data submitted by Managed Care Organizations (MCO) to the State

15

EQR Protocol

State establishes standards for encounter data

State must establish the following standards:– Definition of “encounter”– Types of encounters – Data accuracy and

completeness– Objective standards for data

comparison

16

EQR Protocol

Five core activities1. Review state

requirements

2. Review MCO’s capability

3. Analyze electronic encounter data

4. Review of medical records

5. Submission of findings and recommendations

17

EQR Protocol Attachment A: Encounter Data Tables

Table 2: Data Element Validity Requirements

18

EQR Protocol

Five core activities1. Review state requirements

• Develop understanding of State-specific policies and procedures for collecting and submitting encounter data

• Identify data exchange protocols and layouts• Evaluate encounter data system interchange

flows, including system edits and submission timelines

• Review existing encounter data quality activities, requirements, and performance standards

19

EQR Protocol

Five key activities, continued2. Review MCO’s capability

• Develop, conduct, and review MCO’s Information System Capabilities Assessment– Identification of IS vulnerabilities– Key findings address:

» Data processing and procedures» Claims/encounter processing and system

demonstration» Enrollment

• Key informant interviews

20

EQR Protocol

Five key activities, continued3. Analyze electronic encounter data

• STEP 1 - Develop data quality test plan to determine:– Magnitude and type of

missing encounter data– Overall data quality issues– MCO data submission issues

21

EQR Protocol


• STEP 2 - Verify integrity of encounter data– Macro-level analysis– Encounter file completeness and

reasonableness» Volume and utilization by encounter type and

service setting» Internal field consistency» General field completeness and validity

22

EQR Protocol


• STEP 3 – Generate and Review Analytic Reports– Micro-level analysis– Encounter record completeness and

reasonableness» Follows similar analysis as outlined

in Step 2» Analyzing volume/consistency by

time, provider, service type

23

EQR Protocol


• STEP 4 – Compare findings to state-identified standards– Identification of appropriate benchmark

population

24

EQR Protocol

Five key activities, continued4. Review of medical records

• Verification of the accuracy of coding• Protocol assumptions• STEP 1 – Determine sampling for medical record

review– Identify valid sample size– Encounter- vs. recipient-based samples

25

EQR Protocol

Five key activities, continued4. Review of medical records

• STEP 2 – Obtain and review medical records and document findings– Procurement efficiencies– Abstraction staff and training– Categorization of errors by level, type, and

source– Procurement tracking and abstraction tools

26

EQR Protocol

Five key activities, continued5. Submission of findings

• Narrative report summarizing findings from Activities 1-4

• Actionable recommendations for overall encounter data quality improvement

27

Proto what?

Questions?

28

Objectives




29

Objectives




31

SFY 2013-2014 Encounter Data Validation (EDV) Study

Agency for Health Care Administration

32

VALIDATION OF ENCOUNTER DATA


Review proposed encounter data validation process– Submitted as part of EQR RFP response– Will be conducted in alignment with CMS’ EQR

Protocol 4– Evaluates the accuracy and completeness of encounter

data submitted to AHCA by capitated health plans

33


Background– Experience– Core competency evaluating data

• Information system reviews• Comparative analyses of MCO and State Medicaid data• Medical/clinical record review

– Methodology is constructed to provide an effective validation of the quality of data maintained by State agencies within resource requirements

34


35


Four key steps for conducting successful evaluations

– Project implementation– Study design– Data collection &

analysis– Reporting &

recommendations

36


Project Implementation– Kick-off meeting with AHCA

• Initiated during contract implementation period• Review and define overall scope of project• Discuss anticipated timelines• Define evaluation parameters

– Number of MCOs included– Data requirements and limitations– Implementation procedures to validate AHCA’s encounter data

– Kick-off meeting with participating MCOs• Description of project and finalized study methodology• Expectations for MCO involvement

37


Study design– Prepare draft methodology including:

• Study objectives and research questions• Data source and collection procedures• Measurement methodology • Analytic methods• Timeline

– Review and approval of methodology by AHCA – Develop of detailed analysis plan or technical companion

document methodology

38


Data collection and analysis– Information systems review

• Scope to be defined in collaboration with AHCA

• Identify key encounter data policies and procedures

– Selection of key evaluation fields, service groups, and encounter types

– Identification of existing/proposed standards– Review of processes affecting data quality

• Expected to be limited in scope– Focused on building contextual knowledge of

systems to facilitate development of effective and actionable recommendations

39


Data collection and analysis– Information systems review, continued

• Request for supplemental documents– Encounter data submission process– Previous studies conducted by AHCA

• Documentation will be used to assess encounter data quality• Used of NCQA® Roadmap where appropriate

40


Data collection and analysis, continued– Encounter data source files

• Review of State encounter data file layouts• Prepare data requirements documents • Receive, process, and load encounter data

– Final status encounters from the Florida Medicaid Management Information System and Decision Support System (FMMIS/DSS)

– Final status claims/encounters from MCO adjudication systems– Includes all claim/service types—i.e., inpatient/outpatient, physician

visits, dental, and pharmaceutical

41


Data collection and analysis– Comparative data analysis of State and MCO

encounter data• Evaluates the extent to which encounters submitted by MCOs to

AHCA are accurate, complete, and reasonable• Preliminary file review

– Ensures files are sufficient for processing– Involves the basic checks

» Percentage present» Percentage valid» Percentage valid values

42


Data collection and analysis, continued– Comparison: State data to MCO data

• Indicators to measure degree of completeness and accuracy for each encounter type

– Overall record matching—percentage of state encounters present in MCO files

– Field-level matching—percentage of state encounters with exact value match in MCO file for each select data element

» Standard fields include: date of service, recipient ID, provider ID, primary diagnosis, procedure code(s), and payment fields

43

44


Table X—Diagnosis Code Matching Rates for Institutional Claims

Plan

Total Number of Matched

Claims

Encounter-Level Matching Field-Level Matching: % Correctly Matched

% With All Diagnoses Correctly Matched in

Both FilesIn First Diagnosis

FieldIn Second

Diagnosis FieldIn Third

Diagnosis FieldIn Fourth

Diagnosis Field

In Fifth Diagnosis

Field

Statewide 4,655,817 92.1% 99.1% 82.1% 88.2% 93.0% 94.9%

Plan A 144,090 96.3% 97.8% 99.0% 99.6% 99.8% 99.9%

Plan B 500,980 99.5% >99.9% 99.9% 99.8% 99.8% 99.8%

Plan C 2,429,624 89.1% 100.0% 75.4% 85.0% 91.5% 94.9%

Plan D 737,587 92.3% >99.9% 68.2% 75.3% 84.0% 89.8%

Plan E 224,193 >99.9% >99.9% >99.9% >99.9% >99.9% >99.9%

Plan F 367,800 89.8% 89.8% >99.9% >99.9% >99.9% 89.9%

Plan G 251,543 >99.9% >99.9% >99.9% >99.9% >99.9% >99.9%

44

45


Table Y—Second Diagnosis Field Code Matching Rates for Institutional Claims

MCPTotal Number of Matched Claims

% Correctly Matched in Both Files

% Mismatch Due to:

Diagnosis Omitted in State File

Diagnosis Omitted in Plan File

True Diagnosis Mismatch

Statewide 4,655,817 82.1% 0.9% 12.1% 4.9%

Plan A 144,090 99.0% <0.1% 0.0% 1.0%

Plan B 500,980 99.9% 0.1% 0.0% 0.1%

Plan C 2,429,624 75.4% 0.0% 23.2% 1.4%

Plan D 737,587 68.2% 5.7% <0.1% 26.1%

Plan E 224,193 >99.9% <0.10% <0.1% <0.1%

Plan F 367,800 >99.9% 0.0% <0.1% 0.0%

Plan G 251,543 >99.9% 0.0% <0.1% <0.1%

45

Phew… Questions?

46


Data collection and analysis, continued– Medical record review

• Represents the “gold standard” • Evaluation of service level accuracy

and completeness• Proposed methodology

– Only include MCOs operational as of January 2013– EQRO Contract Years 1, 2, and 3 (7/1/2013-6/30/2016): review one-

third of selected plans each year– EQRO Contract Years 4 and 5 (7/1/2016-6/30/2018): review one-half of

selected plans each year– Procure and abstraction 25 percent of all sampled records each quarter– Minimum 50 cases reviewed per plan– Target professional, dental, and pharmacy encounters

47



• Sample selection methodology1. To generate list of randomly

selected encounters for medical review, HSAG proposes using data files from comparative analyses

2. Two-stage stratified sampling design used to ensure:» Member’s record is selected only

once» Number of encounters included in

final sample covers all encounter types and proportional to total distribution of encounters

48



• Sample selection methodology– Identify all users by encounter type per MCO– Determine required sample size of each encounter type based on total

distribution of users– Randomly select users form each encounter type based on required

sample size– Identify all encounters associated with applicable encounter types for the

selected users– Final sample will consist of 50 cases randomly selected from applicable

encounter types per MCO per year, OR1,200 cases for 1/3 of all MCOs being reviewed per year

– For each encounter type, HSAG will define specific data elements for validation

49



• Procurement of selected sample records– General Process

» Once sample is selected, each MCO to receive list of its study cases

» HSAG will match selected date of service for each sampled member with rendering provider

» MCOs will procure and submit identified medical records to HSAG for review

50

SFY 2013-2014 Encounter Data Validation (EDV) Study Data collection and analysis, continued

– Medical record review• Procurement of selected sample records

– Two-hour technical assistance call with all participating MCOs

– HSAG to review project and procurement protocols– Able to accommodate a variety of procurement

methods:» Faxing» Hardcopy submissions» Electronic submission via secure file transfer protocol

– Note: HSAG applies strict protocols to ensure security and confidentiality of members’ medical records

51



• HSAG procurement and abstraction tool– Data collection, management, and reporting system

• HSAG reviewers are experienced:– Clinical nurses– Nurse coders

• Procurement and abstraction process– Based on established policies and procedures– Continually monitored to ensure validity and accuracy

» Inter-rater reliability testing & Rater-to-standard testing» All reviewers must achieve 95% accuracy rate» Variety of reports will be generated, i.e., medical record compliance

rates

52


Data collection and analysis, continued– Medical record review – analysis of cases

• Compare electronic encounter data to medical record data• Analyze record completeness and the accuracy of coding• Four primary indicators for data completeness and accuracy

1. Medical Record Agreement

2. Medical Record Omission (surplus)

3. Encounter Record Omission (missing)

4. Erroneous

53


Reporting and recommendations– Prepare draft report of findings including:

• Indicator results• Sub-analysis findings• Preparation of supplemental findings

for future evaluation by MCOs

– Presented for statewide and MCO-specific results– Actionable recommendations for improvement

54

Objectives




55

Questions

introduction to encounter data validation presenter: thomas miller, ma executive director, research...

Documents

accurate encounter data

complete data

service data

data validationpresenter

good data

adult medicaid quality

importance of measurement

accurate information