discretionary grants: data collection, processing & reporting 1 back-to-basics - discretionary...

1

Discretionary Grants: Data Collection, Processing & Reporting

Back-to-Basics -Discretionary Grants

Data Collection, Processing & Reporting

Employment and Training AdministrationU.S. Department of LaborFrances Perkins Building200 Constitution Avenue, NWWashington, DC 20210www.doleta.gov

2


Data Collection

Data Processing

Reporting Requirements

Session Objectives

1

2

3

Note: The discretionary grantees that are the focus of this session are those subject to USDOL’s common measures (e.g., High-Growth and Community-Based Job Training Grantees)

3


Reports and Information

Reports and Information

Data Collection

Data Collection

Data Processing

Data Processing

“What matters in the end isCompletion. Performance.Results. Not just making promises, but making good on promises. - President’s Management Agenda -

Performance Management

See Handouts 1 and 2

4


29 CFR Parts

95 & 97

29 CFRPart 37

PublicLaw

107-288

Grant Agreement

Federal Requirements

Data Collection & Reporting Requirements

5


Data Collection—IntroductionData Collection—Introduction

Process used to gather and collect information for the management of the project

To develop a tracking mechanism, must address four fundamental questions:

What information needs to be collected and in what format?

Where does the grantee obtain the information?

When does the information need to be collected?

How does the grantee know the information is accurate/valid?

6


Data Collection—What Information To Collect?Data Collection—What Information To Collect?

What information is needed?

Start with grant proposal and grant award documents

Review goals, objectives, & expected outcomes for your grant

Review Work Plan, Service Flow Chart, Performance Measures, Project Deliverables, etc.

Involve internal staff and external partners in the process of identifying data collection elements

They know what format it comes in and when the information is collected

7


Case Management

Cycle

Application/ Intake/Eligibility

Data Collection—Service FlowData Collection—Service Flow

Service Delivery

Job Development Exit &

Follow-Up

Service Planning

Assessment

See Handout 3

8


Data Collection—Participant CharacteristicsData Collection—Participant Characteristics

Types of individual characteristic information to collect includes:

Participant contact and emergency contact information (e.g., name, address, telephone)

Information to assess a person’s eligibility for the project (tailored to each grantee based on definition/information from the proposal)

EEO information (e.g., age, sex, ethnicity, race, and disability status)

Veteran’s characteristics and status

Continued

9


Data Collection—Participant CharacteristicsData Collection—Participant Characteristics

Continued from previous slide

Education information (e.g., highest school grade completed)

Employment status at enrollment and past/current employment information

Information about supportive service needs and additional reportable characteristics (e.g., offender, runaway, low income, single parent, etc.)

10


Data Collection—Participant ServicesData Collection—Participant Services

Collect information about the services provided to individual participants

Types of services in which the participant is enrolled

Some examples include career guidance services, job search assistance activities, supportive services, customized training, on-the-job training, apprenticeship training, occupational skills training, Adult Basic Education/ English Second Language training

Participation or service enrollment dates

Service completion dates

Training provider information

11


Data Collection—Participant OutcomesData Collection—Participant Outcomes

Common Measures Initiative

Why use common measures?

Ability to describe in a similar manner the core purposes of the workforce system—how many people found jobs? Did they stay employed? What did they earn?

Enhance ability for different programs to collaborate by reducing multiple reporting requirements

Ability to analyze program outcomes and develop effective program strategies

Ability to compare results of similar government programs

12


DOL Common Measures PolicyDOL Common Measures Policy

Common measures is required for ETA formula-funded grants and some discretionary grants

High growth and community-based grantees are subject to the common measures policy

Training and Employment Guidance Letter (TEGL) No. 17-05, dated February 17, 2006 details DOL policy for common measures

See TEGL 17-05

13


What Common Measures Apply to Your Grant?What Common Measures Apply to Your Grant?

Adult Measures

Entered Employment

Employment Retention

Average Earnings

Youth Measures

Placement in Employment or Education

Attainment of a Degree or Certificate

Literacy or Numeracy Gain

14


Required Definitions—ParticipantRequired Definitions—Participant

Participant = An individual determined eligible to participate in the program who receives a service funded by the program in either a physical location (e.g., One-Stop Center) or remotely through electronic technologies

You need to determine what services trigger participation in your grant

You may need to capture information about services provided remotely

15


Required Definitions—Other Terms Related to ParticipationRequired Definitions—Other Terms Related to Participation

Participation Date: Date of first program-funded service

It’s possible to record an earlier date if services were being provided by another partner at the time of participation

Discretionary grantees, however, cannot report a participation date that begins before the actual period of performance in the grant award document

Participation Quarter: Calendar quarter containing the participation date (e.g., If 1st service is 4/22/07 then participation quarter is 2nd CY quarter)

Participant Cohort: Group of individuals who share the same participation quarter

16


Required Definitions—ExitRequired Definitions—ExitExit: A participant who hasn’t received a program or partner-funded service for 90 consecutive days and no future services are scheduled has exited the program

1. Hasn’t received a service

2. No service received for 90 consecutive days

3. No future services scheduled (does not include any follow-up services or circumstances where the participant voluntarily withdraws or drops out of the grant)

For example, if a participant completes training on 2/13/2007 and planned to take one additional specialty course but then failed to do so, the exit date would be retroactive to 2/13/2007.

Discretionary grantees cannot report an exit date that extends beyond the actual period of performance in the grant award document

17


Required Definitions—Other Terms Related to ExitRequired Definitions—Other Terms Related to Exit

Exit Quarter: Calendar quarter containing the exit date

For example, the exit quarter for individuals exiting on 2/13/2007 would be Q1 of 2007

Exit Cohort: Group of individuals who exit during the same calendar quarter.

For example, all participants that exit between 1/01/07 and 3/31/07 are in the same exit cohort

17

18


Last ServiceDate Triggers

Exit Date

Date of 1st

Service Triggers

Participation Date

Gap inServicesAllowed

(see TEGL 17-05Page 22)

No ServicesReceived for

90 ConsecutiveDays

Follow-upDiscussed

in TEGL 17-05(see TEGL 17-05

Pages 20-21)

Service DeliveryService Delivery

Eligible &Receives Grant

Services

ParticipantReceiving Grant

Services

Last ServiceReceived & No

Future Services Scheduled

ParticipantReceivesFollow-upServices

19


Basic Common Measurement — Rules and ConceptsBasic Common Measurement — Rules and Concepts Only participants are included in performance calculations

Specific conditions exclude participants from calculations, such as being employed on the date of participation means excluded from the entered employment measure

Global exclusions if participant becomes (1) institutionalized, (2) invalid or missing SSN, (3) terminates due to health/medical or family care, (4) dies, (5) is a reservist called to active duty, and (6) relocates to a residential or non-residential program (applies to youth).

Calculations are based on exiters

Exception is Youth Literacy/Numeracy because based on one year from participation date

Each measure is based on a specific cohort

For example, a person who was unemployed on their participation date and exits on 4/28/07 would be in the entered employment rate cohort of Quarter 3 of 2007 (July 1-September 30, 2007)

Specific data sources must demonstrate the outcomes

20


Exit CohortsExit Cohorts

Exit Cohorts

Oct 1 -

Dec 31

Jan 1 –

Mar 31

Apr 1 –

Jun 30

Jul 1 –

Sep 30

Jan 1 –

Mar 31

Apr 1 –

Jun 30

Jul 1 -Sep 30

Oct 1 -

Dec 31

Apr 1 –

Jun 30

Jul 1 –

Sep 30

Oct 1 –

Dec 31

Jan 1 –

Mar 31

Jul 1 –

Sep 30

Oct 1 -Dec 31

Jan 1 -Mar 31

Apr 1 –

June 30

Exit Participant Exit Quarter

Q1 1st Quarter After Exit Quarter

Q2 2nd Quarter After Exit Quarter

Q3 3rd Quarter After Exit Quarter

21


Pay stubsProgress ReportsSurveysSelf-attestation formsCopy of DiplomaTraining Certificates InterviewsPublic Agency RecordsStudent ID

Social Security CardDriver’s License/ID CardHospital Records Intake/Eligibility FormsAttendance SheetsSign-In SheetsSchool RecordsActivity FormsAssessment Results

Data Collection—Where to Get the Information?

Source DocumentationSource Documentation

22


Wage Records—Data Source for Outcomes MeasuresWage Records—Data Source for Outcomes Measures

Discretionary grantees that are State and local workforce investment boards or one-stop providers will most likely have access to wage record data

Employers are required to send confidential quarterly reports of wage earners and their wages for the respective calendar quarter

The majority of discretionary grantees, however, will not have access to Unemployment Insurance (UI) wage records

The following websites provide fee-based access to employment verification and wage information, but not all employers provide information to these sources: (1) http://www.theworknumber.com/; (2) https://www.verifyjob.com/; and (3) http://www.usverify.com/incomeverify.html

See Handout 6

23


Wage Records—Data Source for Outcomes Measures (cont’d)Wage Records—Data Source for Outcomes Measures (cont’d)

Wage records are the preferred source of information for calculating common measures

The proposed reporting package for HG/CB job training grantees would require grantees to submit quarterly participant records for each exiter to ETA containing 4 data elements

ETA would then use these records to obtain wage record data and calculate the common measures for grantees

This is proposed!

See Handout 6

24


Supplemental Data SourcesSupplemental Data Sources

Most discretionary grantees will need to use supplemental data sources to verify employment and wages

Copy of a participant’s pay stub

Employer contact—telephone or written verification

Participant contact—telephone, participant surveys or written self-attestation form

Case management notes

All supplemental data sources must be documented in participant case files and are subject to audit

See Handout 6

25


Data Sources for Measures Not Related to EmploymentData Sources for Measures Not Related to Employment

Pertains to all three youth measures

Youth Placement in Employment or Education

Administrative records utilized for placement in education (wage records or supplemental data for employment portion)

Attainment of Degree/Certificate

Administrative records utilized

Literacy and Numeracy Gains

Assessment instrument

25

26


Entered EmploymentEntered Employment

Of those who are not employed at the date of participation:

Number of adult participants who are employed in the 1st quarter after the exit quarter

Number of adult participants who exit during the quarter

27


Entered EmploymentEntered Employment

Employed atParticipationDate?

Employed in1st quarterafter exit?

Excluded from Measure

Numerator

Denominator

YES

NO

NO

YES

28


Employment RetentionEmployment Retention

Of those who are employed in the first quarter after the exit quarter:

Number of adult participants who are employed in both the second and third quarters

after the exit quarter


29


Employment RetentionEmployment Retention

Employed in 1st quarter after exitquarter?

Employed in2nd and 3rd

quarters afterexit quarter?


Numerator

Denominator

NO

YES

NO

YES

30


Earnings MeasureEarnings Measure

Of those adult participants who are employed in the first, second and third quarters after the exit quarter:

Total earnings in the second quarter plus total earnings in the third quarter after the exit quarter


31


Average Earnings Average Earnings

Employed in 1st quarter afterexit quarter?

Employed in2nd and 3rd

quarters afterexit quarter?


Numerator = Wages from quarters 2 & 3, post-exitincluded

Denominator = Number ofadult participants who exitduring the quarter

No

Yes No

Yes

32


Youth Placement in Employment or EducationYouth Placement in Employment or Education

Of those who are not in post-secondary education or employment (including the military) at the date of participation:

Number of youth participants who are employed (including military) or enrolled in post-secondary education and/or advanced training/occupational skills training in the first

quarter after the exit quarter

Number of youth participants who exit during the quarter

33


Youth Attainment of a Degree or CertificateYouth Attainment of a Degree or Certificate

Of those who are enrolled in education--at the date of participation or at any point during the program:

Number of youth participants who attain a diploma, GED, or certificate by the end of the third quarter

after the exit quarter

Number of youth participants who exit during the quarter

34


Youth Literacy and Numeracy GainsYouth Literacy and Numeracy Gains

Of those out-of-school youth who are basic skills deficient:

Number of youth participants who increase one or more educational functioning levels

Number of youth participants who have completed one year of participation in the youth program (one year from date of first program service) plus the number of youth participants who

exit before completing a year in the youth program

35


Data Collection—Is That It?Data Collection—Is That It?

While you are required to track information to report on the common measures, other management information continues to be necessary

Collect information relevant to the day-to-day operations of your project (e.g., information about enrollments and counts on various service dimensions)

May collect information to help case managers, job developers, managers, etc. make programmatic decisions

36


Data Collection—When to Collect the Information?Data Collection—When to Collect the Information?

Important to develop an information management plan addressing:

How will data be collected?

Information collected verbally, with paper documents or electronically

Who is responsible for the data collection?

Assign responsibility for data collection to specific staff to ensure that expectations are clearly understood

In what format will data be collected?

MS Excel spreadsheet, MS Access, Proprietary system

When will data be collected?

Incorporate deadlines & establish timeframes for all collection tasks

37


Data Collection—Is the Information Reliable?Data Collection—Is the Information Reliable?

Ensure data elements are well defined and understood by project staff—this includes contractors

Train all staff on definitions and processing requirements

Note that original source of information more reliable than secondary information

Ensure duplication of effort does not occur in the collection of data

Costly

Lack of Consistency

38


Data Processing—IntroductionData Processing—Introduction

Process used to input data and information into a MIS, find and correct errors in the data, and compile and aggregate the information into a user-friendly format

Grantees must address these fundamental questions:

Who has responsibility for data entry, compilation and processing?

How should they validate the data and ensure its integrity?

What management information system will be used to maintain the data?

39


Data Processing—Who Has Responsibility?Data Processing—Who Has Responsibility?

The grantee is responsible for ensuring that a system is in place to track participant characteristics, services and outcomes

It is highly recommended that grantees maintain access to data processing and reporting at all times

You need to know what is going on with your grant!

The grantee may contract out for services, but should provide input on how the data is maintained and gathered

40


Data Processing—Who Has Responsibility?Data Processing—Who Has Responsibility?

All staff, including sub-recipients staff, need to clearly understand their role in creating, collecting, maintaining, and reporting program data

Understand interrelationship of staff member duties

Understand consequences of missing deadlines, not sharing information, etc.

Reduce errors by training staff and making sure they meet specified qualifications

41


Data Processing—Is the Data Reliable?Data Processing—Is the Data Reliable?

Data needs to be reliable…accurate, complete and consistent

Factors affecting data reliability:

Lack of data collection and data processing policies and procedures

Inaccurate and incomplete data

Insufficient staff training

Differences in definitions

Insufficient system controls

42


Data Processing—Is the Data Reliable?Data Processing—Is the Data Reliable?

Data errors can occur at any point in record keeping. To help maintain data quality, consider the following:

Assign ranges of possible values to each data element A person’s age could range from 16-100, anything outside the

range would be flagged by the system

Check consistency across data elements Same identification number but different race or sex

Look for missing data

Conduct periodic reviews of information keyed into MIS compared with original sources (i.e., sample of case files and MIS cases)

Remember data must be auditable—must be able to track back to original source

43


Data Processing—What System Should We Use?Data Processing—What System Should We Use?

How sophisticated or elaborate does the grantee’s MIS need to be?

It varies, but high-performing organizations have an MIS that produces information/reports to assist staff in addressing issues and improving performance

Examples include MS Access, MS Excel, or a proprietary system such as Client Tracking System

Contract with local workforce investment areas to process and aggregate data

What must grantee MIS be able to do? At a minimum, capture all required data elements, perform any

necessary calculations and report information to the grantee and its partners

In some cases, ETA may provide the required data collection instruments and a system for collecting and reporting the information

See Handout 5

44


Data Processing—What System Should We Use?Data Processing—What System Should We Use?

Key principles of an effective MIS: Enter information only once

Enter information in a timely manner

Enter information at the most cost-effective level of the organization (e.g., line staff versus technical staff)

Information available on demand

Access available to all levels in the organization, as needed

45


Reports and InformationReports and Information

Ability of grantee to use data processing and processing techniques to develop reports

Need for an information system that can give you reports that tell a story about your grant

What other data help tell your story?

What about customer satisfaction data?

Don’t collect data simply to ‘feed the Federal data machine,’ but to inform program design and

management

46


Reports and Information—Federal RequirementsReports and Information—Federal Requirements

Reporting requirements based on Uniform Administrative Requirements in 29 CFR Parts 95.51 and 97.40

Comparison of actual accomplishments with goals and objectives for the project

Reasons why established goals were not met and corrective action being taken

Other pertinent information, including technical assistance needs, best practices or any promising approaches

In some cases, ETA has a required report format for your use with established due dates

47


Reports and Information— Common Measures Reports and Information— Common Measures

Common measures are based on specific cohorts

Cohorts based on calendar quarters January 1 to March 31, 2007 = CY 2007 Q1 April 1 to June 30, 2007 = CY 2007 Q2 July 1 to September 30, 2007 = CY 2007 Q3 October 1 to December 31, 2007 = CY 2007 Q4

If using wages to calculate performance on employment, earnings, and retention there will be a lag in when the wage information is available

48


Reports and Information—Table of Reporting PeriodsReports and Information—Table of Reporting Periods

Reporting Time Periods

Oct 1 -

Dec 31

Jan 1 –

Mar 31

Jul 1 -Sep 30

Apr 1 –

Jun 30

Oct 1 –

Dec 31

Jul 1 –

Sep 30

Jan 1 -Mar 31

Exit Participant Exit Quarter

Q1 1st Quarter After Exit Quarter

Q2 2nd Quarter After Exit Quarter

Q3 3rd Quarter After Exit Quarter

49


Reports and Information—Is the Information Reliable?Reports and Information—Is the Information Reliable?

Employ mechanism(s) to ensure reports generated and outcomes reported are calculated correctly

Useful to have a data entry/MIS handbook Reference tool for system users that details data

collection protocols, definitions of terms, coding assignments, acceptable data sources, operational rules for data entry, available edit checks and report operations

50


Data Collection

Data Processing

Information & Reports

Conclusion

Measuring performance can be a powerful mechanism for managing performance!

Internal managementQuarterly ETA reportsStakeholder reportsCustomer satisfactionEvaluation reportsContractor reports

Internal managementQuarterly ETA reportsStakeholder reportsCustomer satisfactionEvaluation reportsContractor reports

Data EntryOrganizing dataCompiling dataAggregating dataMISExcel spreadsheetsAccess databasesAnalyzing data

Data EntryOrganizing dataCompiling dataAggregating dataMISExcel spreadsheetsAccess databasesAnalyzing data

Intake/Eligibility formsAttendance sheetsActivity formsProgress reportsRecords of completionExit & follow-up formsSatisfaction surveys

Intake/Eligibility formsAttendance sheetsActivity formsProgress reportsRecords of completionExit & follow-up formsSatisfaction surveys

51


Are there any questions?

THANK YOU

discretionary grants: data collection, processing & reporting 1 back-to-basics - discretionary...

Documents

data collection elements

telephone information

discretionary grantees

proposal eeo information

emergency contact information

employment status

grant proposal

service flow chart