larry condelli & amanda duffy american institutes for research … · 2017-06-21 · leap...
TRANSCRIPT
LEAP Again: Updates and Training of Trainers on WIOA Accountability
Larry Condelli & Amanda Duffy American Institutes for Research
National Reporting System Support Project Presented at the Adult Education State Directors’ Meeting
June X, 2017
1
Questions?
Write questions on post-its.
We’ll collect and respond.
2
Purpose of This Session Review and clarify key accountability topics. Present updates to performance reporting requirements. Practice responding to challenging questions about National Reporting System (NRS) table changes.
3
Updates and Clarifications
Periods of Participation (PoPs) and Assessment:
Review procedures. Carry over across PoPs and program years. Implications regarding PoPs. Placement and posttest gains.
Table updates and reminders:
Table 4, Contact hours. Table 4b, Participants who are pre- and post-tested.
4
Periods of Participation (PoPs) and Assessment
PoPs –Reminder of Policy A Period of Participation (PoP) begins when an individual enters the program. A PoP ends at program exit (90 days without service and no scheduled services). A PoP does not end when a program year ends.
6
7
Periods of Participation (POP)Follow-Up Indicators Example
Jul 1st Jun 30th
PoP
12+ Contact Hours
12+ Contact Hours
Jul 1st Jun 30th PY 2016 PY 2017
PoP
Follow-up must occur for PY 2016 exit-based indicators.
Follow-up must occur for PY 2017 exit-based indicators.
Program Entry/ Reentry Program Exit (90 days since last service)
PoPs –Implications Participant Status
1. An individual becomes a participant once he/she has attained 12 hours of instruction within a period of participation (PoP).
2. Participant status is retained until an exit occurs.
3. Individuals must achieve participant status each time a new PoP begins.
8
PoPs –Implications Participant Status (continued)
4. Participants who remain continuously enrolled across multiple program years do not need to requalify as a participant in new program years.
5. Measurable Skill Gain (MSG) is reported at least once for every participant per program year.
9
Periods of Participation (POP) MSG Indicator Example
10
Jul 1st Jun 30th
Program Entry/ Reentry Program Exit (90 days since last service)
PoP
MSG Reported for
PY 2016
MSG Reported for PY 2016
12+ Contact Hours
12+ Contact Hours
Jul 1st Jun 30th PY 2016 PY 2017
PoP
MSG Reported for PY 2017
10 Contact Hours
Participant Participant
Changes to Reporting MSG in a Prior PoP Purpose: To ensure PoPs do not have an impact on proper testing practices, the following clarifications have been made.
Educational Functioning Level (EFL) placement may be carried over from a previous PoP, if the state accounts for this practice in the state assessment policy and the test is still valid according to the test publisher’s guidelines. EFL gain via post-testing may be achieved in a PoP based on post-testing in a subsequent PoP, assuming the test scores are still valid according to test publisher guidelines.
11
Reporting MSG in a Prior PoP: Scenario 1 MSG may be applied to a previous PoP under the following scenario:
◦
◦
◦
A participant exits, has enough hours to posttest (according to state policy and test guidelines), but does not complete a posttest.
The individual reenters the program (PoP2) and is tested at entry.
Since the participant had already qualified to posttest based on state policy and test guidelines, the test given upon reentry may be used as the pretest for PoP2 and EFL gain may be counted for PoP1, if a gain is achieved.
12
Reporting MSG in a Prior PoP: Scenario 1
13
Jul 1st Jun 30th
Program Entry/ Reentry Program Exit (90 days since last service)
PY 2016
POP 1 POP 2
EFL Gain achieved based on POP2 test
Attends 70 Hours
PY Initial
Placement PoP 2
Placement
ABE Lvl. 4 Math Test
ABE Lvl. 5 Math Test
Reporting MSG in a Prior PoP: Scenario 2 EFL gain may also be applied to a previous PoP under this scenario:
◦
◦
◦
◦
A participant exits, does not have enough hours to posttest (according to state policy and test guidelines), and is not post-tested. The individual returns to the program (PoP2), is tested after achieving enough instructional hours (combining hours in PoP1 and PoP2), and achieves an EFL gain. This test in PoP2 serves as the pretest for PoP2 and as a posttest for PoP1. The EFL gain may be counted for PoP1. EFL gain in PoP2 would require another test after the participant receives sufficient instructional hours for a posttest.
14
Reporting MSG in a Prior PoP: Scenario 2
15
Jul 1st Jun 30th
Program Entry/ Reentry Program Exit (90 days since last service)
PY 2016
POP 1 POP 2
EFL Gain achieved based on POP2 test
Placement pulled
forward from PoP1
Attends 12+ Hours Attends 60 Hours
PY Initial
Placement
Placement for POP2
MSG
ABE Lvl. 4 Math Test
ABE Lvl. 5 Math Test
Reporting MSG in Prior PoP: Requirements In order to apply MSG across PoPs via pre-/post-testing:
◦
◦
The participant must have sufficient hours to posttest according to state policy and test guidelines from either PoP1 or combined PoP1+PoP2.
The PoPs must be in the same program year. MSG cannot be retroactively applied to a prior program year.
16
Activity With a partner, write a scenario and response that may happen in a program around periods of participation and assessment. Use the information learned on slides 12 and 14 to support your writing. Quiz another team with your scenario.
Have fun with this! Be creative! This will help you respond to locals’ questions.
17
Questions?
18
Assessment: Placement and EFL Gain
20
Assessment and EFL: Setting Initial Placement
As of July 1, 2017:
Initial Placement is the first student placement assigned during a program year.
Initial placement for a program year may be set using any subject area of pre-tests given. NRS will no longer indicate placement in the lowest subject area of all tests.
The EFL for initial placement should be locked in upon assignment to ensure accurate data reporting.
Assessment and EFL: Reporting based on Initial Placement
All participant data reported on NRS Table 4 will be reported on the row of the initial placement for a given Program Year. Only columns I and J will be updated on NRS Table 4 for
subsequent PoPs. Include total attendance hours for all PoPs in column C.
21
Assessment: Changing How EFL Gains Can Be Made
22
Under WIA
EFL gain was measured with a posttest in the subject area in
which the participant was initially placed.
Under WIOA
Measurable Skill Gain via EFL gain may be measured with a posttest
in any subject area in which a participant was pre-tested.
Assessment and EFL Gain: Example: Jim
23
Initial Placement
Jim enrolls and scores at the ABE 3 level in reading and the ABE 4 level in math. The local program may place him based on either the reading OR math score. His reason for attending the program is to work on math so the program places him at ABE level 4.
EFL/MSG Gain
Jim, in ABE Level 4, took a reading post-test and progressed from ABE Level 3 into ABE Level 4. Even though this was not the subject area for his initial placement, he would be credited with achieving an EFL gain under the MSG indicator.
Reporting Gain on NRS Table 4: Jim’s Example Since Jim was initially placed at ABE Level 4 based on his Math pre-test score, all of his data would be reported on the ABE Level 4 row for NRS Table 4.
Assume Jim took a post-test, made an EFL gain, and exited the program. (PoP1) Assume Jim returned to the program and completed a secondary credential after reentering the program. (PoP2)
Entering Educational Functioning
Level
Total Number Enrolled
Total Attendance Hours for
all participants
Number who achieved at least one
educational functioning level gain
Number who
attained a secondary
school diploma or
its equivalent
Number Separated
Before Achieving
Measurable Skill Gains
Number Remaining in Program
without Measurable Skill Gains
Percentage Achieving
Measurable Skill Gains
Total number of Periods of
Participation
Total number of Periods of
Participation with
Measurable Skill Gains
Percentage of Periods of
Participation with
Measurable Skill Gains
(A) (B) (C) (D) (E) (F) (G) (H) (I) (J) (K) ABE Level 4 ABE Level 5
1 1 152 0 0 0 100% 2 2 100%
24
••
Guiding Questions At your table, discuss one of the following questions: ◦
◦
◦
How might the changes discussed affect program practice? Positively or negatively? Why? Do you have concerns about accurate testing or misuse of tests? What are they and why? What affect might the changes have on data collection or data systems? On performance?
25
Questions?
26
Credential Attainment Rate
27
Credential Attainment Rate
28
Secondary Credential
Participants without a secondary diploma or its recognized equivalent who were in a secondary education program at or above the 9th grade
level and who exited from the program.
A secondary education program refers to enrollment rather than placement.
Postsecondary Credential
Participants who were co-enrolled in a postsecondary education or
training program that leads to a credential and who exited the
postsecondary education or training program.
Exiting an IET program in which the postsecondary education or training leads
to a credential is considered exiting for this purpose.
Credential Attainment Indicator- Calculations
29
Participants who obtained a secondary school diploma or its recognized equivalent during the program or within one year after exit and who were employed or in an education or training program leading to a recognized postsecondary credential within one year after exit.
Participants who obtained a recognized postsecondary credential during the program or within one year after exit.
Secondary Postsecondary
Numerator
Denominator
Participants without a secondary school diploma or its recognized equivalent who were in a secondary education program at or above the 9th grade level and who exited from the program.
Secondary Postsecondary
Participants who were co-enrolled in a postsecondary education or training program that leads to a credential and who exited from the program.
Credential Attainment Indicator- Calculations
30
Numerator
Denominator
Participants who achieve a secondary and postsecondary credential should be reported only once in the numerator for the credential indicator based on the highest credential attained.
Participants who qualify for the secondary and postsecondary credential cohorts should be reported only once in the denominator for the credential indicator.
NRS Tables 4C and 5A
Participant data for all periods of participation are included on distance education tables, if the participant met the state’s definition of a distance education participant during any POP during the program year.
31
Reporting Timeline 2017-2020
32
October 15, 2017, is a Sunday. The reporting date will slide to October 16, 2017.
NRS Table Questions
33
Table Activity
34
Wrap-up
35
NRS Regional Training, Fall 2017 WIOA Accountability: Year One Data Check-Up Pre-Training Activities: ◦ As you review your year end data, identify “problems” and write 1-3
questions that need answers. Example: How do we provide better follow-up? ◦◦
Consider your current data flow. Check your system for validation rules and error checks.
36
Additional Questions? Please contact us at:
Alan Tucker [email protected] Larry Condelli [email protected] Jay LeMaster [email protected]
37