The Data-informed Decision-making Challenge

PVAAS Overview:Evaluating Growth,Projecting PerformanceWebinar

PVAAS Statewide Core TeamFall 2011August 2011

PVAAS Overview1Welcome to PVAAS 2011!

This webinar will provide an overview to PVAAS reporting available to public districts and schools in Pennsylvania.

If you would like this presentation in PowerPoint format, please enter your email address into the chat box on the lower left-hand side of the screen. We will email the PowerPoint file out to you at the conclusion of this session. The PowerPoint used during this session has been pushed out to all webinar participants in PDF format.

More information on any PVAAS-related topic is available on the PDE website or by contacting the PVAAS Statewide Core Team; contact information is provided at the end of this presentation.Goal of SessionTo provide educators with an understanding of the Pennsylvania Value-added Assessment System and the meaning and uses of the following reports.Web-based Scatter PlotsDistrict/School Value-Added ReportValue-added Summary ReportDistrict/School Performance Diagnostic ReportIndividual Student Projection ReportProjection Summary Reports

The purpose of this presentation is to provide educators with an understanding of PVAAS and the meaning and uses for the reports listed on this slide. These are key reports to use for data-informed decision making in districts and schools.

PVAAS provides powerful new data to be incorporated with other data to create a more accurate and meaningful profile of student performance.PVAAS Overview2AgendaWhat is PVAAS?ScatterplotsPVAAS Value-added ReportsGoing Deeper with Value-addedPVAAS ProjectionsLines of InquiryResources and SupportHere is the agenda for todays session.

What is PVAAS?ScatterplotsPVAAS Value-added ReportsGoing Deeper with Value-addedPVAAS ProjectionsLines of InquiryResources and SupportPVAAS Overview3What Is PVAAS?PVAAS Overview4We will now describe what PVAAS is and clarify what PVAAS is not.PVAAS IS:Analysis of existing assessment dataLongitudinal PSSA scores in available content areasMath, Reading, Science, WritingProvides reliable estimates

A measure of a schools effectiveness

Critical Relationship: Growth Data vs. Achievement DataPVAAS Overview5In this high-stakes assessment and accountability climate, we need accurate information to guide us in making appropriate decisions for studentsdecisions about instruction, curriculum, and assessment. PVAAS is one tool to assist schools in this process. PVAAS is an analysis of existing assessment data (PSSA) no other testing is necessary. PVAAS utilizes longitudinal PSSA test scores to produce reliable estimates of growth.

PVAAS is a tool to provide a measure of a schools effectiveness from year to year.

It provides progress data to add to achievement data as schools are analyzing their educational programs for students.

PVAAS offers powerful diagnostic information about school practice by leveraging student achievement data with complex statistical tools. It provides a tool to identify strengths and weaknesses in educational programs. PVAAS:Is not another test

Critical conceptIs not a comparison of single test scores

PVAAS Overview6PVAAS is not another test. PVAAS uses existing assessment data, specifically the PSSA in its statistical models to yield measures of growth. The PSSA is the ONLY common assessment used by all public districts and schools in Pennsylvania. If there are other assessments in the future that all public districts and schools use, they could potentially be added to the PVAAS model.

PVAAS measures growth by comparison of estimates based on multiple test scores. Comparison of 2 single test scores such as scaled scores are snapshots . This is NOT an effective way to measure growth.Review of LiteratureIs there a relationship betweenDemographics vs. Achievement?YES

Demographics vs. Growth?NODemographics are inherently controlled for in PVAAS due to the use of all longitudinal data on students!PVAAS Overview7From the 1960s, it has been established that there is a relationship between student demographics and achievement results. This is NOT the case when considering growth.

A review of the literature indicates there is NO relationship between growth/progress and demographic factors, such as socioeconomic status and racial/ethnic background.

There is significant evidence to this fact in research literature. (Please reference the trainer notes for this slide for some of these references if asked- or tell people you can provide references if they want them)

What does that mean well, it means high achieving schools can make high growth, high achieving schools can make low growth, low achieving schools can make high growth and low achieving schools can make low growth. Pennsylvania schools yield evidence each year to verify this.

References:A single measure of student achievement has inherent limitations due to the fact that achievement is correlated to a students socioeconomic status and past performance (Hershberg, et al.; Olson, 2007; Sanders, 2000).

Fallon (2003) reports that the importance of value-added assessment is that it is based on the experimental design that removes virtually all influence of genetics and socio-economic factors. The design provides a measure of the direct effect of the effectiveness of schools.

One of the advantages of value-added assessment systems is that it calculates the growth of an individual not relative to some generic cohort average but actually against that students own achievement levels in the previous year. (Callendar, 2004; Drury & Doran, 2003; Hershberg, 2005; Hershberg, Lea-Kruger, & Simon, 2004)

[Value-added assessment systems] can remove the effects of factors not under the control of the school, such as prior performance and socioeconomic status, and thereby provides a more accurate indicator of school or teacher influence than is possible when these factors are not controlled (McCaffrey, Lockwood, Koretz & Hamilton, 2003; Ross, Wang, Sanders, Wright & Stringfield, 1999a; Wright, Horn & Sanders, 1997).High AchievementAverage AchievementLow AchievementCurrent Data Collection ApproachesAchievement is based on each students ending point.PVAAS Overview8Lets look at how we have traditionally viewed school achievement.

We give a summative assessment, usually at the end of the school year, and see whether or not students are measuring up to end of year expectations, without taking into account their starting point.

By looking at end of year performance, we can conclude that some students, schools or districts are low, average, or high in terms of student achievement.

However, this does not provide the entire picture as we are trying to make data-informed decisions in our schools and districts.

This does not tell us about student growth, or progress, that may have occurred during that school year because we are not taking the starting point into consideration. PVAAS Adds Another Variable To AchievementGROWTH!Below GrowthStandardAt or AboveGrowthStandardGrowth is based on each students starting point!!PVAAS Overview9PVAAS DOES take into account the starting point.

PVAAS is a different way of looking at the data that adds the critical dimension of growth, or progress.

It is not another test but a comprehensive analysis of achievement trends and progress rates.

With PVAAS we are able to look at BOTH progress AND the level of achievement at a given point in time.

PVAAS is able to provide a measure that illustrates what effect schooling has had on students, regardless of variables such as ethnicity, socio-economic status, ESL and disability.

PVAAS indicates the growth, or progress, these groups of students have made between state assessments (PSSA) on an annual basis.

PVAAS is the only PDE data tool that provides these measures of growth.Scatter Plot Quadrants and Their Meanings

Quad 2 - SlippingAdequate AchievementNegative GrowthQuad 1 - ExcellingAdequate AchievementPositive GrowthQuad 4 - ImprovingBelow Adequate Achievement Positive GrowthQuad 3 - UnderperformingBelow Adequate AchievementNegative GrowthPVAAS Overview10Now, lets get a view of achievement and growth together. On this screen you see a scatterplot. The horizontal axis displays the PVAAS growth Index. A growth Value of 0 indicates the school met the standard for PA Academic Growth. Positive growth is plotted to the right of zero and negative growth to the left of 0. The vertical axis displays the % Proficient and Advanced. Weve highlighted the AYP target in the graph.What does it mean if a school was to be located in any of the 4 quadrants?

These two axis divide the scatter plot into four quadrants. Lets discuss the meaning of each quadrant:If a school has their dot in the first quadrant, it means that they have accomplished adequate achievement and positive growth. The farther the dot for the school is away from the intersection of the boundaries, the higher the achievement and/or the greater the evidence of growth. Some apply the term Excelling to schools in this quadrant.

If a school has its dot in the second quadrant, this indicates that they have adequate achievement but negative growth. This negative growth should be of great concern since continued adequate achievement may be in jeopardy. Some apply the term Slipping to schools in this quadrant.

If a school has its dot in the third quadrant, this indicates that the school has both below adequate achievement and negative growth. This situation suggests that substantial investigation and remediation is necessary since the students are not achieving and are headed the wrong direction. Some apply the term Underperforming to schools in this quadrant.

If a school has its dot in the fourth quadrant, this indicates the school has not achieved at or above the AYP target level; however, their growth is positive. This situation may indicate that interventions implemented are producing positive results. Some apply the term Improving to schools in this quadrant.Example: Regional Scatter Plot

PVAAS Overview11This scatter plot displays measures of both achievement and growth of 4th Grade Math for all the elementary schools in a district. These data are from an actual Pennsylvania district.

It appears that only four schools in the district did not reach the 56% achievement target for this particular year. Also, four did not meet the standard fro PA Academic Growth.

Notice there are two schools (4th quadrant) that exhibited positive growth but did not meet the 56% target. These schools are headed the right way and should consider continuing their efforts. The other two schools that did not make the 56% target exhibit negative growth. This is an indication that there is a need for substantial investigation and intervention.There are also two schools (2nd quadrant) that met the achievement target but exhibited negative growth. This is a warning sign that their students are slipping in their achievement and the conditions at those schools require some attention.

It is likely that personnel from the four schools may want to have a conversation with the personnel from several of the other schools that DID have similar achievement levels, but are making positive growth.

Example: Regional Scatter Plot: Same Schools, Different Subject

PVAAS Overview12This scatter plot displays the measures of achievement and growth in 4th Grade Reading from the same district as the Math scatter plot on the previous slide.

These are the same schools, same students, same communities, etc.

This display suggests that there is perhaps a more systemic issue to address in Reading throughout the district.

Remember what the math scatterplot looked like for these same schools!

Web-Based Scatter Plots

August 201113New this year are web-based scatter plots!PVAAS Overview13NEW! Web-based Scatter PlotsAugust 201114

The user may now create their own scatter plots for public districts or schools to which they have access. This feature is available at the top of the orange Reports tab. Various options are available to the user including selections for:

Grades 4-8 or Grades 9-11 Math, Reading, Science or WritingPlotting growth for all grade levels combined for a whole school or district growth measure, or each grade level individuallyPlotting growth for the current year, or the 3-year-average growth measure

Again the access is determined by your account access privileges as determined by your local education agency. Additionally, current PA legislation does not allow for grade level data to be reported publicly hence you would only have access to the grade level data for your LEA.

PVAAS Overview14NEW! Web-based Scatter Plots!August 201115Users may choose to plot different variables against achievement and/or growth!This includes demographics such as:% Tested Economically Disadvantaged% Tested Limited English Proficiency% Tested Special Education% Tested Minority

Users may choose to plot different variables against achievement and/or growth!

This includes demographics such as:

% Tested Economically Disadvantaged% Tested Limited English Proficiency% Tested Special Education% Tested Minority

PVAAS Overview15ExampleAugust 201116

This is an example of a new scatter plot showing percentage of students Proficient or Advanced against the PVAAS Growth Index.

Area summaries in the corner help bring meaning to the data plotted in each quadrant. Here you can see that schools plotted in the first quadrant and school that have a higher percent of Proficient and Advanced students and a higher PVAAS Growth Index. You can also view the percentage of schools falling in each scatter plot quadrant.

PVAAS Overview16ExampleAugust 201117

This is an example of a new scatter plot showing percentage of students tested in all tested grades that are identified as economically disadvantaged plotted against the PVAAS Growth Index for the school.

Notice that there is no correlation between growth and socioeconomic status. A similar visual occurs when showing all schools in Pennsylvania.

This section sums up a big picture view of achievement and growth using PVAAS as a lens for the conversation.

Now lets talk about the specific reporting in the PVAAS system. PVAAS Overview17TWO Types of PVAAS InformationTodayLooking Back/EvaluationValue-added Growth Reports For Groups of StudentsLooking Forward/PlanningPVAAS Projection Reports For Individual Students and Groups of StudentsPVAAS Overview18There are two main types of data from PVAAS:

The Value-added, or growth, information looks backit helps schools to evaluate the effectiveness of the school. How much growth did students make in the past school year?

The Projection information looks forwardit helps schools plan for the future for students. Are students on a path to proficiency or higher?

Both serve different purposes, and both are equally important for continuous school improvement.

We are going to start by discussing the value-added growth reports- and how measures of growth are determined in Pennsylvania. Value-addedWouldnt it be great to know if your district/school met the standard for PA Academic Growth for students tested in grades 4-8? And in Grades 9-11?PVAAS Overview19The answer to this question on the slide is YES of course!

In this portion of the presentation, we will discuss important considerations that provide a foundation for the statistical methodology that produces the growth measures of PVAAS.

You do not need to be a statistician to use PVAAS data however we want to give you a conceptual view of how growth measures are yielded in PVAAS so that your use of PVAAS can be more meaningful.

PVAAS Growth MethodologiesTestingSubjects GradesMethodologiesPSSA in consecutive years Mathematics 4-8 Reading 4-8Growth StandardPSSA not in consecutive yearsWriting 5, 8, 11Science 4, 8, 11Math & Reading 11PredictivePVAAS utilizes two methodologies for computing growth measures based on whether the assessments are administered in consecutive grades or not.

The Growth Standard Methodology is used for consecutive grades of data. (Grades 4-8, Reading and Math)

The Predictive Methodology is used for non-consecutive grades of data. (Grades 11 Reading and Math; Science and Writing)

PVAAS Overview20Growth Standard MethodologyMath & Reading Grades 4 through 8Definition: A group makes one years growth when they maintain their relative achievement level from one year to the next in relation to the statewide distribution (2006 baseline). Lets start with the Growth Standard Methodology for grades 4-8, reading and math. It is about maintaining the position of a group of students in the statewide distribution of test scores from one grade level to the next.

So what does that mean??? Is there a new target each year???PVAAS Overview21A Childs Growth ChartThis growth chart shows a little girl who is at the 15th percentile on her 1st birthday.

Her growth is typical if she stays at the 15th percentile in later years.

Well, lets use the growth of a child as an analogy.

This growth chart shows a little girl who is at the 15th percentile on her 1st birthday. Her growth may be considered typical if she stays at the 15th percentile in later years- if she were to maintain her position over time.PVAAS Overview22Now at 3 years old

XXXGrowth Spurt!Typical GrowthGrowth Concern?This slide depicts three different scenarios for the little girl who was in the 15th percentile.

If the little girls height at 3 years old is above the 95th percentile substantially, we would think she experienced a growth spurt.

If her height was about the 15th percentile, we would say that her growth was normal and typical since she maintained her place in the distribution of all little girls of those age groups.

If, on the other hand, her height at 3 years old is significantly below the 15th percentile, we would have indication that something MAY be of concern and further investigation would be warranted.

The same concept is applied to measuring academic growth. Is it at least reasonable to maintain the achievement level of a group of students? Yes!

Now that may or may not be enough to meet your achievement goal, but it the standard for PA Academic Growth as defined in PVAAS. Maintaining the position of the group for grades 4-8,k reading and math, means you have met the standard for PA Academic Growth .PVAAS Overview23A Challenge Comparing Scores Across YearsMath & Reading Grades 4 through 81000 1100 1200 1300 1400 1500Scaled Score UnitsAugust 2011240 10 20 30 40 50 60 70 80 90 100NCE (Normal Curve Equivalent) UnitsNext comes the challenge of how to compare scores over time.

The averages and distribution of scores of PSSA exams change annually. This is normal and expected. So PSSA scaled scores from different years for different grades cannot be compared grade to grade. It is necessary to convert PSSA scaled scores to a common scale for grades 4 through 8.

PVAAS rescales each of the PSSA distributions from each year using a Normal Curve Equivalent Scale (NCE) a common scale for comparisons over time.

NCEs are a common language for the scores to look at growth over time.

Once a set of test scores has been rescaled, they now can be compared across different grades and years. PVAAS reporting for grades 4 through 8, reading and math uses NCE units to allow for these kinds of comparisons for the calculation of longitudinal growth. These conversions to NCEs are based on the base year of 2006--- so the answer to the one question is I recently asked.does the target change each year? - the answer is NO!

The distributions from 2006 are used to determine position of a group to themselves- just like the height charts in the doctors office. They were established in a particular year- then each year you can compare a child to himself/herself to see if they have changed position- the same is true for academic growth as measures by PVAAS for grades 4-8, reading and math.PVAAS Overview24What Value-Added Growth Measure Means!Growth Standard MethodologyMath & Reading Grades 4 through 8August 2011250 10 20 30 40 50 60 70 80 90 100NCE ScaleState Mean = 50Achievement 5th grade47.6Achievement 6th grade52.7PVAAS Growth Measure Change in position from 5th to 6th grade 52.7 47.6 5.1 NCE unitsThis slide displays what the Value-added Growth Measure means.

If you are into statistics you may want to know that the NCE Scale sets the State Mean at 50 with a standard deviation of 21.06 so that nearly all data will fall within the range of 0 to 100. It is important to remember that this is simply about rescaling so that scores from different years (different scales) can be compared! [CLICK]

Now, assume that a group of students has an estimated achievement at the end of 5th grade that equals 47.6. (Remember, the estimated achievement value of 47.6 is not based on one score; it is an estimate based on all longitudinal data and has been converted to NCEs)

Suppose that the group of students estimated achievement at the end of 6th grade was 52.7.

We will then complete the displayed calculation to estimate a growth of 5.1 NCE units for the 6th grade school year.The group improved their position by 5.1 NCE units. This means that the school has fostered an improvement in position for these students by 5.1 NCE units from the previous year. While this is a simple explanation of the concept behind PVAAS, it is a much more robust statistical process than simply comparing 2 different scores.[CLICK]

PVAAS Overview25August 201126Example of Growth Measure=0 for alow-achieving district/schoolExample of Growth Measure=0 for ahigh-achieving district/school

This is an example of two schools meeting the standard for PA Academic Growth maintaining the position of the group of students.

This may be enough for a high achieving school.

However, low achieving schools need to make more than one year of growth to raise achievement.

We also know some high achieving schools that want more student to move from proficient to advanced performance- maintaining the position of the group would not be enough to meet this goal.

SO, that is the concept of how growth is calculated for grades 4-8, reading and math.PVAAS Overview26PVAAS Growth MethodologiesTestingSubjects GradesMethodologiesPSSA in consecutive years Mathematics 4-8Reading 4-8Growth StandardPSSA not in consecutive years Writing 5, 8, 11 Science 4, 8, 11 Math & Reading 11PredictiveNow we turn our attention to the PVAAS Value-added Reports for Writing, Science and grade 11 Math and Reading- remember these are not tested in consecutive grades.PVAAS Overview27Predictive MethodologyMath & Reading Grades 11Writing Grades 5, 8, 11Science Grades 4, 8, 11Definition:A group meets the growth measure when their actual achievement meets their predicted achievement - based on the average schooling experience.The Predictive Methodology is about comparing the observed mean to the predicted mean and determining if the group of students performed as expected.PVAAS Overview28PVAAS Value-added ReportsWriting, Science & Grade 11 Math and Reading Predictive Methodology

A predicted score for each student;

Mean of predicted scores for the group of students;

Mean of observed scores for the group of students.

Growth Measure Observed Mean - Predicted MeanPVAAS Overview29The process for determining growth for grade 11 reading and math, as well as science and writing is as follows:

A predicted score is calculated for each student;Then a Mean or average of the predicted scores is calculated for the group of students;Then the student actually take the PSSA, and a mean of observed scores for the group of students is calculated.The growth measures is determined by comparing the observed mean to the predicted mean. Did the group meet, exceed or not meet the predicted mean- what was expected.Every measure has error- therefore a standard error is applied as well which you will see on the reports.

This is how growth is calculated for grade 11 reading and math, as well as science and writing.

PVAAS Growth ReportingDistrict/School Value-added Report

Value-added Summary Report

District/School Performance Diagnostic Report

Performance Diagnostic Report for subgroupsNow lets get to the reports! Here is a list of the PVAAS value-added reports that we will discuss in this webinar. Remember there is the growth reports in PVAAS and the projection reports- we will start with the value-added growth reports. PVAAS Overview30NEW! PVAAS Value-addedGrowth Descriptors August 201131DBSignificant evidence that the district/school exceeded the standard for PA Academic GrowthLB

Moderate evidence that the district/school exceeded the standard for PA Academic GrowthG

Evidence that the district/school met the standard for PA Academic GrowthYModerate evidence that the district/school did not meet the standard for PA Academic GrowthR

Significant evidence that the district/school did not meet the standard for PA Academic GrowthPVAAS Overview31In case you are not aware of this change-The PVAAS Value-added Growth Descriptors are now aligned so that the colors and interpretations are the same for all subjects and all grade levels, despite the difference in methodologies that we spoke of earlier. This was not the case in previous years.

The Ratings on the PVAAS Value-added District/School Reports are color-coded to assist you with quick recognition of the rating. While some people want to know where all of the numbers come from and how they are calculated, most people want to know what the report means and how to use it for local decision-making.

To provide more information to schools about their effectiveness in meeting the needs of groups of students, this new color legend is being introduced in Fall 2011 that will assist schools in differentiating their growth in both directionsmore and less than the standard for PA Academic growth.

Dark Blue (DB) - Significant evidence that the district/school exceeded the standard for PA Academic Growth

Light Blue (LB) Moderate evidence that the district/school exceeded the standard for PA Academic Growth

Green (G) Evidence that the district/school met the standard for PA Academic Growth

Yellow (Y) - Moderate evidence that the district/school did not meet the standard for PA Academic Growth

Red (R) Significant evidence that the district/school did not meet the standard for PA Academic Growth

District/School Value-added ReportGrades 4 through 8: Math & ReadingAugust 201132

Question for Educators:Are all grade levels meeting the standard for PA Academic Growth?PVAAS Overview32This is an example of a Value-added Report that is available at both the district and school level. It is important to remember that this report is about the effectiveness of the school or district! The red rectangle at the top of the report highlights the growth measures for each grade level 4 through 8 in 2011. In this example, grade 5 has a Red rating indicating significant evidence that the group of students did not meet the standard for PA Academic growth. However, the Dark Blue rating in grades 4, 7, and 8 indicates significant evidence that those students, as a group, made more than the standard for PA Academic growth- they improved the position of the group. The Light Blue rating in grade 6 indicates moderate evidence that the group of students made more than the standard for PA Academic growth. Again, as this is a report for grades 4-8, reading and math- we know this means they improved the position of the group.

Notice also that the 3-year Average Growth Measure for each grade is calculated. These values are useful when looking for patterns of growth within the same grade level. It is also useful to consider the values in the grade-level columns. Users should always keep in mind that when you are considering one grade levels values over several years, the student groups typically change each year. In the case of 8th grade in this report, we see three years of significantly positive growth. This indicates that there may be a systemic condition in 8th grade in this subject that is supporting growth regardless of the group of students; the 3-year average in that grade level is also Dark Blue. Generally if we see three years of all positive or three years of all negative, we have evidence that there are system conditions not related to the particular groups of students who went through that grade level over the 3-year span.

At the bottom of the report, you will see that we have highlighted the achievement values on the diagonal for the 2011 8th grade. These are reported in NCE units for grades 4-8 in reading and math. This diagonal view may provide insights into the pattern of growth over time for a particular group of students.

District/School Value-added ReportGrades 9-11 Math & Reading; Science; & WritingAugust 201133Question for Educators:Are all grade levels meeting the standard for PA Academic Growth?PVAAS Overview33This is a copy of the Value-added Science Report for 2011 which uses the Predictive Methodology. This is also what the PVAAS report looks like for Grade 11 Reading and Mathematics AND for Writing reporting. These are reported in PSSA scaled score points, not NCEs.

The Avg PSSA Score column indicates the observed mean score of a particular group of students on the previous PSSA assessment- hwo did they actually perform on the assessment.

The Avg Predicted PSSA Score column indicates the predicted mean score of the same group of students based on their academic testing history. Remember, the predicted scores is based on the entire longitudinal history of these students, NOT just on their last PSSA.

This report indicates that the 4th grade group of students (tested in 2011) performed moderately above expectations on the Science exam. However, the 2011 8th grade group of students performed moderately below expectations on the Science exam, while the 2011 11th grade group of students performed significantly above expectations on the Science exam!

District Value-added Summary ReportGrades 4 through 8: Math & ReadingQuestion for Educators: Which schools at which grade levels are exceeding the standard for PA Academic Growth?

This report(Value-Added Summary Report) displays the growth status of all schools and grade levels 4-8 in reading and math, in the selected subject. The values are the Growth Measures and the cells are color coded just as they are on the Value-added report. This report is helpful to get a quick view across schools in a district.

This report provides the district with an overview of the Growth Measures for all schools and grade levels to identify strengths and weaknesses in growth across buildings and grade levels.

Each of the school names are hyperlinked to the School Value-Added Report for that particular school.

If we select Cotton Candy Elementary, we will see the School Value-Added Report for the students in Cotton Candy Elementary in Math. If you are on the website, you can change the subject to Reading using the Subject tab at the top of the screen.

PVAAS Overview34District Value-added Summary ReportGrades 9-11 Math & Reading; Science; & Writing

Question for Educators: Which schools at which grade levels are exceeding the standard for PA Academic Growth?This report displays the growth status of all schools for both Math and Reading in grades 9-11. The values are the Growth Measures and the cells are color coded just as they are on the Value-added report. These are reported in PSSA scaled score points.

This report provides the district with an overview of the Growth Measures for all schools in both Math and Reading to identify strengths and weaknesses in growth across buildings and subjects.

Each of the school names are hyperlinked to the School Value-added Report for that particular school.

If we select Orange Yellow High School, we will see the School Value-added Report in Math for the students at Orange Yellow High School. PVAAS Overview35Lines of InquiryMain Questions Regarding PVAAS ReportsAre we serving all students adequately?Are we serving all students equitably?

Questions to Ask Regarding Value-added ReportsAre students in all grade levels across all schools meeting or exceeding the standard for PA Academic Growth in all subjects?Is the pattern of growth at the same grade level consistent from school to school? Or, does growth differ from school to school within the same grade level?For each school, is the pattern of growth at the same grade level consistent from year to year? Or, has growth differed within the same grade level across the last three years?

August 201136PVAAS Overview36These are some of the key questions that the Value-Added Reports at the district and school level may help answer for you.

< Read 1-2 of the questions>

Now lets dig deeper into the PVAAS district and school reports.

Going DeeperPerformance Diagnostic ReportAugust 201137Blue Bar Current YearGold Bar Previous YearsMissing Bar Insufficient Number of StudentsWhisker Margin of Error on Growth ValueQuestion for Educators:Are all groups of students making adequate growth?This is an example of a grade level Performance Diagnostic Report. This report can be generated at a district or school level. The purpose of this report is to provide insight into the growth of students who are predicted to perform at the Below Basic, Basic, Proficient or Advanced levels on the current years exam in this subject. This prediction is based on the students previous performances.

The report consists of two sections a graph and a table.

On the graph there are blue bars representing the growth of the predicted PSSA performance group over the last tested year. The gold bars represent the growth of the three prior years groups of students in this predicted PSSA performance level.

You will notice that we are missing a blue bar for students predicted to be Basic in this example. This is due to the fact that PVAAS will provide an estimate only for a group of students with 5 or more students. Otherwise the error becomes too large for the estimate to be useful and therefore it is not reported.

The error is represented by the red whisker (or red I bar) at the end of each blue and gold bar. This is a margin of error whisker that says that the true value of growth is likely to be within the whisker. The whisker plays a critical part in the interpretation of the growth value as is indicated in the next slide.PVAAS Overview37PVAAS Performance Diagnostic Growth Descriptors - Interpretation

August 201138Zero (0) LineGrowthMet the standard for PA Academic GrowthGreen

Did not meet the standard for PA Academic GrowthPinkExceeded the standard for PA Academic GrowthBlueWhat the whiskers tell us38The position of the whisker relative to the green zero (0) reference line is the key to the interpretation of the Performance Diagnostic Report.

On this slide, we display the interpretation of the whiskers for this report

The green line represents the standard for PA Academic Growth.

The key is where the red whiskers or I-bars are located relative to the green line.

If the whisker or I-bar intersects the green line at all, we conclude that the group of students has met the standard for PA Academic Growth.

If the whisker or I-bar is completely below the green line, we conclude that the group of students did not meet the standard for PA Academic Growth.

If the whisker or I-bar is completely above the green line, we conclude that the group of students exceeded the standard for PA Academic Growth.

So for grades 4-8 in Reading and math, the whiskers tell us whether the group made more than a year of growth, a year of growth, or less than a year of growth.

For Science, Writing and Grade 9-11 Reading and Math, the whiskers tell us whether the group performed better than expected, as expected or less than expected.

Lets Look at Our Example to Practice InterpretationAugust 201139

Questions for Educators:

Are all types of learners showing a years growth or meeting expectations?

Are we serving all students adequately?

Are we serving all students equitably?Lets go back to the report example we were previously viewing. Now that we have a better understanding of how to interpret this report, lets try to interpret this report for Orchid Elementary, 4th grade math.

What would you say about 4th grade students who were predicted to be Below Basic? Predicted to be Proficient? Predicted to be Advanced?What can we say about students predicted to be Basic in this grade level group of students?

[As a facilitator, you may also want to ask participants if they would feel confident in saying that it appears that the students predicted to be Below Basic made more growth than those predicted to be Advanced. In this case, the answer is probably yes since the red whiskers for those groups do not overlap each other. If the red whiskers overlap when placed side-by-side, we cannot say that one group grew more than anotherno matter what size of bar you may see.]PVAAS Overview39

August 201140Link to all students reported for this subject.Links to students in selected predicted PSSA performance categoryPVAAS Overview40There are a number of hyperlinks, indicated in blue and underlined, on this report. We will address a few key links here.

To access a list of the students predicted to be in each PSSA performance level, users should click on the hyperlink (or hot key) for the particular group of interest to see all students in that predicted PSSA performance level. If viewing a report for grades 4-8 in math and reading, this will produce a list of students, their state NCE scores and actual PSSA performance levels. If viewing a report for grades 9-11 math and reading, or science, or writing, this will produce a list of students, their predicted PSSA scaled score, their actual PSSA scaled score, and actual PSSA performance level. (Note: This must be done from a school level report. You cannot access student level data from a district level report. Additionally you must have permission to access student level data.)

If you click on the hyperlink (or hot key) for the subject area (in this example, Math), you will receive a list of ALL students (regardless of predicted PSSA performance level) included in the report and their state NCE scores and actual PSSA performance levels (as this is a grade 4 report).

Student Lists from Performance Diagnostic ReportsAugust 201141

Current PSSA Performance level for each studentPVAAS Overview41This slide displays the student list in the selected categories from the Performance Diagnostic Report with their individual 2011 state NCE scores and their observed (actual) 2011 PSSA performance levels. All data in this report are sortable by clicking on the column title of the field for which the sort is desired.

In addition, by clicking on the individual student name, you will be directed to the Student Report, which we will discuss shortly.

Patterns of GrowthAugust 201142ABCQuestions for Educators:

What patterns of growth do we see at the various grade levels?

What implications does this have for future achievement levels of students?

What implications does this have for closing he achievement gap? PVAAS Overview4242 The POWER of the PVAAS Performance Diagnostic Reports are the PATTERNS that they can display about your school/district. These patterns serve as an indicator of whethere you are headed in the right direction in terms of growing students and where you may have areas for improvement. The meaning of the pattern of growth should be viewed through the context of the achievement goals of your school. What are your schools goals for achievement? And, as a result where do you need to meet or exceed the standard for PA Academic growth?

This slide displays some common patterns that we see in Performance Diagnostic reports:A: Tent PatternThis pattern occurs when school learning experiences benefit students in the middle of its population more than their lower-achieving or higher achieving peers. This trend can be reversed by attending to students at the extremes of the grade levels population.

B: Uniform PatternThis pattern occurs when school learning experiences benefit all students in approximately the same way. In this example, all students in all predicted PSSA performance levels demonstrated positive and significant growth. A uniform pattern with all bars below the green zero (0) reference line would obviously indicate that none of the learners demonstrated positive growth.

C: Downward Shed PatternThis pattern occurs when school learning experiences benefit lower achieving students more than their higher-achieving peers. For many reasons, the system is not appropriately addressing the needs of all students, but rather has a focus on the lower- achieving students.Lines of InquiryQuestions to Ask Regarding Grade Level Performance Diagnostic ReportsAre students at all predicted PSSA performance levels meeting or exceeding the standard for PA Academic Growth in all grades and subjects?What type of growth pattern do we see for each grade level? And, what are the implications for achievement if this growth pattern continues?Where are we doing well? Which groups are showing positive growth?Where are we facing challenges? Which groups are not showing positive growth?Is there a change from previous years?Are other buildings within the district seeing similar growth patterns?What does this tell us about the standards-aligned system, specifically around curriculum, instruction, resources, and interventions?

August 201143PVAAS Overview43This is a list of possible questions to consider when using the PVAAS Performance Diagnostic data.

Performance Diagnostic Subgroup ReportsAugust 201144

PVAAS also provides Performance Diagnostic Reports for demographic subgroups as reported on the PSSA file.

This is a display of the subgroup selection pop-up window used to select the demographic subgroup(s) you wish to investigate.

The categories on this screen reflect the categories on the statewide PSSA student level file.PVAAS Overview44August 201145Performance Diagnostic Subgroup Report

PVAAS Overview45This is an example of a Performance Diagnostic Report for the subgroup selected: Male students only.

Best to Compare Subgroup to Entire Grade LevelAugust 201146Same School Same Grade Same Subject

Question for Educators: How do our subgroups compare to the entire grade level?SubgroupEntire grade levelThe real power of the Performance Diagnostic for subgroups becomes clear when you compare the growth of the whole grade level to the growth of the subgroup. You should be aware that this report on the whole group DOES also include those in the subgroup.

PVAAS Overview46Lines of InquiryQuestions to Ask Regarding Diagnostic Reports for SubgroupsDid the students in the selected subgroup achieve acceptable growth? What about those students predicted to be Below Basic? Predicted to be Basic? Predicted to be Proficient? Predicted to be Advanced?Is the growth pattern for subgroups similar to the growth pattern for all students in the same grade level?How do these growth patterns compare to subgroup growth patterns in other grade levels and subject areas?Is there a change from previous years?Are other buildings within the district seeing similar growth patterns with subgroups?What does this tell us about the standards-aligned system, specifically around curriculum, instruction, resources, and interventions?

August 201147PVAAS Overview47These are example of questions to ask when using the PVAAS Performance Diagnostic Report for subgroups.

This concludes the section on the value-added growth reports from PVAAS. TWO Types of PVAAS InformationTodayLooking Back/EvaluationValue-added Growth Reports For Groups of StudentsLooking Forward/PlanningPVAAS Projection Reports For Individual Students and Groups of StudentsPVAAS Overview48Now, we will focus on the PVAAS projection reports- a look to the future of student performance.Student ProjectionsWouldnt it be great to know the likelihood that a student will be proficient on afuture PSSA?PVAAS Overview49The answer to this, of course, is YES!!!!

PVAAS estimates the chance that a students performance on a future PSSA will reach the Basic, Proficient, or Advanced performance levels.

Projections provide realistic insights into possible future performance if conditions remain the same for the student(s). While not everything stays the same, this is a highly reliable indicator of future performance.Student Projection MethodologyIs this the same methodology as the one used to estimate growth?No, the projection methodology is a separate modeling process that focuses on individual students.

What data are used in this methodology?ALL available longitudinal data in both Reading and Math are used in projection calculations for all four subjects.

What students have projections? In what subjects?Projections are available for all students who have a minimum of 2 years of historical data. The next slide details the grade levels and subjects in which projections are reported.August 201150PVAAS Overview50The Student Projection Methodology is a separate modeling process that focuses on individual students although projections are available for both individual students and groups of students.

Most important to consider is that the projection methodology uses ALL of the available data in both Reading and Math in its projection calculations. This can be done because students performance in Reading is highly correlated to their performance in Math (and the reverse is also true).

Projections are available for individual students who have a minimum of 2 years of historical data, as well as summaries of projections for groups of students.

PVAAS Fall 2011 Reporting: Projections to Basic, Proficient, Advanced Math and Reading: 3 4 or 5 4 5 or 6 5 6 or 7 6 7 or 8 7 8 8 11Writing: 3 5 4 5 5 8 6 8 7 8 8 11

Science: 3 4 4 NA 5 8 6 8 7 8 8 11

August 201151Grade PSSA Last TakenGrade PSSA Last TakenGrade PSSA Last TakenPVAAS Overview51This slide shows you what projections are available for students with sufficient data in fall 2011 reporting.

Note that the projections available are dependent upon the grade level in which the PSSA was last taken, meaning students last taking the PSSA in grades 3-8. This is important to remember for middle and high schools where they will need to access projections for students last tested in grade levels not in their building. For example, high schools serving students in grades 9-12 will need to access middle school data to obtain projections for those students last tested in grade 8 in the middle school(s).

Unless your district has submitted currently enrolled information on its students. In that case, users can retrieve projections on students in the current grade level in which they are enrolled instead of where they were last tested. Information and instructions on doing this will be sent to districts in the fall

PVAAS Projection ReportingStudent Projection Report

District/School Projection Summary ReportHere is a list of the PVAAS projection reports that we will discuss in this webinar. We recommend that educators consider these reports in this order when reviewing PVAAS data to be integrated with other data sources for instructional decision-making.

PVAAS Overview52

PVAAS Student Projection ReportAugust 201153PVAAS Overview53This is what a Student Projection Report looks like. You access this from the Projections tab when you are on the website.

The red dots show the students observed performance in Reading at grade levels 3, 4, 5, 6 and 7. The yellow square to which the red arrow points indicates the projection for 8th grade Math. This says that based on this students history and his/her achievement pattern (taking both Reading and Math into account) the student is projected to be in the 44th state percentile on the 8th grade Math assessment. The calculation of this projection assumes two important criteria:Conditions that were in place for the data that are used to create the model continue to be in force;Where possible, the calculation adjusts the probabilities based on the school most likely to attend for the particular student. If there are not sufficient data to make this adjustment, the projective value will assume the average school experience.The red, blue and green lines on the graph indicate the percentile boundaries for the Basic, Proficient and Advanced categories for the specified subject area based on the most recent PSSA testing. These values are utilized in the calculation of the probability of a student score being in an indicated category.The center table highlights the probabilities that this projected score will be in the Basic category or above (92.9%), in the Proficient category or above (72.0%), or in the Advanced category (26.5%).A table of interpretations and recommendations based on the probabilities are also provided.

For your information:Some districts are sharing these reports with parents.PDE has a parent letter template available to accompany this report if your district or school chooses to do that systemtically.We will move to the next slide to enlarge the sections on the bottom of the report for better viewing.

PVAAS Student Projection ReportAugust 201154

PVAAS Overview54The first table in the center of the report highlights the probabilities that this projected score will be in the Basic category or higher (92.8%), in the Proficient category or higher (72.0%), or in the Advanced category (26.5%).

These are risk indicators for the educational team.

A table of interpretations and recommendations based on the probabilities are also provided at the bottom of the report.

More examplesAugust 201155Report AReport BHere are two examples of PVAAS student projection reports.

Report A: this student has a high probability of at least reaching a proficient level.

Report B: this student is at risk of not reaching a proficient level of performance. PVAAS Overview55The Quality of Projections for Decision MakingThe projection is precise, and is created using a model that has been reviewed and approved by four different peer review panels and the GAO (US Government Accountability Office).

PAs growth model proposal to USDOE includes information regarding the statistical model and projection reliability study. (http://www.ed.gov/admins/lead/account/growthmodel/pa/index.html)

Recent studies have confirmed that the PVAAS projections (even as far as 3 years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA score.August 201156How good are the projections???

Read info on slidePVAAS Overview56

PVAAS Projections for Groups of Students

Student SearchAugust 201157Question for Educators: What are students probabilities of reaching Proficient or Advanced levels on the next PSSA?PVAAS Overview57Now you may want a quicker way to get to he projections on students. You can access projections for groups of students by using the Student Search capability under the Reports Tab.

Currently the default setting for all schools is to access students in the grade last tested. Starting last school year, schools have the option of accessing students by grades currently enrolled if their district submits currently enrolled data to SAS in the fall. We anticipate this option will be provided again this year.

Participants may also identify specific demographic information or PSSA performance levels, and PVAAS will search for students meeting those criteria.

The Search for Students also allows teachers and administrators to generate a list of students whose probabilities of achieving a selected PSSA performance level fall within a specified range of probabilities. You can have as many as four search criteria for each search.

Identified Students Meeting Search RequirementsAugust 201158

Question for Educators: How can we better plan for this year now that we know the achievement probabilities of these students?PVAAS Overview58When the parameters of the search have been submitted, a report is generated that lists the students that meet the specified criteria outlined in the student search.

All demographics for the included students are displayed.

Each of the columns are sort-able by clicking on the column header.

Each student is listed by name and PASecureID along with their probability of reaching the selected PSSA performance level on the specified PSSA exam.

Student Projection SummariesWouldnt it be great to have a summary of all of the probabilities that students in one school or in the entire district will be proficient on a future PSSA?PVAAS Overview59The answer to this question, of course, is YES!!!!

PVAAS provides Projection Summary Reports at both the District and School levels!Projection Summary ReportsWhat are they?This is a report that summarizes the numbers and percentages of students in various likelihood ranges of performing at a proficient level on a future PSSA exam.

How might a school use these reports?Intervention Planning Resource Allocation Strategic Planning School Improvement Planning

CautionsThis report provides ONE indicator about likelihood of future performance. Additional data should be used with the projection for better decision-making.August 201160PVAAS Overview60PVAAS includes Projection Summary Reports a report that summarizes the numbers and percentages of students in likelihood ranges of performing at the proficient level on a future PSSA exam.This gives you a big picture view of where students are headed on the PSSA in the future.

District/School (Single Grade) Projection Summary ReportAugust 201161

Questions for Educators:

Do we have a high percentage of our students likely to reach proficiency?

Do we have interventions and enrichment opportunities available for the appropriate students?PVAAS Overview61The School (Single Grade) Projection Summary Report is displayed on this slide summarizing the projections of all students at that grade level whose records contain sufficient data to calculate projections.

Note that you may select 1 year projections or 2 year projections for longer range projections.

The number and percentage of students who are likely to be Proficient are summarized as Likely probabilities between 70% and 100%;Marginally likely probabilities between 40% and 70%; andUnlikely probabilities between 0% and 40%Notice also that the number of students in each category is hyperlinked, or is a hot-key. This will lead to a complete list of students comprising that probability range.

Lines of InquiryQuestions to Ask Regarding ProjectionsWhere are we headed as a district or school?Do we have a high percentage of our students likely to reach proficiency?Do we have large numbers of students at risk of not reaching proficiency?Do we have interventions and enrichment opportunities available for the appropriate students?What is the schedule of courses? Is it fitting students needs consistently across schools?How are students placed in courses? AP courses? Enrichment? Support?Are students accessing the appropriate interventions and resources?What does this tell us about the standards-aligned system, specifically around curriculum, instruction, resources, and interventions?

August 201162PVAAS Overview62These are possible questions to consider when using the PVAAS projection reports. Projection Summary ReportsSummary ReportDescriptionDistrict Grade SpansReport details probabilities of future performance of all students in the district by grade spans.District Single GradeReport details probabilities of future performance for all students in the district in a selected grade level.School Grade Last TestedReport details the probabilities of future performance on next PSSA exam for all students in all grades in a selected school.School Single GradeReport details probabilities of future performance for all students in a selected school in a selected grade level.This is a summary for you of all of the Projection Summary Reports available in PVAAS. These reports can all be accessed at the bottom of the orange Report Tab.PVAAS Overview63TWO Types of PVAAS InformationTodayLooking Back/EvaluationValue-added Growth Reports For Groups of StudentsLooking Forward/PlanningPVAAS Projection Reports For Individual Students and Groups of StudentsPVAAS Overview64Remember, the two PVAAS methodologies really look at two different areas of student performance.

The Value-added, or growth, information looks backit helps schools to evaluate the effectiveness of the school. How much growth did students make in the past school year?

The Projection information looks forwardit helps schools plan for the future. Are students on a path to proficiency or higher?

Both serve different purposes, and both are equally important for continuous school improvement.

Fall Professional Development Opportunities

Check out what is being offered this school year by visiting the PVAAS login page!Now, what supports are available to you as you use your PVAAS data?

Discuss professional development offerings.PVAAS Overview65PVAAS Help Menus

Please be aware that you can access PVAAS Help Menus if you need assistance with any report within the system.

The Help Menus are designed around Frequently Asked Questions (FAQs).

There are over 500 pages of Help Menus that are specific to the report you are viewing at that time.

They can be printed and used during a team meeting or when you are individually reviewing a report.

Each help menu is organized into 5 common sections.Navigating: designed to help users navigate through the report and access all available features.Understanding: designed to increase users understanding of the key pieces of the report, as well as general assistance with correct interpretation.Interpreting: designed to help users gain a deeper understanding of the interpretation of the reports, including tips on analyzing the report for patterns and trends.Using: designed to help users gain an understanding of how the report can be used with other data sources for various decision-making purposes.Assistance: provides general information on the viewing and printing of reports, as well as state and IU points of contact66PAs Intent for PVAASAnother Tool for Continuous Improvement

Measure of Growth to Add to Status/Achievement Measures

Not to Be Used in Isolation

PVAAS Overview67In summary, PVAAS is another tool as part of the cadre of analysis tools provided by PDE for districtssuch as eMetric PSSA Data Interactive and PA AYP.

PVAAS is a measure of growth/progress to add to achievement/status measures.

PVAAS should not be used in isolation, but rather as part of collaborative dialogue in districts and schools.

Questions:

PVAAS Materials or Statewide Implementation

Email: pdepvaas@iu13.orgPhone: 717-606-1911

PVAAS Reporting Web Sitehttps://pvaas.sas.comPVAAS Overview68Questions related to the statewide implementation of PVAAS or PVAAS materials can be directed to pdepvaas@iu13.org or 717-606-1911.

Remember to go to https://pvaas.sas.com to access your district and school PVAAS reports.

www.pde.state.pa.us

PVAAS Overview69PVAAS is provided by the Department of Education as a data tool for continuous school improvement.