ideas for leveraging district data - schedschd.ws/hosted_files/wabe2015/89/leveraging district...

Leveraging District Assessment Information for ELL Peer Data Analysis WABE 2015 Kristyana Mansfield, Shannon Lemieux, Kristi Dominguez, Lynne Skerry Bellingham Public Schools

Ideas for Leveraging District Data

• Identify individual students (or groups of students) whose needs may present instructional challenges to the classroom teacher and/or other specialists and provide coaching or other supports to meet those needs. Or, use that same data to seek professional development and learning about how to provide high quality instruction for those students (e.g., ELLs with communication disorders, internationally adopted students, etc.).

• Find “stand out” areas in the data that might demonstrate that the instructional program is not working for a group of students (e.g., newcomers, migrant students, students in a particular language group, intermediate (L2) ELLs). Once you have found the anomaly, investigate further to try to find the cause.

• Use assessment data, such as WA Kids data, to develop intervention plans that are aligned with the English Language Proficiency Standards.

• Require peer analyses for any ELL whose performance is causing great concern, or whose

performance appears extraordinarily high. These analyses can be presented to classroom teachers or multidisciplinary teams, and should be prepared as part of a pre-referral process to SPED or HCP.

Kindergartener

Leveraging District Assessment Information for ELL Peer Data Analysis

Kristyana Mansfield, Shannon Lemieux, Kristi Dominguez, Lynne SkerryBellingham Public Schools

The presentation slides are available online at:

www.tinyurl.com/WABEpeeranalysis

Invite audience to write down link.The style of this presentation does not represent best practice. It is text-heavy because we wanted you to have a resource to use, with some of the research base and validation from OSPI. The PPT is text rich because it is meant to serve as a stand-alone support document that people can reference after the presentation.This presentation is available online for you to download later.

The Performance of ELLs

Peer analyses can help us decide, in the context of other factors, if an ELL is making adequate progress and performing within range for their current level of proficiency.

SHANNONELLs often do not match their grade level peers on grade level assessments. This may or may not be cause for alarm. So, how can we tell when an ELL’s performance is alarming? Peer analysis is one tool to help us learn about individual student growth along with whole cohort data.

That said…

Everything our team unearthed says to compare ELL students against his/her peers. But most references to peer analysis in the literature simply said, “do a peer analysis,” with little description of what that entailed.

We needed a procedure, so we created our own to meet our district and student needs.

SHANNONWe know data gathering is the best way to develop a hypothesis around why a student may be struggling and then target any needs that may be impacting learning. Within data and information gathering, we started to see a trend of requesting a peer data analysis. This made perfect sense to us, yet we couldn't find enough information on what this was, how to do it, and how to apply what we find. We searched and researched and still came up with little. The information on peer data analysis for ELL was thin. Therefore, we decided to take what we did find and create our own definitions and procedures on what, how, and why. We are going to share what our team is doing, but we are also hoping through this we can learn what other districts are doing.

Agenda● Purposes for peer analysis

○ Examples: District data revelations● How to do a peer analysis

○ Peer analysis usage (brief case studies)● Lingering questions● Communicating about data● Recommendations

SHANNONThis is the agenda, and the presentation will be dense. We’d be more than happy to answer questions at the end and we’ll give you our contact information.

Purposes for Peer AnalysisFrom OSPI:

“The purpose of the Peer Analysis is to inform the concern about limited English proficient students’ possible need for referral for Special Education

evaluation….The Peer Analysis compares a LEP student to all other LEP students who came the same year, grade and placement level as the student in question. This ensures a comparison with the progress of similarly situated

students from across the state.”

We think there is more you can do with peer data.

SHANNONTimed turn and talk: What would you like to do with peer data in your district or school? What do you think you could learn about your students?

Purposes for Peer AnalysisWe should do peer analyses to learn about ELL subgroup performance on district and state assessments so that we can:1. set expectations with regard to 2nd language acquisition

a. set expectations for ELLs’ performance and growth trajectories on assessments (e.g. 3rd grade L2s should be reading in range _ to _)

b. consider and analyze the impact of 2nd language acquisition on literacy development;

2. decide when and where to intervene and/or collaborate for ELLs who might be academically at risk; and

3. develop a more accurate picture of an individual ELL’s performance prior to a SpEd or HCL referral.

SHANNONThere are many things we can do with a peer analysis. It is an intense data gathering system, but it provides useful information to understand and set learning expectations (always keep them high!), support classroom teacher’s understandings of language acquisition, decide upon interventions, develop a true representation of students skills within a group prior to SpEd or HCL referral.

“Monolingual children are clearly not the most appropriate group for norm referencing dual language children’s performance...[T]he development of norms based on dual language children’s performance for existing tests can be more feasible. In fact, some school districts might already have the data needed to develop such local norms using test results for dual children that have been conducted at regular intervals in the past according ages/grades.”

(Paradis, et al., 2010)

1a) Understand and set expectations for ELLs’ performance

YANA

One of the issues we have with interpreting ELLs’ assessment scores is that it is difficult to determine when the students’ performance is concerning or exceptional. One way to find out is to develop district norms for ELLs to set the context for the performance of ELLs by proficiency level and grade level, perhaps eventually with information about years in program. This information is frequently asked for by school psychs, but has not been available to us. With access to data, we can begin to see trends and start the process of setting norms for the performance of ELLs.

Might be important to note that Paradis, et al also said that “Interpreting local test scores according to a set of local norms is not without its limitations. Such norms may not meet all the stringent psychometric criteria used in developing published standardized test norms, and they may not be sensitive to variations in children’s language exposure. Nevertheless, local norms are more likely to be informative than monolingual norms.”

“L2 [second language] reading acquisition is different from L1 [first language] reading acquisition...1) students learning to read in an L2 usually come from different sociocultural backgrounds, 2) they know and use another language, and 3) they are still learning the L2. All three factors can influence the rate at which L2 students learn to read…”

“L2 [second language] learners obviously differ from students learning to read in their L1 because they are still learning important aspects of oral language and, in particular, they are still acquiring advanced-level morphosyntactic and discourse skills of the type that underpin academic language and reading comprehension.”(Paradis, et al., 2010)

1b) Consider and analyze the impact of language acquisition on literacy

YANAWith regard to literacy, we know that ELL students frequently have a different profile - students may struggle with phonemic awareness, accuracy (pronunciation, reading word endings, etc.) and/or comprehension more than their grade level peers. We know this, but many classroom teachers and reading specialists need help interpreting ELLs’ performance on literacy assessments and classroom work.

Using peer data can help identify areas that might need additional support when we consider ELLs’ first language, proficiency level, literacy in the first language, etc.

1a & 1b) Performance expectations & impact of 2nd language acquisition on literacy

YANATo create this trajectory, we pulled longitudinal data from our universal screener (BAS) as follows:

● All ELLs who placed as L1 in Kindergarten● All ELLs with a score at each assessment window (Fall Grade 1, etc.)● We converted the BAS letter level (A-Z) to a number and then found the

median and average score for each grade level and assessment window● The median and average scores were plotted against the Benchmark score

(expected or “at standard” score) for each grade level and assessment window

This data tells us that students who place at L1 in Kindergarten start below and continue to read below the benchmark expectation for their elementary school years. However, these students do, for the most part, make progress that mirrors the benchmark trendline - just below grade level. We should be careful about how we interpret this data; it may tell us that L1 students make adequate progress that is below grade level due to their English language development...it may also be telling us that we have a lot of work to do as a system to close the gap by 5th grade. This is the first year we’ve had readily accessible data on a single reading assessment for K-5, so we’re not 100% sure what the trend lines mean.

1a & 1b) Performance expectations & impact of 2nd language acquisition on literacy

This data was collected the same way as the L1 data and, again, this is the first year we’ve been able to pull data for reading like this.

It is interesting to note that our L2 students seem to track very closely against the benchmark until 4th grade, which may indicate that we have some work to do. We’d need to do a deeper dive in order to find out what is pulling down students’ scores at 4th grade but, knowing what we know about the reading system we’re using, it would make sense for it to be comprehension - texts at 3rd grade get much more difficult (sentence length and complexity, vocabulary, multisyllabic words, unfamiliar content).

There is great promise […] in using an RTI approach for many reasons.

First, the universal screening and progress monitoring called for in the RTI process allow for comparison of students to other similar or “true” peers in their local cohort rather than to national norms.

Second, an effective RTI model requires collaboration among all educators (e.g. speech and language therapists, school psychologists, counselors, English as a second language/Bilingual specialist) thereby providing increased opportunities for professional dialogue, peer coaching, and the creation of instructional models integrating best practices of the various fields of education and related services.(Brown and Doolittle, 2008)

2) Decide when and where to intervene and/or collaborate

SHANNONSo, how can we use the data we just showed you? One way is to examine individual students’ scores using the L1/L2 trendlines as context. For example, a classroom teacher might be concerned that her 2nd grade student is reading at Level I in Winter when the expectation is Level K. One factor (and this is just one factor) we can use is the trend lines - the median score for a 2nd grader who entered the system as an L1 in Kindergarten is a Level I. So, in this context the student’s performance is not wildly inappropriate. However, if the student had entered Kinder as an L2, the ELL performance trend line would tell us that this student is performing 2 levels below her peers’ median score. This second scenario might raise more flags.

One thing we (Shannon and Yana) are trying now is to use the trend lines to help determine if a student might need an ELL-specific intervention before or in addition to a reading intervention. So, for example, if an ELL is performing 2 or more levels below the median score for the peer group then perhaps the student might need an ELL intervention to develop particular language skills that would support literacy development. In these cases, we would do more assessment to determine each student’s particular area of need (is it using complete sentences? understanding past tense constructions? pronouns? etc.). Any additional assessment information would be used for intervention planning and teacher/specialist support (e.g., coaching on specific strategies that would help the student).

Use your available data. For example, WaKids:● Is a universal assessment of all Kinders required

statewide in Fall (BPS also does Winter & Spring)● Is developmentally appropriate● Measures growth over time● Enables teachers, schools, district to look for standout

areas & trends so students receive additional support● Supports goal setting using the data


Another area where data collection is pretty solid in Bellingham Schools is with WA Kids.

KRISTI TALKS - talking points?

For ELL Specialists, WA Kids helps to...● Monitor language development and intervene early

across the system● Highlight the importance of oral language

development in support of literacy development● Inform coaching needs for Kindergarten teachers● Assist with peer analysis reports so we can see how

students are doing compared to other children with the same home language


KRISTI

“Educators often take a ‘wait-and-see’ approach to the difficulties of struggling L2 readers, on the assumption that their difficulties will resolve once they have been in school longer and have advanced in the L2 proficiency...Although this may be true for some L2 learners, it is not likely for many...The problems that result from delaying intervention can be avoided, or at least minimized, if a preventive classroom-based approach is used because intervention can begin early (Al Otaiba et al., 2009).”

Again, “L2 learners obviously differ from students learning to read in their L1 because they are still learning important aspects of oral language” (Paradis, et al., 2010)

WA KIDS allows us to start looking for students in Kindergarten whose oral language proficiency needs early intervention.


KRISTI


KRISTI


Kindergartener

KRISTI / YANAThe WA Kids data can be used in conjunction with the new ELPs to design intervention plans for students with the greatest need for language development. For example, this intervention plan was written for a student who entered as L1 but continues to score much lower than other L1s in the building on several areas of language, cognitive, and literacy development as measured on WA Kids. Her intervention plan was written to support the specific areas that she needs help in, which also easily align with the ELPs. Once the needs are identified, instructional strategies and materials can be selected, and this plan can be shared with the teacher so that instruction is aligned.

When an ELL student becomes a focus of concern, the instructional program itself must be examined to determine the match between the demands of the curriculum and the child’s current level of proficiency in the language of instruction. It is important to examine the achievement of the student’s “true peers” (similar language proficiencies, culture and experiential background) to see if they are excelling or not. If several “true peers” are struggling, this is an indication that the instruction is less than optimal for that group of students. (Brown and Doolittle, 2008)


SHANNONGiven the trajectories you will see for a group of peers, you may notice that the student you are focusing on is not outside of the norm for that group. You may also notice that the group in general is performing below grade level. This can be expected for some of our ELL students, but a peer analysis will can highlight when a whole group is underperforming. Many times these are groups we already know that are at risk.

Migrant Students entered as L1 Spanish Speakers in Math

Is the instructional program meeting the needs of these students?


Red = Well below benchmark

Yellow = Below benchmark

Green = At benchmark

Blue = Above benchmark

SHANNONWe did a peer analysis because there was a student of concern, but when the data came back we saw that most students qualified as migrant (the student’s true peers) were struggling in math. We can see an indication that the instructional program is not meeting the needs of students qualified as migrant, which could be based on a number of factors: language acquisition, mobility, etc. We don’t want to set our expectations lower for these student, but we could find ways to adjust the instruction to meet their needs. ELLs will often perform below the normed peer group (English speaking grade level peers), but we should be thinking about what we can do differently.

If the student’s CALP scores do not differ significantly from, and evidence similar growth across grade levels with, those of his or her LEP peers in the same grade, then this finding might indicate that his or her language abilities are developing normally.

Rhodes, Ochoa, and Ortiz (2005)

3) Develop a more accurate picture of an individual ELL’s performance prior to a SPED or HCL referral

If, however, the student’s CALP scores do differ significantly from, and do not evidence the same degree of growth across grade levels with, those of his or her LEP peers in the same grade, then the normal second-language acquisition process might not be the cause of the student's academic or language difficulties.”

SHANNONAn additional consideration is the student’s rate of growth. We want to see growth, and similar growth among peers. If a student is significantly underperforming or outperforming peers over a period of time, then we should narrow our focus and make decisions about how to positively impact that student’s learning.

3) Develop a more accurate picture of an individual ELL’s performance prior to a SPED or HCL referral

SHANNONThis is the same data as before, but one of the things that we noticed in this migrant student data set was that there was one student who was significantly outperforming his peers. This is where doing a peer analysis in your district can have multiple benefits. By focusing on one student in this group, it was then discovered this cohort needed some additional attention, and another student was brought to focus for an HCP referal.

This school year, our Highly Capable Program began using peer analyses to:● identify ELLs who should probably be tested, and● confirm HCP referrals and test results because the HCP tests are not normed

for ELLs.




SHANNONThis is where we are in the agenda now.

How do you do a peer analysis?AGAIN: Everything our team unearthed says to compare ELL students against his/her peers. But most references to peer analysis in the literature simply said, “do a peer analysis,” with little description of what that entailed.

We needed a procedure, so we created our own to meet our district and student needs.

SHANNONWe cannot stress enough that this is our process to address this need, but it is a work in progress, and we are always seeking more information in this area.

How to do a peer analysis

1. Setting up a peer analysis2. Peer group selection 3. Troubleshooting peer group selection issues4. Leveraging district data5. Assembling and analyzing the data6. Case studies

SHANNONThere are many things we can do with a peer analysis. It is an intense data gathering system, but it provides useful information to understand and set learning expectations (always keep them high!), support classroom teacher’s understandings of language acquisition, decide upon interventions, develop a true representation of students skills within a group prior to SpEd or HCL referral.

1) Setting Up a Peer Analysis

To whom is the ELL student being compared?A peer analysis is critical in determining if the student’s performance is atypical.

The ideal peer group are ELLs, same language background, same time in program, same grade of entry in school. Scour district longitudinal data and find as large a peer group as possible.

[We suggest that, if needed, you group language minority students with similar language peers (e.g., use Punjabi speakers as peers for Hindi speakers; Russian speakers as peers for Ukrainian speakers, etc.)]

Example: Russian speaking students who were placed in program in Kindergarten, at Level 2.

YANA

What data should I look at for the peer comparison?

Years in program Entry grade WLPT-II [WELPA] levels WASL [MSP/HSPE] scores Mobility Parent input There is always more to find out…

1) Setting Up a Peer Analysis

YANA

Begin with a student information system, or Excel spreadsheet with at least the following data on all ELL students: name school current grade placement grade placement ELP level native language

Extract the students’ scores for all state and local assessments, and download them to an Excel spreadsheet. This will help you to deduce whether you have enough students with enough scores to be useful in comparison.

Troubleshoot, with the goal of finding as many peers as possible. -To increase group size, add other language groups or add other current grade levels to the sort. Note: current grade is only important in that districts change assessments over time, so same or similar grade level students will have had more of the same assessments to compare. -To decrease the size of a large group, try sorting within only one school, or a set of demographically similar schools.

Sort to find students whose placement grade level and placement ELP levels match those of the student of concern. True peers will also share the same language and current grade level, although others may need to be included if the initial group is less than about 10-15..

2) Peer Group Selection Process

YANA

2) Peer Group Selection ProcessFind student’s “true peers” - those who placed at the same grade and ELP levels, and who speak the same first language. If your district has changed tests often, it also helps if they’re at the same current grade level, so that their data will be aligned.

District policy gapTitle I/ non-Title

difference

YANAFinding a group that has the same set of assessment data is important. You may have students in the peer group with large gaps in their assessment data due to absences or gaps in the district assessment policy, so after your initial attempt you will need to comb through the spreadsheet to see if you have enough peers. Due to shifts in assessment policy over the years, using students in the same grade level cohort is helpful when creating a peer analysis - the students will, generally speaking, have had the same assessment experience.

As for gaps in the assessment policy, peer analysis makes this problem clearer. Your district, like ours, may have different assessment procedures for different schools. In our district, there are differences in how Title I schools and non-Title schools are required to assess students. For example, this year Title I schools must use the SRI as a universal screener (reading assessment) in grades 3-5 but non-Title schools do not; they use reading MAP. At Title I schools, all 2nd graders are given the BAS reading assessment 3 times per year but non-title schools only assess NYAS 2nd graders in Winter. When looking for and at data for ELLs across the district we found that it was sometimes difficult to find peers that had an adequate number of data points because the peers attended schools where students are assessed less frequently.

3) Troubleshooting: Peer Group SizePeer group n is too small (less than 10 or 15):● add students from other current grade levels, starting

with one above and one below, or two above, and expand even more, as needed.

● add students from other similar language groups or, if necessary, from all language groups.

● consider using both the expanded group and the smaller, “true peer” group with your team, simultaneously.

YANAIt is possible to use 2 peer groups, a small and an expanded peer group, so get a better picture of how a particular student is doing. A larger peer group is better because it captures a broader set of educational experiences and can help even out the data. For example, if you only have a peer group of 7 then it is quite possible that in that group of 7 are a few students are doing exceptionally well or quite poorly; having a group of 15 helps you to find a more normalized pattern of achievement - your perception of student performance is less likely to be skewed by a couple of outliers. It is not a bad idea to pull data on a small peer group, then search for more peers and pull data on the larger group to compare and see if there is any considerable difference in peer performance.

A small, “true peer” group might be needed to help provide better context for an individual student’s performance.

This peer group was expanded to include all languages, because in older kids it becomes more important to stay with similar current grades and there were only 3 or 4 in the true peer group, Russian.

3) Troubleshooting: Peer Group Size

YANA -

Because if the students are at very disparate grade levels, they will not have taken many of the same tests. Current high school students have never had a BAS, SRI at elementary level, etc.

3) Troubleshooting: Peer Group Size

Peer group size is too large:● such that analyzing the data would be too cumbersome

and graphs unreadable...○ consider compiling the data anyway, but then

narrow the group for closer analysis.○ consider using only students from the same school,

or schools with similar demographics (e.g., Kinder L2 Spanish speakers in Title schools)

YANA

3) Troubleshooting: Student needs closer peers

Special populations: Limit search to students in a special category (e.g.,

migrant, Speech & Language IEPs, internationally adopted) when attempting to analyze the performance of a student whose language development may be significantly impacted by other circumstances.

Why? Some factors should weigh heavily when examining a student’s performance and growth trajectory on district assessments. For example, ELLs identified with communication disorders will most likely significantly underperform their peers because the disorders impact their ability to hear, process, and/or produce language. A better peer group would be limited to students with communication-only IEPs.

YANAMany times, we find that students have additional or distinctive characteristics that can influence their performance on assessments and in the classroom. For example, my school has a very high number of ELLs with communication disorders (articulation and language) and a high number of homeless students. We have a couple of schools in our district with a high number of internationally adopted students. Shannon’s school has a large number of students qualified under the migrant program. These factors all influence first and second language acquisition, and can have a significant impact on student achievement in school. If there is a student of concern who has a characteristic that you believe is strongly influencing his or her performance, it would be worthwhile to do a peer analysis using peers who have that same characteristic. This peer analysis can be supplemental to a regular peer analysis that follows the selection process we described.

4) Leveraging District Data & AssessmentsDistrict data we use:

Language Literacy Math

K-2 WELPAWA Kids

WA Kids Fountas & Pinnell Phonological Awareness ScreenBAS (Fountas & Pinnell Benchmark Assessment System)K-Screen Literacy

WA KidsK-Screen Math

3-5 WELPA BASSRI (Scholastic Reading Inventory)MSP MAP (Measures of Academic Progress)

MSPMAP

6+ WLPT-II BAS (some schools)MSPMAP

MSPMAP

YANAThink about what data is available in your district. What universal screeners are given at each grade level or band? Where is that data stored, and how could you access it?

The data from our assessments is stored in Homeroom, a data warehouse BPS purchased a couple of years ago. Most of the data for the assessments shown here is stored and accessible from Homeroom. However, it took a considerable effort on the part of several folks in the district to get the data moved from our old student information system (SIS) into Homeroom, and there is still a lot of ongoing work to collect and clean up data before uploading to Homeroom at the close of each assessment cycle.

Other data that can be gathered and considered:

Based on Rhodes, Ochoa, & Ortiz (2008) “It is also important to ascertain the degree to which the student’s language abilities differ from those of other LEP students’ language abilities on informal language measures.”

Therefore, we might do a classroom SOLOM observation, or other data collection from classroom CFA’s, on just ELL peers in a class. This requires there are enough similar peers in the classroom.

4) Leveraging District Data & Assessments

SHANNONNow the Peer analysis may be important district wide, but we can also narrow this to the classroom focus. Depending on the number of ELL students in the classroom, and how many have the same language profile, classroom assessments and observations may be used in a more local peer analysis. We don’t want to solely rely on the WELPA as a language profile, as it is only done once a year. Gathering other language data is also helpful. Think of the language assessments at your school. Are there other ways for you to gather language or addtional data for a peer group?

We also use the State Language Peer Analysis to get a different picture of how the student is performing. This can be found on the EDS website under the assessment tab.

4) Leveraging District Data & Assessments

Getting there...

➔ Home➔ My Applications➔ Limited English Proficiency➔ Reports➔ Assessments➔ Peer Analysis

Info: https://eds.ospi.k12.wa.us/LimitedEnglishProficiency/Reports/LEP%20Peer%20Analysis.pdf

Note: OSPI says they have fixed the database behind its Peer Analysis tool. Additionally, consider your student’s unique characteristics when using this tool. Use this data wisely.

SHANNONWe are just going to touch briefly on this tool provided by OSPI. It is a peer analysis that compares ELL students to like peers across the state on state assessments. This is helpful to learn about language growth in comparison and comparison on large scale testing like MSP. This tool was down for a bit. but you probably just received an email that it is back up and running. You can access it as follows:

https://eds.ospi.k12.wa.us/LimitedEnglishProficiency/Reports/LEP%20Peer%20Analysis.pdf




5) Assembling the Data for Analysis

After the peer group has been identified:1. Use initials or alphabetic lettering (a, b...z, aa) to

remove student identities2. Collect historical and current assessment data for

previous 2-3 years (as possible) for each peer3. Create data tables for each data set (e.g., WA Kids

Language, WELPA, BAS)4. Graph data to show performance trends over time5. Highlight the student who is the focus of concern

SHANNONTHere are a few key things to remember once you pull the data for your student of focus. Make sure to honor confidentiality. Collect as much data as you can, also make sure it is relevant to the area of concern. Use Excel or a similar program to create data sets, and subsequent graphs of the different assessments. Lastly, highlight the student in each graph so it is easy to find. THere are tools in excel to do this. We have found that this helps when looking at a larger group’s graph.

5) Analyzing the DataQuestions to ask about the data:● What is the student’s performance relative to peers? (top 25%,

mid-range, bottom 25%)● What is the growth trajectory for the student in question relative

to the peer group? Is the student gaining ground, losing ground, remaining on the same trajectory relative to the peer group?

● What barriers or concerns are there, if any, for ELLs taking this particular assessment? (Do the assessment norms include ELLs? Is the assessment measuring fluency and not comprehension?)

● What kind of assessment data is being used? (Annual, quarterly, universal screener?) Give more weight to more frequent measures when analyzing growth.

SHANNONWe’re not using a bell curve. We just decided to use quartiles to get a rough picture of a student’s performance relative to peers. We might see data where most peers are performing right at grade level, and we’d be happy about that. This is not about finding standard deviations or developing strict norms for ELLs in the district because each student’s profile, while similar to peers, will be different based on the other factors that we know contribute to language acquisition.

5) Analyzing the Data for Referral“Overall, educators conducting a formal assessment of an ELL for special education, including within the RTI process, must keep in mind the following general testing characteristics:1. ELLs generally perform lower than non-ELLs on content-based

assessments (i.e., math, science, social sciences) even though they might not actually know less.

2. English language proficiency affects instruction and assessment.3. Language background variables may confound ELLs’ content-based

assessment outcomes.4. Assessments for ELL students have lower validity and reliability,

particularly for those at the lower end of the English proficiency spectrum (Abedi, 2004a).

5. Language factors may be a source of assessment error, affecting validity and reliability (Messick, 1989).”

SKIP - resource only

Source: Klingner, Janette K. Why Do English Language Learners Struggle with Reading?: Distinguishing Language Acquisition from Learning Disabilities. Thousand Oaks, CA: Corwin, 2008. 95. Print.

6) Case Study - Group Analysis

Spanish L2s

SHANNONThis graphs is a representation of our fall WaKIDS language data from some of our schools. This is a peer group of all Spanish speaking L2 students (Students were place with WELPA in the fall as well). Now, this can represent the range of language abilities within L2 speakers, but it also lets us see which students might need more scaffolding within their sheltered instruction. If we were to revisit this data in the winter, then we could also see which students have made growth or may need intervention in language. This type of data collections allows us to target diagnostic assessments on students on the lower edge of the peer group.

6) Case Study - Special Population

● An internationally adopted student was a focus of concern, and was being referred to SPED.

● As viewed from the classroom and school perspective, this student had lower academic achievement than other students who were receiving the same instruction.

● Our program conducted a peer analysis using data only from other internationally adopted students in the district, as they were closer peers.

YANA

Using the graphs from the peer analysis of internationally adopted students (next slide), which student appears to be the focus of concern?

6) Case Study - Special Population

YANA

YANAThe student represented by the green line in each of these graphs is the internationally adopted student who was a focus of concern. She first learned to speak Tamil, a language that is not commonly spoken in our community and that is not spoken by her adoptive parents. Because she has no language peers in the community or at home, this student went through language attrition after arrival. This student entered the school district as a Level 1 ELL in September of 1st grade. At the time this peer analysis was done, the student was one of only 4 ELLs in the school.

Paradis et al (2011, p159) cite a meta-analysis of internationally adopted students, conducted by Van IJzendoorn et al (2005), which “found that adopted children, both domestic and international, had significantly better IQs and better academic outcomes than children from similar backgrounds who remained in institutions or in their birth families. IA [internationally adopted] children often did not differ from their nonadopted peers in the their adoptive environments on IQ. However, the language skills and school performance of IA children were significantly poorer than those of their nonadopted peers in the adoptive environment, and they exhibited more learning problems. This pattern of results was evident for both domestic and international adoptees.” This Bellingham IA case study and Van IJzendoorn’s meta-analysis demonstrate the need for specialized peer analyses for special populations, which have the potential to be more informative about a student’s performance than a comparison to English-only grade level peers and even ELL like-language peers.

SPED disclaimer

CAUTION: If a student is in the bottom 25% of the peer group, this is NOT an automatic SPED referral!!

SHANNONTurn and Talk: WHY?

6) Case Study: Is the data supportive of a referral?

These graphs show data from our district. In each graph, the highlighted line shows the performance of a student who was considered a focus of concern.

After we show each graph, turn and talk to you your neighbor about what you think the data says.

SHANNONTo show you the power of a peer analysis in a collaborative setting. We are going to show you two peer analyses that were done in our district. By now, you know your neighbor well, so its time to process what you see in each graph.

Does this data support a referral?

SHANNONThe highlighted student is the student of focus. What does this suggest to you for a referall? We decided to do more intervention with this student.

Does this data support a referral?

YANAWe decided to refer this student, who had received a couple of years of intervention in reading, ELL, and math. This student is currently being assessed by the school psychologist.




YanaThis is where we are in the agenda now.

Lingering questions about peer group creation and data analysis:

● When and why is it better to limit peers to those with the most similar educational experiences (such as single school, Title I or non-Title only, etc.)? Should we always be looking district-wide?

● Which district assessments give us the best data?● What do we do when there is limited data (e.g., K-2)

or there are significant gaps in the data for peers?

SHANNON OR YANA

Communicating About Data

It is important to help classroom teachers, other collaborators, and parents understand how to read and understand the data.● CAUTION: If a student is in the bottom 25% of the peer group, this is

NOT an automatic SPED referral!!● Do not allow educators to use the data to set lower expectations, but

rather to set realistic expectations guided by an understanding of second language acquisition and development. The long-term goal is ALWAYS to move a student toward CALP in English and grade level achievement in content areas.

● If a peer analysis shows that the student is performing at the middle or high end of the peer group, help other educators reset (raise) their expectations for the student.

YANA

It is important to help classroom teachers, other collaborators, and parents understand how to read and understand the data.

● Talk honestly with parents about their child’s performance. Show the graphs if you think it appropriate, but be careful. Help the parents understand that the point of the data is to get a better picture of their child’s performance and to make decisions about what the school can do next.

● Again, if a child is performing at the low end of the peer group, do not allow parents to jump straight to a SPED referral. Think carefully with the family about what other school-based interventions can be put into place to support the student.

Communicating About Data

YANA

Recommendations

Get data! Make it happen...● For example:

○ We decided that all ELLs would be assessed on the BAS three times a year, regardless of “at standard” status on other assessments (SRI, MSP, MAP). This ensures we see a truer picture of the performance of all ELLs, not just the NYAS students.

○ Our Early Childhood Coordinator, Kristi Dominguez, decided that WA Kids would be required 3 times a year; this is above the state-requirement.

YANA

RecommendationsUse the data! For example:● Our district invested in Homeroom, which is a data warehouse connected

to Skyward. We use Homeroom to pull data for peer analyses.● We set up basic peer groups in Homeroom (e.g., all students who placed

as L1 in grades K, 1, 2, 3; all students who placed as L2 in grades K, 1, 2, 3…). This makes getting the initial data set easier.

● We are just beginning to use the data in collaboration with reading teachers and classroom teachers to decide which intervention is most appropriate for a given ELL.

● Find standout areas in the district or school-level data and work with teachers to address the identified need.

YANA

Contact information:

Kristyana Mansfield, ELL [email protected]

Shannon Lemieux, ELL [email protected]

Kristi Dominguez, Director of Teaching & Learning, Early Childhood Education

[email protected]

mailto:[email protected]






Begin with a student information system, or Excel spreadsheet with at least the following data on all ELL students: name school current grade placement grade placement ELP level native language

Extract the students’ scores for all state and local assessments, and download them to an Excel spreadsheet. This will help you to deduce whether you have enough students with enough scores to be useful in comparison.

Troubleshoot, with the goal of finding as many peers as possible. -To increase group size, add other language groups or add other current grade levels to the sort. Note: current grade is only important in that districts change assessments over time, so same or similar grade level students will have had more of the same assessments to compare. -To decrease the size of a large group, try sorting within only one school, or a set of demographically similar schools.

Sort to find students whose placement grade level and placement ELP levels match those of the student of concern. True peers will also share the same language and current grade level, although others may need to be included if the initial group is less than about 10-15..

Peer Group Selection Process

Leveraging District Assessment Information for ELL Peer Data Analysis WABE 2015Kristyana Mansfield, Shannon Lemieux, Kristi Dominguez, Lynne Skerry Bellingham Public Schools

We also use the State Language Peer Analysis to get a different picture of how the student is performing. This can be found on the EDS website under the assessment tab.

Leveraging District Data & Assessments

Getting there...

➔ Home➔ My Applications➔ Limited English Proficiency➔ Reports➔ Assessments➔ Peer Analysis

Info: https://eds.ospi.k12.wa.us/LimitedEnglishProficiency/Reports/LEP%20Peer%20Analysis.pdf

Note: OSPI has been fixing the database behind its Peer Analysis tool. Additionally, consider your student’s unique characteristics when using this tool. Overall, use this data wisely.

Leveraging District Assessment Information for ELL Peer Data Analysis WABE 2015Kristyana Mansfield, Shannon Lemieux, Kristi Dominguez, Lynne Skerry Bellingham Public Schools


Peer Analysis Planning Sheet The purpose of a peer analysis is: The basic elements, at minimum, of a peer analysis are:

How can you build understanding among teachers, specialists, and other staff about the rationale for doing peer analyses and interpreting the data?

What data does your district collect on student performance? (What assessments are given and what kind of data do they produce?)

Which assessments are universal? Which assessments are given in which grade bands?

Where is your district data housed? (For example, Bellingham Schools stores its data in Skyward and Homeroom, but WA Kids data is stored on its own website.) If your school district is considering purchasing a data warehouse, can you or another key influencer participate in the selection process?

Leveraging District Assessment Information for ELL Peer Data Analysis WABE 2015 Kristyana Mansfield, Shannon Lemieux, Kristi Dominguez, Lynne Skerry Bellingham Public Schools Who has access to the data? How would someone in charge of peer analyses get permissions to access and download that data? Who are the key influencers and collaborators in the district who can help develop processes and procedures for doing peer analyses? What district resources can be dedicated to creating peer analyses? (For example, Bellingham Schools’ elementary ELL team has one member who has some time dedicated to accessing the data and creating Excel tables and graphs.) What additional professional development or resources do I need to increase my understandings about peer analyses? (Excel, data warehouse, SPED referrals, RTI, collaboration time?)

© 2013 Teaching Strategies, LLC, Bethesda, MD; www.TeachingStrategies.comThese materials may not be duplicated without the express written permission of Teaching Strategies, LLC.

5/13: GOLD_HNDT_ODL Objectives and Dimensions (WaKIDS)

Teaching Strategies GOLD® Assessment System

Social–Emotional1. Regulates own emotions and behaviors b. Follows limits and expectations c. Takes care of own needs appropriately2. Establishes and sustains positive relationships c. Interacts with peers d. Makes friends

Physical

4. Demonstrates traveling skills5. Demonstrates balancing skills6. Demonstrates gross-motor manipulative skills7. Demonstrates fine-motor strength and

coordination a. Uses fingers and hands b. Uses writing and drawing tools

Language9. Uses language to express thoughts and needs a. Uses an expanding expressive vocabulary b. Speaks clearly c. Uses conventional grammar d. Tells about another time or place10. Uses appropriate conversational and other

communication skills a. Engages in conversations b. Uses social rules of language

Cognitive11. Demonstrates positive approaches to learning c. Solves problems d. Shows curiosity and motivation e. Shows flexibility and inventiveness in thinking12. Remembers and connects experiences a. Recognizes and recalls13. Uses classification skills

Literacy15. Demonstrates phonological awareness a. Notices and discriminates rhyme b. Notices and discriminates alliteration c. Notices and discriminates smaller and smaller

units of sound16. Demonstrates knowledge of the alphabet a. Identifies and names letters b. Uses letter–sound knowledge17. Demonstrates knowledge of print and its uses b. Uses print concepts18. Comprehends and responds to books and

other texts a. Interacts during read-alouds and book

conversations b. Uses emergent reading skills c. Retells stories19. Demonstrates emergent writing skills a. Writes name b. Writes to convey meaning

Mathematics20. Uses number concepts and operations a. Counts b. Quantifies c. Connects numerals with their quantities21. Explores and describes spatial relationships

and shapes b. Understands shapes 22. Compares and measures

GOLD Objectives and Dimensions (WaKIDS)

Note. These 19 objectives are a subset of the 38 Teaching Strategies GOLD® (TSG) objectives. The number associated with the objective corresponds with the TSG objective; numbers are missing when the associated TSG objective is not part of WaKIDS.


Resources

The presentation slides are available online at:

__________________________________________________________

Bender, Franklin. "The Continuum of Evaluating and Serving ELL Students for SPED Services." www.NationalCEU.com. National CEU, LLC, 1 Jan. 2007. Web. 24 Feb. 2015. <http://www.ode.state.org.us/teachlearn/conferencematerials/sped/bender_ell.pdf>.

Brown, Julie Esparza, and Jennifer Doolittle. "A cultural, linguistic, and ecological framework for response to intervention with English language learners." (2008).

Gill, Steve, and Ushani Nanayakkara. The ELL Critical Data Process: Distinguishing between Disability and Language Acquisition. 1st ed. CreateSpace Independent Platform, 2014. Print.

"GOLD Objectives and Dimensions (WaKIDS)." Teaching Strategies, LLC, 1 Jan. 2013. Web. 1 Mar. 2015. <http://www.k12.wa.us/WaKIDS/pubdocs/GOLD_HNDT_Objectives.pdf>.

Klingner, Janette K., John J. Hoover, and Leonard M. Baca. Why Do English Language Learners Struggle with Reading?: Distinguishing Language Acquisition from Learning Disabilities. Thousand Oaks, CA: Corwin, 2008. 95. Print.

OSPI Peer Analysis information: https://eds.ospi.k12.wa.us/LimitedEnglishProficiency/Reports/LEP%20Peer%20Analysis.pdf

OSPI Staff, Migrant & Bilingual Education (2009). When is it Appropriate to Refer an ELL to Special Education? [PowerPoint slides]. Retrieved from https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&cad=rja&uact=8&ved=0CDYQFjAD&url=http%3A%2F%2Fwww.k12.wa.us%2Fspecialed%2Fprogramreview%2Fmonitoring%2FStudentPerformance%2FWhenIsItAppropriatetoReferanELL.ppt&ei=g0wDVfbHBITmoAT3vYLIDw&usg=AFQjCNGe8VdeMhHmCplhwOZ9dNBqooAJLw&sig2=8lDZcM8IhAsZpvh3tm5Yhg

Paradis, Johanne, Fred Genesee, and Martha Crago. Dual Language Development and Disorders: A Handbook on Bilingualism and Second Language Learning. 2nd ed. Baltimore: Paul H. Brookes Pub., 2010. 221-222. Print.

Rhodes, Robert L., Salvador Hector Ochoa, and Samuel O. Samuel. Assessing Culturally and Linguistically Diverse Students: A Practical Guide. New York: Guilford, 2005. Print.

ideas for leveraging district data - schedschd.ws/hosted_files/wabe2015/89/leveraging district...

Documents