Responsive Evaluation: Implementation of Saxon Phonics and Spelling - K

A REPORT TO: ABC Elementary School

March 27, 2013

Elementary schools have the important role of laying the groundwork for creating successful students. Elementary school teachers must educate students on many competencies including social, behavioral, and instructional skills. Schools are responsible for teaching students fundamental skills in Pre-Kindergarten and Kindergarten which will carry them throughout life. One of the most essential skills that young children need to develop is that of reading. How does a teacher know if the program they use to teach reading is effective? How can one gauge the successful implementation of a reading program? These questions are answered as an investigation is conducted at ABC Elementary School.

Various methods and programs are used to teach reading to students. One program that is used at ABC Elementary School is Saxon Phonics and Spelling K (Saxon Publishers, 2005). To gain insight into the implementation of Saxon Phonics and Spelling K (Saxon Phonics) at ABC Elementary School, focus group meetings were conducted with Kindergarten teachers who have been using this program since the 2009-2010 school year and with related instructional personnel who support the Kindergarten teachers’ classroom implementation of the program (C. Gravely, personal communication, October 5, 2012) . Information provided by focus groups indicated strengths and areas for probable improvement in the implementation of the program. This evaluation investigated the classroom practices in regard to program implementation. Also, student data was reviewed to determine the effectiveness of the program.

An evaluation was led at ABC Elementary School to determine the effectiveness of a supplemental phonics and reading program which teaches phonemic awareness—Saxon Phonics and Spelling K. In the pages which follow, a detailed program description of Saxon Phonics is detailed, which includes the background of the program implementation at ABC Elementary School. Then, a review of the literature on fundamental reading skills and approaches are outlined. These skills are organized into four themes: (a) pre-literacy skills, (b) phonics approach, (c) whole language approach, and (d) eclectic approach. Finally, the proposal for carrying out the evaluation is shared, as well as a report of the researchers’ findings.

ABC Elementary School

Responsive Evaluation: Implementation of Saxon Phonics and Spelling – K

March 27, 2013

“Multi-sensory approach engaging visual,auditory, and tactile senses”

THE EVALUANDThe evaluand is the implementation of Saxon Phonics and Spelling – K at ABC Elementary School.

“The primary goal of the [Saxon Phonics and Spelling K] program is for all kindergartners to be able to recognize and know the primary sounds of all the letters of the alphabet” (Simmons, p. 7). The program is designed to be implemented early in the school year. All students are screened using the pre-assessments to identify each child’s phonological and phonemic awareness. There are 140 lessons spread across 35 weeks. The instructional overview of the program states that students need to review all letters of the alphabet. Ensuring that students have time to practice these skills every day is crucial to the validity of the program. The Saxon Phonics program includes (a) the controlled vocabulary and practice, (b) the annotated bibliography, (c) coding, (d) spelling, (e) assessments, (f) pacing, (g) handwriting, and (h) reading fluency (Simmons, 2006).

The controlled vocabulary uses letters, letter clusters, and sounds that have been instructed throughout the program and reinforces concepts as they are taught. The annotated bibliography helps the teacher find appropriate literature resources to reinforce concepts taught in a lesson. Coding is used so readers can identify common vowel patterns and clusters. Spelling is a part of the Saxon Phonics program that teaches spelling rules as well as spirals throughout the duration of the lessons. The program also teaches simple sight words and words that do not follow the typical spelling rules. Assessments are used to determine how well students are learning concepts and should be used as diagnostic tools to guide remediation. Pacing can vary amongst classrooms based on the pre-assessments. Teachers can slow or accelerate the pace of the program as needed. Handwriting is a component of the program; however, the Saxon Phonics and Spelling K program accommodates varying handwriting styles. Reading fluency is necessary for students to develop reading comprehension skills. The program provides a separate fluency instruction booklet to guide how this should be taught (Simmons, 2006).

Saxon Phonics and Spelling K has several components to support its implementation. It includes the (a) teacher manual, (b) classroom materials, (c) student materials, and (d) optional components. The teacher manual is broken into three volumes and is housed in binders. Two of the binders include the scripts for the lessons as well as the resources to provide extra practice. The third binder is the teacher resource binder. The teacher’s manual also includes lesson booklets for each of the 140 lessons. The lessons have been designed in a sequential order so it is imperative that a lesson is not skipped or taught in a different order than stated. The classroom materials include decks of review cards that are available that are used for students to practice daily during teacher directed activities. The review decks include letter cards, picture cards, spelling cards, and sight word cards. Additional classroom materials include three kid card decks used for individualized practice and remediation, alphabet wall cards, syllable division wall cards, vowel rule wall cards, spelling rule wall cards, color and number posters, and classroom management posters. An audio cassette tape that allows the student to hear letter and letter cluster sounds is also available. Student materials include laminated alphabet handwriting strips that students can refer to as a reference. Letter tiles for students to use to practice spelling, letter recognition, and alphabetizing are available. Worksheets and homework are provided to introduce concepts and reinforce learned skills. Decodable readers are introduced once every other week to teach print awareness, vocabulary, and to check children’s understanding of taught skills. Initially they are used in small or whole group and then students use them to read independently (Simmons, 2006). There are several optional components available but not all are used by ABC Elementary School. The components used are the handwriting masters which provide practice in forming capital and lower case letters. Leveled fluency readers are used to engage students in fluency practice.

CURRENT ISSUE/PROBLEM

Problem Statement/Current Issue or Situation

Research was conducted at ABC Elementary School to determine the effectiveness of a supplemental phonics and reading program which teaches phonemic awareness—Saxon Phonics. ABC Elementary School uses Saxon Phonics to assess, teach, and reinforce phonemic awareness and spelling. Teachers can adjust the pace of their curriculum based on the needs of the classroom as determined by pre-assessments. Optional components are not consistently used in the kindergarten classrooms at ABC Elementary School. However, the core instructional components of Saxon Phonics are implemented consistently across the grade level. Saxon Phonics is funded through textbook funds and the Title I grant. It is overseen by the school Principal, the Director of Instruction, and the Division Superintendent of ABC Public Schools. This program is supplemental to the basal reading program.

The target audience at ABC Elementary School is the entire kindergarten population. The stakeholders include (a) ABC County School Board, (b) division leaders, (c) principal, (d) teachers, (e) students, (f) parents, and (g) related instructional personnel. ABC County School Board and division leaders are involved by overseeing the results of the program. The school principal is responsible for overseeing the implementation of the program. The teachers implement the program at the classroom level with fidelity. The students receive the benefits of the program and parents are involved with the homework and reinforcement. Finally, the related instructional personnel provide instructional support to the classroom teachers during implementation (C. Gravely, Personal Communication, October 5, 2012). The research to be conducted will help determine the effectiveness of the implementation of the Saxon Phonics program at ABC Elementary School.

“Systematically distributed instruction, practice, and assessment”

Historical Considerations

Saxon Phonics was originally purchased and piloted in kindergarten during the 2008-2009 school year to remediate and teach target skills in the areas of phonemic awareness and phonics. Full implementation across the school division began during the 2009-2010 school year. Prior to that, it had been implemented in a special education classroom as well as a second grade classroom by teachers who were concerned about the low reading levels of their students. The gains noticed in student performance after using this program warranted pilot implementation at lower levels. There was training provided to all teachers in 2009 by the Houghton-Mifflin Company which owns the Saxon Phonics program. Follow up training was also provided during the year to ensure proper program implementation. Saxon Phonics instruction is used simultaneously with the basal reading program. Two directors of instruction have overseen the program during the pilot and implementation phases. Saxon Phonics (2005) has been implemented at ABC Elementary School since 2009-2010.

ABC Elementary School is located in the small rural community of Alphabet and is one of three elementary schools within the school division. Due to budget cuts, two community schools - ABC Elementary School located in Alphabet and XYZ Elementary School located in Letterland – were reconfigured to combine the student population of Alphabet and Letterland serving different grade spans. This grade reconfiguration was in effect for the 2010-2011 school year. ABC Elementary School serves students in Pre-Kindergarten, Kindergarten, Early Childhood Special Education, and Head Start (C. Gravely, Personal Communication, October 5, 2012).

“New skills introduced by building upon previously taught concepts”

LITERATURE REVIEW

Research on fundamental reading skills can be organized into four themes: (a) pre-literacy skills, (b) phonics approach, (c) whole language approach, and (d) eclectic approach.

Pre-Literacy SkillsPre-literacy skills are necessary to lay a strong foundation for reading. Research reveals some

fundamental elements that are important in reading and also various instructional approaches for learning how to read. These skills include but are not limited to (a) alphabet knowledge, (b) concepts about print, (c) phonological awareness, and (d) expressive vocabulary. Explicit and systematic instruction based on the needs of the students is an important instructional approach to teaching reading (Faust et al., 2011; Irwin et al., 2012; Menzies et al., 2008; Sanchez et al. 2010; Saxon Publishers, 2004). Children must have a complete understanding of the pre-literacy skills to become good readers. Kindergarten is a critical school year for teaching these skills to students.Alphabet Knowledge

Research suggests that alphabet knowledge is an important fundamental skill needed to develop strong reading skills. Irwin et al. (2012) suggest children may begin this process by learning their name and the letters which compose it. Having children identify the letters in their name helps them to understand that the letters in their names have meaning. Introducing children to alphabet books will help them develop alphabet knowledge (Irwin et al., 2012). Concepts about Print

Irwin et al. (2012) suggest children who can discriminate between letters, numbers, words, and pictures understand print concepts. Understanding that words are read left to right is an important skill when learning to read. These concepts can be taught when reading with students and pointing out the different parts of the book. Concepts of print also include holding the book in the correct position for reading. Ball and Gettinger (2009) suggest that providing children with various types of print material will allow teachers the opportunity to offer language and print-rich environments for students.Phonological Awareness

Phonological awareness progresses from large phonological units to small phonological units. Phonological awareness refers to the ability to identify, perceive, and manipulate the sounds of spoken words (Brunswick et al., 2011; Sanchez et al., 2010). Phonological awareness is a broad topic in which different areas of phonological awareness peak at different times during a child’s development (Brunswick et al., 2011, p. 269). Early intervention instruction in the area of phonemic awareness allows children to gain a strong foundation in the alphabet principle (Saxon Publishers, 2004). Meta analyses of phonological awareness studies conclude that teaching phonemic and alphabetic skills in kindergarten is essential to early literacy development (Vadasy & Sanders, 2012).Expressive Vocabulary

Research suggests that expressive vocabulary is a strong indicator of reading success (Irwin et al., 2012). Children with a limited vocabulary often struggle in school because explicit instruction in this area does not begin until later in a student’s career and by then it is too late. Students need this pre-literacy skill in order to become fluent and successful readers. Rationale for Skills

Pre-literacy skills are imperative in order to become a successful reader. These skills best prepare children to be good readers at a later date. Irwin et al. (2012) suggest instruction should be systematic and sequential. Saxon Publishers developed their Saxon Phonics and Spelling – K program based on these skills. Faust and Kandelshine-Waldman (2011) attack pre-literacy skills through the approaches to reading which include phonics, alphabet principle, and whole language. Each of the components is reflective of the skills children need to be successful at reading early in their school career. Irwin et al. (2012) understand the importance in promoting emerging literacy in young children with the belief that early success in attainment of reading skills will lead to later successes as the learner grows.

Phonics ApproachThe phonics approach to learning how to read emphasizes the important components of a word and

memorizing the sounds of the letters through a bottom-up process (Faust & Kandelshine-Waldman, 2009). Phonics is one approach to reading suggested through this review of literature. Heide (2005) suggests research does not support isolated phonics instruction even though phonics instruction has been shown to improve reading achievement (p. 32). Teaching children the relationships between letter-sound relationships and how to apply their skills in meaningful texts is an essential component of the phonics approach to teaching reading. Hammill and Swanson (2006) suggest phonics instruction is structured, highly-sequenced, and focuses primarily on the alphabetic principle (p. 17). The phonics approach to reading includes aspects of phonemic awareness, guided oral reading, and instruction in strategies for comprehension (Heide, 2005, p. 32). Research supports the need for explicit and systematic phonics instruction in order to increase a student’s reading comprehension (Saxon Publishers, 2004). The phonics approach to reading promotes early reading interventions for those beginning readers. A substantial amount of research supports early reading intervention for kindergarten students who are at-risk (Vadasy & Sanders, 2012). A study completed by Hammill and Swanson (2006) found that students instructed using the phonics approach made significant growth in reading when compared to those students with no phonics instruction.The Alphabetic Principle

The alphabetic principle refers to understanding that each letter makes its own sound and using relationships between letters and phonemes to assist readers in pronouncing unfamiliar words. This includes words with irregular spelling patterns. Research suggests phonics instruction places emphasis on words and syllables and teaches children to read and spell words (Hammill & Swanson, 2006). Vadasy and Sanders (2012) suggest systematic phonics instruction beginning in kindergarten will allow students to read and spell words by applying knowledge of the alphabetic principle. Phonemic Awareness

Children who are given opportunities to gain phonemic awareness will allow them to become better readers (Saxon Publishers, 2004). A study conducted by McGeowan, Johnston, and Medford (2012) suggests phonics instructed children have a better letter-sound knowledge, phonemic awareness, and word reading skills (p. 363). Brunswick et al. (2011) found through a longitudinal study there was a significant correlation between visual-processing skills, phonological awareness, phonological memory, and emergent reading ability (p. 282). Research shows a strong link between phonological knowledge of phonemic awareness and phonics as an important fundamental element in learning to read (Faust et al., 2011; Irwin et al., 2012; Menzies et al., 2008; Sanchez et al., 2010; Saxon Publishers, 2004). Vandasy and Sanders (2012) found positive significant effects on kindergarten phonological awareness and comprehension outcomes as described by the intervention year study completed. Teaching students at the phoneme level helps them to understand how to blend letter sounds and form words. Brunswick et al. (2012) state phonological memory supports the identification of unfamiliar words during the application of the grapheme-phoneme rules and contributes to the comprehension of written text by helping children remember words they already know (pp. 269-270).

Whole Language ApproachThe whole language approach to learning how to read

emphasizes the use of literature to teach literacy skills. The whole language approach has become controversial as new ways of teaching reading have emerged. Jeynes and Littell (2000) discovered through a meta-analysis study that whole language instruction is appealing because it emphasizes (1) the use of whole pieces of literature, (2) student choice versus whole class assignments, and (3) integrated language experiences. The review of articles led to the evolution of the definition of the whole language approach and an examination of the pedagogy involved in its implementation, positive implications, and the limitations of the approach.

Defining Whole LanguageSimply stated, the whole language approach can be

described as a contrast to learning to read using a phonics approach. Yet, research on the whole language approach exposes that there is not a clear definition of this approach. In reviewing the literature, it is apparent that the description of the whole language approach to learning to read has evolved over time and it is difficult to pinpoint an exact definition (Cambourne & Turbill, 2007; Jeynes & Littell, 2000). Jeynes and Littell (2000) constructed their view of the whole language approach from research which discussed (1) constructing meaning, (2) the writing process, (3) use of literature, including trade books, and (4) student-centered choice and language. Cambourne and Turbill (2007) examined the history of whole language development in order to define it, beginning in the 1960s. They describe the beliefs about whole language for each era to help define the term: (a) 1960s: children must control their individual skills before they can read and write; student created literature was used to teach reading; (b) 1970s: emphasis was placed on the use of language in meaningful situations; whole language became a thematic approach to teaching different concepts using literature; Sustained Silent Reading and Drop Everything And Read programs emerged; (c) 1980s: the term literacy emerged from the whole language approach as reading and writing were integrated; students stopped writing “stories” and started writing “narratives, descriptions, procedures, and reports” (d) 1990s: the term “whole language” was marginalized and literacy was more clearly defined to include speaking, listening, reading, and writing; and (e) 2000s: the term “whole language” was replaced by “literacy;” literacy was recognized as a complex set of skills needed for pleasure reading and writing, needed for formal writing and comprehension of written materials, and used in social settings for communication. Faust and Kandelshine-Waldman (2009) explain whole language as an approach which extracts meanings of words from context while implementing a top-down process of exploring sentences and text.

Whole Language:

Use whole pieces of literature Student choice Integrated language

experiences

Eclectic Approach

Much like an eclectic approach to decorating a house is based on merging different styles and patterns to create a unique look, the eclectic approach to teaching reading uses varying approaches to meet students’ needs. Specific components of an effective reading program should include portions of both the phonics and the whole language approaches. This balanced approach to reading programs is synonymously called “an eclectic” approach. The review of literature on the eclectic approach helps one to understand the various instructional practices used in the eclectic approach, theoretical basis for why this approach works, and implications for students taught using the eclectic approach.

Instructional PracticesA balanced reading instructional program combines the use of different instructional

approaches. The eclectic approach “cannot be interpreted monolithically because of the various ways that whole language and phonics can be taught and combined” (Calais, 2008, p. 1). Calais (2008) proposes that an eclectic approach offers an alternative to phonics only or whole language only programs by combining instructional methods of each to meet the needs of an array of learning styles with the goal to teach students to read fluently. Faust and Kandelshine-Waldman (2009) explain the eclectic approach as incorporating the use of bottom-up phonics instruction with top-down whole language approaches to teach the alphabetic principle. Research conducted by McGeowan et al. (2012) described the eclectic approach as combining a “whole-word, guess-from-context approach” with the slower paced phonics approach (p. 361). Menzies et al. (2008) conducted a study in which they used an eclectic model which had the following components added to the regular language arts program: (1) ongoing assessments to measure progress and skill attainment; (2) high intensity instruction delivered in small groups; (3) explicit instruction for children who lacked knowledge of phonemes and the alphabetic principle; and (4) collaborative teaching with a literacy coach. The eclectic approach proposes balanced instructional practices of phonics and whole language.

Implications for Using the Eclectic ApproachPositive implications for using an eclectic approach are evident based on the studies examined

in the literature review. The eclectic approach was successful due to the students’ vocabulary knowledge through the use of big books and story time activities (whole language), letter-sound knowledge as children were taught at the phoneme level to blend sounds (phonics), and rhyming skills as students used phonological word recognition and whole word skills (eclectic) (McGeown et al., 2011). In the study completed by Menzies et al. (2008), all students showed growth after implementing the eclectic approach in the already existing language arts program. Menzies et al. discovered that even students who began the study with considerably lower scores on phonemic segmentation assessments made progress that may not have been attained in the previous setting. Faust and Kandelshine-Waldman (2009) uncovered that top-down practices (whole language) were more effective with higher level readers while bottom-up practices (phonics) were more beneficial to readers who lacked phonologically based decoding skills. Thus, an eclectic approach is able to address varying needs of all readers. Finally, Calais (2008) implies understanding the theory of how children learn will strengthen the argument that a balanced approach to learning how to read will allow the teacher to concentrate on the student needs and beneficially meet those needs.

Literature Review ConclusionsBased upon this review of literature, certain conclusions can be made for early intervention

reading programs. Pre-literacy skills are essential for establishing a strong foundation for learning to read. The literature provided three different approaches for teaching reading. Each approach offers an argument for the need to use the specific method for instruction.

Pre-literacy skills encompass visual perception skills, awareness of skills, and perceptions of print. Alphabet knowledge helps children understand that letters have meaning and are used to build words. Concepts of print are important skills for students to be able to assemble letters, numbers, words, and pictures into meaningful relationships. A limited vocabulary impedes a child’s ability to understand complex text. Use of expressive vocabulary helps children build their literacy skills through fluency and enunciation. Phonological awareness is a broad skill that is important for students to learn so they have the ability to apply decoding when learning new words.

The phonics approach in learning to read emphasizes the understanding of the alphabetic principle, which is the understanding of how each letter makes its own sound. The phonics approach teaches students that parts of words are combined to make whole words. Students who are taught using the phonics approach have better-letter sound knowledge, an understanding of phonemic awareness, and strong word reading skills. Research indicates a very strong link between phonological knowledge of phonemic awareness and phonics as an important foundation for learning to read.

The whole language approach is a complex approach to learning to read which has evolved over the last forty years. It typically is understood to be an opposite approach to phonics based instruction. The whole language approach emphasizes using various genres of literature to construct meaning from words based on student choice and language. Whole language in some aspects has been replaced by the word literacy which involves a very complex set of skills needed for reading for pleasure, writing, and exploring the meaning of text.

The eclectic approach merges the phonics and whole language approaches to create a unique style of learning to read. Since whole language instruction can vary depending on the teacher as well as the students, the pedagogy of the eclectic approach will also vary. This balanced approach for learning to read utilizes literature resources such as big books and story time activities as well as phonic skills to teach letter sound knowledge. Using rhyming text in phonological word recognition and whole word skills is an important aspect of the eclectic approach.

Each of the three approaches of learning to read utilizes the knowledge of pre-literacy skills to establish the building blocks needed to become successful readers. It is apparent through the research there is a valid argument for each approach to create productive readers. The literature review provided documentation that the eclectic approach to learning to read provides an environment to meet the needs of a diverse student population. The eclectic approach encompasses the phonics approach to meet the needs of lower skilled readers whereas the whole language approach is used to meet the needs of the stronger readers.

Evaluation Design

After meeting with staff for focus group meetings at ABC Elementary School, it became evident that parts of the Saxon Phonics program are being implemented with fidelity, where other areas still need to be developed. Concerns about the implementation of the Saxon Phonics program are partly due to emerging technologies, a lag time in initial training and current needs, and a lack of supplemental fluency materials (Focus Groups, personal communication, October 5, 2012). Within the values branch, the evaluator’s role is to be the communicator that promotes dialogue that is meaningful (Mertens & Wilson, 2012). During the evaluation, the evaluators were engaged in meaningful conversations with the stakeholders for effective communication. The evaluators remained aware of power differences that exist within the organization and how power may influence the evaluation. Using leadership roles through the leadership process, the evaluators bring to light possible conflicts and provide a safe, comfortable setting for possible discussions about the implementation of Saxon Phonics (Mertens & Wilson, 2012). In addition to interviews, the evaluators conducted classroom observations, reviewed Saxon Phonics program documents, reviewed student documents, and dissected student performance data. This section will examine the theoretical lens in which the evaluation proposal was developed, the focus for the evaluation, the data collection tools used during the evaluation, and the targeted populations who were involved in the evaluation.

FOCUS OF THE EVALUATIONThe focus for the evaluation of Saxon Phonics was the implementation of the Saxon Phonics

program in Kindergarten classes at ABC Elementary School. The principal at ABC Elementary is interested in determining how effective the implementation of the Saxon Phonics program is based on the fact that some components of the program are not being used. The following questions were answered as a result of the responsive evaluation.

1. How does the classroom implementation of all the components of Saxon Phonics impact the effectiveness of the program?

2. How has the lack of on-going professional development for the Saxon Phonics program affected the fidelity of the program?

The evaluation employed different methods to collect data including observations, interviews, program documents, student documents, and student data. Each of the questions posed for the evaluation were answered through the data collection process. Question one refers to the classroom implementation of the Saxon Phonics program. The evaluators used observations, interviews, and student data to determine how the varying implementation of the program between classes impacts the effectiveness of the program, if at all. Question two refers to the professional development for the Saxon Phonics program. The evaluators used documents, observations, and interviews to determine how professional development since the beginning of the program implementation has had an effect on the fidelity of the program. Student data was limited to measuring achievement of phonics and literacy attainment.

LOGIC MODEL

Figure 1. The logic model demonstrates how the input and output components of the evaluation of the Saxon Phonics program.

Steps Taken to Enhance CredibilityStrategies to enhance credibility with the qualitative data collection process included substantial

engagement, persistent observations, member checks, peer debriefing, and use of multiple data sources. Substantial engagement in the observation process required that the evaluators spent sufficient time understanding how the Saxon Phonics program was being implemented to draw appropriate conclusions. Persistent observations were necessary to understand the scope of the Saxon Phonics implementation. For example, one lesson should be implemented over an entire week. It was necessary for the evaluators to observe more than one day. The evaluators used member checks to share the data collected with the participants. This helped to determine the accuracy of the data collected. The two evaluators used peer debriefing to discuss the research process at different stages. This allowed for the evaluators’ viewpoints to be shared with each other and explained when misunderstandings may have occurred. The evaluators used multiple data sources, including observations, interviews, and checklists, to strengthen the credibility of their findings. This allowed for triangulation of data which led to a conclusion about the implementation of Saxon Phonics.

Steps Taken to Enhance DependabilityData collection happened during a two week period. Interviews were held with the teachers which lasted

approximately one half hour each with the four teachers who implemented the Saxon Phonics and Spelling – K regularly. Notes were taken on their responses and the evaluator notes were reviewed prior by them. The interviewees approved their responses as accurate. Classroom observations were completed in each of the four kindergarten classrooms while the Saxon Phonics lesson was being taught. The classroom teachers completed a document checklist indicating the components available for their use. The two evaluators completed observation checklists separately and met to debrief after the observations were completed. All checklists and data collection tools can be found in the appendices.

Data Analysis

During the evaluation process the evaluators used the following qualitative data measures: (1) observations, (2) interviews, and (3) documents. Concurrently quantitative data were collected. The qualitative data provided the evaluators information about how the stakeholders interact with the Saxon Phonics program. The qualitative data provided the evaluators insight as to how the program components were being implemented on a weekly basis within the regular classroom setting. The quantitative data provided evidence of student achievement of letter and sound recognition, concept of word, rhyming, and spelling. The qualitative data measures provided more information to answer the questions posed by the research being conducted on the Saxon Phonics and Spelling – K program.

Observations Observations were conducted over a consecutive four day time frame. Each of the four regular

Kindergarten classes was visited to observe which components were used in the weekly lesson delivery of Saxon Phonics. Interviews

Interviews were conducted with stakeholders. Interviews allowed for stakeholders to speak freely in regard to their implementation of the Saxon Phonics program in their classrooms. Interviews provided an opportunity for staff to express their views of the program, as well as their expectations for program outcomes. Documents

Program documents, as well as documents kept by the teachers, were reviewed. The evaluators viewed classroom teachers’ lesson plan books for documentation of the components used from the Saxon Phonics program. Classroom materials and student work sample documents were reviewed to determine the level of implementation of the program.

Figure 2. Classroom observations and review of classroom documents portrayed how many teachers use the various components of the Saxon Phonics and Spelling- K program.

Quantitative MeasuresStudent data was collected as a part of the program evaluation. Six-week assessments on upper and lower

case alphabet recognition and letter sounds were reviewed as well. Student achievement was compared using baseline data from the beginning of the school year and on mid-year assessments. Data from the Phonological Awareness and Literacy Screening (PALS) assessment was collected from the beginning of the school year and mid-year for students identified as risk for learning phonological concepts. All Kindergarten students took the PALS assessment at the beginning of the year, but only students who were identified as at-risk by the pre-assessment were tested using the PALS mid-year assessment. The alphabet knowledge and PALS data was reviewed by classroom to determine the impact of the implementation of Saxon Phonics by classroom and to identify whether or not there was an effect on student achievement.

DiscussionThe evaluation process provided the researchers with sufficient information to make conclusions about the

classroom implementation of the Saxon Phonics program. The following questions were answered as a result of the evaluation:

1. How does the classroom implementation of all the components of Saxon Phonics impact the effectiveness of the program?

2. How has the lack of on-going professional development for the Saxon Phonics program affected the fidelity of the program?

Upon discussing the findings, the evaluators will provide limitations of the evaluations and recommendations forfuture evaluations.Question 1: Classroom Implementation by Teacher

Observations provided evidence of the program components being used on a daily basis. Interviews further examined how teachers are using the components of the program as well as changes they would like to see how the program is implemented. Based on the data, teachers are generally using the same components of the program. However, there are some instances where some teachers are using a specific component and other teachers are not. Teachers are individually weighing the importance of the components used in their classrooms and it is not consistent across all classes. For example, two teachers showed no evidence of using the letter tiles, one teacher showed some evidence of use, and one teacher showed ample evidence of use as indicated by the observations. Upon further investigation during the interviews, it was apparent that the teachers who did not use the letter tiles did not see the value of that component in their classroom. The other two teachers did see the value of the component because one teacher stated she has used them but more in the beginning of the school year and less during the second semester. One teacher continues to use the letter tiles.

Recommendation related to classroom implementation. It is recommended that all teachers are included in the planning phase for which components will be utilized in the program. This should be addressed at the beginning of each school and reviewed at least every six weeks. During this time, teachers should also review student achievement data. By looking at the component utilization and comparing it with student achievement data, teachers can make adjustments to which components should be continued to be implemented in the program or if any changes should be made. Question 2: Professional Development

During the interviews, it was noted by all of the kindergarten teachers that they had only had one day of staff development training in regard to the implementation of Saxon Phonics and Spelling – K. This training day was at the beginning of the implementation of the program over three years ago. Teachers feel that they did not receive enough training to fully implement all the components of Saxon Phonics successfully. One teacher stated she would like training on how to effectively incorporate all components into the daily lesson. All four kindergarten teachers feel that they need additional training to implement the Saxon Phonics program with fidelity.

Recommendation related to professional development. It is recommended that division and school leadership provide additional training for all teachers using Saxon Phonics. As new teachers are hired or transferred, retraining should occur to ensure proper implementation of the program. Professional development opportunities should also be offered to expand upon teachers’ knowledge of letter and sound recognition, spelling, and concept of word.

ConclusionThe evaluand is the implementation of Saxon Phonics and Spelling – K at ABC Elementary School. The

program is designed to be implemented early in the school year. All students are screened using the pre-assessments to identify each child’s phonological and phonemic awareness. There are 140 lessons spread across 35 weeks. The instructional overview of the program states that students need to review all letters of the alphabet. Ensuring that students have time to practice these skills every day is crucial to the validity of the program. The Saxon Phonics program includes (a) the controlled vocabulary and practice, (b) the annotated bibliography, (c) coding, (d) spelling, (e) assessments, (f) pacing, (g) handwriting, and (h) reading fluency (Simmons, 2006).

The evaluators were able to obtain information about the implementation of Saxon Phonics through interviews, observations, program documents, and student achievement data. The interviews, observations, and program documents provided the information needed to answer the two questions posed by the research of the implementation of the program. The quantitative student achievement data provided information as to the result of the implementation of the program; however, did not offer any additional information as to the actual classroom implementation of the program components.

Limitations of this evaluation. The data collection process took place over a one week time period. This allowed the researchers to see a complete weekly unit for one letter that utilizes the various program components. This in itself was a limitation because instead of taking one class period to implement a lesson it takes a complete week. Time constraints were a limitation for the researchers. Classroom interruptions occurred during three of the observations which may have impacted how the teacher implemented the lesson. One day during the observations, a substitute teacher was employed in one of the classrooms which impacted how the lesson what taught. Another limitation was a student teacher taught the lesson during the week of observations. As evaluators, we know the substitute and student teachers would not have had any training as to how the Saxon Phonics program should be implemented. Individual teaching styles are also limitations as how the program components are implemented especially with the lack of training. Finally, the study was limited to only four classroom kindergarten teachers in the division.

Recommendations for future evaluations. Two recommendations were discussed by the evaluators for future considerations: (1) an additional evaluation, and (2) expanding the study to another division school. After ABC Elementary School has implemented the recommendations provided by the evaluators, a re-evaluation of the implementation of the components of the Saxon Phonics program should be completed. This could be done utilizing the observation checklists provided in Appendix B. It is also a recommendation that the study be expanded to include the only other elementary school in the division which serves kindergarten students. This will allow for comparison of the program implementation horizontally across the school division.

ReferencesBall, C., & Gettinger, M. (2009). Monitoring children's growth in early literacy skills: Effects

of feedback on performance and classroom environments. Education & Treatment of Children (West Virginia University Press), 32(2), 189-212.

Brunswick, N., Neil Martin, G., & Rippon, G. (2012). Early cognitive profiles of emergent readers: A longitudinal study. Journal of Experimental Child Psychology, 111(2), 268-285. doi: 10.1016/j.jecp.2011.08.001

Calais, G. J. (2008). Employing Siegler's overlapping waves theory to gauge learning in a balanced reading instruction framework. FOCUS on Colleges, Universities & Schools, 2(1), 1-10.

Cambourne, B., & Turnbill, J. (2007). Looking back to look forward: Understanding the present by revisiting the past: An Australian perspective. International Journal of Progressive Education, 3(2), 8-29.

Donat, D. (2006). Reading their way: A balanced approach that increases achievement. Reading & Writing Quarterly, 22(4), 305-323. doi: 10.1080/10573560500455745

Faust, M., & Kandelshine-Waldman, O. (2011). The effects of different approaches to reading instruction on letter detection tasks in normally achieving and low achieving readers. Reading and Writing: An Interdisciplinary Journal, 24(5), 545-566. doi: 10.1007/s11145-009-9219-1

Hammill, D. D., & Swanson, H. L. (2006). The national reading panel's meta-analysis of phonics instruction: Another point of view. The Elementary School Journal, 107(1), 17-26. doi: 10.1086/509524

Heide, S. (2005). Meaningful phonics instruction: Engaging students through poetry and word study folders. Illinois Reading Council Journal, 33(4), 32-36.

Hurry, J., Sylva, K., & Riley, J. (1999). Evaluation of a focused literacy teaching programme in reception and year 1 classes: Child outcomes. British Educational Research Journal, 25(5), 637.

Irwin, J., Moore, D., Tornatore, L., & Fowler, A. (2012). Expanding on early literacy: Promoting emerging language and literacy during storytime. Children and Libraries: The Journal of the Association for Library Service to Children, (2012, Summer/Fall), 20-23. Retrieved from http://www.haskins.yale.edu/Reprints/HL1695.pdf

Jeynes, W. H., & Littell, S. W. (2000). A meta-analysis of studies examining the effect of whole language instruction on the literacy. Elementary School Journal, 101(1), 21.

McGeown, S. P., Johnston, R. S., & Medford, E. (2012). Reading instruction affects the cognitive skills supporting early reading development. Learning & Individual Differences, 22(4), 360-364. doi: 10.1016/j.lindif.2012.01.012

Menzies, H., Mahdavi, J., & Lewis, J. (2008). Early intervention in reading: From research to practice. Remedial and Special Education, 29(2), 67-77. doi: 10.1177/0741932508315844

Mertens, D. M., & Wilson, A. T. (2012). Program evaluation theory and practice: A comprehensive guide.New York, NY: The Guilford Press.

Sanchez, M., Magnan, A., Ecalle, J. (2010). Knowledge about word structure in beginning readers: What specific links are there with word reading and spelling? European Journal of Psychology of Education, 27, 299-317. doi: 10.1007/s10212-011-00071-8

Saxon Publishers. (2005). Scientific research base for Saxon Phonics and Spelling K-3: Foundational research and program efficacy studies. Retrieved from http://steckvaughn.hmhco.com/HA/correlations/pdf/s/saxon_phonics_research.pdf

Simmons, L. (2006). Saxon phonics and spelling K: Instructional overview. Austin: Saxon Publishers, Inc.

Virginia Department of Education (2012). ESEA flexibility.  Retrieved from http://www.doe.virginia.gov/federal_programs/esea/flexibility/index.shtml

Vadasy, P. F., & Sanders, E. A. (2012). Two-year follow-up of a kindergarten phonics intervention for English learners and native English speakers: Contextualizing treatment impacts by classroom literacy instruction. Journal of Educational Psychology, 104(4), 987-1005. doi: 10.1037/a0028163

Yarbrough, D. B., Shulha, L. M., Hopson, D. B., & Caruthers, F. A. (2011). The programevaluation standards: A guide for evaluators and evaluation users. Los Angeles, CA: SAGE.