Report on the

America’s Legislators Back to School Program Evaluation Louisiana and Wyoming Phases

September 2008

Prepared by

Table of Contents

Abstract ................................................................................................................. iv Executive Summary ................................................................................................v

Method .........................................................................................................v Results....................................................................................................... vii

Introduction..............................................................................................................1

Purpose and Scope .......................................................................................1 Purpose.............................................................................................2 Scope................................................................................................2

Limitations ...................................................................................................3 Delimitations....................................................................................3 Other Limiting Conditions...............................................................3

Methods....................................................................................................................4 Evaluation Design........................................................................................4 Participants...................................................................................................6 Instruments...................................................................................................8

Representative Democracy Test ......................................................8 Classroom Monitor Observation Form ............................................9

Procedure .....................................................................................................9 Parental Consent ............................................................................10 Mail-Outs and Test Administration ...............................................10 Classroom Monitoring ...................................................................11

Results of the Data Analysis ..................................................................................11 Legislators’ Visits ......................................................................................11 Attention to Back To School Program Objectives.....................................16 Extent of Implementation ..........................................................................18 Student Gains .............................................................................................18 Intervention and Control Group Differences .............................................21 Test Score Gains and Characteristics of the Intervention..........................34

Conclusion .............................................................................................................34 References..............................................................................................................37

ii

List of Tables

Table A Implicit ALBTS Program Objectives Identified through Examination of Program Materials and Interaction with NCSL Staff ................................ vi

Table 1 Implicit ALBTS Program Objectives Identified through Examination of Program Materials and Interaction with NCSL Staff ..................................2

Table 2 Experimental Design....................................................................................5

Table 3 Teachers, Classes, and Students Participating in the America’s Legislators Back to School Program Evaluation ............................................................7

Table 4 Classroom Visits in Which Legislators Used Specific Lesson Types Recommended by NCSL ...........................................................................12

Table 5 Extent of Legislators’ or Teachers’ Use of Your Ideas Count Materials ..12

Table 6 Students’ Level of Attention During Legislators’ Visits .......................... 13 Table 7 Students’ Level of Participation During Legislators’ Visits......................14

Table 8 Content of Main Points Made by Legislators During Classroom Visits ...16

Table 9 Test Areas and Item Clusters With Significant Gains From Pretest to Posttest for Intervention Groups: Louisiana and Wyoming Phases...........20

Table 10 Paired Samples T-Test Results Showing Significance of Difference in Pretest and Post-test Scores for Intervention and Control Groups ............23

Table 11 Test Areas and Item Clusters in Which Intervention Groups’ Posttest Scores Exceeded Those of Control Groups: Louisiana and Wyoming Phases.........................................................................................................27

Table 12 Mean Louisiana and Wyoming Intervention and Control Group Pretest, Posttest, and Change Scores and ANCOVA Results Showing Significance of Differences in Posttest Knowledge and Perception Area Scores While Controlling for Differences in Pretest Scores ............................................29

List of Figures

Figure 1 Legislators’ Use of Classroom Visit Time (in Minutes)............................15 Figure 2 Students’ Understanding of Representative Democracy: Pretest and

Posttest Knowledge Scores, Intervention and Control Groups; Louisiana and Wyoming.............................................................................................19

Figure 3 Students’ Appreciation of Representative Democracy: Pretest and Posttest Perception Scores, Intervention and Control Groups; Louisiana and Wyoming....................................................................................................19

iii

Abstract

This study examined the impact of the America’s Legislators Back to School (ALBTS) program on participating students’ understanding and appreciation of representative democracy. Evaluators used a randomized pretest/posttest experimental design to examine the ALBTS programs in Louisiana (fall 2006) and Wyoming (spring 2008). Participating middle school social studies classes in each state were assigned to intervention and control groups. Legislators visited intervention classes for one class period and teachers used program materials in these classes as time permitted. No visits or ALBTS instruction on the part of teachers occurred in control group classes during the study period. Pretest and posttest administrations of a test on representative democracy were used to measure students’ knowledge and perceptions prior to and one to two weeks following completion of legislators’ visits. Classroom monitors accompanying legislators on classroom visits used a form to record observations on materials used, topics addressed, and other conditions surrounding legislative visits. Students in Wyoming’s intervention group—classes visited by legislators—demonstrated a higher level of overall understanding of representative democracy than did Wyoming students in control group classes, as measured by posttest average scores in the test’s knowledge area. In examining students’ achievement on individual item clusters making up the overall knowledge score, evaluators found that average posttest scores of Wyoming students in classes visited by legislators were significantly higher than those of control group students on six of the nine item clusters related to understanding representative democracy. Although the overall average score of Louisiana’s intervention group on the knowledge section did not significantly exceed that of the control group, Louisiana classes visited by legislators scored significantly higher than the control group on an item cluster measuring their understanding of one of the nine representative democracy item clusters on the knowledge section of the test. For the Wyoming intervention group, average posttest scores on three of the four appreciation item clusters, as well as on the overall appreciation section of the test, significantly exceeded the posttest appreciation scores of the control group. Average posttest scores of intervention group students in both states exceeded those of their respective control groups on one item cluster: appreciation of the right of every citizen to be heard by his or her representatives. Pretest to posttest gains on this “right to be heard” cluster also were significant for intervention groups in both states. No significant knowledge or perception gains from pretest to posttest were found for either state’s control group. This evaluation shows that the America’s Legislators Back to School program results in significantly higher levels of understanding and appreciation of representative democracy among middle school students. These higher levels of achievement and appreciation are possible when legislators and teachers use the recommended lesson types and materials, and address the intended program objectives.

iv

Executive Summary

The America’s Legislators Back to School Program was initiated in 1999 to “provide a contemporary approach for engaging the American student population in understanding the value of democracy” (National Conference of State Legislatures, 2004). Funded by the National Conference of State Legislatures’ Trust for Representative Democracy, the program annually sponsors state legislators’ visits to schools. The National Conference of State Legislatures (NCSL) provides educational materials on representative democracy for legislators’ use in their presentations to K-12 classes. NCSL also provides training, promotional strategies, and organizational support for state coordinators and legislators.

In 2005, NCSL contracted with Turcotte Public Administration Consulting and Training, LLC (TPACT) to conduct a controlled study of the program’s impact. NCSL selected the state of Louisiana as the site of the study’s first phase. NCSL program managers subsequently expanded the study to include an examination of the effectiveness of the America’s Legislators Back to School intervention in a second high-implementation state, Wyoming. The purpose of the ensuing study was to evaluate the impact of the America’s Legislators Back to School Program on participating students’ understanding and appreciation of representative democracy. Method

The evaluators collaborated with NCSL staff to identify the specific areas of knowledge and appreciation on which the study would focus. An examination of program materials and interaction with NCSL staff through meetings and correspondence yielded the nine program objectives for student understanding and four objectives for student appreciation listed in Table A.

The evaluators used a randomized pretest/posttest experimental design in both phases of the study. For each participating school, evaluators randomly assigned participating classes to the intervention or control group. Use of random assignment reduced the likelihood that systematic bias entered into the composition of the intervention and control groups.

Legislators visited intervention classes and did not visit control classes. An exception to this procedure occurred in cases in which legislators visited control classes after the completion of the study. Visits following administration of the posttest would not have affected the internal validity of the study. To further enhance the study’s internal validity, the evaluators asked teachers of control classes to refrain from exposing these classes to America’s Legislators Back to School Program materials before or during the study period (i.e., prior to the administration of the posttest).

v

Table A

Implicit ALBTS Program Objectives Identified Through Examination of Program Materials and Interaction With NCSL Staff Objectives on knowledge of representative democracy: Students will demonstrate their understanding—

A. of the lawmaking process B. that new laws begin when citizen groups want changes that will help improve

government or meet their needs C. that the procedures a state legislature uses to resolve differences about public

policy include debate, negotiation, and compromise D. of the difference between a state legislature and the U.S. Congress (i.e., that a

state legislature makes laws that affect individuals within a state, as opposed to the nation as a whole)

E. that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues

F. that representative democracy derives its authority from the people G. that representatives are elected from the citizenry H. of checks and balances in government I. of the need for compromise as a part of the legislative process

Objectives on appreciation of representative democracy: Students will appreciate— J. the right of every citizen to be heard by his or her representatives K. the need for citizens to be actively involved in state issues L. legislators’ interest in meeting the needs of the people M. the need for checks and balances in government

The objectives listed above in Table A formed the basis of a representative

democracy test developed by the evaluators and administered by participating teachers on a pretest and posttest basis. This test operationalized the representative democracy knowledge and appreciation constructs examined in the study. The evaluators also developed an observation form for use by classroom monitors accompanying legislators on classroom visits. The observation form was used to collect information on the context of each visit, the amount of time spent on various activities, the duration of the visit, the extent to which the legislator used certain standard program activities, levels of students’ attention and participation, legislators’ main points, and questions asked by students.

Twelve legislators visited the 15 Louisiana classes whose members fully

participated in the study’s intervention group. Full participation by a class included completion and return of pretests and posttests completed by students whose parents had formally agreed to allow their children to participate in the study. Eight Wyoming legislators visited the 10 fully participating intervention classes in that state. An additional 14 Louisiana classes and 10 Wyoming classes fully participated as members of the control group.

vi

Following the study period in each state, the evaluators assessed the extent to which the America’s Legislators Back to School Program design had been implemented by examining three conditions for each monitored visit—the legislator’s use of at least one of the lesson types recommended by NCSL, distribution or presentation of NCSL’s Your Ideas Count material in booklet or video form, and presentation of information on or discussion of Back to School program objectives (Table A). Evidence from reports submitted by monitors supports a conclusion of full implementation by Louisiana and Wyoming legislators participating in the study.

Results

Results of this two-phase study demonstrates that legislators’ Back to School Program classroom visits strengthened students’ understanding and appreciation of several elements of representative democracy. After legislators had visited Wyoming’s intervention group, students in that group demonstrated a higher level of overall understanding of representative democracy than did the Wyoming students in the control group. Average scores of students in classes visited by legislators also were significantly higher than those of control group students on six of nine concepts related to representative democracy. These included item clusters measuring the extent to which students understood

• that new laws begin when citizen groups want changes that will help

improve government or meet their needs; • that the procedures a state legislature uses to resolve differences about

public policy include debate, negotiation, and compromise; • that representative democracy is a system in which individuals vote to

elect representatives to an assembly that represents the people in deciding issues;

• that representatives are elected from the citizenry; • the concept of checks and balances in government; and • the need for compromise as a part of the legislative process.

Although the overall average score of Louisiana’s intervention group on the

knowledge section of the representative democracy test did not significantly exceed that of the control group, Louisiana students in classes visited by legislators scored significantly higher than Louisiana’s control groups on their understanding of the difference between a state legislature and the U.S. Congress.

Students in Louisiana and Wyoming classes visited by legislators also showed

higher levels of appreciation of representative democracy following their legislative visits. The overall appreciation of representative democracy by Wyoming students whose classes were visited exceeded the appreciation levels of the control group. The Louisiana intervention group’s posttest scores significantly exceeded those of the control group on their appreciation of the right of every citizen to be heard by his or her representatives.

vii

Students in Wyoming’s intervention group also scored significantly higher than those in the control group on specific measures of appreciation of representative democracy. These included the items and clusters measuring appreciation of

• the right of every citizen to be heard by his or her representatives (the item on which Louisiana’s intervention group posttest scores also exceeded those of the control group),

• legislators’ interest in meeting the needs of the people, and • the need for checks and balances in government.

In all comparisons of the Louisiana and Wyoming intervention groups’ posttest

scores with the posttest scores of their respective control groups, the evaluators used a procedure that controlled for differences in the pretest scores of the intervention and control groups. These controls were particularly important in examining posttest results in test areas and item clusters in which intervention group scores were significantly higher or lower than those of the control group at the outset of the study.

Additional support for a hypothesis that the program significantly improved

Louisiana and Wyoming students’ understanding of representative democracy were in the significantly higher overall pretest to posttest gains seen in intervention group students’ understanding of representative democracy, as well as in the absence of significant overall knowledge gains on the part of the control group. The average pretest to posttest gains of students in both intervention groups—Louisiana’s and Wyoming’s—were statistically significant on at least one of four item clusters measuring students’ appreciation of representative democracy. The item cluster on which both states’ intervention groups gained significantly was the cluster related to the program objective on the right of every citizen to be heard by his or her representatives. For control group students in Louisiana or Wyoming, no significant gains occurred in understanding of concepts specified in the program’s nine knowledge-related objectives, or on any of the program’s four perception-related objectives.

This evaluation of the America’s Legislators Back to School Program shows that

the program, when implemented by legislators and teachers who use the recommended lesson types and materials and address the intended program objectives, results in significantly higher levels of understanding and appreciation of representative democracy among middle school students. The program’s long-term impact on students’ knowledge and perceptions of representative democracy may warrant attention in future studies.

viii

Introduction

The America’s Legislators Back to School Program was initiated in 1999 to “provide a contemporary approach for engaging the American student population in understanding the value of democracy” (National Conference of State Legislatures, 2004). Funded by the National Conference of State Legislatures’ Trust for Representative Democracy, the program annually sponsors state legislators’ visits to schools. The National Conference of State Legislatures (NCSL) provides educational materials on representative democracy for legislators’ use in their presentations to K-12 classes. NCSL also provides training, promotional strategies, and organizational support for state coordinators and legislators.

NCSL’s qualitative program evaluations have yielded evidence of enthusiastic support from legislators, teachers, and students (National Conference of State Legislatures, 2005). However, prior to the study described in this report no controlled studies had been conducted to examine the program’s success in increasing students’ understanding and appreciation of representative democracy. In 2005, NCSL contracted with Turcotte Public Administration Consulting and Training, LLC (TPACT) to conduct a controlled study of the program’s impact. NCSL selected the state of Louisiana as the site of the study’s first phase, originally scheduled for fall 2005. This site was selected because NCSL considered Louisiana’s America’s Legislators Back to School Program to be among the nation’s most fully implemented. NCSL postponed the study shortly after Hurricane Katrina devastated parts of Louisiana, severely affecting several participating school districts. In the fall of 2006, TPACT evaluators conducted the Louisiana study described in this report.

NCSL program managers subsequently expanded the study to include an

examination of the effectiveness of the Legislators’ Back to School intervention in a second involved or active state, Wyoming. This report describes the design, implementation, and results of the study’s Louisiana and Wyoming phases. Purpose and Scope of the Evaluation

The purpose of this experimental study was to evaluate the impact of the America’s Legislators Back to School Program on participating students’ understanding and appreciation of representative democracy. Participants in the Louisiana phase of the study (fall 2006) included 12 legislators who visited the 15 intervention classes that fully participated in the study. Intervention classes are those visited by legislators following administration of the pretest and prior to posttesting. Control classes were not visited at all, or were visited by legislators after students had taken the posttest. Classes were considered full participants if they provided pretest and posttest data for students whose parents had consented to the inclusion of their children’s test results in the data analysis. Louisiana’s 29 fully participating intervention and control classes were under the instruction of 24 teachers. In the Wyoming phase (Spring 2008), eight legislators visited ten fully participating intervention classes. Wyoming’s intervention classes and control classes were taught by five middle school and junior high school teachers.

1

Purpose. The study was designed to examine the impact of the America’s Legislators Back to School Program on students’ understanding and appreciation of representative democracy. The evaluators collaborated with NCSL staff to identify the specific areas of knowledge and appreciation on which the study would focus. An examination of program materials and interaction with NCSL staff through meetings and correspondence yielded the nine program objectives for student understanding and four objectives for student appreciation listed in Table 1.

Table 1

Implicit ALBTS Program Objectives Identified Through Examination of Program Materials and Interaction With NCSL Staff Objectives on knowledge of representative democracy: Students will demonstrate their understanding—

A. of the law making process B. that new laws begin when citizen groups want changes that will help improve

government or meet their needs C. that the procedures a state legislature uses to resolve differences about public

policy include debate, negotiation, and compromise D. of the difference between a state legislature and the U.S. Congress (i.e., that a

state legislature makes laws that affect individuals within a state, as opposed to the nation as a whole)

E. that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues

F. that representative democracy derives its authority from the people G. that representatives are elected from the citizenry H. of checks and balances in government I. of the need for compromise as a part of the legislative process

Objectives on appreciation of representative democracy: Students will appreciate— J. the right of every citizen to be heard by his or her representatives K. the need for citizens to be actively involved in state issues L. legislators’ interest in meeting the needs of the people M. the need for checks and balances in government

Scope. The geographic scope of the study was limited to one state in the study’s first phase, which focused on the effectiveness of the program’s fall 2006 implementation in Louisiana. NCSL later expanded the geographic scope to include Wyoming’s spring 2008 implementation of the America’s Legislators Back to School Program.

2

Limitations This section describes limitations imposed by the evaluators, which are considered delimitations. Also described in this section are several limiting conditions that were not fully under the control of the evaluators.

Delimitations. The object of the evaluation (the entity studied) was the America’s Legislators Back to School Program in its fullest form, which included delivery of recommended visit content and use of recommended classroom materials. The proposed study was designed to minimize involvement by participants who were unlikely to follow the program’s protocol, a source of experimenter bias that is known as lack of treatment fidelity (Gall, Gall, & Borg, 2003). To minimize this source of bias, the evaluators limited the study to states identified by NCSL program managers as having highly effective statewide program coordination. America’s Legislators’ Back to School Program coordinators in these states closely adhere to NCSL guidelines for the program. In addition, the evaluators limited to the extent possible the pool of participating legislators to those who previously had participated in the program at or near full implementation level by adhering to recommended content guidelines and material use. The level of legislators’ previous involvement was determined by state program coordinators. These coordinators also assisted in promoting treatment fidelity by encouraging use of NCSL instructional material and approaches by legislators and teachers.

Another design delimitation was restriction of posttest data collection to a period

no longer than two weeks following a legislator’s visit to an intervention class. Measuring the effectiveness of an intervention soon after its completion is essential as a first step in examining a program’s effectiveness. Gains or losses in knowledge or perception may occur at various intervals following an intervention, but no assumptions can be made concerning the direction or strength of subsequent changes based on initial student outcomes. Other Limiting Conditions. Other limitations resulted from circumstances that evaluators could not fully control. Although the evaluation design included the treatment fidelity provisions mentioned above, legislators and teachers were free to use their own discretion in selecting instructional approaches and content. Members of the evaluation team monitored program implementation to determine the extent to which participating legislators and teachers used standard program materials and addressed designated program objectives. Evaluators used information from monitors’ reports to examine relationships between level of implementation and program outcomes. In addition, the quality of pretest and posttest data might have varied based on the extent to which the testing instrument was administered as intended. For example, the evaluators asked teachers to provide sufficient time for students to read and respond to all items; to prohibit students from discussing the items or their responses during the testing period; and to minimize distractions throughout the testing period. The evaluators asked participating intervention and control group teachers to adhere to these pretest and

3

posttest administration protocols. On-site verification of teachers’ adherence to these protocols was not possible, but the evaluators received no comments that suggested that abnormalities had occurred in test administration. The rate of class and student participation in the evaluation is another area over which the evaluators had relatively little control. Participation depended to a great extent on rates at which students returned signed parental consent forms. The evaluators elicited teachers’ and students’ cooperation in seeking parental signatures on consent forms by offering incentives for class participation. Another factor affecting rates at which students’ parental consent forms were available to evaluators—and, as a result, rates at which students’ test results were available for inclusion in the analysis—was teachers’ adherence to protocols for collecting these forms and returning them to the evaluators. Illnesses among teachers and their family members also imposed limitations on the availability of classes for full participation in the study. Finally, the evaluators were limited in the extent to which they could develop a test that would address validity criteria while remaining within an administration time frame that was acceptable to participating teachers. Particularly challenging was development of a brief pretest and posttest with enough items to fully measure accomplishment of each program objective. This limitation is discussed in the Instruments section below, as well as in the Results section.

Methods This section describes the design of the study and provides information on participants, including legislators, classes, and students. The two instruments used in the study are described and procedures are listed. Evaluation Design The evaluators used a randomized pretest/posttest experimental design in both phases of the study. Table 2 shows the design of each phase. For each participating school, evaluators randomly assigned participating classes to the intervention or control group. Random assignment of classes to treatment condition (intervention or control) enhanced the study’s validity. That is, use of random assignment reduced the likelihood that systematic bias entered into the composition of the intervention and control groups. Intervention and control groups formed in this way are likely to have included, for example, comparable proportions of students who were enthusiastic about learning civics, as well as comparable proportions of students who lacked enthusiasm for civics education. Similarly, as a result of random assignment, characteristics of teachers of both groups (intervention and control classes) are likely to have been distributed in equivalent proportions within both groups. For example, both groups of teachers (those who taught intervention group and control group classes) are likely to have ranged from those who were highly motivated to promote in their students an appreciation for representative democracy, to those who were moderately motivated or not motivated to do so. Without random assignment, it is possible that instructors of intervention group classes would

4

have been those who had sought out an opportunity to expand their students’ understanding of representative democracy by participating in the Legislators’ Back to School program. Such teachers are likely to have sought out similar opportunities for

Table 2

Experimental Design Louisiana Phase (Fall 2006)

Group Assignment Classes Students Pretest Treatment Posttest

Intervention Random 15 214 RDTa Visits by 12 legislatorsb

RDT

Control Random 14 166 RDT No visitc RDT

Wyoming Phase (Spring 2008)

Group Assignment Classes Students Pretest Treatment Posttest

Intervention Random 10 71 RDTa Visits by 8 legislatorsb

RDT

Control Random 10 91 RDT No visitc RDT

a Representative Democracy Test, a measure of students’ understanding and

appreciation of representative democracy b One visit per intervention class; evaluation team monitors observed visits on-site c Legislators visited some control classes after data collection ended

their students, thereby introducing a confounding variable (availability of related learning opportunities to intervention students, but not to control students). If assignment had been based on intervention group teachers’ requests to participate in the program, students in the control group might have been those whose teachers were somewhat less motivated to provide opportunities to convey to their students the importance of representative democracy.

Legislators visited intervention classes and did not visit control classes. An

exception to this occurred in cases in which legislators visited control classes after the completion of the study. Visits following administration of the posttest would not have affected the internal validity of the study. To further enhance the study’s internal validity, the evaluators asked teachers of control classes to refrain from exposing these classes to

5

America’s Legislators Back to School Program materials before or during the study period (i.e., prior to the administration of the posttest).

Participation in the America’s Legislators Back to School program was the

independent variable in this study (i.e., the variable expected to cause a difference in participants’ outcomes). The two levels of this variable were intervention (students’ participation in the program) and control (students’ temporary exclusion from participation). The intervention condition included a legislative visit and participation by the teacher of an intervention class. Classroom visits were made by one or more legislators who volunteered to visit a class participating in the study. These legislators, as well as others participating in America’s Legislators Back to School Program, had access to Web sites describing the program. Material specific to their state programs and instructional material on representative democracy also were available on the Web. Teachers participating in the program had access to Web-based instructional material on representative democracy designed specifically for their use, as well as material for distribution to students.

Students in intervention and control classes took a pretest prior to the period

during which legislators visited classes and a posttest during the two-week period following legislators’ visits. Teachers of intervention group classes were asked to refrain from using Legislators Back to School Week material or providing any other instruction related to the program until their intervention group students had taken the pretest. Teachers of control group classes were asked to refrain from doing so until their students had completed the pretest and posttest. Scores on these tests, which measured students’ understanding and appreciation of representative democracy, served as the dependent variable (i.e., the outcome variable).

The evaluators used a variety of methods to promote and document participants’

adherence to program specifications and research protocols. These procedures ensured that the object of the evaluation was the Legislators Back to School program in its intended form and that tests were administered validly. Evaluation monitors observed program implementation at participating schools and completed forms describing the content of legislators’ sessions, including materials distributed, activities conducted, and legislators’ main points. Monitors also reported on conditions surrounding the visits, such as length of visits, individuals present, materials distributed by teachers, and levels of student participation. The evaluators provided detailed written instructions on test administration to promote uniformity in testing conditions.

Participants Legislative staff members assisted the evaluators in identifying teachers who were willing to make social studies classes available for random assignment to the intervention or control group. Louisiana’s Legislators Back to School Program typically occurs over a two- or three-week period in the fall. The state coordinator of the Wyoming program arranges visits to schools at various times throughout the academic year. Twenty-four Louisiana and 21 Wyoming teachers originally agreed to participate in the evaluation and provided scheduling information on their classes. Incentives used in promoting teacher

6

participation included funding provided by TPACT for purchase of classroom materials. In the Louisiana phase, the evaluators used class and teacher information to create a list of all available classes, then selected classes at random for assignment to the intervention and control groups. This process ended when enough classes had been assigned to the intervention group to accommodate participating legislators. Legislative staff members helped the evaluators coordinate match-ups of Louisiana legislators with intervention classes available at times that were feasible for legislators’ visits. When legislators’ schedules precluded visits to some classes that had been placed in the intervention group, evaluators selected classes at random from a pool of unassigned classes they had held in reserve for substitution. This process resulted in assignment of 18 Louisiana classes to the intervention group and 16 classes to the control group. In Wyoming, geographic distances and limitations in the pool of potential evaluation monitors were factors in selecting school districts for participation. The Wyoming legislative staff member who coordinates the state’s ALBTS program identified four school districts and nine teachers offering two or more classes for inclusion in the study. In Wyoming, curriculum resources from the Center for Civic Education and student bracelets with the message “Make Your Voice Heard” were offered as incentives for class participation. A senior evaluator used random assignment to place each available class in the intervention or control group. For each participating teacher, at least one class was assigned to each group. This process resulted in placement of 17 Wyoming classes in the intervention group and 17 classes in the control group. Several classes in each state subsequently became unavailable for participation due to local circumstances, such as teachers’ illnesses or administrators’ reluctance. Table 3 shows the number of classes and students participating in various stages of data collection and the number taking part in the program at the full participation level. Table 3

Teachers, Classes, and Students Participating in the America’s Legislators Back to School Program Evaluation

Total Intervention Group

Control Group

Louisiana

Classes Classes for which pretests were

returned 32 17 15

Classes for which posttests were returned

32 16 16

Classes for which consent forms were returned

32 15 17

Classes with pre- and posttests and consent forms

29 15 14

7

Total Intervention

Group Control Group

Students Total pretests returned 633 357 276 Total posttests returned 635 344 291 Total consent forms returned 557 277 280 Students with usable pre- and

posttests and parental consent 380

214

166

Wyoming

Classes Classes for which pretests were

returned 28 14 14

Classes for which posttests were returned

25 12 13

Classes for which consent forms were returned

25 12 13

Classes with pre- and posttests and consent forms

20 10 10

Students Total pretests returned 434 206 228 Total posttests returned 240 114 126 Total consent forms returned 239 117 122 Students with usable pre- and

posttests and parental consent 162 71 91

Although many other legislative visits occurred during the Louisiana phase, 12 Louisiana legislators visited one or more of the 15 fully participating intervention classes. In Wyoming, eight legislators visited the ten fully participating intervention classes, as well as two additional classes for which the full set of pretests, posttests and parental consent forms were not available. Some Wyoming legislators chose to form teams of as many as four legislators who worked together in planning and conducting classroom visits. In some cases, Wyoming legislators visited groups consisting of two intervention classes participating in the same visit. Instruments

The evaluators developed a Representative Democracy Test for use in measuring middle school students’ knowledge of and perceptions about representative democracy. They also developed an observation form for use by classroom monitors accompanying legislators on classroom visits.

Representative Democracy Test. The objectives listed previously in the Purpose

and Scope section formed the basis of the Representative Democracy Test developed by the evaluators and administered by participating teachers on a pretest and posttest basis. This test operationalized the representative democracy knowledge and appreciation

8

constructs examined in the study. Scores on the posttest served as the study’s dependent variable.

To construct the test the evaluators identified several relevant items from NCSL’s

Back to School materials and developed additional test items to be used in measuring students’ attainment of the 13 Back to School objectives. To ensure construct validity, two middle school social studies teachers rated the strength of each item’s relationship with the corresponding objective. Teachers also rated the clarity and difficulty of each item for middle school students. They administered the test items on a pilot basis to a total of 150 middle school students in 10 classes. The participating teachers noted that the typical student completed the test in 5½ to 9 minutes. Louisiana teachers who had been consulted about participating in the first phase of the study warned that a test that would use more than a few minutes of class time would suppress teacher interest in participation in the evaluation. This pressure for brevity was balanced against test validity concerns calling for substantial numbers of test items per objective. Based on teachers’ comments in the pilot administrations and students’ responses, the evaluators refined test items to improve their clarity and the strength of their relationship with the objectives whose accomplishment they were designed to measure. The final version of the test included one or more items per objective for a total of 20 items. The evaluation team and an NCSL staff member reviewed the test prior to the first administration in the fall of 2006.

Classroom Monitor Observation Form. To provide the information needed in

examining factors associated with the program’s impact, the evaluators developed a Monitor Observation Form with sections requesting information in the following areas: context of the visit (name of the school, the visiting legislator, and the teacher; date of the legislator’s visit; number of students attending class at time of the legislator’s visit; others present during the visit; and grade level of the majority of students in the class), the amount of time spent on various activity types (lecture, eliciting and answering students’ questions, etc.), the duration of the visit, the extent to which the legislator implemented certain standard program activities, levels of students’ attention and participation, legislators’ main points, and questions asked by students.

The senior evaluators trained Louisiana monitors on use of the observation form in a telephone conference just prior to the visiting period. In the Wyoming phase a senior evaluator discussed use of the monitoring form with the legislative staff member who coordinated the evaluation for that state. This staff member subsequently used the form in monitoring legislative visits to five participating classes. She then trained a school district staff member, who monitored the balance of Wyoming’s legislative visits. Monitors returned completed forms within 14 days of their classroom observations.

Procedure

TPACT evaluation site coordinators, overseen by TPACT senior evaluators, managed program implementation and data collection in each of the study’s two phases. Implementation of the study in each phase began with recruitment of middle school teachers and classes by legislative coordinators of the Back to School program. The Louisiana evaluation site coordinator created a Web site with information on the

9

evaluation for participating teachers, as well as for monitors who had been recruited by the evaluation site coordinator to record information on classroom visits. The Wyoming evaluation site coordinator, who also was the legislative staff member responsible for coordinating the Wyoming program, included information on the ALBTS evaluation on the Wyoming Legislators Back to School Program Web site. The evaluation site coordinator in each state identified all classes that were available for participation in the study. As noted above in the Evaluation Design and Participants sections, a senior evaluator then used random assignment to place classes in the intervention and control groups, substituting additional randomly selected classes as needed to replace intervention group classes taught at times that were not feasible for visiting legislators. This process yielded 18 intervention and 16 control classes in Louisiana and 17 intervention and 17 control classes in Wyoming. Parental Consent. In Louisiana the evaluation site coordinator contacted participating teachers by e-mail and telephone to provide instructions on retrieval of the parental consent letter from the project Web site, copying of the letter, and distribution to students for use by parents in designating approval or withholding approval for students’ participation. The letter informed parents that participation was voluntary and that test results of students who did not wish to participate or whose parents withheld consent would not be used in the study. The letter also included a section to be completed and signed by the parent and the student. In Wyoming the site coordinator copied and mailed parental consent forms to participating teachers.

Mail-Outs and Test Administration. In Louisiana the evaluation site coordinator mailed consent form return envelopes to intervention and control group teachers and labeled and mailed packets containing tests and detailed instructions on test administration. Wyoming’s site coordinator also mailed test materials, detailed instructions, and clearly labeled return envelopes for teachers’ use in returning pretests and posttests.

Teachers of all participating classes (intervention and control) administered the

Representative Democracy Test approximately one week before the legislative visit. In Louisiana teachers administered the pretest approximately one week before the state’s Back to School week; in Wyoming, where legislative visits that were included in this study occurred throughout a three-month period, teachers in each participating school administered the pretest to intervention and control classes approximately one week before the date on which a legislator was scheduled to visit the school’s intervention classes. Legislators conducted their Back to School visits, which were monitored by trained evaluation team members, and teachers subsequently administered the Representative Democracy Test on a posttest basis. Teachers used pre-addressed envelopes provided by the site coordinator to mail consent forms and completed pretests and posttests to the evaluators.

Classroom Monitoring. After participating in a monitor training conference call

or an individual training session, monitors accessed the project Web site to download forms on which they were to record information on classroom visits or received

10

monitoring forms from the site coordinator. Monitors helped coordinate legislative visits as needed and recorded notes on their observations during classroom visits. Soon after each visit, monitors completed their forms electronically and returned the forms to a senior evaluator as e-mail attachments.

Results of the Data Analysis

This section begins with descriptive information on length and content of legislators’ visits. The section also presents a summary of students’ test gains and results of statistical tests examining the significance of differences in intervention and control group outcomes. Legislators’ Visits Monitors returned forms reporting on 16 Louisiana visits and 12 Wyoming visits. Of the 16 Louisiana monitors’ forms, three reported on visits that were not included in the study because teachers of these classes did not return one or more of the three forms or data types (parental consent forms, pretests, or posttests) needed for inclusion of students in the study. As a result, 13 of the Louisiana monitoring reports were usable and three were not usable because a component of the student data set was not available for the class. The usable monitoring reports provided information on legislators’ visits to 13 of the 15 Louisiana intervention classes for which all three data types were available (parental consent forms, pretests, and posttests). Similarly, two of the Wyoming monitors’ forms reported on visits that were not included in the study because a component of the student data set was missing for those classes. The remaining 10 Wyoming monitors’ reports (one for each of the participating intervention classes) were included in the analysis.

Monitors’ reports show that materials and approaches used by participating legislators varied in the extent to which they were aligned with NCSL’s program design and materials. Many legislators used lessons recommended by NCSL to convey relevant information about representative democracy. As shown in Table 4, the recommended lesson type that legislators used most often in Louisiana was the simulated committee hearing, which was used in nine of the 13 monitored Louisiana visits. Of the 10 monitored Wyoming classes, six participated in simulated committee hearings or floor debates.

The Your Ideas Count booklet and video are key components of the set of instructional material on representative democracy developed and made available through NCSL’s Trust for Representative Democracy. Monitors reported that legislators or teachers provided one or both forms of the Your Ideas Count material to nine of the 13 monitored intervention classes in Louisiana and five of the 10 intervention classes in Wyoming (Table 5). The evaluators asked monitors to rate the level of students’ attention during each legislative visit. Students’ attention levels typically were rated in the attentive to very attentive range in Louisiana, and in the attentive range for Wyoming students (Table 6).

11

Table 4 Classroom Visits in Which Legislators Used Specific Lesson Types Recommended by NCSL1

Recommended Lesson Types Louisiana (N = 13) Wyoming (N = 10)

Not at All Some A lot

Not at All Some A lot

Legislative simulation—conducted a simulated committee hearing or floor debate

4 1 8 4 1 5

Solved a class problem—class debated and negotiated an agreement to resolve a problem

7 3 3 10 0 0

A day in the life of a legislator—shared a page from their calendar

10 1 2 6 2 2

The perfect chocolate chip cookie (or pizza)—class debated and arrived at a compromise

8 2 3 10 0 0

1 Frequency of use of the Your Ideas Count booklet or video is shown in Table 5. Table 5

Extent of Legislators’ or Teachers’ Use of Your Ideas Count Materials

Louisiana (N = 13)

Wyoming (N = 10)

Use of Your Ideas Count Materials Yes No Yes No

Teacher or legislator showed the Your Ideas Count video to the class or distributed the Your Ideas Count booklet

9 4 5 5

12

Table 6

Students’ Level of Attention1 During Legislators’ Visits

Louisiana Wyoming

Student Attention

3

Very Atten-tive

2

Atten-tive

1

Some-what In-

atten-tive

0

Very Inat-ten-tive

3

Very Atten-tive

2

Atten-tive

1

Some-what Inat-ten-tive

0

Very Inat-ten-tive

Monitors’ ratings of students’ level of attention during legislators’ visits

5 7 1 0 0 7 3 0

1 Definitions of ratings of students’ level of attention during the legislator’s visit

Very attentive Virtually all students appeared to be paying attention throughout the visit.

Attentive A large majority of students appeared to be paying attention throughout the visit.

Somewhat inattentive Some students were inattentive (e.g., talking, reading, playing) at various times during the visit.

Very inattentive Many students were inattentive throughout the visit.

The evaluators also asked monitors to rate students’ level of participation during legislators’ visits. Table 7 shows that monitors assigned the highest rating, “high level of participation,” for a plurality of class visits in both states. Louisiana legislators used the lecture mode of instruction in conducting more than 1/3 of the typical visit (36 percent) and used questioning and discussion methods for an additional 35 percent of the typical visit (Figure 1). Wyoming legislators collectively devoted a comparable proportion of time (37 percent) to lecture and spent 41 percent of the typical visit asking questions and leading discussions.

13

Table 7

Students’ Level of Participation1 During Legislators’ Visits

Louisiana

Wyoming

Students’ Participation

2

High level of partici-pation

1

Mod-erate level

0

Low level

2

High level of parti-

cipation

1

Mod-erate level

0

Low level

Monitors’ rating of students’ participation during legislators’ visits

6 5 2 4 3 3

1 Definitions of ratings of students’ participation during legislators’ visits

High level Many students (50% or more) participated in discussions or asked questions.

Moderate level Some students (25%-49%) participated in discussions or asked questions.

Low level Few or no students (fewer than 25%) participated in discussions or asked questions.

14

Figure 1

Legislators’ Use of Classroom Visit Time (in Minutes)

Louisiana

Eliciting and answering

questions, 10, 19%

Asking questions and

leading discussion,

18, 35%

Other, 5, 10% Lecture, 19, 36%

Wyoming

Lecture, 18, 37%

Eliciting and answering

questions, 11, 22%

Asking questions and

leading discussion,

20, 41%

Other, 0, 0%

Note: Legislators' average use of time is shown in minutes (between commas) by mode of instruction and time use by mode as a percent of total visit duration (Louisiana’s Average Duration of Visit = 52 minutes, N = 13; Wyoming’s Average Duration of Visit = 49 minutes, N = 10)

15

Attention to Back to School Program Objectives The evaluators analyzed monitors’ notes on legislators’ main points to examine the alignment between the content of legislators’ presentations and Legislators Back to School Program objectives. The evaluators coded entries in monitors’ lists of main points, compiled the results across all monitors and tabulated frequencies across visits. Table 8 shows the results of this analysis. A classroom visit in which a legislator discussed a concept directly aligned with a program objective was recorded as one classroom visit in which a legislator addressed that objective (the “Number” column entry). The maximum frequency of occurrences in each row of Table 8 is 13 for Louisiana and 10 for Wyoming (one per monitored classroom visit). As shown in Table 8, Louisiana legislators described the lawmaking process in at least nine (69 percent) of their classroom visits and discussed the importance of citizen participation in state issues in at least six (46 percent) of their visits. Similarly, Wyoming legislators discussed the lawmaking process (100 percent of visits) and noted the need for citizens to be actively involved in state issues (60 percent of visits). Other topics discussed in more than half of Wyoming legislators’ visits included the right of every citizen to be heard by his or her representatives (60 percent) and legislators’ interest in meeting the needs of the people (60 percent). Note that the data in Table 8 includes monitors’ perceptions of legislators’ “main points.” The absence of a monitor’s entry listing a concept as a “main point” does not suggest that the topic was not discussed during the visit. Table 8

Content of Main Points1 Made by Legislators During Classroom Visits (Louisiana N = 13 Classroom Visits; Wyoming N = 10 Classroom Visits; Total N=23)

Classroom Visits in Which Legislator Addressed Objective

Louisiana Wyoming Total Knowledge Area Number Pct. Number Pct. Number Pct.

Students will demonstrate their understanding—

A. of the lawmaking process 9 69.2 10 100.0 19 82.6 B. that new laws begin when

citizen groups want changes that will help improve government or meet their needs 1 7.7 3 30.0 4 17.4

C. that the procedures a state legislature uses to resolve differences about public policy include debate, negotiation, and compromise 3 23.1 5 50.0 8 34.8

16

Classroom Visits in Which Legislator Addressed Objective

Louisiana Wyoming Total Knowledge Area Number Pct. Number Pct. Number Pct.

D. of the difference between a state legislature and the U.S. Congress (i.e., that a state legislature makes laws that affect individuals within a state, as opposed to the nation as a whole) 0 0.0 5 50.0 5 21.7

E. that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues 1 7.7 1 10.0 2 8.7

F. that representative democracy derives its authority from the people 3 23.1 0 0.0 3 13.0

G. that representatives are elected from the citizenry 2 15.4 2 20.0 4 17.4

H. of checks and balances in government 4 30.8 0 0.0 4 17.4

I. the need for compromise as a part of the legislative process

2 15.4 4 40.0 6 26.1 Perception Area Students will appreciate— J. the right of every citizen to

be heard by his or her representatives 3 23.1 6 60.0 9 39.1

K. the need for citizens to be actively involved in state issues 6 46.2 6 60.0 12 52.2

L. legislators’ interest in meeting the needs of the people 4 30.8 6 60.0 10 43.5

M. the need for checks and balances in government 2 15.4 1 10.0 3 13.0

1 As reported in the “Legislator’s Main Points” section of the Classroom Monitors’ Report. A senior evaluator coded main points by ALBTS objective.

17

Extent of Implementation

The evaluators assessed the extent to which the America’s Legislators Back to School Program design had been implemented by examining three conditions for each monitored visit—legislators’ implementation of at least one of the recommended lesson types (described in Table 4), distribution or presentation of the Your Ideas Count material in booklet or video form (Table 5), and presentation of information on or discussion of Back to School program objectives (Table 8). In 10 of the 13 monitored Louisiana visits, legislators implemented at least one of the recommended lessons or the class received the Your Ideas Count material in booklet or video form, or both of these implementation conditions were met. All three of the remaining monitored Louisiana visits included information on or discussion of at least three of the Back to School program objectives. Wyoming legislators’ implementation levels were comparable to those of participating Louisiana legislators. Presentation of at least one recommended lesson type or distribution of the Your Ideas Count material occurred in nine of the 10 Wyoming classes. Both conditions were met in four of these nine classes. Legislators visiting the remaining class addressed five of the program’s objectives. This evidence supports a conclusion of full implementation by Louisiana and Wyoming legislators participating in the study.

Student Gains

The evaluators examined pretest and posttest score differences for the intervention and control groups to determine whether either group showed evidence that their understanding or appreciation of representative democracy had changed from the period just prior to the intervention to the period immediately following the visit. (See Figure 2.) The Louisiana intervention group’s overall scores on the section of the test measuring knowledge of representative democracy increased significantly (gain = 8.0 percentage points; t (213) = 3.55, p < .001) from pretest to posttest, as shown in Table 9. The evaluators found no significant difference between the Louisiana control group’s pre- and posttest overall knowledge scores (gain = 1.8 percentage points; t (165) = -.80, p = .426). Neither of the Louisiana groups’ overall scores on the representative democracy appreciation section of the test changed significantly (intervention: gain = -.03; t (213) = .50, p = .616; control: gain = -.01; t (164) = .20, p = .839). (See Figure 3.) Thus, the average overall knowledge score of students in classes visited by Louisiana legislators increased significantly while average control group overall knowledge scores remained essentially the same from pretest to posttest.

Results for Wyoming classes were somewhat similar. Like the Louisiana

intervention group’s average pretest score, the Wyoming intervention group’s average overall knowledge score increased significantly from pretest to posttest (Wyoming gain = 6.5 percentage points; t (70) = -2.82, p = .01). The Wyoming control group’s average score declined slightly (gain = -1.4 percentage points; t (90) = .63 , p = .53.), but not significantly. The Wyoming intervention group’s pretest to posttest change in overall appreciation of representative democracy was positive, but did not reach statistical significance (gain = .12 score points on a scale ranging from 0 to 3; t (70) = -1.28, p =

18

.21). However, the control group’s average appreciation score declined significantly (gain = -.12 score points; t (90) = 2.25, p = .03).

47.7%

55.7%50.7% 52.5%

62.2% 68.7%

50.6% 49.2%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Louisiana--Intervention

Louisiana--Control Wyoming--Intervention

Wyoming--Control

Figure 2Students' Understanding of Representative Democracy: Pretest and Posttest Knowledge Scores, Intervention and

Control Groups; Louisiana and Wyoming

Louisiana--Intervention

Louisiana--Control

Wyoming--Intervention

Wyoming--Control

2.80

2.77

2.44

2.562.74

2.622.77

2.76

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Figure 3Students' Appreciation of Representative Democracy:

Pretest and Posttest Perception Scores, Intervention and Control Groups;

Louisiana and Wyoming

19

In addition to examining changes from pretest to posttest in the two groups’ overall knowledge and appreciation of representative democracy, the evaluators examined the significance of pre- to posttest score changes on items in each of the 13 program objective clusters. The evaluators found no significant control group changes from pretest to posttest for any item cluster in the Louisiana phase, but significant gains emerged on item clusters related to six objectives for the Louisiana intervention group. Table 9 summarizes results for each phase.

Table 9

Test Areas and Item Clusters With Significant Gains From Pretest to Posttest for Intervention Groups: Louisiana and Wyoming Phases1

Test Area Cluster Statistically Significant Gain by Intervention Group in- Louisiana Knowledge Entire

area Overall understanding of representative democracy

Knowledge A Understanding of the law making process Knowledge C Understanding that the procedures a state legislature uses

to resolve differences about public policy include debate, negotiation, and compromise

Knowledge D Understanding of the difference between a state legislature and the U.S. Congress (i.e., that a state legislature makes laws that affect individuals within a state, as opposed to the nation as a whole)

Knowledge E Understanding that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues

Knowledge F Understanding that representative democracy derives its authority from the people

Perception J Appreciation of the right of every citizen to be heard by his or her representatives

Wyoming Knowledge Entire

area Overall understanding of representative democracy

Perception J Appreciation of the right of every citizen to be heard by his or her representatives

1 Neither state’s control group gained significantly on either overall test area (knowledge or perception) or on any item cluster. Although the Wyoming intervention group’s overall knowledge gains were statistically significant, as noted earlier in this section, none of the intervention group average score gains on the individual knowledge item clusters making up the overall

20

score were found to be statistically significant (Table 10). These average score changes ranged from no change on the difference between a state legislature and the U.S. Congress to a gain of 11.3 percentage points on understanding the lawmaking process, a substantial gain that approached statistical significance (p = .09). The Wyoming control group’s changes from pretest to posttest also varied widely among the nine knowledge objectives, ranging from a statistically significant control group decline of 17.5 percentage points (p = .01) on the group’s understanding of the role of citizens’ groups in helping to initiate change (Objective B) to gains of 9.9 percentage points on each of two objectives, neither of which was a statistically significant gain. As noted previously, score gains from pretest to posttest for Wyoming’s intervention group were not significant overall in the area of perception of representative democracy. Nevertheless, the average intervention group score on one perception objective, the right of every citizen to be heard by his or her representatives (Objective J), increased significantly (gain = .56 points on a scale ranging from 0 to 3, t(70) = -3.17; p <.001). The intervention group’s average score gain approached but fell short of significance on the perception objective “appreciates legislators’ interest in meeting the needs of the people” (gain = .24 points t(70) = -1.85, p = .07). No Wyoming control group gains approached or reached significance on any of the perception objectives. In summary, intervention group participants in both states gained significantly from pretest to posttest in their overall understanding of representative democracy while the control groups’ overall understanding of representative democracy remained about the same from pretest to posttest. Although neither of the states’ intervention group gains in overall appreciation of representative democracy reached statistical significance, both intervention groups gained significantly in their appreciation of the right of every citizen to be heard by his or her representatives (Objective J). Intervention and Control Group Differences

In addition to examining changes from pretest to posttest within each group (see Student Gains section above), the evaluators compared posttest scores of the intervention and control groups while controlling for differences in pretest scores. Intervention and control group posttest means were compared directly to test a hypothesis stating that mean posttest scores of one group (e.g., the intervention group) would be significantly different from those of the other group. Controlling for differences in pretest scores is essential because even an experimental design with random assignment can yield groups that differ to some extent on the variables of interest—in this case knowledge and appreciation of representative democracy—at the outset of the study. Such differences must be statistically controlled to ensure internal validity. That is, the analysis must control for pretest differences to ensure that any posttest differences that might be found are restricted to those that can be attributed to the effectiveness of the intervention and not to differences already present in the groups’ knowledge and perceptions at the outset of the study. Controlling for pretest differences also can ensure that comparatively low initial scores on the part of the intervention group will not mask program effects that might be present. In an analysis of knowledge and appreciation pretest scores the evaluators found that the Louisiana intervention group had scored significantly lower (t

21

(365) = .252, p = .012) on the pretest (mean = 55.6 percent) than had the control group (mean = 68.1 percent) on items related to Objective B (citizens’ groups as change agents), confirming the need for a procedure that would control for pretest differences. Conversely, the Wyoming intervention group scored significantly higher (mean = 62.2 percent) than the control group (mean = 50.6 percent) on the knowledge pretest (mean difference = 11.6 percentage points; t(160) = -3.23; p = .001). The Wyoming intervention group’s pretest scores also were significantly higher than those of the control group on the item clusters related to Objectives C, D, and I. No significant differences were found between either state’s intervention and control group scores on pretests measuring students’ appreciation of representative democracy. The evaluators used Analysis of Covariance to examine differences in intervention and control group representative democracy posttest means while controlling for pretest differences. No significant differences were found in the two Louisiana groups’ overall mean posttest knowledge or appreciation scores. (See Table 11 for a summary of these results and Table 12 for more detailed information.) The evaluators also examined intervention and control group posttest score differences on item clusters related to the program’s 13 objectives. In the Louisiana phase, these analyses yielded significant differences between the average scores of the intervention and control groups on one representative democracy knowledge cluster (Objective D—the difference between a state legislature and the U.S. Congress) and one representative democracy appreciation cluster (Objective J—the right of every citizen to be heard by his or her representatives). The Louisiana intervention group scored significantly higher than the control group on the posttest items in these clusters when controlling for differences in initial scores. The control group’s posttest scores were not found to have exceeded those of the intervention group in any item cluster. Although the overall knowledge and appreciation posttest score means of Louisiana’s intervention group did not differ significantly from those of the control group, Wyoming’s intervention group scored significantly higher than did the control group on the knowledge and appreciation areas of the posttest, as well as on knowledge and appreciation item clusters related to individual objectives. Table 11 lists the overall test areas and objectives related to item clusters on which the Louisiana and Wyoming intervention groups’ average posttest scores exceeded those of the control group.

22

Table 10

Paired Samples T-Test Results Showing Significance of Difference in Pretest and Posttest Scores for Intervention and Control Groups

10 a: Louisiana 1 2 3 4 5 6 Area Objective Intervention Group¹ Control Group¹

T df p t df p Knowledge Area

-3.55 213 0.000 *** -0.80 165 0.426

Students will demonstrate their understanding—

A. of the lawmaking process -2.88 212 0.004 ** 0.99 161 0.325 B. that new laws begin when citizen groups

want changes that will help improve government or meet their needs -1.18 212 0.240 1.58 165 0.115

C. that the procedures a state legislature uses to resolve differences about public policy include debate, negotiation, and compromise -3.34 213 0.001

*** -0.28 165 0.780

D. of the difference between a state legislature and the U.S. Congress (i.e., that a state legislature makes laws that affect individuals within a state, as opposed to the nation as a whole) -3.24 208 0.001

*** -0.83 165 0.406

E. that representative democracy is a system in which individuals vote to elect representatives to an assembly that -1.99 211 0.047 * -0.72 163 0.475

23

10 a: Louisiana 1 2 3 4 5 6 Area Objective Intervention Group¹ Control Group¹

T df p t df p represents the people in deciding issues

F. that representative democracy derives its authority from the people -2.10 212 0.037 * -0.44 165 0.664

G. that representatives are elected from the citizenry -0.19 212 0.851 -1.01 164 0.313

H. of checks and balances in government -1.47 213 0.142 -0.91 165 0.363 I. the need for compromise as a part of the

legislative process -0.49 213 0.623 -1.59 164 0.113 Perception Area 0.50 213 0.616 0.20 164 0.839 Students will appreciate— J. the right of every citizen to be heard by his

or her representatives -2.11 209 0.036 * -0.46 161 0.644 K. the need for citizens to be actively involved

in state issues 0.83 211 0.408 0.98 164 0.329 L. legislators’ interest in meeting the needs of

the people 0.83 211 0.405 0.14 164 0.889 M. the need for checks and balances in

government 1.48 211 0.140 -1.27 161 0.208

¹ Pretest and posttest means are listed in ANCOVA table. * Group posttest mean is significantly higher than the same group’s pretest mean at the .05 level (p ≤ .05). ** Group posttest mean is significantly higher than the same group’s pretest mean at the .01 level (p ≤ .01). *** Group posttest mean is significantly higher than the same group’s pretest mean at the .001 level (p ≤ .001).

24

10 b: Wyoming 1 2 3 4 5 6 Area Objective Intervention Group¹ Control Group¹

t df p t df p Knowledge Area -2.82 70 .01 ** .63 90 .53 Students will demonstrate their

understanding—

A. of the lawmaking process -1.73 70 .09 -.62 90 .54 B. that new laws begin when citizen groups

want changes that will help improve government or meet their needs -.26 70 .80 2.68 90 .01**

C. that the procedures a state legislature uses to resolve differences about public policy include debate, negotiation, and compromise -1.56 70 .12 -1.11 90 .27

D. of the difference between a state legislature and the U.S. Congress (i.e., that a state legislature makes laws that affect individuals within a state, as opposed to the nation as a whole) 0.00 70 1.00 -1.53 90 .13

E. that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues -.69 70 .50 1.82 90 .07

F. that representative democracy derives its -1.35 70 .18 .33 90 .74

25

10 b: Wyoming 1 2 3 4 5 6 Area Objective Intervention Group¹ Control Group¹

t df p t df p authority from the people

G. that representatives are elected from the citizenry -1.09 70 .28 1.52 90 .13

H. of checks and balances in government -1.59 70 .12 .89 90 .37 I. the need for compromise as a part of the

legislative process -1.35 70 .18 -1.58 90 .12 Perception Area

-1.28 70 .21 2.25 90 .03* Students will appreciate— J. the right of every citizen to be heard by his

or her representatives -3.17 70 <.001 *** -.33 90 .75

K. the need for citizens to be actively involved in state issues .82 70 .42 1.13 90 .26

L. legislators’ interest in meeting the needs of the people -1.85 70 .07 1.89 90 .06

M. the need for checks and balances in government -1.51 70 .14 1.26 89 .21

¹ Pretest and posttest means are listed in ANCOVA table. * Group posttest mean is significantly higher than the same group’s pretest mean at the .05 level (p ≤ .05). An exception

is in the results for the Wyoming control group. That group’s perception area posttest mean was significantly lower than the same group’s pretest mean.

** Group posttest mean is significantly higher than the same group’s pretest mean at the .01 level (p ≤ .01). *** Group posttest mean is significantly higher than the same group’s pretest mean at the .001 level (p ≤ .001).

26

The evaluators also examined intervention and control group posttest score differences on item clusters related to the program’s 13 objectives. In the Louisiana phase, these analyses yielded significant differences between the average scores of the intervention and control groups on one representative democracy knowledge cluster (Objective D—the difference between a state legislature and the U.S. Congress) and one representative democracy appreciation cluster (Objective J—the right of every citizen to be heard by his or her representatives). The Louisiana intervention group scored significantly higher than the control group on the posttest items in these clusters when controlling for differences in initial scores. The control group’s posttest scores were not found to have exceeded those of the intervention group on any item cluster. Although the overall knowledge and appreciation posttest score means of Louisiana’s intervention group did not differ significantly from those of the control group, Wyoming’s intervention group scored significantly higher than did the control group on the overall knowledge and appreciation areas of the posttest, as well as on knowledge and appreciation item clusters related to individual objectives. Table 11 lists the overall test areas and objectives related to item clusters on which the Louisiana and Wyoming intervention groups’ average posttest scores exceeded those of the control group. Table 11

Test Areas and Item Clusters in Which Intervention Groups’ Posttest Scores Exceeded Those of Control Groups: Louisiana and Wyoming Phases1

Test Area Cluster Intervention Group Scored Significantly Higher Than

Control Group in – Louisiana Knowledge D Understanding of the difference between a state legislature

and the U.S. Congress Perception J Appreciation of the right of every citizen to be heard by his

or her representatives Wyoming Knowledge Entire

area Overall understanding of representative democracy

B Understanding that new laws begin when citizen groups want changes that will help improve government or meet their needs

C Understanding that the procedures a state legislature uses to resolve differences about public policy include debate

E Understanding that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues

G Understanding that representatives are elected from the citizenry

H Understanding of checks and balances in government

27

Test Area Cluster Intervention Group Scored Significantly Higher Than Control Group in –

I Understanding of the need for compromise as a part of the legislative process

Perception Entire area

Appreciation of representative democracy

J Appreciation of the right of every citizen to be heard by his or her representatives

L Appreciation of legislators’ interest in meeting the needs of the people

M Appreciation of the need for checks and balances in government

1 Neither control group’s average posttest score significantly exceeded that of the respective intervention group on either overall test area (knowledge or perception) or on any item cluster.

In summary, the evaluators found significant intervention and control group posttest differences in both phases of the study. Although intervention and control group differences in overall knowledge and appreciation of representative democracy were not significant in the Louisiana phase, average intervention group scores in Louisiana exceeded average control group scores on the knowledge area item cluster related to understanding the difference between a state legislature and the U.S. Congress. Significant differences also occurred in the appreciation item cluster related to the right of every citizen to be heard by his or her representatives. Wyoming’s average intervention group posttest scores exceeded those of the control group in the overall knowledge area, as well as in item clusters related to students’ understanding of six major concepts in the knowledge area. Wyoming’s average intervention group scores also exceeded those of the control group in the overall perception area and on three item clusters measuring appreciation of representative democracy. These included the right of every citizen to be heard by his or her representatives (which also was significant in the Louisiana phase).

28

Table 12

Mean Louisiana and Wyoming Intervention and Control Group Pretest, Posttest, and Change Scores and ANCOVA Results Showing Significance of Differences in Posttest Knowledge and Perception Area Scores While Controlling for Differences in Pretest Scores Table 12 a: Louisiana 1 2 3 4 5 6 7 8 9 Test Area

Objective Inter-vention Group PretestMean

Inter-vention Group PosttestMean

Inter-vention Group Mean

Change

Control Group PretestMean

Control Group PosttestMean

Con-trol

Group Mean

Change

F df be-

tween |

within

p

Knowledge Area 47.7% 55.7% 8.0% 50.7% 52.5% 1.8% 1.86 1 | 377 .17 Students will demonstrate their

understanding—

A. of the law making process 39.4% 52.1%

12.8% 47.6% 43.2% -4.4% 3.16 1 | 372 .08

B. that new laws begin when citizen groups want changes that will help improve government or meet their needs 55.6% 61.0%

5.5% 68.1% 60.8% -7.2% .09 1 | 376 .77 C. that the procedures a state

legislature uses to resolve differences about public policy include debate, negotiation, and compromise 50.9% 63.6%

12.6% 58.1% 59.3% 1.2% .95 1 | 377 .33 D. of the difference between a state

legislature and the U.S. Congress (i.e., that a state legislature makes 43.1% 58.4% ** 15.3% 40.4% 45.2% 4.8% 6.60 1 | 372 .01

29

Table 12 a: Louisiana 1 2 3 4 5 6 7 8 9 Test Area

Objective Inter-vention Group PretestMean

Inter-vention Group PosttestMean

Inter-vention Group Mean

Change

Control Group PretestMean

Control Group PosttestMean

Con-trol

Group Mean

Change

F df be-

tween |

within

p

laws that affect individuals within a state, as opposed to the nation as a whole)

E. that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues 59.7% 69.3%

9.6% 64.5% 68.3% 3.8% .04 1 | 373 .83 F. that representative democracy

derives its authority from the people 50.9% 60.6%

9.6% 50.6% 53.0% 2.4% 2.16 1 | 376 .14 G. that representatives are elected

from the citizenry 45.8% 46.9%

1.1% 36.1% 41.2% 5.1% 1.34 1 | 375 .24 H. of checks and balances in

government 42.1% 47.4%

5.3% 44.6% 47.9% 3.3% .01 1 | 377 .92 I. the need for compromise as a part

of the legislative process 44.0% 46.7%

2.7% 44.6% 53.3% 8.8% 1.61 1 | 376 .20 Perception Area1

2.80 2.77 -0.03 2.77 2.76 -0.1 .001 1 | 376 .99 Students will appreciate— J. the right of every citizen to be

heard by his or her representatives 2.56 2.83 ** 0.30 2.33 2.40 0.07 7.98 1 | 369 .01

30

Table 12 a: Louisiana 1 2 3 4 5 6 7 8 9 Test Area

Objective Inter-vention Group PretestMean

Inter-vention Group PosttestMean

Inter-vention Group Mean

Change

Control Group PretestMean

Control Group PosttestMean

Con-trol

Group Mean

Change

F df be-

tween |

within

p

K. the need for citizens to be actively involved in state issues 2.80 2.76

-0.05 2.89 2.82 -0.07 .05 1 | 374 .83 L. legislators’ interest in meeting the

needs of the people 2.92 2.85

-0.07 2.82 2.81 -0.01 .002 1 | 374 .96 M. The need for checks and

balances in government 2.67 2.52

-0.17 2.55 2.74 0.19 3.53 1 | 371 .06

12 b: Wyoming 1 2 3 4 5 6 7 8 9 Test Area

Objective Inter-vention Group PretestMean

Inter-vention Group PosttestMean

Inter-vention Group Mean

Change

Control Group PretestMean

Control Group PosttestMean

Con-trol

Group Mean

Change

F df be-

tween |

within

p

Knowledge Area 62.2% 68.7% *** 6.5% 50.6% 49.2% -1.4% 18.1 1|159 .001

Students will demonstrate their understanding—

A. of the law making process 50.7% 62.0% 11.3% 42.9% 47.3% 4.4% 2.87 1|159 .092 B. that new laws begin when citizen

groups want changes that will help improve government or meet their needs 76.1% 77.5% *** 1.4% 65.9% 48.4% -17.5% 13.32 1|159 .001

31

12 b: Wyoming 1 2 3 4 5 6 7 8 9 Test Area

Objective Inter-vention Group PretestMean

Inter-vention Group PosttestMean

Inter-vention Group Mean

Change

Control Group PretestMean

Control Group PosttestMean

Con-trol

Group Mean

Change

F df be-

tween |

within

p

C. that the procedures a state legislature uses to resolve differences about public policy include debate, negotiation, and compromise 70.4% 77.5% ** 7.1% 49.5% 53.8% 4.3% 6.59 1|159 .011

D. of the difference between a state legislature and the U.S. Congress (i.e., that a state legislature makes laws that affect individuals within a state, as opposed to the nation as a whole) 57.7% 57.7% 0.0% 34.1% 44.0% 9.9% .96 1|159 .330

E. that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues 74.6% 78.9% * 4.3% 74.7% 64.8% -9.9% 4.29 1|159 .040

F. that representative democracy derives its authority from the people 52.1% 60.6% 8.5% 48.4% 46.2% -2.2% 3.11 1|159 .08

G. that representatives are elected from the citizenry 54.9% 62.0% * 7.1% 53.8% 45.1% -8.7% 5.21 1|159 .024

H. of checks and balances in government 55.6% 64.1% ** 8.5% 50.5% 46.2% -4.3% 9.4 1|159 .003

32

12 b: Wyoming 1 2 3 4 5 6 7 8 9 Test Area

Objective Inter-vention Group PretestMean

Inter-vention Group PosttestMean

Inter-vention Group Mean

Change

Control Group PretestMean

Control Group PosttestMean

Con-trol

Group Mean

Change

F df be-

tween |

within

p

I. the need for compromise as a part of the legislative process 64.8% 73.2% ** 8.4% 36.3% 46.2% 9.9% 6.07 1|159 .015

Perception Area12.62 2.74 ** .12 2.56 2.44 -0.12 7.72 1|159 .006

Students will appreciate—

J. the right of every citizen to be heard by his or her representatives 2.34 2.90 ** .56 2.27 2.33

0.06 8.03 1|159 .005

K. the need for citizens to be actively involved in state issues 2.64 2.54 -0.1 2.54 2.43 -0.11 .22 1|159 .642

L. legislators’ interest in meeting the needs of the people 2.62 2.86 ** .24 2.62 2.45 -0.17 10.08 1|159 .002

M. The need for checks and balances in government 2.8 3.04 * .24 2.74 2.57 -0.17 6.05 1|159 .015

1 Perception scale ranges from 0 (low level of appreciation) to 3 (high level of appreciation).

* Intervention group posttest mean is significantly higher than the control group posttest mean at the .05 level (p ≤ .05). ** Intervention group posttest mean is significantly higher than the control group posttest mean at the .01 level (p ≤ .01). *** Intervention group posttest mean is significantly higher than the control group posttest mean at the .001 level (p ≤

.001)

33

Test Score Gains and Characteristics of the Intervention

To determine whether growth in intervention classes’ understanding or appreciation of representative democracy was related to specific characteristics of the intervention, the evaluators examined correlations between score gains and intervention variables. Using the Spearman’s Rho procedure (a nonparametric test), the evaluators examined the correlation between knowledge or appreciation pretest-to-posttest score gains and the following intervention variables:

Your Ideas Count status (whether the class received these materials) Time (minutes) the legislator lectured Time the legislator spent eliciting and answering questions Time the legislator spent asking questions and leading a discussion Duration of visit (minutes) Lesson on simulation (extent to which this type of instruction was used—

some, a lot, or not at all) Lesson on solving a class problem (extent to which this type of instruction

was used—some, a lot, or not at all) Lesson on a day in the life of a legislator (extent to which this type of

instruction was used—some, a lot, not at all) Lesson on perfect cookie/pizza (extent to which this type of instruction was

used—some, a lot, not at all) Students' attention (all, majority, some, or none paid attention) Students' participation (many, some, few participated)

No consistent patterns of relationship emerged between average class gains in

knowledge or appreciation (the criterion variables) and specific intervention characteristics (the predictor variables). This outcome is likely to have been attributable to the limited sample size (N = 13 classes and N = 10 for Louisiana and Wyoming, respectively) for these tests. Data at the class level, rather than the student level, were analyzed because specific intervention characteristics were at the class level, rather than at the level of the individual student.

Conclusion

Results of this two-phase study demonstrate that Louisiana and Wyoming legislators’ Back to School Program classroom visits strengthened students’ understanding and appreciation of several elements of representative democracy. After legislators had visited Wyoming’s intervention group, students in that group demonstrated a higher level of overall understanding of representative democracy than did the Wyoming students in the control group classes. Average scores of Wyoming students in classes visited by legislators also were significantly higher than those of control group students on six of nine concepts related to representative democracy. These included item clusters measuring the extent to which students understood

• that new laws begin when citizen groups want changes that will help

improve government or meet their needs;

34

• that the procedures a state legislature uses to resolve differences about public policy include debate, negotiation, and compromise;

• that representative democracy is a system in which individuals vote to elect representatives to an assembly that represents the people in deciding issues;

• that representatives are elected from the citizenry; • the concept of checks and balances in government; and • the need for compromise as a part of the legislative process.

Although the overall average score of Louisiana’s intervention group on the

knowledge section of the Representative Democracy Test did not significantly exceed that of the control group, Louisiana students in classes visited by legislators scored significantly higher than Louisiana’s control group on an item cluster measuring their understanding of the difference between a state legislature and the U.S. Congress.

Students in Louisiana and Wyoming classes visited by legislators also showed

higher levels of appreciation of representative democracy following their legislative visits. The overall appreciation of representative democracy of Wyoming students whose classes were visited exceeded the appreciation levels of the control group. The Louisiana intervention group’s posttest scores significantly exceeded those of the control group on an item cluster measuring appreciation of the right of every citizen to be heard by his or her representatives.

Students in Wyoming’s intervention group also scored significantly higher than

those in the control group on specific item clusters measuring appreciation of representative democracy. These included the items and clusters measuring appreciation of

• the right of every citizen to be heard by his or her representatives (the item on which Louisiana’s intervention group posttest scores also exceeded those of the control group)

• legislators’ interest in meeting the needs of the people, and • the need for checks and balances in government.

In all comparisons of the Louisiana and Wyoming intervention groups’ posttest

scores with the posttest scores of their respective control groups, the evaluators used a procedure that controlled for differences in the pretest scores of the intervention and control groups. These controls were particularly important in examining posttest results in test areas and item clusters in which intervention group scores were significantly higher or lower than those of the control group at the outset of the study.

Additional support for a hypothesis that the program significantly improved

Louisiana and Wyoming students’ understanding of representative democracy can be seen in the significantly higher overall pretest to posttest gains seen in intervention group students’ understanding of representative democracy, as well as in the absence of significant overall knowledge gains on the part of the control group. Further, when the evaluators examined the significance of the each state’s intervention group gains on item

35

clusters related to students’ understanding of basic principles of representative democracy, they found significant improvement in the Louisiana intervention group’s knowledge of concepts in five of these nine item clusters.

The average gains of students in both intervention groups—Louisiana’s and

Wyoming’s—were significant on at least one of four appreciation item clusters, the right of every citizen to be heard by his or her representatives. No significant gains occurred in understanding of concepts specified in the program’s nine knowledge-related objectives, or on any of the program’s four perception-related objectives for control group students in Louisiana or Wyoming.

The evaluators examined monitors’ reports on legislators’ visits and analyzed that

information in relation to student outcomes to determine whether any specific methods used by legislators or other classroom conditions were directly associated with students’ growth in understanding and appreciation of representative democracy. Monitors’ reports were useful in confirming that legislators and teachers had fully implemented the intervention. However, the evaluators’ detailed examination of monitors’ data yielded no consistent patterns of relationship between student outcomes and characteristics of the intervention for either phase of the study.

This evaluation of the America’s Legislators Back to School Program shows that

the program, when implemented by legislators and teachers who use the recommended lesson types and materials and address the intended program objectives, results in significantly higher levels of understanding and appreciation of representative democracy among middle school students. The program’s long-term impact on students’ knowledge and perceptions of representative democracy may warrant attention in future studies.

36

REFERENCES National Conference of State Legislatures. (2004). America’s Legislators Back to School

Week. Retrieved 2/13/2005 from http://www.ncsl.org/public/trust/bksdesc.htm National Conference of State Legislatures. (2005). Lawmakers receive high marks from

educators! Retrieved 2/13/2005 from www.ncsl.org/public/trust/ BTSW04evalpromo.htm

ABOUT THE RESEARCHERS The principal researchers of TPACT, LLC, have a unique combination of expertise about the state legislatures, program evaluation, and civic education. John Turcotte is president of Turcotte Public Administration and Consulting, LLC, and director of the Program Evaluation Division of the North Carolina General Assembly. He has thirty years’ experience as a state legislative staff director and program evaluator. Turcotte was director of the Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review (PEER) from 1977 to 1995. From 1996 until 2003, he was director of the Florida Legislature’s Office of Program Policy Analysis and Government Accountability (OPPAGA). During his career, he directed 768 performance audits and program evaluations. Turcotte holds both a BA with highest honors and an MA in political science from the University of Southern Mississippi and is a graduate of the U.S. Federal Executive Institute. In 1994, he served as staff chair of the National Conference of State Legislatures and represented more than 30,000 state legislative staff nationwide. While staff chair, he initiated a multi-year project to boost civic education in America. Turcotte was a founding member of the National Legislative Program Evaluation Society and served as Society Chairman in 1979. Kathleen Sullivan, associate professor and director of the Center for Educational Research and Evaluation at the University of Mississippi since 1998, is an accomplished state legislative program evaluator, education researcher, and methodologist. She holds a BA in mathematics from Marquette University, an MS in research and evaluation from Emory University and a PhD in higher education and student personnel from the University of Mississippi. From 1984 until 1998, she served as evaluation division manager and methodologist for the Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review (PEER). Sullivan is the author of numerous books, PEER program evaluations, journal articles, and monographs. Jim Pelligrini, former deputy legislative auditor, State of Montana, has thirty years’ of experience in managing, planning and conducting program, economy, and efficiency audits. He has strong expertise in government processes, legislative decision-making, analytical techniques and controls. He is an accomplished consultant and seminar leader for various organizations including government and private group and has made numerous contributions to government operations leading to increased performance, internal efficiencies and enhanced internal controls.

37