Out-of-School STEM Opportunities

Out-of-School STEM Opportunities

Out-of-School STEM Opportunities

Out-of-School STEM OpportunitiesA review of current and best practices in out-of-school STEM opportunities

Y of Central Maryland Contract # NOO189-09-P-1837 Anna Kavetsky Deputy Project Director Ken Barsa STEM Project Director

Table of ContentsIntroduction5Purpose of this Report5Literature Review6Synthesis of Literature7Literature Search7Methodology8STEM in Central Maryland8Program Director9Program management and program administration9Staff experience and training10Family and community involvement10Community partnerships11Evaluation11Recommendations12County STEM Coordinators/Principals13Programs13Teacher involvement13Curriculum and professional development14Assessment and evaluation14Recommendations14Site Staff15Program environment15Instructional features17Program organization19Preference questions19Recommendations20Parent Interest21Findings22Insight Recommendations25The Recommendations for Best Practices25Recommendation 1:26Recommendation 2:27Recommendation 3:27Recommendation 4:28Recommendation 5:28Recommendation 6:28Summation29References30Appendix A Program Director Survey32Appendix B STEM Coordinator/Principal Survey34Appendix C Site Staff Survey35Appendix D Parent Interest Survey39Appendix E Program Descriptions40Destination Imagination40Future City Competition40National Engineers Week40Sea Perch41WIE-KEYS Empowering Youth Program41WIE-KEYS Stepping Stones to your Future41Baltimore Museum of Industry-Engineering Challenge42Mad Science42Project Lead the Way-Gateway to Technology42Maryland MESA43USNA Summer STEM Program43NASA BEST (Beginning Engineering, Science and Technology) Club43LETS GO (Leadership, Engineering, Technology & Science Generating Opportunities at the Boys & Girls Club)44National Partnership for After-School Science 2(TASI)44Materials World Modules (MWM)44Engineering is Elementary45National Defense Education Program (NDEP)45Techbridge45Wonderwise 4-H46SPICE (Students with Potential and Interest, Considering Engineering)46WIE Pre-College Elementary School: iEngineer @ UMCP46Up, Up and Away47Afterschool Science PLUS (AS+)47Afterschool Universe47Girls at the Center (GAC)48MARS (Math and Related Sciences) Camp48Operation SMART48SciGirls49VDP (Virginia Demonstration Program)49STEM RAYS49Summer LEGO Robotics Camp50First Lego League (Girls Scouts of America)50Toy Challenge50Appendix F Advisory Board51Dr. Benjamin S. Carson51William Stewart52John Hoey53Robert J. Brosmer53Robert Kavetsky54Retired Colonel Kenneth O. McCreedy54Christine Ader-Soto55David Fries56Amy Warner56

Out-of-School STEM OpportunitiesA review of current and best practices in out-of-school STEM opportunitiesIntroductionIn an effort to address the current national crisis of the diminishing quantity of qualified U.S. science, technology, engineering and mathematics (STEM) workers, the Y of Central Maryland (the Y) in collaboration with the Carson Scholars Fund was supported by a contract from the Department of Defense to conduct initial research in out-of-school STEM programming both at the local and national level in grades three through eight. This project was envisioned as the first phase of a two phase project. Phase one incorporated two tasks:Task 1: Construct a searchable database of STEM activities in the central Maryland region to include Anne Arundel County, Baltimore County, Baltimore City, Harford County, Howard County and Queen Annes County. Conduct a survey to determine currently existing STEM programs, how could they be tailored, and what gaps exist? Bring educators involved in STEM together to share best practices and identify needs/gaps. Search the internet to locate exiting STEM activities. (Refer to Appendix E for existing STEM activities at the local, state, and national level.)Task 2: Compile a database of best practices at the national level, especially in the areas of assessment, evaluation and curriculum development.The outcome of the first phase was to create a report summarizing the results of this study along with an electronic and hard copy version of the searchable database. After several months of research, utilizing an array of research methodologies, multiple out-of-school programs were identified with only a couple of programs employing the best practices criteria that are identified in the methodologies section of this report.Purpose of this ReportThe purpose of this report is to provide a summary of the findings of this research, including identification of current practices in central Maryland, an analysis of the gaps that currently exist in out-of-school programming, and insight into some best practices for assessment, evaluation, and curriculum development at the elementary and middle school level for STEM in the out-of-school setting. In addition, this report will provide recommendations as to best practices in developing out-of-school STEM programs as well as identify benchmarks and key indicators that could be used as a set of standardized expectations for out-of-school STEM programming. A synthesis of literature was conducted to extract these common key indicators and benchmarks mentioned in the literature and are provided in the literature review in the following section.While the research for this report was being conducted, Governor Martin OMalleys office published a final report from the STEM Task Force in August of 2009, Investing in STEM to Secure Marylands Future. Many of the recommendations in this report support and expand upon the recommendations given in this final report of the P-20 plan. Literature ReviewResearchers have found that STEM education prepares all students for the challenges and opportunities present in the 21st century economy. STEM provides education and economic opportunity, both now and into the future. (National Governors Association Center for Best Practices, 2007). As technology becomes more sophisticated, and as more nations compete with the United States in the global economy, the next generation of students must be well-grounded in STEM to succeed economically. The future of our economy and our national security posture depends on cutting edge, innovative workers with high end STEM knowledge and skills. STEM education is a key building block to that innovation process. In the 21st century global economy, a highly skilled, technologically savvy workforce is increasingly in demand. Survey after survey has made it clear that todays employers, regardless of industry, need STEM-literate workers. The economic success of states and nations is grounded in people who arent just smart, but who also are creative, analytical, and willing to take risks. Innovation is the key to our nations long-term economic success, as the way that new ideas enter the economy and change what is produced, how it is produced, and the very process of production itself. Rigorous and relevant STEM education supports innovative thinking in all students. (National Governors Association Center for Best Practices, 2007).Researchers at the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) at the University of California, Los Angeles (UCLA) have also found that out-of-school programs are beneficial to student outcomes in three critical ways. First, they provide children with supervision during afterschool hours a time period where research has found rates for both victimization of juveniles and juvenile crimes peak. (U.S. Department of Education & U.S. Department of Justice, 2000; Snyder & Sickmund, 1995). Secondly, they provide experiences that may benefit students social skills and work habits (Fashola, 1998.) Finally, afterschool programs may help improve academic achievement through tutoring and enrichment activities (Fashola, 1998.) Quality afterschool programs can provide these basic benefits to students, as well as additional opportunities to acquire new skills and broaden their educational experiences (4). Therefore, it is important to recognize the potential of afterschool and out-of-school programs to positively impact both the social and academic development of students, while also realizing the importance of assessing the critical factors that are necessary for providing a high quality out-of-school STEM program. (Huang, D. et al, 2008).Ongoing research in out-of-school programming has suggested that three critical elements in creating high quality programs include providing child safety, opportunities for positive social development, and academic enrichment. (U.S. Department of Education & U.S. Department of Justice, 2000.) Building from these basic elements, a more comprehensive view is provided of the factors necessary for increasing the quality of out-of-school programs. Based on the overall literature review, it was determined, through the CRESST report on best practices, that there are three major areas that can determine effectiveness. These areas include (a) program organization, (b) program environment, and (c) instructional features, of which each has been identified and surveyed in the following survey reviews. (Huang, D. et al, 2008).Synthesis of LiteratureSimilar to a meta-analysis of literature, defined as a type of systematic review that uses statistical methods to combine and summarize the result of several primary studies, the STEM project team decided to use a synthesis of literature approach for this study. A large body of literature was uncovered in researching out-of-school STEM programming, and therefore, it was decided that this would be the preferred method of study. It should be noted that there were very few studies with qualifying quantitative data or empirical evidence thereby creating limitations of this process in the ability to draw formal inferences to the larger populations. (Huang, D. et al, 2008).Literature SearchThe initial literature search began in July of 2009 and has continued through November 2009. Searches were conducted using ERIC (Education Resources Information Center) and the Google search engine. These searches were conducted using the terms STEM, after school STEM, STEM in after school, STEM summer camps and STEM outreach as keywords or descriptors, and with the following parameters: 2001-2009, elementary and/or middle school age group, and English language only. This produced multiple resources varying from peer-reviewed journals, to books, journal articles, conference papers, and websites alike.Many of the abstracts of these resources were reviewed and discussed by members of the STEM project team and their advisory board. (See Appendix F.) Certain titles indicated they should be excluded from the study, for example, those that included high school, undergraduate, and post-graduate programming. The criteria were then established for inclusion into the study: Studies that referred to out-of-school programs in STEM for kindergarten through eighth grade, specifically focused more on third through eighth grades, and Studies that commented on or addressed the quality indicators for out-of-school STEM programs or out-of-school programs in general. Specifically, the study could have either identified characteristics of out-of-school programs or reviewed literature that summarized that quality indicators based on existing literature.The STEM project team obtained the full articles, if, and only if, the abstracts met all of the criteria mentioned above. From all of the previously mentioned resources, the project team identified numerous resources to be included in this study. These resources included books, websites, statistics, review articles, summaries and evaluation reports, and were often written by researchers, educators, or subject matter experts who had extensive experience in the fields of either out-of-school programming, STEM education, and/or assessments and evaluations. Methodology According to a study of best practices in science education, the Bayer Corporation found that in order for all students, regardless of age, gender, cultural or ethnic background, disabilities, aspirations, or inspiration and motivation in science, to achieve in science, they must have access to highly-skilled professional teachers, adequate classroom time dedicated to science learning and quality science learning materials. Given this information, for the purposes of this report and identification of best practices in out-of-school programming in STEM, programs were decided to be a best practice if, and only if, they met all of the following four criteria (criteria based on guidelines by Building Engineering and Science Talent (BEST) Commission, National Science Education Standards, and National Science Resources Center):1. The program must have an inquiry based, experiential curriculum that is both clearly defined and understood;2. The program must provide an environment where teachers and their students work together as active learners;3. The program must have clearly identified goals with success measured against them; and4. The program must have strong leadership and sufficient resources including continuity of program funding as well as school and/or school district support. (Bayer Corporation, 2006).STEM in Central MarylandOn July 1, 2009 research began on the current practices in STEM programming in the central Maryland region. A framework first had to be developed to determine exactly how to identify programs in each of the targeted districts, including the city of Baltimore along with Anne Arundel, Baltimore, Carroll, Harford, Howard and Queen Annes counties. After reviewing literature on other out-of-school research, it was determined that creating and distributing surveys to county program directors, STEM coordinators, principals, site staff and parents alike would be a valuable and efficient method of identifying both current practices and gaps in programming. In addition, an analysis of literature through online research would be conducted to gather additional information and insight into the out-of-school programming in each district. Each individual countys website would be surveyed to identify current practices in STEM at the elementary and middle school levels. Interview sessions would also be utilized to collect data based on surveys developed for site staff at STEM programs. It was determined that the combination of these methods would provide a thorough review of current programming in central Maryland. However, multiple districts provided little to no response to multiple phone, email, and survey attempts, which ultimately resulted in reliance primarily on the review of online literature to determine current practices in central Maryland. The following table lists the participants in the study by role and area of programming. (See Appendix E for a listing of current practices in STEM in central Maryland, the state of Maryland, and at the national level.)Table 1 Study Participants by Role and Area of ProgrammingParticipantSurvey

Program Directors (Total)5

Maryland Engineering Challenges1

Title One Extended Day1

MD Mathematics, Engineering, Science Achievement (MESA)1

Virginia Demonstration Program (VDP)1

The After School Institute (TASI)1

STEM Coordinator/Principal (Total)3

District Coordinator1

County Title One Representatives1

Lead Teachers @ STEM High School1

Site Staff (Total)16

Program DirectorThe program director survey was designed to examine key indicators of program organization including items on management, administration and sustainability, staff experience and training, family involvement, community partnerships, and evaluation. Five programs (see Table 1 above) were surveyed through interviews with the program director(s) utilizing data collection instruments that were developed by the STEM project team. The surveys that were developed (see Appendix A) were completed through interviews along with site visits at each programs location. The following will provide a brief summary of the results of the survey followed by some recommendations based on data results and feedback from program directors. Program management and program administration Effective program management is necessary for quality-based afterschool programs. (Huang, D. et al, 2008). More specifically, it is important to have leadership articulate a shared mission statement and program vision that motivates staff, provides a positive organizational climate that validates staff commitment to these goals, as well as open the communication channels between afterschool, day school, parent and community. (Huang, D. et al, 2008). Sixty percent of the programs interviewed revealed that their program has a strong and clear mission statement that accurately reflects the vision of their program. The planning process in developing a highly effective out-of-school program requires input from multiple sources including staff, parents, and students alike. Staff input was considered in sixty percent of these programs in regards to planning activities. Student input, on the other hand, was not as actively considered as indicated by twenty percent of the programs stating that they strongly disagreed with allowing student input in the planning process. Administration is also a key entity in obtaining buy-in from the school and thereby obtaining buy-in to the program from teachers, parents and students. As clearly identified by eighty percent of the programs, school principals were also highly involved in most of the program planning and in promoting innovation at individual schools. Research has indicated that a strong leadership team and committed staff must plan for program sustainability through effective administration including systematic organization of student records, program attendance, resource needs, program budget, a future financial plan, and marketing (Huang, D. et al, 2008). All of the program directors either agreed (40%) or strongly agreed (60%) that there was a long-term financial plan in place at their individual programs providing for sustainability, maintenance of program quality, and potential for future growth. All agreed that the program budget was and would be periodically adjusted based on resource needs. Most of the directors also agreed (20%) or strongly agreed (60%) that all possible resources were being utilized to the make the programming as affordable as possible for the schools and students. For example, programs would utilize community resources or fundraising opportunities to provide scholarship assistance or assistance with purchasing materials of instruction. While all of the programs have workable budgets, the number of students that were able to be affected by the program was strictly budget limited due to lack of funding. Only 20% of those interviewed felt that the resources were not being exhausted to make programming affordable to all students. Finally, all of the programs had some type of marketing plan in place to publicize the program whether it was via website, email, newsletters, conferences, etc. Staff experience and training In order to enhance staff efficacy, the staff must have the appropriate experience and training in working with afterschool students. (Huang, D. et al, 2008). After interviewing all of the program directors it became very clear that staff experience and training was of utmost concern and value in the central Maryland region. All of the directors indicated that instructors were well qualified to deliver and ensure high quality instruction while also being well-trained to work with school-aged students in an afterschool setting. It was also indicated by all directors that their program staff were very sensitive to the cultural and social influences that impact how students learn and relate to others. Eighty percent of the directors strongly agreed that their staff were competent in the core academic areas, as appropriate. To ensure high quality instruction, staff members should be consistently provided with opportunities for professional development. (Huang, D. et al, 2008). Out of the five programs interviewed, eighty percent indicated that instructors and staff participated in professional development together in order to allow for a relationship to build and provide a better out-of-school environment for the children. While this statistic is acceptable, there was still a concern issued about the quality of professional development available and the amount available to accommodate all teachers. Finally, program directors also received training on program management to better enhance the overall management of the program at all levels. Family and community involvement Research on afterschool programs consistently associates family and community involvement with program quality. (Huang, D. et al, 2008). By providing a steady, open line of communication between the family and the program, feedback and collaboration can best be provided to obtain the highest level of quality programming for students. Four out five of the programs interviewed strongly agreed that their program had a clearly defined plan for parental involvement and also welcomed family members to visit their program at any time. (This was identified to be based on the time of day the programming took place as many directors felt it was inappropriate to have family members visit during the school day, but afterschool time was more than acceptable.) Orientation sessions are another way to provide outreach to families and let them know exactly what type of activities their children are involved. Unfortunately, only sixty percent of the programs provided a strong orientation for families, while twenty percent provided a basic orientation and twenty percent did not provide any type of orientation. Parental input should be sought and encouraged at the most basic levels of programming to keep them actively involved in their childrens education. Sixty percent of the programs strongly agreed and actively sought parental input for program planning. Twenty percent only sought parental input on occasion and twenty percent did not consider it at all. Providing extra-curricular learning opportunities for parents at their childrens schools can be an excellent way to get parents involved in the activities of the school and provide an opportunity for the child and parent to learn together. Sixty percent of the programs interviewed provided some sort of learning opportunity for the parents through the out-of-school program, while twenty percent disagreed and another twenty percent strongly disagreed about providing these opportunities. A recent research finding on quality afterschool programs is that connections matter. Relationships among staff, schools, families, youth, and communities are crucial. (NIOST, 7.)Community partnerships Local communities can be an invaluable resource to out-of-school programs by providing subject matter expertise, facilities, financial assistance, etc. Research has shown that high quality out-of-school programs have established partnerships with local community resources that have provided assistance in planning and funding. (Huang, D. et al, 2008). All of the program directors interviewed indicated some level of community partnership with sixty percent agreeing and forty percent strongly agreeing that community partners should be involved in program planning. Eighty percent of the programs indicated some type of partnership with local community organizations such as libraries, businesses, and/or colleges/universities, with twenty percent indicating a strong partnership with these types of organizations. In return for the community partnerships, many times students will be expected to participate in community service projects, which can instill a sense of pride and honor among their community. (Huang, D. et al, 2008). In the programs identified, sixty percent encouraged youth representation in local community organizations through mentorship opportunities. While programs encouraging mentorship opportunities were in the majority, only twenty percent of the programs encouraged opportunities for students to give back to the community through community service projects. This was a major area of concern among all of those interviewed and something that the program directors hoped to address in the near future. Evaluation As an instrument to inform continuous self-improvement, periodic evaluations are critical for the sustainability of afterschool programs. Furthermore, having evidence of programs outcomes is essential for continued and or increased funding and support. (Huang, D. et al, 2008). Evaluations, both internal and external, are critical to the success of any high quality out-of-school program. Having stated this, program directors were asked a series of questions addressing their programs evaluation protocol. The results left much to be desired and clearly identified an area of weakness among local STEM programs. In general, the majority of programs provided some type of internal method for evaluating program activities with sixty percent agreeing and twenty percent strongly agreeing to this approach. Sixty percent also strongly agreed that there was an internal method for evaluating staff performance, while only twenty percent disagreed. When asked about evaluating the level of student engagement, forty percent of the program directors agreed and twenty percent strongly agreed that there was an internal method of measurement of this criterion. Students academic improvement and social skills development can be especially important in documenting program outcomes. (Huang, D. et al, 2008). Academic performance was among the top criterion measured internally with sixty percent of the programs strongly agreeing to the use of these measurement tools. Socials skills, on the other hand, were not as highly valued with a total of sixty percent of the programs either disagreeing or strongly disagreeing to the presence of this measurement in their internal or external evaluations. The majority of programs (sixty percent) surveyed both students and parents interest and satisfaction with the individual programs as well as sixty percent surveying staff members to identify areas of improvement. Sixty percent of the programs did say they used the evaluation findings to improve the program while the other forty percent strongly disagreed to this utilization of evaluation findings. When asked if evaluation reports were made readily available to the general public, including parents, the community and all other stakeholders, sixty percent of the programs either disagreed or strongly disagreed with sharing this information. The overall synopsis received from program directors indicated that most evaluations currently utilized are attitudinal with no development of any longitudinal study to determine long term effects of programming. If evaluations are to be effective, they must be consistently reviewed and made readily available in order to examine the progress of the program. (Huang, D. et al, 2008).Recommendations Through conducting the series of interviews with multiple program directors, the STEM project team has developed a list of recommendations based on feedback and data from the program director surveys. The recommendations are as follows:1. Out-of-school programs should emphasize a variety of activities that cultivate developmental assets that ultimately help the children to succeed. This is important in order to extend rather than duplicate the school day experience. (Huang, D. et al, 2008). It has been shown that programs that have a rigid duplication of the school day curriculum habitually have lower attendance rates and therefore less of an opportunity to influence the children. These programs should focus on developing social and work habits and ultimately learning assets that can be carried back into the school day curriculum to ultimately affect the overall level of achievement of the student. Increasingly, among the educational community, there is a call for the development of the whole child.The child development literature also describes the whole child approach to cultivate the students intellectual, social, and emotional well-being in order for them to achieve their full potential. (Huang, D. et al, 2008).2. Programs should be conducted under a positive environment that encompasses safety, student well-being, adequate physical space, and positive relationships. Peer mentoring should also be included in promoting a positive environment.3. Student input should be included in the evaluation process as programming should be tailored to students various interests. 4. A framework for a longitudinal study should be developed to track the long-term learning effects of programming and use evaluation results to better the program. These results should be tracked consistently and made readily available so as to provide for the most efficient and effective examination of program progress. It appears this will be a focus of the Governors P-20 STEM Task Force implementation plan.5. Programs should place an emphasis on community and family partnerships as these are critical in the success of any program. With open communication between families and the community, stakeholders will feel more comfortable engaging with the staff to ultimately enhance the overall objectives and outcomes of the program.6. Program staff must be highly competent in the subject field as well as motivational to the students. Principal recommendation and an interview process would ensure the placement of a highly qualified and motivating teacher. Appropriate compensation can assist in reducing the turnover rate of high-quality program teachers and staff.County STEM Coordinators/PrincipalsState and county STEM coordinators and local principals were given the opportunity to comment on current out-of-school programming opportunities available in their specific school or county to include general instructional practices, activities and assessment procedures. All of the surveys were conducted either in person, by telephone, or using online interviews based on the surveys developed by the STEM project team. (See Appendix B) The survey was designed to identify programs available in central Maryland counties, level of teacher involvement, curriculum, and assessment and evaluation procedures. Of the six districts of central Maryland that were specifically targeted and contacted via multiple phone calls and email attempts, only three returned the solicitation. These three representatives included a county STEM coordinator, representatives from a local Title One program, and several lead teachers from a local STEM high school. In general, the interviews lasted approximately two hours covering the contents of the survey along with brief overviews of programs available in each site. It should also be noted that a brief presentation was delivered at a bi-annual state STEM coordinators meeting on October 20, 2009. The surveys that the project team had developed were handed out with self-addressed stamped envelopes along with a free five day pass to any local Y Health & Wellness facility. Of the thirty five surveys that were handed out, the STEM project team received zero responses.Programs Through literature research and the interviews conducted with several representatives, it was determined that there is a multitude of programs available in the central Maryland region in STEM programming for during the school day, out-of-school, and summer programming. Many of these programs are happening during the school year with the ultimate goal to influence more students to enter the sciences. Most of these programs are part of the extended day programming available at local schools, with several available as part of Saturday morning academies. Some of the counties indicated that their programs are strictly run by their local Parent Teacher Associations (PTA). In general, programs meet one day per week for approximately fifteen twenty weeks with a culminating competition. The percentage of those students in STEM activities that actually attended the competitions was not available. Teacher involvement STEM programs at various schools are started in many different ways and have various methods of selecting who is in charge on running the program. Through the interviews, it was determined that most of the teachers in the areas examined are selected by the administration based on the skill content required by the designated STEM activity. The activities themselves are selected mainly on the recommendation from the local jurisdiction but did not exclude a personal interest from a teacher to conduct their own activity. Salary was categorized as a major issue in obtaining teacher interest and involvement. Generally, staff receives a salary of approximately $25 per hour or they receive a yearly stipend. The administration reiterated the need to recruit teachers that were highly motivational as well as knowledgeable in their subject field for out-of-school programs. They also reiterated the need for quality student-teacher relationships that foster trust and exemplify excellent character traits. Curriculum and professional development Professional development for the out-of-school environment is erratic at best. The training opportunities and requirements range from zero to one or two week summer workshops. The preferred method of training is through in-house professional development. Overall, the type of professional development offered was in the subject areas of math or science with very little addressing self-evaluation or assessment. Almost one hundred percent of those interviewed indicated that there was little to no correlation from the out-of-school programming to the school day curriculum. There appeared to be an overwhelming feeling that out-of-school activities are strictly viewed as extracurricular and not academic. Assessment and evaluation The general consensus among districts was that internal methods of evaluation were the most accepted form of evaluating any program. These evaluations generally took on the form of a basic pre/post assessment that was usually identical in form. External evaluations were felt not to be cost effective to programs and therefore are not used by the majority. The only programs that were identified to utilize external evaluations were those that were specifically grant funded and had external evaluations as a component of the grant. A specific example of such a program would be Project Lead the Way at both the middle and high school levels. Recommendations Based on the limited interviews conducted, the STEM project team has developed several recommendations as follows:1. There is an overwhelming need to centralize all findings and efforts from each district. Each of the districts appears as a separate entity with little to no collaboration throughout the state leading to the constant recreation of programs and the so-called recreation of the wheel. To simplify, and ultimately better achieve the national goal of producing more United States citizens entering STEM career fields, there needs to be collaboration and cooperation at the county, state and national level rather than the current competitive and secretive attitude encompassing STEM programming today. Competitions for STEM grant money has been indicated as one of the many reasons for this separation and privatization of information. The lack of surveys being returned may be a key indicator of this, keeping in mind that surveys took a maximum of twenty minutes to complete. (It should be noted, as well, that while each county was required to submit an evaluation report to the Maryland State Department of Education of their current STEM program in order to justify funding, the STEM project team was unable to obtain access to all but one of these reports to use strictly for research purposes.) In order to ultimately affect the students and achieve the ultimate outcomes of these programs, a collaborative and cooperative effort is a necessary fact. This recommendation correlates to Recommendation Seven of the Governors P-20 STEM Task Force report for creating a STEM innovation network and bringing together all STEM education stakeholders. (2009).2. Teachers and administrators should have greater opportunities to be exposed to the multiple STEM activities available, specifically at the elementary and middle school levels. 3. While the just have fun attitude has been acceptable and desired, a need has developed for a rigorous, replicable and effective assessment and evaluation framework to be developed and initiated at all levels to ultimately affect the outcomes of the programs. Without such a framework, programs will not be able to improve nor will they be able to share their successes with others.4. Most assessments currently are individually teacher generated. There are some programs, e.g. MD Engineering Challenges, that have developed evaluations per activity, but they are still very basic and mostly attitudinal. Quantitative data, e.g. MSA scores, report cars, etc., need to be collected and followed in order to begin and maintain a longitudinal study to determine the effect of programs, long-term, and determine which ones are truly the best practices out there.5. Mentorship has been indicated as a key issue as well, leading to the STEM project team proposing an element of mentorship to each program enabling the enhancement of students self-esteem and promoting quality leadership skills. In the development of a program, the project team has proposed incorporation of the Carson Scholars Alumni as peer leaders/mentors in each of the programs implemented eventually through the Y.6. Finally, there is a concern that these culminating competitions that are held at the end of each year may be promoting the lack of cooperation among the districts. At some venues, it has been proposed and recommended that these competitions be turned into showcases that give students an opportunity to demonstrate their project, but not necessarily have to compete for a prize. This may assist in preventing some self-esteem issues among children with fewer resources to have to be involved in a winner or loser environment. Site Staff The site staff survey was developed to examine key indicators of program environment, instructional features and program organization identified as indicators of high quality out-of-school programs by the National Partnership for Quality Afterschool Learning (NPQAL). The survey included items on health and safety, code of conduct, staff/student relationships, overall program climate, activities and goals, academics and enrichment, instructional methods and strategies, personal responsibility, self-direction and leadership, collaboration with day school, staff support, and staff experience and training. In addition, a series of preference questions were asked, through in-person interviews, to determine areas of best practices and concerns in afterschool programming. Interviews were conducted with a total of sixteen teachers/site staff members representing five districts to include Anne Arundel County, Baltimore City, Baltimore County, Howard County, and Queen Annes County in a variety of settings which included focus groups, in-person interviews, phone interviews, emails, and conferences. Interview sessions usually lasted about one hour. (It should be noted that the STEM project team attended the Technology Educators Association of Maryland conference on October 16, 2009 in Howard County and handed out approximately thirty site staff surveys with self-addressed stamped envelopes and a free five day pass to any local Y Health & Wellness center, to teachers from multiple districts. Of the thirty that were delivered, only one was returned.)Program environment As part of the analysis of the program environment, site staff were interviewed on questions relating to the programs having a safe environment, providing for the student health and well-being, having a well-equipped and suitable physical space, and promoting positive relationships. Figure 1 illustrates the benchmarks and indicators for this program component.

Program Environment

Physical SpacePositive RelationshipsHealthSafety

Staff: student relationshipStudent: student relationshipStaff: staff relationshipSufficient spaceProper layout of spaceProvision of multiple activitiesClean & SecurePrevention strategiesProper supervisionsPromote healthy habitsNutritious snacksMinimize health risksSafe equipment

Figure 1. Indicators for program environment (Huang, D. et al, 2008).The program environment focuses on how the structure of the afterschool program creates an atmosphere conducive to positive academic achievement and self-esteem for youth; they are attractive, affective contexts for youth development. The four main elements of the program environment, which are consistently referenced by research, include (a) safe environment, (b) student health and well-being, (c) well-equipped/suitable physical space, and (d) positive relationships. (Huang, D. et al, 2008). (See Figure 1 above.) In order to maintain a safe program environment a program must have a system in place to keep unauthorized people from taking students and provide proper security measures as need as well as provide appropriate supervision of the children. (Huang, D. et al, 2008). Eighty percent of those interviewed verified that students are always under adult supervision and one hundred percent agreed that each of the programs was operating in a site with sufficient security. Eighty percent also agreed that there was an effective system in place to keep unauthorized people from taking students from the program. As part of providing a safe and productive learning environment, it is necessary to have a well established program code of conduct that is easily identified and well understood. Ninety percent of the programs interviewed indicated that their program has established clear participation and attendance expectations for students while eighty percent of the programs stated that their staff appropriately and consistently applies rewards and consequences for student behavior. One hundred percent of the site staff agreed that their students understood the rules for behavior. The emotional climate of the program environment is characterized by warm, supportive relationships between the staff members and students, among the students themselves, and between staff members. (Huang, D. et al, 2008). Out of the sixteen interviews conducted only twenty percent indicated that their program(s) lacked clear written guidelines about expectations for staff-student relationships. One hundred percent of the interviewees agreed that the activities help to foster better relationships between the staff and students. Eighty percent agreed that students generally interacted with each other in positive ways.The overall program climate among high quality out-of-school programs should not only offer and safe and healthy environment, but a nurturing environment for all participants as well. (Huang, D. et al, 2008). The overall program climate of the programs interviewed provided healthy feedback indicating that one hundred percent of the students enjoyed in the programs in which they participated. Eighty percent strongly agreed that the programs facilitated students self-confidence. Sixty percent of those interviewed agreed that acceptance of alternative viewpoints was encouraged at the programs while another sixty percent strongly agreed that staff built respect for differences among students.Instructional features While many of the out-of-school programs vary in their area of emphasis, there is ongoing reference in instructional research to several key elements of instruction, including (a) activities and program goals, (b) academics and enrichment, (c) instructional methods and strategies, (d) personal responsibility, self-direction, and leadership, and (e) collaboration with the school day. Employing a variety of research-proven teaching and learning strategies can help staff members to increase engagement among students with different learning styles. Furthermore, a failure to design activities that meet the needs and interests of students may result in reduced program attendance. (Huang, D. et al, 2008). One hundred percent of the site staff interviewed agreed that activities were offered that reflected the different learning styles of the students to include styles such as auditory, kinesthetic, etc. Sixty percent of those interviewed also agreed that the activities reflected the interest of the students in the program.The next benchmark provided for whether the programs provided a balance between academics and enrichment. (Huang, D. et al, 2008). One of the key indicators identified for providing a balance in academics and enrichment was to determine if the students were given an opportunity to socialize, of which one hundred percent were. Research has also supported that students should be provided with opportunities to develop critical thinking skills, of which eighty percent of the programs strongly indicated was a part of their program. The last element in establishing balance is in the program addressing the physical needs of the students, of which eighty percent, again, strongly agreed that their programs addressed this dimension. The following table indicates the different types of activities offered through the programs interviewed.Table 2: Statistics on Activities OfferedActivityPercentage Offering Activity

Math100%

Science80%

Technology80%

Engineering60%

Study Skills20%

Tutoring40%

Physical Fitness20%

Instructional methods and strategies were examined over the course of these surveys to identify what was being taught in the programs and how the information was being delivered. Of those interviewed, eighty percent strongly agreed that the activities in their program were designed to take into account students strengths and weaknesses. Eighty percent also strongly agreed that the instruction is differentiated to meet the needs of all students, including those who have special needs. Twenty percent disagreed on this question. Finally, eighty percent of the site staff strongly agreed that students were given multiple opportunities to develop new skills. Site staff was then polled on what different types of instructional practices they utilized in their out-of-school programming and were given a list of strategies to check off on. The following table indicates the results.Table 3: Percentage of Instructional Practices UsedInstructional PracticePercentage

Project-based learning practices (students work on projects spanning several days)80%

Experiential learning (opportunities for hands-on experience, reflection an action)80%

Utilizing real-world contexts80%

Teaching math using learning games80%

Providing students with opportunities to complete math projects40%

Using technology or computers as a tool to teach math80%

Having students complete hands-on science investigations40%

Having students practice scientific inquiry40%

Having students work in small groups80%

The next benchmark analyzed whether the program provides opportunities for development of personal responsibility, self-direction, and leadership. (Huang, D. et al, 2008). There were four indicators that were identified as critical for fulfilling this component including: 1) The program highlights student work through activities such as displaying student work; 2) Enables participants to develop life skills, resiliency, and self-esteem via activities; (Huang, D. et al, 2008). 3) Program allows for students choice and self-direction; and 4) Program promotes the development of leadership skills. Of those surveyed, sixty percent agreed while forty percent strongly agreed that the program highlights students work in various ways. Sixty percent also agreed that the activities promoted self-esteem, self-direction, as well as leadership abilities for the students. The following table lists percentages of ways students participated in the programs interviewed.Table 4: Types of Student ParticipationTypes of Student ParticipationPercentage Participating

Providing feedback about the activities offered in the program60%

Suggesting activities for the program60%

Helping staff to plan activities20%

Playing a leadership role during activities60%

Reflecting about their experience in the program20%

Acting as peer mentors60%

The final benchmark in the program component of instructional features examines the collaboration of the out-of-school program with the school day curriculum. This benchmark ad two key indicators that were examined including, 1) the maintenance of a flow of information between the out-of-school program and the school day teachers; and 2) whether the day school and out-of-school programs share staff development opportunities. While the majority (sixty percent) did indicate that there was some maintenance of flow of information between the day school and out-of-school programs, a disturbing forty percent indicated that they were in disagreement with this statement. Only forty percent of those interviewed agreed that staff development opportunities were shared among day school and out-of-school staff.Program organization For the program organization, two key benchmarks were extracted from the synthesis of literature: (a) Staff Support, and (b) Staff Experience and Training. Three key indicators were identified for the benchmark of staff support based on a synthesis of literature through studies of the CRESST report on best practices in out-of-school. These three indicators included (a) Staff is well paid; (b) Staff is provided with performance feedback and an opportunity to provide their own feedback; and (c) Staff receives an orientation before working with youth. Regretfully, eighty percent of the staff disagreed with being paid well for the work they complete in their programs. Many of the staff only received payment for hours instructing versus receiving compensation for planning time as well. Only twenty percent strongly agreed that they were provided with a written job description outline their responsibilities. Again, only twenty percent strongly agreed that they were given preparation time for their program. Even more disturbing than the previous statistics was that sixty percent of those that responded indicated that they disagreed with the statement that with the current staff: student ratio, staff is able to give sufficient attention to all students. Finally, another forty percent indicated that there was not a plan in place to provide adequate staff coverage in case of emergencies or staff absences. The following table was meant to examine professional development opportunities available to staff of out-of-school programs and determine the availability of different types of programming.Table 5: Types of Professional DevelopmentTypes of Professional DevelopmentPercentage Offered

Promoting the safety, health and nutrition of youth40%

Promoting students self-esteem20%

Working with families40%

Designing activities that support program goals40%

Planning activities that support students developmental assets20%

Working with English Language Learners0%

Content specific training (e.g., math, science, etc.)60%

Assessment20%

Classroom management and discipline0%

Other0%

No professional development was offered0%

Preference questions Preference questions were designed by the STEM project team, based on a synthesis of literature, to determine best practices as viewed by the staff directly involved with the children and the programming. Nine questions were asked ranging from preference for time of day for programs to obstacles that would inhibit staff members from teaching and/or learning to benefit the learning of the children. The goal of these questions was again to determine best practices in programming with a future goal of allowing other programs to utilize these best practices in developing new and innovative programs in STEM. The following table illustrates the criteria indicated by the site staff surveys on best practices in out-of-school programming.Table 6: Site Staff Preference QuestionsTopicAnswer

1. Best time of day for out-of-school programmingRight after school and ending with activity bus schedules

2. Willingness to commit to two paid days per week for one and a half hours Yes, if the salary were higher than $25 per hour; 6 week sessions were considered more favorable than year-long programming; teachers do not want a program that is identified more as day-care

3. Willingness to work Saturday mornings Yes, although time of year would be a factor due to competition with school sports. The idea of running this type of program in January or February was mentioned.

4. Willingness to attend a workshop to discuss ideas on afterschool programming in STEM100% agreed to be willing to attend a paid workshop

5. Obstacles to teaching math or science to benefit the learning of the children80% agreed there would be obstacles specifically in the areas of funding including salaries and materials of instruction; graduate classes that teachers would be taking, and; teacher content including teacher quality, availability and allotted planning time

6. Involvement in collaborative meetings/activities outside of the school environmentYes, if the principal allows, although it was rare prior to MSA testing. Some had worked with churches and/or youth groups and attended conferences. Some had also attended a Northrop Grumman summer academy as well as trainings at Booz Allen, NASA, and Lockheed Martin.

7. Knowledge/introduction to STEM education80% indicated that they had been exposed in some way to STEM education although most elementary teachers indicated that there was very limited exposure to STEM, although it is increasing. Many of the teachers could relate very well once they were exposed to the STEM concerns in the country.

8. Use of cooperative learning strategies in the classroom80% said they used vertical teaming between the high school, middle school, and elementary school

9. Experience in co-teaching environment100% said they had worked in a co-teaching environment, but mainly with special education teachers

Recommendations After conducting the sixteen different interviews with site staff members at various programs, the STEM project team proposes the following recommendations in regards to site staff:1. When conducting afterschool programs, the highest attendance rates will be gained by running programs immediately after school beginning in late October and commencing early April. This timing indicator is determined based on activity bus scheduling and fall and spring sport schedules. Saturday programming should be scheduled on a case-by-case basis determined by student interest levels. 2. In order to obtain and retain the most qualified and motivating site staff for out-of-school programs, salary should be determined on a per diem basis rather than a strict hourly rate. Site staff should be compensated for planning time as well as instructional time. In addition, principal recommendations along with personal interviews should be utilized during the selection process of out-of-school program leaders. This recommendation correlates back to the Governors P-20 STEM Task Force report second recommendation for enhancing the STEM preparation and aptitudes for elementary and early childhood teachers. (2009.)3. One hour to one hour and forty five minutes, two times per week for six weeks is the recommended time and length of out-of-school programs operating during the school year.4. The physical location of the out-of-school activity must be in a safe atmosphere and conducive to a good learning environment.5. All activities need to be interactive and inquiry based allowing for experiential learning for ALL students. 6. Student: site staff rapport should be a high priority.7. Communication and feedback between out-of-school program and the day school should be regular and consistent to enhance the programs and open them up to a wider audience. 8. Family involvement is highly recommended as it will enhance any program, regardless of the role of the family member.9. Partnerships are strongly encouraged with local churches/Ys/schools/businesses/etc.10. Staff development should address the cooperative teaching model between STEM experts and the classroom teacher.Parent InterestA parent interest survey (see Appendix D) was distributed to parents of children ages five through seventeen that were currently affiliated with the Y of Central Maryland, either through membership or program participation, to gauge their overall interest in a STEM program. The STEM project team employed two different methodologies, including online and hard copy surveys, to gather the broad base of data that was needed in establishing credibility of the survey results. The results of the two surveys were then merged and analyzed in an aggregated format. It should be noted that all questions pertaining to the childs area of interest (hobbies) and job goals, were answered by the parent and not directly by the child. Out of a total universe of approximately 7800 member and program participants, the STEM project team received 650 total responses or 8 percent. The breakout based on the childs grade level is as follows:

Table 7: Sample breakout of child's grade levelGrade LevelPercentage

3rd grade24%

4th grade24%

5th grade25%

6th grade14%

7th grade9%

8th grade6%

Note: For data reliability, the overall goal was a completion of 366 respondents at a 95% confidence level with a confidence interval of five points. While the results fell slightly short of this goal, the data is reliable at a 94.6 confidence level with a five point confidence interval.

Findings Overall, as the following figure indicates there is sufficient interest/commitment among parents surveyed to participate in such a program.

Figure 2: If an extended day learning opportunity were offered at your child's school, would you commit to two days per week for six weeks?

The next figure, although solely based on response rate by location, highlights center/regional locations where there is strong commitment/interest. In this case, Dancel, Ward and Catonsville appear to be the areas where interests is greatest and suggests possible locations for a pilot program (demonstration).

Figure 3: Interest/Commitment level by location

When asked about their childrens hobbies, the diversity indicated appeared to be as diverse as the children themselves. As the following table highlights, reading, playing video games and playing sports are at the top of the list. Although it can be argued that all these hobbies have either a direct or indirect correlation to math and science (i.e. Art = imagination - teaches one to think outside the box), one-third of parents indicate their child is interested in Electronics/Model Building, which is encouraging for a future in STEM education.Table 8: Child's Area of Interest (Multiple Mentions)Childs Area of Interest% (n=321)

Reading46%

Playing Video Games43%

Playing Sports39%

Art (drawing/design/painting/graphic arts/photography)38%

Music (Listening to)37%

Watching TV37%

Music (Playing an Instrument)32%

Board Games31%

Puzzles29%

Card Games20%

Writing18%

Theatre Arts (Acting/Dancing)17%

Electronics15%

Model Building (Airplanes/Erector Sets/Boats/Characters/Etc.)14%

Photography12%

My child has not interests or hobbies0%

Other27%

It appears that the childs interests, in-terms of careers they are currently interested in, are quite varied, with teacher/professor at the top of the list. One-third of respondents indicate professions that require a great detail of math and science related courses ranging from the medical field to the engineering/science related fields.Table 9: Childs Career Aspiration (Parent's Perspective)Career Aspiration% (n=385)

Teacher/Professor17%

Dentist/Doctor/Nurse14%

Veterinarian12%

Scientist/Biologist/Zoologist/Chemist/Archeologist9%

Engineer/Inventor7%

Law Enforcement/Police Officer6%

Professional Athlete5%

Actor/Actress/Singer/Model4%

Artist4%

Fire Fighter4%

Lawyer3%

Singer/Model3%

Astronaut2%

Chef2%

Fashion Designer2%

Architect1%

Author/Writer1%

Train Engineer1%

Carpenter1%

Military1%

Music Teacher1%

Waitress1%

Parents were also polled on the level of their childrens interest in discussing and learning about science related topics as well as asking the why on how things work. Overall, as the figure below indicates, the majority of parents indicated that their child(ren) had an interest in this area.Figure 4: Child's Interest Level in Science Topics (Parent's Perception)

Parents were then polled on their childs interest level in math. They indicated that their childs interest in math is about a one-third less strong than their interest in science appeared to be.

Figure 5: Child's Interest Level in Math (Parent's Perception)

Insight RecommendationsAlthough the Y of Central Maryland has a limited before and after school programming presence within Ellicott City, there still appears to be strong interest within this area. The sample for this market is largely from members and program participants with age appropriate children and not from school aged services program at Owen Brown (sixteen respondents from this area). Also indicated through the parent survey is that children at this age do not appear to make the connection between math and science. Any program that hopes to be effective in ultimately affected the overall interest level and entrance in STEM careers must incorporate all elements of STEM and clearly show how mathematics and science are interrelated in real-world, hands-on experiences. Based on the survey responses from the Y of Central Maryland parents, there is sufficient evidence to proceed with a STEM initiative, as there was a great deal of interest/commitment expressed among Y members and program participants. The Recommendations for Best PracticesTo complete the focus of phase one of this project, the STEM project team, through research, literature review, interviews, and survey compilation, studied and developed recommendations based on feedback from program directors, STEM county coordinators, principals, site staff, and parents alike. The team then reached a broad consensus on the most essential elements of best practices in out-of-school STEM programming, grounding its recommendations in evidence based research reports and analysis of local, state and national data. The STEM project team offers the following six recommendations for best practices in out-of-school STEM programming: 1. Align out-of-school STEM curriculum with elementary, middle, high school and college requirements, as well as with workplace expectations in order to prepare ALL students for postsecondary success.2. Ensure that all program staff has the knowledge, skills, and abilities, especially in mathematics and science, to help all students successfully complete the elementary, middle and high school, college, and career-ready curriculum.3. Provide adequate compensation for both planning and instructional time for staff.4. Incorporate a plan for engaging youth with STEM professionals.5. Establish a robust evaluation framework utilizing a mixed methods approach of measuring qualitative, quantitative, attitudinal, and longitudinal data. 6. Create a STEM Community of Practice to make STEM out-of-school resources available to all STEM stakeholders.Recommendation 1:Align out-of-school STEM curriculum with elementary, middle, high school and college requirements, as well as with workplace expectations in order to prepare ALL students for postsecondary success.

According to Achieve, Inc.,In the last decade, research . . . has shown a strong convergence in the expectations of employers and colleges in terms of the knowledge and skills high school grads need to be successful, especially in English and mathematics . . . Being college and career ready ultimately means that students are prepared for their next steps, that all doors remain open to them as they continue to pursue their education and their careers. (Achieve Inc., 2009.) According to the STEM Task Force Report, the Bohanan Commission in Maryland has forcefully supported the adoption of a statewide P-12 school day curriculum that is aligned with global and workforce standards. The curriculum should have a strong emphasis on STEM; should provide a seamless transfer into post secondary education; and should include a definition of standards for teaching, writing, mathematics, and science. (2009) As such, the STEM project team is committed to taking this a step further by proposing to develop out-of-school curriculum that is directly aligned with a statewide STEM curriculum that prepares students for their post-secondary success. In addition, the STEM project team has recognized the need to supplement the school day STEM curriculum by providing a variety of activities in the out-of-school environment that cultivate developmental assets and provide hands-on, real-world learning experiences. According to the Search Institute, developmental assets are the building blocks for raising healthy children and youth. Since its creation, the Search Institutes Framework has become the most widely used approach to positive youth development in the United States. (2009.) In providing programs that cultivate these developmental assets, programs can provide for not only technologically capable students, but for positive youth development as well. Providing hands-on, real-world learning experiences for the students is an invaluable criteria of program development as well. These types of experiences allow for children to connect the theoretical experiences they learn in the classroom with the practical requirements of daily work in the field. Experiences could include field-trip opportunities, co-op experiences, and/or opportunities to work with subject matter experts brought into the out-of-school environment. Recommendation 2:Ensure that all program staff has the knowledge, skills, and abilities, especially in mathematics and science, to help all students successfully complete the elementary, middle and high school, college, and career-ready curriculum.Research has established that the quality of P-12 mathematics and science teaching is the single most important factor in improving students mathematics and science achievement. (STEM Task Force, 2009.) There have been several national reports that have pointed to the importance of improving the content-knowledge and pedagogical skills of all P-12 mathematics and science teachers, including those involved in out-of-school programming. Rising Above the Gathering Storm emphasized that there is a solid link from teacher excellence to student interest and performance in mathematics. (2006). Two out of three of the actions submitted by the report refer to the preparation of teachers of mathematics and science for grades P-12 as well as to the professional development of existing teachers in these fields for grades 6-12. According to research in An American Imperative, effective teacher preparation programs are reviewed and revised on a regular basis. They should require an undergraduate degree in the content area to be taught, and should emphasize development of age-appropriate pedagogical skills built on continuing research on how students learn. The programs also need to be aligned with state curriculum and include supervised classroom experience. (2007.)As such, the STEM project team proposes to develop a plan for obtaining highly qualified staff for instruction of out-of-school STEM programs and to develop a professional development program that allows for emphasis of pedagogical skill development, alignment with state curriculum, and supervised out-of-school program experiences. Recommendation 3:Provide adequate compensation for both planning and instructional time for staff.As the STEM project team conducted their series of interviews and surveys among multiple programs and staff members in the central Maryland region, a central theme recurred of out-of-school staff not receiving adequate payment for their time. In general, all instructional staff in out-of-school programs received compensation on an hourly basis for time spent in the classroom. Staff members did not receive compensation for planning time, although many indicated spending an equal amount of planning time for programs as they did on instruction. This has resulted in a lack of interest among potential site staff to participate in additional, if any, out-of-school programs. The STEM project team proposes to develop a compensation plan for site staff that incorporates both planning and instructional time for out-of-school STEM programs. This plan will incorporate a pay scale of allowing a half hour of planning time for every hour of instruction. When this plan was introduced to staff members of current out-of-school STEM programs it was overwhelmingly received as a pro-active approach to obtaining and retaining quality staff members.

Recommendation 4:Incorporate a plan for engaging youth with STEM professionals.The role that professionals play in engaging with youth around STEM and workforce development is one of immense importance. Programs, such as the Baltimore Museum of Industrys Engineering Challenges, utilize a mentorship role by engaging engineers from the Engineering Society of Baltimore to provide mentoring opportunities to students involved in the challenges. (The Baltimore Museum of Industry, 2010). This, in turn, gives the students an opportunity to work with professionals that are in the field giving the children a positive role model to look up to.As part of the pilot program that the STEM project team proposes, a mentoring/leadership role will be developed to provide opportunities for STEM professionals to become role models for elementary and middle school students. This part of the program will also utilize alumni of the Carson Scholars program as peer mentors working along with the STEM professionals to provide positive, motivating experiences for all children involved and interested in STEM.Recommendation 5:Establish a robust evaluation framework utilizing a mixed methods approach of measuring qualitative, quantitative, attitudinal, and longitudinal data. A central theme that arose during the research of current programming is the lack of a robust evaluation framework to indicate effective STEM out-of-school programs in the central Maryland area. Evaluations are a necessary component of any program in order to provide for ongoing evaluation and self-assessment to ensure program effectiveness and continual program improvement. A multi-method approach will allow for an overall review of program quality from the students perspective to the site staff and program directors perspective. Also important to consider is the parental perspective, as involving parents in out-of-school programming is an essential component to any successful programming. The STEM project team proposes to develop a robust and all-encompassing evaluation framework that is easily replicable to various forms of out-0f-school programs. This framework will utilize a multi-methods approach and include qualitative, quantitative, attitudinal and longitudinal data. This framework will ultimately assist in providing for a method of evaluating the overall effectiveness and quality of out-of-school STEM programs. Recommendation 6:Create a STEM Community of Practice to make STEM out-of-school resources available to all STEM stakeholders.Comprehensive physical and virtual STEM networks are becoming a common investment among many states allowing for the promotion of collaboration among all STEM education stakeholders. These stakeholders include P-12 teachers, high education, business and community leaders, along with government and homeland security leaders as well. A best practice identified among the STEM networks in the North Carolina STEM Community Collaborative. While still in developmental stages, the community has engaged regional and national networks of innovation and policy for the purpose of developing and implementing a sustainable STEM education-workforce-research-economic development strategy for the state. (Report: Investing in STEM, 2009). The Maryland Business Roundtable, as part of the STEM Task Forces recommendations to the governor, is developing a STEM innovation network that will have established very well-aligned partnerships. They have realized that there are pockets of excellence that exist in schools, school districts, and college and universities, but unless and until those bright spots are linked into a synergistic network, the state will continue to miss opportunities to build on our success. (Report: Investing in STEM, 2009). The STEM project team proposes to develop an augmented approach to the STEM innovation network by developing a Community of Practice that is specifically targeted to STEM education stakeholders in the central Maryland region. This network will provide for both a comprehensive physical and virtual network allowing for a more intimate approach to targeting all stakeholders in the immediate area. SummationPublic education is potentially our countrys most valuable asset, yet our system has too long ignored the development of critical teaching and workforce skills. (National Academies, 2006.) The STEM project team examined a number of out-of-school programs for STEM education and identified key indicators of best practices in out-of-school STEM programming. In addition, the study was able to identify a few out-of-school STEM programs that could be identified as best practices based on the key indicators identified prior in this report. The main tasks identified and accomplished in this study were to: Construct a searchable database of STEM activities in central Maryland and identify what is out there that is working already, what gaps exist, and how these programs could be tailored, and Compile a database of best practices at the national level, especially in the areas of assessment, evaluation and curriculum development.Under these guidelines, a synthesis of literature was conducted and key benchmarks and indicators were established to identify high quality programs in out-of-school STEM. These key indicators and benchmarks can be further developed to assist program directors, site coordinators, and site staff in assessing and evaluating their current programs and augmenting them to provide for continuous self evaluation and improvement. With the current discussion and concern with STEM education and the future of our country in regards to STEM careers, data and evidence-based programs are essential for establishing and assessing program outcomes to improve program quality. When programs are consistently evaluated and improved, they can yield the most benefits for their student populations and garner more support from local, state, and federal constituencies. (Huang, D. et al, 2008).

ReferencesAchieve, Inc. (2009). Closing the expectations gap: Fourth annual 50-state progress report on the alignment of high school policies with the demands of college and careers. Washington, DC: Author.Bayer Corporation. (2006). Planting the Seeds for a Diverse U.S. STEM Pipeline: A Compendium of Best Practice K-12 STEM Education Programs. Author.Business Higher Education Forum. (2007). An American Imperative: Transforming the Recruitment, Retention, and Renewal of our Nations Mathematics and Science Teaching Workforce. Washington, DC: Author.C.S. Mott Foundation Committee of After-School Research and Practice. (2005). Moving towards success: Framework for after-school programs. Washington DC: Collaborative Communications Group.ERIC Development Team. (1998). Urban after-school programs: evaluations and recommendations. ERIC/CUE Digest, 140. New York: ERIC Clearinghouse on Urban Education.Huang, D. (2001). An after-school evaluation system for middle and high school programs. NAASP Bulletin, 85(626), 45-61.Huang, D., La Torre, D, Harven, A., Huber, L.P., Jiang, L., Leon, S., et al. (2008). Identification of Key Indicators of Quality in Afterschool Programs. Los Angeles, California: National Center for Research on Evaluation, Standards, and Student Testing.Huang, D., Cho, J., Mostafavi, S., & Nam, H. (2008). What Works? Common Practices in High Functioning Afterschool Programs Across the Nation in Math Reading, Science, Arts, Technology, and Homework A Study by the National Partnership. Los Angeles, California: National Center for Research on Evaluation, Standards, and Student Testing & the Center for the Study of Evaluation.The Baltimore Museum of Industry. (2010). Maryland Engineering Challenges. Baltimore, MD. Retrieved October 10, 2009 from http://www.thebmi.org/index.cfm/cID/630National Academies. (2006, December). Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future. Washington, DC: National Academies Press.National Governors Association. (2007). Innovation America: Building a Science, Technology, Engineering and Math Agenda. Washington, DC: Author.National Governors Association Center for Best Practices. (2007). Promoting STEM Education: A Communications Toolkit (Science, Technology, Engineering and Mathematics). Washington, DC: Author.Project Kaleidoscope. (2005). What Works: Building Engineering and Science Talent (BEST). Washington, DC: Author.Report: Investing in STEM to Secure Marylands Future: Final Report of the Governors STEM Task Force, Presented to Governor Martin OMalley. (2009, August.) Retrieved September 9, 2009 from http://www.governor.maryland.gov/documents/090806stemReport.pdfSearch Institute. (2008.) What are Developmental Assets? Retrieved November 25, 2009 from http://www.search-institute.org/content/what-are-developmental-assets

Appendix A Program Director SurveyThank you for participating in this survey for program directors. Your input is especially important for us to understand your perceptions about your after school programs organizational structure. Please be aware that your answers will be kept confidential and will not be associated with either your name or after school site in our report.

Please mark the extent to which you agree or disagree with each of the following statements regarding your after school program.

I. Program Management (planning & policy-making)Strongly disagreeDisagreeAgreeStrongly Agree

1We have a clear mission statement that explains our programs vision.|_||_||_||_|

2Staff input is considered in program planning.|_||_||_||_|

3Student input is considered in program planning.|_||_||_||_|

4We work with the school principal to promote the use of innovation.|_||_||_||_|

II. Administration and Program SustainabilityStrongly disagreeDisagreeAgreeStrongly Agree

1There is a long-term financial plan in place for sustaining the program and maintaining program quality.|_||_||_||_|

2All possible resources (e.g., community resources, funding) are used to make program affordable for all families.|_||_||_||_|

3The program budget is periodically adjusted based on resource needs.|_||_||_||_|

4There is a marketing plan in place to publicize the program.|_||_||_||_|

IV. Staff Experience and TrainingStrongly disagreeDisagreeAgreeStrongly Agree

1We have qualified instructors available to ensure high quality instruction.|_||_||_||_|

2Our staff is well-trained for work with school-age kids in the after school setting.|_||_||_||_|

3Our staff is sensitive to the cultural and social influences that impact how students learn and relate to others.|_||_||_||_|

4Our staff is competent in core academic areas, as appropriate.|_||_||_||_|

5Our instructors participate in professional development along with staff.|_||_||_||_|

6Your program directors receive training in program management.|_||_||_||_|

V. Family InvolvementStrongly disagreeDisagreeAgreeStrongly Agree

1Our program has a clearly defined plan for parental involvement.|_||_||_||_|

2Family members are welcome to visit any time throughout the day.|_||_||_||_|

3We hold orientation sessions for new families.|_||_||_||_|

4Parent input is considered in decisions about after school programming.|_||_||_||_|

5Learning opportunities are provided for parents through the after school program (e.g., ESL, computer, literacy workshops.)|_||_||_||_|

VI. Community PartnershipsStrongly disagreeDisagreeAgreeStrongly Agree

1We involve local community partners in program planning.|_||_||_||_|

2We have established partnerships with local community organizations (e.g., libraries, businesses, colleges/universities.)|_||_||_||_|

3Our program encourages youth representation in local community organizations.(mentorship)|_||_||_||_|

4We encourage our students to give back to the community through community service projects.|_||_||_||_|

VII. EvaluationStrongly disagreeDisagreeAgreeStrongly Agree

1We have an internal method for evaluating program activities.|_||_||_||_|

2We have an internal method for evaluating staff performance.|_||_||_||_|

3We have an internal method for evaluating student engagement.|_||_||_||_|

4Students academic improvement is evaluated.|_||_||_||_|

5Students social skills development is evaluated.|_||_||_||_|

6Students are surveyed to measure their interest in the program.|_||_||_||_|

7Parents are surveyed to measure their satisfaction with the program.|_||_||_||_|

8Staff is surveyed to identify areas for program improvement.|_||_||_||_|

9Evaluation findings are used to improve the program.|_||_||_||_|

10Evaluation findings are available to the general public (e.g., parents, community.)|_||_||_||_|

Appendix B STEM Coordinator/Principal SurveyThe Carson Scholars Fund in a joint effort with the Y of Central Maryland are conducting a survey of county STEM Coordinators/Principals to assist in developing programs to enrich their after school day care. Your feedback is greatly appreciated as is any input.County: Name: Email Address: Phone: Please fill in the following table with STEM programs that are currently available in your county for grades K-8. (Please check off whether the program is extended day, summer camp, or during the school day.) If this information is already available please add/email as an attachment in other formats as applicable.Program NameGrade Level (K-8)Extended DaySummer CampSchool Day

1. Is parental support sought and encouraged? Are some programs strictly run by parents?2. How are teachers, tutors and mentors selected and trained?3. How much pertinent training do program instructors have and what level of credential is required?4. Is there a correlation between what is being taught during the school day to what the children are learning in the above listed STEM activities? Yes or No5. Has salary been an issue when trying to get teachers involved and/or having teacher participate in training? Yes or No6. Are there currently any Saturday programs available in STEM? If yes, please list.7. How well defined are the outcomes of the programs listed above?8. Is there an evaluation and/or assessment program in place that monitors and provides feedback on the program? If yes, please list types of evaluation (e.g., attitudinal surveys, pre-post assessment.) Is there a contact person who does the evaluations?9. Is there a cost associated with any of these programs to the students? To the school? If yes, please list who the cost applies to and type of cost (e.g., materials of instruction, janitorial, transportation, etc.)10. How are the STEM clubs selected to be run at each school? (Do teachers initiate interest or are the schools solicited by external programs?)11. How often does each club meet? For how long?12. What, if any, gaps do you in the implementation of STEM programs in your county?

Appendix C Site Staff SurveyThank you for participating in this survey for site staff. Your input is especially important for us to understand your perceptions about your after school programs environment, instructional features, and organization. Please be aware that your answers will be kept confidential and will not be associated with either your name or after school site in our report.I. Background Questions

How many years have you worked as a staff member in this after school program? ______

II. Program Environment

Healthy and Safe EnvironmentThink about the environment in which you work. Please check all that apply:

Students are always under adult supervision. The program is operating in a site with sufficient security. An effective system is in place to keep unauthorized people from taking students from the program.

Program Code of ConductPlease check all that apply:

Program has established clear participation and attendance expectations for students. Staff appropriately and consistently applies rewards and consequences for student behavior. Students understand the rules for behavior.

Relationships among Staff and StudentsPlease mark the extent to which you agree or disagree with each of the following:

Strongly disagreeDisagreeAgreeStrongly Agree

1Program has clear written guidelines about expectations for staff-student interactions.1234

2Activities help build relationships between staff and students.1234

3Students generally interact with each other in positive ways.1234

Overall Program ClimatePlease mark the extent to which you agree or disagree with each of the following:

Strongly disagreeDisagreeAgreeStrongly Agree

1Students enjoy participating in the program.1234

2The program facilitates students self-confidence.1234

3Acceptance of alternative viewpoints is encouraged.1234

4Staff builds respect for differences among students. 1234

III. Instructional Features

Activities and Program GoalsPlease mark the extent to which you agree or disagree with each of the following:

Strongly disagreeDisagreeAgreeStrongly Agree

1Activities are offered that reflect the different learning styles of the students (e.g., visual, auditory, kinesthetic).1234

2Activities reflect the interests of the students in the program.1234

Academics and EnrichmentPlease mark the extent to which you agree or disagree with each of the following:

Strongly disagreeDisagreeAgreeStrongly Agree

1Students are given opportunities to socialize.1234

2Students are provided opportunities to develop critical thinking skills.1234

3The program addresses the physical needs of the students.1234

During the past year, which of the following types of activities were offered by your program? Please check all that apply? Math Science Technology Engineering Study Skills Tutoring Physical FitnessInstructional Methods and StrategiesPlease mark the extent to which you agree or disagree with each of the following:

Strongly disagreeDisagreeAgreeStrongly Agree

1Activities are designed to take into account students strengths and weaknesses.1234

2Instruction is differentiated to meet the needs of all students, including those who have special needs.1234

3Students are given multiple opportunities to develop new skills.1234

Which of the following instructional practices did you use during the past school year? Please check all that apply.

Project based learning practices (students work on projects spanning several days.) Experiential learning (opportunities for hands-on experience, reflection, and action.) Utilizing real world contexts. Teaching math using learning games. Providing students with opportunities to complete math projects. Using technology or computers as a tool to teach math. Having students complete hands-on science investigations. Having students practice scientific inquiry. Having students work in small groups.

Personal Responsibility, Self-Direction, and LeadershipPlease mark the extent to which you agree or disagree with each of the following:

Strongly disagreeDisagreeAgreeStrongly Agree

1Staff highlights student work (e.g., displays work, has students present work to class.)1234

2Activities promote self-esteem.1234

3Activities promote self-direction for the students.1234

4Activities promote leadership abilities.1234

In which of the following ways have students participated in the after school progra