technical proposal - office of superintendent of public instruction

PRESENTED BY: Wireless Generation 55 Washington Street Suite 900, Brooklyn, NY 11201 T: 212.213.8177

RFP Number 23 Smarter Balanced Assessment Consortium Digital Library with Formative Assessment Practices and Professional Learning Resources for Educators

Original

THIS PROPOSAL CONTAINS CERTAIN CONFIDENTIAL AND/OR PROPRIETARY INFORMATION OF WIRELESS GENERATION THAT IS EXEMPT FROM PUBLIC DISCLOSURE. SUCH INFORMATION SHOULD NOT BE USED FOR ANY PURPOSE OTHER THAN THE EVALUATION OF THIS PROPOSAL, NOR SHOULD IT BE DISCLOSED TO ANY THIRD PARTY, EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW.

© 2012 Wireless Generation, Inc. For use by CONSORTIUM only. (10/12/12)

© 2012 Wireless Generation, Inc. For use by CONSORTIUM only. (10/12/12) Page 11

Table of Contents Letter of Submittal ................................................................................................................................................ 2

Certification and Assurances Form (Exhibit A) ................................................................................................. 3 Vendor Intake Form (Exhibit E)......................................................................................................................... 8

Technical Proposal .............................................................................................................................................. 13 1.0 Project Approach/Methodology ............................................................................................................ 14 2.0 Innovative Thinking to Develop the Digital Library Technology Application ......................................... 24

2.1 Open Licensing ................................................................................................................................... 27 2.2 Highly-Available and Scalable System ................................................................................................ 28 2.3 Security............................................................................................................................................... 28 2.4 Appropriate Use Policies .................................................................................................................... 31 2.5 User Comments.................................................................................................................................. 31 2.6 Minimal Impact on State and Local Systems...................................................................................... 31 2.7 Member State Cost of Ownership...................................................................................................... 32 2.8 Varying Levels of State and Local Technology Capability................................................................... 33 2.9 System Flexibility................................................................................................................................ 34 2.10 Accessibility ........................................................................................................................................ 35 2.11 Robust Governance Model ................................................................................................................ 35 2.12 Recognition of Highly-Effective Contributors .................................................................................... 37 2.13 Moderated Collaboration Groups ...................................................................................................... 37

3.0 Work Plan and Schedule......................................................................................................................... 38 3.1 Comprehensive Development Strategy ............................................................................................. 38 3.2 Digital Library Application .................................................................................................................. 41

3.2.1 Functional Requirements........................................................................................................... 46 3.2.2 Technical Systems Requirements .............................................................................................. 47 3.2.3 Application Design ..................................................................................................................... 48 3.2.4 Application Development .......................................................................................................... 49 3.2.5 Quality Assurance and Testing................................................................................................... 50 3.2.6 Data Transfer and Input ............................................................................................................. 58 3.2.7 Application Deployment ............................................................................................................ 58 3.2.8 Systems Integration, Quality Control, and Defect Resolution ................................................... 59 3.2.9 Application Maintenance and Support ...................................................................................... 60 3.2.10 Application Hosting.................................................................................................................... 60 3.2.11 Knowledge Transfer, Transition, and Turnover ......................................................................... 61 3.2.12 Our Development Process ......................................................................................................... 61

3.3 Resource Tagging and Uploading System .......................................................................................... 69 3.3.1 Resource Tagging System .......................................................................................................... 69 3.3.2 Resource Uploading and Removal System ................................................................................ 71

3.4 Quality Criteria Policies ...................................................................................................................... 72 3.4.1 Assemble and Moderate a National Panel of Experts ............................................................... 72

3.5 Sustainable Leadership System .......................................................................................................... 82 3.5.1 Recruiting State Leadership Teams and State Networks of Educators...................................... 84 3.5.2 Robust Governance System ....................................................................................................... 88 3.5.3 Ongoing Governance Plan Implementation .............................................................................. 92

3.6 Smarter Balanced Formative Assessment Practices and Professional Learning Resources .............. 96 3.6.1 Professional Learning on Assessment Literacy .......................................................................... 97 3.6.2 Exemplar Instructional Materials............................................................................................. 104 3.6.3 Using Smarter Balanced Score Reports and Aligned Educator Resources............................... 109 3.6.4 Develop Training Materials and Training for State Leadership Teams .................................... 114


3.7 Currently Available Resources ......................................................................................................... 118 3.7.1 Comprehensive Inventory of Resources Currently Available .................................................. 118 3.7.2 Identification and Recommendation of Supplemental Resources .......................................... 127

3.8 Records of Decision Making ............................................................................................................. 128 3.9 Contractor Meetings ........................................................................................................................ 129 3.10 Project Schedule............................................................................................................................... 130

4.0 Deliverable Matrix ................................................................................................................................ 131 5.0 Outcomes and Performance Measurement......................................................................................... 144 6.0 Risks...................................................................................................................................................... 145


1.0 Project Approach/Methodology

There is no doubt that the work outlined in the Smarter Balanced Assessment Consortium (CONSORTIUM) RFP #23 is essential to the ultimate success of the CONSORTIUM’s work. If educators are not equipped with the appropriate resources and training, there is a high risk that the CONSORTIUM assessments will not be properly utilized or achieve the desired results.

The work effort detailed in this RFP involves many moving parts that must work well together to produce the high quality learning resources and training necessary for the Consortium states’ educators to drive fundamental changes in how they understand and utilize the Consortium assessments, ultimately ensuring that all students leave high school prepared for college or career.

"Technology is just a tool. In

terms of getting the kids

working together and

motivating them, the teacher

is the most important."

—Bill Gates, Microsoft

Corp. co-founder

• Wireless Generation and our partner, Public Consulting Group (PCG), are uniquely positioned to

assist the Consortium with this endeavor. For more than a decade, Wireless Generation has been responsible for fundamental, system-wide change in states, districts, and schools. Since the release of our first software products, we have been working with educators and families at all levels to drive the adoption of formative assessment and the use of data to personalize instruction. As technology tools and software have evolved, our commitment to supporting the human infrastructure that puts these tools into practice has remained one of our core strengths.

• We have the proven track record of successfully managing the delivery of large scale systems and process implementation of the scale outlined in the RFP.

• We bring years of experience developing technology solutions similar to the Digital Library and IT systems to support the scope of this RFP for diverse stakeholders within complex governance structures. Working with the largest school districts, state agencies and multistate organizations, we understand design, delivery and management in contexts similar to the Consortium.

• We have a team of instructional designers, educators, and editors whose sole function is to produce professional learning materials and instructional modules for teachers, and will provide expertise in the Common Core State Standards.

• PCG, with its experience in delivering statewide engagements across the US has a long history of helping organization develop strong and sustainable governance processes.

To elaborate further on the complexity of the engagement outlined in the RFP and, as a framework to present our understanding of the project and our approach, we present the image of this project as a factory in the graphic on the next page.


ewireless generation•

CONSORTIUM State Leadership

and Involvement

Existing formative assessment practices and

professional learning packages"

55 Washington Street, Suite 900

: Brooklyn,NY 11201-1071

T 212.213.8177 F 212.213.8749

www.wirelessgeneration.com

High Quality Learning Resources

Available to CONSORTIUM Educators

http://www.wirelessgeneration.com/


We depict the work detailed in this RFP as a factory, powered by the continuous, well governed flow of leadership and involvement of the CONSORTIUM states’ State Leadership Teams and State Networks of Educators. This factory has many processes or work streams that work in concert to produce the learning resource “packages” that get stored in the Digital Library. At the heart of this factory is a “power plant” comprised of the involvement and leadership from the CONSORTIUM states. It is this power plant that delivers learning resource “packages” to be utilized by the educator community of the CONSORTIUM states.

As illustrated below, the active, coordinated participation of the various CONSORTIUM state teams supplies the energy to this initiative. This participation is funneled through a leadership and governance system that the Wireless Generation/ PCG team will work with the CONSORTIUM to develop. It is critical that a strong governance system gather and harness the power of the states’ involvement and focus it on the areas where it is most needed.

In order to create this governance system, the Wireless Generation/ PCG team, in conjunction with the CONSORTIUM leadership will recruit the State Leadership Team and State Network of Educators (SNE). As we discuss in this proposal, we intend to do this by leveraging existing CONSORTIUM structures while collaboratively establishing new recruiting policies and procedures that SLTs can implement in each state. We provide options for both decentralized and centralized recruiting, and will work with the CONSORTIUM to determine the right schedule to taper our oversight as SLTs increase their knowledge


and experience with the process. In addition to recruiting and compensating these teams, the Wireless Generation / PCG team will lead the effort to develop and implement the required governance system that will keep the “flow of power” through the factory necessary to provide for the continuous operations of the professional resource manufacturing system for the CONSORTIUM. We believe the best structure for this phase of the CONSORTIUM’s growth is a matrix model of governance in which roles and responsibilities are clearly articulated and it is evident to all SNEs who have decision-making authority in their particular states. To set up such a structure, we will provide written guidance via an easy-to-read “handbook” that outlines the governance structure, short- and long-term goals, processes, and policies. Having this strong governance posture around the creation and management of professional learning resources will keep this factory operating well after the engagement is concluded.

All factories need a warehouse to store the output of the manufacturing process. For this project, that component is the Digital Library Application coupled with the Resource Tagging System. We recommend using the agile approach to build the IT systems needed for this effort. We are experts in agile software design, development, and full lifecycle maintenance, having successfully applied the agile process in over a dozen software products in active development and in the creation and maintenance of “custom solutions” for clients, including the development of large scale content management solutions similar to the CONSORTIUM Digital Library and Resource Tagging System. This agile approach aligns our expertise and over a decade of experience perfectly with the CONSORTIUM’s chosen style of work for the software development components of the project as indicated in the RFP.

The initial requirements for the Digital Library and Resource Tagging System will become agile development stories which will drive initial mock-ups and early iterations of functionality for testing, and


allow for progressive refinements through successive development sprints with regular demonstrations to continually accept feedback from the CONSORTIUM for refinement. Iterative development allows for a broad set of stakeholders and allows for a seamless feedback process designed for efficient and effective feedback collection, analysis, and decision-making across a broad audience of reviews and reviewers. It also allows for natural incorporation of applicable feedback to adjust the technology in future iterations. Because this process is our standard development methodology, one in which we have focused our extensive experience and expertise, we can naturally support a broad audience including CONSORTIUM members and external stakeholders.

Prior to any “package” getting into the Resource Tagging System and Digital Library warehouse, it must pass through a Quality Control function. In order to provide this function, we will, within the first thirty (30) days of the engagement, convene and moderate a National Panel of Experts to develop the quality control policies that will define the criteria for the development and selection of resources for the Digital Library. We have extensive experience in preparing for and leading large events and workgroup meetings like this, having conducted focus groups and professional development sessions all over the country. We will bring this experience to bear in both preparing the content and managing the logistics for the meetings.

We will convene three face-to-face meetings with the Panel of Experts, each two days in length. Using our Taking Action with Data framework, we will work with the CONSORTIUM to construct agendas that will elicit the experience of our diverse Panelists in constructing policies for quality criteria. We will also prepare background slides to lead the discussion, and templates to lead brainstorming and sharing activities. Further details of our ideas around these meetings are detailed later in this proposal.


For this factory, the packages that hit the assembly line come from a number of sources. The first source is from the Wireless Generation led effort to inventory and harvest existing formative assessment practices and professional learning “packages” that exist in the market already. We are well positioned to create a comprehensive inventory of existing royalty-free formative assessment, instructional, and curriculum resources. Our experienced trainers, and many of our staff, are former teachers and school leaders with an average of close to two decades of experience. They are located all over the country, including in almost all CONSORTIUM states, and have taught in classrooms throughout those states. We will leverage our first-hand experience as they provide suggestions for resources culled from lists they've compiled and previously used with success. Working with the CONSORTIUM teams, based on the quality control policies developed, we will determine which of these packages should get placed on the assembly line of our factory for inclusion in the Digital Library.

The second source of formative assessment practices and professional learning “packages” for inclusion as raw materials of our factory are those that we will develop on behalf of the CONSORTIUM as part of our partnership under this engagement. To meet the requirements of this RFP, we will leverage our extensive experience to develop assessment literacy professional learning “packages” that address how to build, administer, score, evaluate, and interpret/use the data from formative, interim, and summative assessments. We will develop grade-level specific, as well as English Language Learner and Special Education customized, packages within our factory that will provide educators access to examples that are specific to their standards and templates and tools that will help them immediately implement these practices and protocols. We will also provide exemplar instructional modules and packages, again focused on each grade band for ELA/Literacy and mathematics. These packages will show exemplar applications of the Common Core State Standards (CCSS), providing resources to help teachers incorporate skills into their practice. Next, we will create packages that walk all stakeholders, including


parents/families and students, through navigation and interpretation of the relevant data on the Smarter Balanced score reports. These packages—developed by Wireless Generation and put on the assembly line with the existing, harvested packages—will be subject to the same quality assurance criteria and CONSORTIUM review before being stored in the Digital Library warehouse.

A final source of packages will be from the CONSORTIUM itself. This may ultimately be one of the most important sources of raw materials for the factory as this source will continue well after our engagement finishes. The governance / leadership process developed and discussed above will not only produce the “power plant” of the factory, but will also produce professional learning modules (“packages”) over time that should clearly be considered for inclusion in the Digital Library. The policies, procedures, and systems that are being developed by Wireless Generation and PCG during this engagement will be developed such that these CONSORTIUM developed “packages” can be put on the assembly line of our factory, in the same manner as the other packages, and be subject to the same QA and review process such that the high quality resources can be warehoused in the Digital Library.


Once the “packages” are produced, they need to be delivered to the educators in all CONSORTIUM states. The delivery mechanism is critical to ensuring that teachers, administrators, parents, and students understand the available packages and know how to best use them to meet their needs. As outlined in the proposal, we will provide a series of 2-day sessions to the State Leadership Teams (SLTs) to prepare them to work with their SNEs to roll out these new resources. We will conduct sessions in each of five locations across the country to best serve the SLTs in each state with minimal travel requirements. These sessions will be interactive, and will include the development of communication and training strategies for how to train SNEs to generate excitement for and adoption of the Digital Library and its resources.

Of course, all complex factories, as well as complex projects such as this scope, depend on a strong “foreperson” or project manager. This foreperson will look across all the inputs, processes, machining, systems, etc. to make sure everything is working in a coordinated way and not blocked from producing the outputs desired. The foreperson, or project manager for this engagement, will keep the timeline and key milestones on track to ensure that the Digital Library warehouse is successfully stocked with “packages” and learning resources for the 2013-14 school year. Wireless Generation will operate as the foreman of this factory during its construction and initial implementation and will be committed to the CONSORTIUM to support its long term operation to whatever degree the CONSORTIUM decides our involvement should be post launch.


As part of the standard management of the construction of the factory and infrastructure (timeline and key milestones), the foreman will also manage all the interactions of components and various personnel to ensure each is working effectively together to the ultimate success of the engagement. We will set up a formal project management structure and act as the single point of contact for all project management concerns of this engagement. We have proven experience implementing successful engagements such as this on behalf of school districts and states across the United States. We are 100% focused on K–12 with vast educational experience across all levels of our organization; we understand the culture of education and how to manage projects with that culture. In partnership with New York City Public Schools we launched the Achievement Reporting and Innovation System (ARIS) on time and on budget, with the initial release occurring only six months after inception. The multi-phase release approach exhibited on ARIS embraces the fluid nature of large, complex projects and allows the final solution to ultimately best meet the needs of the client.

Lastly, as foreman, we will develop the other aspects of the Comprehensive Development Strategy requested in the RFP, in collaboration with the CONSORTIUM, setting the goals of the Digital Library and communicating this to the broader stakeholder community as well as setting the measurable benefits and indicators of a successful development strategy. We are proposing that we use a monthly Comprehensive Development Strategy dashboard to provide a continuous reporting mechanism for the progress of the engagement over time. Using a dashboard approach with mutually agreed upon key performance indicators of the operations of the factory, we as foreman can continually know where to focus our energies to keep the project on track for success and the CONSORTIUM can know what areas may require increased involvement to drive the factory’s “power plant”.


In order to successfully implement and manage the Comprehensive Development Strategy, we will need to collaborate across the CONSORTIUM stakeholders. We are experienced in working with a wide variety of collaborators and stakeholders, across geographic and technological challenges and are experienced in doing so knowing that we will need to approach various areas of the project with various stakeholders in different ways. We expect to implement everything from formal status meetings to more collaborative workshops and design charettes. We value the collaboration qualities of in person meetings, but realize that virtual sessions can often be equally productive and with wider reach. In either case, we would ensure that the right people participate in key collaboration sessions. We are experienced at running webinar, video conferences and phone conferences. Our team leads understand how to organize an agenda, present ideas and gather feedback in an organized manner.

As indicated in the RFP, “Central to the CONSORTIUM’s vision for the Formative component is the active involvement of educators in the creation of digital library resources, the ongoing oversight of additions to library resources, and the training of fellow educators to maximize the use of library resources.” Wireless Generation and our partner PCG believe that this factory example illustrates that we fully understand the complexities of all the moving parts that go into positioning the CONSORTIUM to meet this vision. It is this factory that will produce the high quality learning resource packages desired as well as deliver the IT systems components to house these packages and the policy, practices and procedures necessary to ensure long term success and the high quality output desired.


2.0 Innovative Thinking to Develop the Digital Library Technology Application

For more than a decade, Wireless Generation has been responsible for fundamental, system-wide change in states, districts, and schools. With the creative application of technology and proven methodologies we have developed solutions that support our mission to improve student achievement through effective instruction in Kindergarten through 12th Grade. Building on the insights we’ve gathered from teachers, learning experts and our own experience in education technology, we have created custom solutions using a collection of proprietary software and open source technologies that not only help educators understand the levels of student achievement, and apply the appropriate instructional action that should be taken based on that achievement in order to help student master standards, but to collaborate with each other and the surrounding community to make create an optimal learning environment for their students . We will draw on this experience to provide the innovative thinking needed to address the key priorities identified by the CONSORTIUM to ensure that the Digital Library technology application meets the needs of its stakeholders and will serve the states.

We envision the overall project as being composed of three major areas:

• Overall program management and Digital Library development strategy • Overall content creation and preparation for Smarter Balanced Digital Library • System and tool development to create, host, and deliver the Digital Library as part of the

shared service platform

The systems will support the following Digital Library business processes during operation:

• Digital Library Resource Creation Process • Content Management Process • Smarter Balanced Professional Development Process

During the project, the system will also support the Digital Library programs and development strategy in the following area:

• Governance and Oversight • Overall Program Management • Sustainability Program • Implementation and Integration Process • Quality Control, Change Management and Operation Process

At the system level, we divide the system into four subsystems to support these processes:

• Library Portal (Portal)

o The overall entry point into the Digital Library


o Shared User Interface Components o Authentication and Single –Sign-On Components

• Content Creation and Collaboration (CMS) o Digital Library Tagging, Rating, Commenting o Importing and Exporting o Curating, Approving, Versioning, Collaboration, Tracking o Forum and Discussion Groups

• Content Repository (Library) o Resource Access API and URI reference o CCSS Taxonomy and Learning Registry LRMI integration o Support search through Digital Library Learning Object Repository search index and

LRMI tagged content • Advanced Content Delivery (Optional)

o Package selected Library content and resourced into Smarter Balanced course modules for either instructor-led or self-training

o Support the CONSORTIUM’s PD Courses to be delivered both online or offline through Coursera.org integration


e

wireless generation• 55 Washington Street, Suite 900

Brooklyn,NY 11201-1071

T 212.213.8177 F 212.213.8749 www.wirelessgeneration.com

Sesson and User Profile Manager

Shared Communication and Servi ces

PortalU! Configura tor WSRP or ot her portlet interface

PortaI Stylesh eet Configur ator Mobile Page (Optiona l)

Login and License Accept ance

SSO and Security Configura tion

Sma rter Balanced

Single-Sign- On

St atic Pages Smarter Ba lanced Header/

Footer

Federated Search

Localization

Digital Library Portal (Could be integrated into overall Smarter Balanced Portal)

CMS DigitalLibrary Discussion Group Library Catalog and

Personalization Authoring,Tagging and Forum Navigat ion Library Archiving Library Search

and Uploading

Dig it a l Library Course Delivery

CMS RBAC and CMSWorkflow Learning Object

Library Export Recommendation Course Content Instruction

Adminsi tration Import/Export Engine (Optional) Packaging Management

Meta-data Management

(TaxonomyTag,Map

CMS Reporting

Learning Registry

Regsi tration

Metadata and Library

Index

PO Delivery Reporting

Course Arc hiving

Certific ation (Optional)

CMS Digital Library Object Repository

Advanced Course Delivery

Data Sou rces LRPN Search/

Slice

cess Taxonomy

External Hosted Content a nd Video Library

( Youtube,WG:B;._H:_'-"e-t c".)- _

External



2.1 Open Licensing

At launch, Wireless Generation will provide a perpetual license to the Digital Library core solution, built on open-source components. As states adopt this platform, the large user base will create an environment for a healthy developer community. After full adoption, we will release the source code to the CONSORTIUM using an open-source license. The community can then modify and adopt the code, so that the platform can be enhanced for future needs.

To support the sustainability model of the CONSORTIUM, we also design our solution to interoperate with the best-of-breed open platforms. For example, beyond the core Library content repository, we also want to provide the capability for PD courses to be created and delivered in a highly scalable manner. By default, the content created would be exportable in IMS Common Cartridge format, so that the contents could be bundled into course modules and then delivered using open course learning management systems or course management platforms. For this purpose, we propose an optional integration with coursera.org, a leading online education course platform provider that partners with the top universities in the world to offer courses online for anyone to take, for free. By producing and packaging the Library content into PD courses offered on Coursera.org, the CONSORTIUM can both deliver the content much more effectively and make the long-term PD programs more sustainable than just offering a repository of content.

We have contributed to the open source and open access community on an ongoing basis and describe some examples below. The list does not include individual contributions made by employees.

Our Past Projects

We have a history of creating open-source content for educators and managing the associated digital rights of this content. In 2007, we launched FreeReading.net, a high-quality, free, open-source, reading intervention resource addressing literacy development for grades K-3. Leveraging the collective wisdom of researchers, teachers, reading coaches, and other education and industry professionals, FreeReading provides a high-quality, cost-effective alternative to static materials. By establishing a foundation of hundreds of research-based lessons and materials that users can download and use for free, FreeReading has created the framework for intervention programs supporting literacy. The activities are targeted toward skills that would normally be mastered in the elementary grades; but in the case of students who are below grade level, the collective wisdom within FreeReading is invaluable. Schools and teachers can use the research-based intervention program or the library of lessons to supplement development of phonological awareness, phonics, vocabulary, comprehension, and writing.

Additional open- licensed projects we created include:

• github.com/wgen/simpleversions: An API for dealing with version strings • github.com/wgen/rover: Rover code collector app

Additional Contributor Projects

We also contributed to the following projects:

http://freereading.net/index.php?title=Integrated_Program%3A_Intervention_A_for_K-1_%28IPID%3A_101%29

http://freereading.net/index.php?title=Find_Activities


• pep8: Style checker for python code • celery: Python API for working with message queues • beaker: Python API for caching • bcfg2: Host configuration management • pylibmc: Python API for accessing Memcached • pybreaker: Python API implementing the circuit breaker design pattern • funkload: Application for load testing • mongodb: Document-based database used by Websync

2.2 Highly-Available and Scalable System

Following the overall architectural framework laid out by the CONSORTIUM, the Digital Library is architected as a highly available and scalable platform. The main system is designed to have its primary site and disaster-recoverable site on the highly available Amazon Cloud infrastructure. The core Library repository system, which supports the long-term CONSORTIUM online operation, has a distributed cache system that can support high-volume access from all the member states and can have additional instances to support peak and expanded access need. If the CONSORTIUM determines it is in its best interest to continue the conversation and ultimately accepts our Coursera delivery approach, this online PD advanced delivery system can enhance the high-availability and scalable nature of the solution as Coursera supports from 100 to 100,000 simultaneous online sessions.

2.3 Security

2.3.1 Wireless Generation Information Security and Privacy Overview

As a provider of K–12 education technology, we understand the importance of adhering to and maintaining information security and privacy policies. Most of our customers must comply with FERPA and/or equivalent state regulations.

Our Privacy Policy sets forth a number of basic privacy practices developed to ensure that we maintain our ability to communicate with and support our customers and protect the privacy and security of educational data. We have also adopted an information security policy that strongly restricts the handling of sensitive data. We employ these processes in all of our use models to protect the privacy of more than 3 million students and more than 150,000 educators.

We employ best practices in the information security field to ensure that data is properly protected. This includes the use of SSL to secure the customer interface with the system as well as numerous other technical, policy, and training measures.


2.3.2 Information Exposure

In general, the Digital Library does not store any student or staff PII. Depending on the type and scope of a given business process, state entities, districts, or individual schools may access the following types of information through Digital Library that might have privacy implications:

• Teacher and educator profile data (names, e-mail addresses, telephone and fax numbers, grades; state, school, school district, and classroom associations);

• Observational assessment records or scores; • Comment and notes on performance/needs and instructional strategies.

2.3.3 Use and Disclosure of Information

Wireless Generation employs data that is input into our systems to provide our educational services and products and to support teachers and other education officials’ activities. For example, we may access personally identifiable information residing on our systems to support our users, to protect the integrity of data or improve the functioning of our systems, and to otherwise provide our products and services. We also may use this information to conduct other legitimate business activities, such as complying with legal requirements or with a legally permitted inquiry by a government agency or regulator, and preventing misuse of our systems.

In general, the Digital Library does not contain student PII. For staff and educators, we will follow the CONSORTIUM’s instruction to decide which third parties have access to personally identifiable information stored on our systems — for example, by granting administrator permission to access specific content management process.

Additionally, we may use the services of agents or independent contractors (e.g., computer technicians) who assist us in providing or improving our services. We bind our agents and independent contractors who may obtain access to personally identifiable information by contract to protect this information in a manner that is consistent with this Privacy Statement by, for example, not using the information for any purpose other than to carry out the educational support services they are performing for Wireless Generation and the CONSORTIUM.

2.3.4 Information Security

Wireless Generation takes reasonable and appropriate data security measures to protect information from loss, misuse, unauthorized access, disclosure, alteration, and destruction. Such measures include physical, electronic, and managerial controls. We have an executive-level information security director who oversees all information security measures at the company. We regularly conduct both internal and external audits of our information security and invest substantial resources in information security systems, training, and development.

All databases are backed up daily, and tapes are stored off-site. The central transactional database is replicated in real-time to a pair of stand-by systems for rapid fail-over. Wireless Generation servers are


hosted at a network operations center that includes tight physical security, comprehensive environmental controls, redundant connectivity, and redundant power and battery back-up.

2.3.5 Access Controls

The following table lists key elements of our information security measures:

Wireless Generation Security Measures

Access Control

Electronic Storage

Emailing (both internally and externally)

Disposal

Network and Host Security

All access requires valid credentials in the form of a username and password. Strong passwords are enforced (at least 8 characters and with a mix of letters and numbers).

Access is restricted to employees and contractors with a business need to access data. All data resources containing PII will not be allowed to be stored in Digital Library. The acknowledgement of the policy is enforced through the policy acceptance step for registration and login process.

Content Management System and Content repository of Digital Library will have Role-Based Access Control enforcement. The ACL and Role Definition will be defined by the CONSORTIUM stakeholders basing on the content and resource categorization and CONSORTIUM content policy.

Unauthorized users will immediately have their accounts disabled and lose access rights to the systems. We regularly audits access to its internal IT systems to confirm that authorized parties have the appropriate level of access.

All security related data on servers is stored in secure restricted locations and is deleted when no longer needed.

Wireless Generation does not email reports with customer PII and uses an enterprise Secure File Transfer Service instead. The Secure File Transfer Service encrypts all transactions and provides an audit trail of all files exchanged.

Wireless Generation degausses or securely deletes all electronic media with PII prior to disposal.

The Wireless Generation network is protected using firewalls and other network security measures. Devices on the network are securely configured and appropriate security patches are applied.


2.3.6 Application Security

Wireless Generation will employ various measures to provide appropriate security within the Digital Library application. The login process and all subsequent interactions will occur exclusively over SSL using 128-bit encryption. An account lockout function is triggered after a certain number of failed logins; locked accounts can only be re-enabled by Wireless Generation support staff. We hire external computer security companies to periodically review the security configurations of production systems and the security of source code to identify and resolve potential vulnerabilities.

Customers can manage account access to their data through an administrative interface allowing them to add or remove individual staff members.

2.4 Appropriate Use Policies

As part of the user registration process, the Digital Library application will disseminate the acceptable use policies and track user acknowledgement of the acceptable use policies. The process is enforced in the License and Use Policy Acknowledgement module. This applies to both registered and unregistered users. For registered users, the module will re-certify user acknowledgement upon user login if the acceptable policies change after previous acknowledgement. For unregistered users, access to the resources will be limited and restricted.

2.5 User Comments

The Digital Library is “an interactive teacher professional development tool to monitor professional learning goals. Teachers will primarily use this component to access resources for their own professional development. This will include resources such as documents, videos, guides, sample summative/interim tests and responses, and forums. Teachers can customize their content, post their reflections, and monitor their progress on implementing new practices. Additionally, the Digital Library also contains a work area to guide teachers to identify and use the best resources for their needs. The system may also be able to use the teacher’s interaction with the system to suggest additional resources.”1 User comments are an integral feature and process to allow users to improve the quality of content and to communicate effectively through the collaborative authoring process. Users will be able to search/query previously entered comments as part of search and filter operations. The user comments associated with specific learning resources can also be exported as part of a digital asset package.

2.6 Minimal Impact on State and Local Systems

As stated in the RFP, “[t]he CONSORTIUM’s vision is to minimize the impact on and required updates to state, district, and local school-specific systems (e.g., networks, servers, and Digital Library PCs). This includes efforts to minimize the technical footprint required of desktop PCs used for the Digital Library,

1 Smarter Balanced Architecture Report 03/11/2012


downloading of new software and add-ons to servers and PCs, and additional data storage requirements.” The Wireless Generation Digital Library solution will treat this vision as part of our overall sustainability objective. We believe that the solution needs to be both free-standing and mindful of the total cost of ownership (TCO). We will achieve this goal by ensuring the following:

• The overall Digital Library is a fully web-based, Application Program Interface (API) enabled,

open application. • The Digital Library is cloud hosted (our current default hosting platform provider is Amazon EC2,

with sites in US West and DR in US East. In the future, hosting could be easily migrated to either a State site or supported as a shared infrastructure in the cloud with much lower hosting and support cost than a normal on-site infrastructure.

• The user interface follows the Drupal MVS design paradigm and thus can allow State level customization following the open source release.

• The optional Coursera.org hosted advanced course delivery system allows minimal technology investment in the long run and will be free to end users after the 2013-2014 hosting period. A potential business model centered on certification or secure assessment might actually benefit the CONSORTIUM and member States in the long run. This option is discussed in Section 2.7.

• The Wireless Generation Digital Library solution is designed with an interoperability framework that involves a standard-based, loosely coupled integration. The interfaces will adopt the leading open standards such as:

o AIF (Assessment Interoperability Framework, a framework jointly developed by the Schools Interoperability Framework (SIF) and IMS Global) - includes Automatic Private IP (APIP)/ Question and Test Interoperability (QTI) for assessment items and SIF/Common Education Data Standards (CEDS) for reporting and learning standard association;

o Learning Resource Metadata Initiative (LRMI)/Dublin Core for tagging; and o IMS Common Cartridge for object import/export/reference.

It can support Security Assertion Makeup Language (SAML) for federated Single Sign-on (SSO), OAuth for registered Restful API authentication, Web Services for Remote Portlets (WSRP) for portal integration. All of the above standard adoption and supports will help the States to adopt, expand and integrate the Wireless Generation Digital Library solution during and after the contract period.

2.7 Member State Cost of Ownership

The highest priority of our solution’s architectural goals is to create a system that is highly sustainable and has an optimal total cost of ownership (TCO). In addition to the design options illustrated in the Section 2.6, we will also take into account of the overall business model and processes needed to support our architectural design.

Our solution is designed to support several key business models:


• Open Source business model - All the core components are built upon open source technology and will be released as open source software to allow the community to sustain long-term platform optimization and extensions.

• Open Access Content mash-up model – The Wireless Generation Digital Library solution is designed to both offer and consume open access content open-access content to create mashed-up content. By following standard interfaces as mentioned Section 2.6, our solution allows content to be “mashed up”, to make the content much more effective for multiple usage scenarios and usage patterns.

• Micro-content Crowd-sourcing model – The Wireless Generation Digital Library solution allows portions of content, or micro-content, to be generated into new items by individual members of the community, or groups of individuals (i.e., crowd-sourcing) throughout the content management and delivery process.

• Software as a Service (SaaS) model – The Wireless Generation Digital Library solution is designed to use cloud computing and minimize on-premise hosting footprint to allow the member States to incur minimum infrastructure cost both upfront and in the long-term. This tactics also allows the States to have long-term options to either operate the service within the State or across multiple States in a shared model.

• “Freemium” model – The optional Advanced Course Delivery component of the Wireless Generation Digital Library solution is achievable through a potential freemium model for future revenue opportunity for the CONSORTIUM, PD vendors and platform providers such as Coursera.org. In the freemium model, basic services and functionality are provided to the user for free, while other premium and derivative services will be offered at a fee to the user. Coursera.org is operating under the freemium model with long-term potential for derivative services such as future course certification, secure assessment of course participants, etc. By allowing such integration, the Wireless Generation Digital Library solution can allow long-term sustainability of the operation much beyond the 2 years of the contract period.

By using technology as a platform to enable these business models, the Wireless Generation Digital Library solution will help both the CONSORTIUM and member states achieve sustainable effectiveness by maximizing the benefits of the investment and seeding future ecosystems beyond the platform itself.

2.8 Varying Levels of State and Local Technology Capability

We recognize that, within the CONSORTIUM, there is considerable variance in technology, from low to high levels of capability. Some states and their associated districts and local schools may not have the ideal processing power of modern PCs/devices, and LAN and network bandwidth may be limited.

The Wireless Generation Digital Library solution addresses this concern by building technology components that can be deployed and managed centrally at the CONSORTIUM level but also can be extended and integrated at the state level. Since the solution will be open-source, the member states or


LEAs can replicate the Digital Library instance to further customize the application for their need. Regarding the planned rich content included in the Digital Library, such as video, audio, and other rich media learning resources, the system will allow export of the resources in portable formats. As a result, the resources can either be cached or directly imported into the LEA’s local learning content management system, LMS system, PD system or other portals for local resource delivery. We plan to have export options that follow the IMS Global Common Cartridge Specification to allow maximum resource portability. We also design the solution to be fully web based and delivered through the public cloud to make access as easy as accessing public web sites on the internet. Lastly, for any public content referenced within the Digital Library, such as WGBH, YouTube or Coursera hosted content, we will simply link the resources to the appropriate location and take advantage of the origin system’s caching, network optimization and CDN (content delivery network) capability.

2.9 System Flexibility

The Digital Library is part of the Shared Services of the Smarter Balanced platform. As such, it is fully integrated into the overall architecture of Smarter Balanced Technology Platform as illustrated below.

Tagging

Uploading

Interfacing

First of all, by adopting the componentized approach within the Wireless Generation Digital Library solution, we could offer both robust functionality for each business process involved and the flexibility and extensibility for innovation and creativity. The core Digital Library functions provided will be built around a highly customizable foundation of Drupal social CMS framework, the most widely used open source stack for such purpose. For the content creation and library repository functions and features, such a framework offers the balance between the out-of-box fit to the common social content management processes and the adaptability to implement the unique Digital Library processes.


Secondly, the Wireless Generation Digital Library solution will adopt homogenous and standard aligned system interfaces to fit into the larger picture of Smarter Balanced Shared Services as well as the overall Smarter Balanced Technology Platform. (Please also refer to section 2.6 about our standard adoption). Such Service Oriented Architecture conformity to the overall platform allows the plug and play options not only for functionality reuse between Digital Library and other Smarter Balanced components, but also for extension opportunities for future business processes such as content recommendation, reference mapping between assessment report and curriculum resources through the Digital Library, and advanced PD course delivery through the integration with cloud platforms such as Coursera.org.

Last but definitely not least, we adopt an agile development process with demand-driven requirement and adaptive release. We recognize the need to be adaptive and innovative and we applaud the CONSORTIUM’s goal-oriented rather than fixed-feature-set RFP. By working on the overall project of policy, sustainable strategy, content development and operation, we will be able to iterate though agile release with SLT and CONSORTIUM stakeholders to best serve the goal for the overall Smarter Balanced program to meet the demands for a flexible and scalable Digital Library.

2.10 Accessibility

We recognize the need for the Digital Library and the hosted educator resources to support access for all Smarter Balanced constituents. On the system side, we are adopting the best practices for large scale learning community and learning content management systems by provide flexible access options basing on a combination of the end user persona, usage scenario and originating device/applications. We will fully support the 508 access options.

2.11 Robust Governance Model

Since the Wireless Generation Digital Library solution is composed of complex program management, policy and strategy, content creation and system building activities, we propose a governance model that can accommodate such requirement and the system would support the governance model to function during and after the implementation. The following diagram illustrates this layered approach during the contract period. The details of the governance approach are further discussed in Section 3.5.2.

This governance model is by no means final and unchangeable. It is based on our understanding of the complex, multi-faceted governance process and will be adjusted and improved during the strategic development strategy definition phase. As mentioned above, Wireless Generation and PCG possess strong expertise in complex large-scale innovation efforts and our approach will reflect the alignment of the strategic, content and system building processes in order to optimize the results.


Note: Not all lines of authority, control, and communication are shown in the outline system diagram

Key: CONSORT = CONSORTIUM

CONSORT PMO = CONSORTIUM assigned Project Manager and Staff Green color – Final Approval Yellow color – Accountable for deliverables Blue color – Responsible for decision making


2.12 Recognition of Highly-Effective Contributors

The Wireless Generation Digital Library solution is built with long-term sustainability in mind. We believe that the key to maintaining high quality content and ensuring continuous motivation for content contribution is to enable a healthy community where contribution is valued, self-regulated, recognized and fostered. We feel that a “big-brother” approach of traditionally managed content production process would be unsustainable. Instead, we plan to track system-generated metadata and paradata, such as usage statistics, user behavior trace, moderator actions, group participation rate, links, quotations, references and relationships among resources. All of this will be tracked in addition to user- generated content such as comments, ratings of content submitted by users, and like/unlike counts. We can then utilize such data to derive and facilitate the contributor rating, kudos, recognitions and even more formal CONSORTIUM-wide communal feedback. The current Digital Library application is designed to collect and track all the data and we will work with the SLT and CONSORTIUM to refine the algorithm as well as business process for evaluation and recognition. The processes for doing so will be outlined as part of the Sustainable Leadership System work stream.

2.13 Moderated Collaboration Groups

The Digital Library is critical to providing educators with useful formative assessments, and instructional and professional learning resources. It is equally important in furthering collaboration among the educators in CONSORTIUM states, reducing isolation of best practices and making sure that every educator has access to best practices and materials. We believe that there are many types of collaboration that are critical, such as asynchronous discussions, the ability to annotate documents, virtual work groups/PLCs, and live moderated discussions to be hosted by SNEs or state professional development networks.

The communities of moderated groups will first be seeded with subject and grade groups from SNE to maintain initial coverage, for example, based on the CCSS taxonomy. Each major subject and grade level will have default moderated group for discussion, resource evaluation, as well as other type of collaboration. The system will also allow more dynamic grouping similar to the Wikipedia type of content focused group. We have experience building similar systems and processes for NYC (ARIS) and DC Public Schools as well as New York State (EngageNY), which is an ongoing project.

To further support this idea of cross CONSORTIUM collaboration, we will co-moderate the system with members of CONSORTIUM State Leadership Teams (SLTs) and State Networks of Educators (SNEs) to support state ownership and make instructional resource decisions as close to the classroom as possible. The selected members from each CONSORTIUM state will receive their training during SLT and SNE trainings. We will gradually release all moderated collaboration responsibility to the CONSORTIUM SLT and SNE members to ensure a smooth transition at the end of the contract without disruption to individual school environments.


3.0 Work Plan and Schedule

3.1 Comprehensive Development Strategy

The Comprehensive Development Strategy forms the backbone of the entire engagement and level sets the stakeholders with regards to the scope and methodology of the project. As defined in the RFP as a “well-crafted strategic plan to guide the development, implementation, and long-term management of the Formative Assessment Practices and Professional Learning component of the Smarter Balanced Assessment System both in the near term and beyond 2014”, this strategy is a key component of the overall Engagement Management artifacts and of the ongoing Communication Plan to the stakeholders. Within this section we will highlight the approach we will use to:

• Describe the Comprehensive Development Strategy • Communicate the Strategy to Smarter Balanced Stakeholders • Evaluate the Success of the Implementation of the Comprehensive Development Strategy

This Strategy document is essential in that it clearly defines the goals and vision of the solution that is mutually agreed to by Wireless Generation and the CONSORTIUM and clearly documented within the deliverable. It also outlines a clear set of mutually agreed upon measurements against which the benefits of the solution will be measured as well as clarifies the key timelines and milestones associated with those measurable benefits. Lastly, as it relates to the actual delivery of the engagement by Wireless Generation on behalf of the CONSORTIUM, it documents the key milestones and project management standards against which both Wireless Generation and the CONSORTIUM can measure if we are on track to deliver the project on time and within budget.

We will use a five phase Strategy Consulting methodology to develop the deliverables for this section. The five phases of the proposed approach are:

1. Initiation 2. Discovery / Vision 3. Analyze / Refine 4. Agree / Plan 5. Document / Execute

In the Initiation Phase, we will meet with the CONSORTIUM leadership to validate the scope and deliverables to be produced from this work stream and plan the touch points with key CONSORTIUM stakeholders involved with the process of developing these deliverables. We will convene an initial meeting with the Formative Work Group and the Director of Professional Learning to begin to get high level input from this team to be used to guide the initial strategy development as well as to provide Wireless Generation a sense from this Core team the depth into, and with whom specifically, they expect to be involved with the final formulation and approval of the Comprehensive Development Strategy. Work products of this Initiation phase include the various meeting notes from Initiation Phase meetings that are held, an initial schedule for the follow-up phases and meetings, and a document


outlining any mutual agreements that affect scope of work related to the Comprehensive Delivery Strategy development.

As this work moves into the Discovery / Vision Phase, the schedule that was agreed to during the Initiation phase will be executed and the various CONSORTIUM stakeholders will be engaged. The goal of this phase is to document the following:

• Ideas and related process maps associated with the key sections of the Comprehensive

Development Strategy. These include initial thoughts and ideas for the goals of the Digital Library, the potential measurable benefits to educators of the Digital Library and the processes to be employed in creating the professional learning content, the indicators of success as well as thoughts around the key milestones that must be made in the project. During these conversations and the meetings that will be held, Wireless Generation and PCG will share with the CONSORTIUM our ideas and best practices that are relevant to the discussion and from our experience working with other school districts and state departments of education.

• Key themes that are uncovered in the ongoing conversations relative to goals and value statements around the professional development / content initiative we are undertaking in this project. Identification of key themes will potentially lead us to other key performance indicators of success that we may want to include in the ongoing evaluation process.

• Potential challenges and issues that we may encounter and initial risk mitigation strategies for each challenge / issue that can help inform both the Comprehensive Development Strategy document as well as how we manage the overall project to completion.

• Vision of each stakeholder as to their perception of success of the initiative and how they might measure it, again searching for themes of relevance that can be utilized later.

Work products of this Discovery / Vision phase include meeting notes as well as modifications to the document outlining and mutual agreements that affect scope of work. Additionally, while not a formal contract deliverable, we will consolidate these initial findings into a documented form that can be shared with the CONSORTIUM for review prior to exiting this phase and moving into the next phase of the methodology. Input into this document, assuming it is timely, can greatly enhance the eventual Comprehensive Development Strategy document.

The next phase of the work in developing the Strategy is the Analyze / Refine Phase. In this phase, Wireless Generation and PCG, in collaboration with the CONSORTIUM stakeholders, State Networks of Educators and the National Panel of Experts (if confirmed and available and, as appropriate for the work), will perform secondary analysis of the findings of the Discovery / Vision phase. We will organize the thoughts of this phase, assimilate the findings with findings from other similar work we have undertaken, prioritize findings, and prepare an initial draft version of the Comprehensive Delivery Strategy deliverable that will be presented for an initial feedback cycle of Work Groups per Exhibit O. As part of this phase we will reconvene the Formative Work Group and the Director of Professional Learning to review this document, the feedback from the initial feedback cycle and validate that the Strategy meets the needs of the CONSORTIUM and the engagement.


The Comprehensive Development Strategy deliverable for this Analyze / Refine phase will focus on the goals of the Digital Library, the measurable benefits to educators and the indicators of successful development strategy sections of the deliverables. We will include our initial thoughts as to the timeline and key milestones based on the initial input we receive from the CONSORTIUM but it isn’t until the next phase that these will be planned out and scheduled based on acceptance by the CONSORTIUM of the deliverable of the Analyze / Refine Phase.

Once that deliverable is accepted by the CONSORTIUM, the team will work with the Project Management Team as well as the leaders of the other project work streams to lay out the final schedule of the engagement as it relates to the timeline and key milestones of the Comprehensive Delivery Strategy. This is done in the Agree / Plan phase of the methodology proposed. This section of the deliverable will be enhanced in the Comprehensive Development Strategy deliverable. Additionally, as a result of this phase will be a proposed executive dashboard that we will use to document on a periodic basis the evidence of success and progress against the goals of the Digital Library, the measureable benefits, the indicators of a successful development strategy and attainment of the timeline key milestones. This artifact becomes part of the ongoing Project Management portfolio that will be made available to the CONSORTIUM for ongoing tracking of project progress.

At the completion of the Agree / Plan Phase we enter the Document / Execute Phase whereby we put into action the Comprehensive Development Strategy with the governing Project Management Teams, the Formative Work Group and the Directory of Professional Learning. It is also this final version of the Comprehensive Development Strategy deliverable that:

• is sent for final approval by the Formative Work Group and Director of Professional Learning and one feedback cycle of Work Groups per Exhibit O;

• becomes the baseline for developing the informational newsletter describing the comprehensive development strategy, the role of the State Leadership Team, and their expected contribution to the Comprehensive Development Strategy; and

• is published on the Smarter Balanced public website.

The table below illustrates the high level view of the schedule to complete the Comprehensive Development Strategy outlined in this section.

Methodology Phase # Weeks Scheduled Month

Initiation Phase 1 -2 weeks January 2013

Discovery / Vision Phase 4–5 weeks January 2013–February 2013

Analyze / Refine Phase 4–5 weeks February 2013–March 2013

Agree / Plan Phase 2–4 weeks March 2013–April 2013

Document / Execute Phase 2 weeks April 2013

Document / Execute Phase Ongoing Throughout Project


Evaluating progress towards meeting the goals of the Digital Library will be an ongoing effort. Wireless Generation will provide a semi-annual evaluation report which will restate the original goals outlined in the comprehensive development strategy, and will provide evidence of progress towards those goals. We will collect and present several different forms of evidence to show progress of the engagement. These forms of evidence will depend on the goals, but may include some or all of the following:

• Direct usage evidence, such as click-throughs and access logs, that show CONSORTIUM members are accessing the Digital Library;

• Status reports that illustrate the number of online formative assessment, instruction, and professional learning resources completed and available;

• Attendance at professional development sessions; • Involvement of educators in the creation of Digital Library resources; • Perception data, via online educator surveys and training evaluations, that shows both the

implementation fidelity of the state networks of professional development and the value of the resources to educators;

In the event that any of the agreed-upon metrics are not on track for a successful implementation, the evaluation report will include documentation of the root causes and mitigation strategies for those metrics.

Finally, the evaluation report will summarize the overall status and success of the comprehensive development strategy implementation by reviewing all key milestones and timelines. Implementation project plans will be updated and reviewed to make sure that all past development phases have been completed and that current and next phase are on track and unblocked. Timelines will reflect any changes in scope or delivery. Post production rollout, evaluation of the development strategy can be compiled through stakeholder polling or focus groups which can provide real world insight into system benefits.

3.2 Digital Library Application

In order to achieve the overall RFP goal, we will develop the Digital Library system to create, curate, share, host and deliver the contents and resources created for overall CONSORTIUM operation process. The Digital Library site is an online, one-stop social site with a wide variety of resources and professional development opportunities to learn more about formative assessment practices and professional learning. The analogy is that the Digital Library site for CONSORTIUM community is like the Microsoft Developer Network (MSDN, msdn.microsoft.com) for Microsoft technology practitioners.

The Digital Library will be designed as a fully integrated part of Smarter Balanced IT systems’ technology solutions and is designed and built in accordance with established CONSORTIUM-approved IT systems and standards. It will be developed using agile methods, incorporating feedback from a wide range of CONSORTIUM stakeholders, including teachers and state representatives.

The hosting and maintenance for the application is provided through September 2014, with transfer and turnover planning consistent with the post-2014 sustainability plan.


The Digital Library search, navigation, and tagging system is coordinated with an item tagging system to allow for a link between score results that may be reported by grade level, content area, claim, and target on the interim assessment and educator, parent, or student resources that may be related to the same grade level, content area, claim, and target. In addition, users are able to search by key words and tags aligned to national resource networks.

Our proposed solution leverages tried and true technologies and techniques to achieve the CONSORTIUM’s goals. We will primarily use open source technology and open platforms to architect and implement the Digital Library application. This will dramatically reduce the cost and also make our proposed solution much more sustainable and extensible. It is important to note that, because Wireless Generation has extensive experience to implement and integrate large scale Content Management and Collaboration platforms using open source technologies, we are uniquely positioned to provide the CONSORTIUM the best possible scalability and response times in an open source model. Please refer to our experience in section 7.B for more information on our work providing content management and collaboration platforms with New York City Department of Education and District of Columbia Public Schools. We are also a subcontractor on the chosen vendor team selected to provide the NY State Education Data Portal Content Management and System Services Solution.

All the software provided and recommended in our proposal can be run on commodity hardware or in the cloud so that operational costs can also be managed. The open source software components selected have significant followings in the open source community to support ongoing development and enhancements.

Overall Architecture

The Digital Library is an integral part of the overall Smarted Balanced Platform as illustrated in the consortium architecture diagram.


An integral part of the Smarter Balanced Technology Platform

The following diagram provides a high-level view of the overall Digital Library architecture. As the diagram illustrates, this platform provides full content management capabilities, including tagging, full- text and taxonomy-driven search, content integration to connect to external content repositories, and advanced content delivery integration. We will support both native authentication through the Digital Library authentication and optionally, SSO integration with a CONSORTIUM authorized Single-Sign-On (SSO) Identity provider.


e

wireless generation• 55 Washington Street, Suite 900

Brooklyn,NY 11201-1071

T 212.213.8177 F 212.213.8749 www.wirelessgeneration.com

Sesson and User Profile Manager

Shared Communication and Services

Portal Ul Configurator

PortalStylesheet Configurat or

WSRP or other portlet int erface

Mobile Page {Optiona l)

Login and License Acceptance

SSO and Security Configurat i on

Smarter Ba lanced

Single-Sign- On

Stat ic Pages Smarter Balanced Header/

foot er

f eder ated Search

Localization

Digital Library Portal (Could be integrated into overall Smarter Balanced Portal)

CMS Digital Library Discussion Group Library Ca talogand

Personalizat ion Authoring. Tagging and Forum Naviga toi n Library Ardhiving Library Search and Uploading Digital Library Course Delivery

CMS RBAC and CMS W orkflow Learning Object

Library Export Recommendation Course Content Instruct ion

Administration Import/Export Engine (Optional) Packa ging Mana gement

Meta-da ta Management

(TaxonomyTag,Map

CMS Reporting

Learning Registry

Registration

Metadata and Library

Index

PD Delivery Reporting

Course Archiving

Certification {Optional)

Data Sources

Slice

CMS

cess Taxonomy

Digital Library Object Repository

Advanced Course Delivery

External



Technologies

As one would expect, there are a vast number of technologies that need to be carefully integrated and balanced to support this type of fully integrated collaborative platform. These choices become even more critical when this platform needs to support the type of scale and flexibility that the CONSORTIUM is envisioning. As we have already explained, the majority of the components provided in this solution will come from Wireless Generation and partners’ technology collections along with open source solutions. We envisioned that the overall solution to be fully open source at the end of the project on delivery to the CONSORTIUM.

The open source platform components below have been chosen for this proposal because of their capabilities and compatibility with the overall solution framework.

• CMS and collaboration framework: Drupal • Public Facing Read-only Caching Proxy: Varnish • General Cache: Memcached • Relational Database: MySQL • Search Engine: Solr • Operating System: Linux

A brief overview of some of the more important of these platform components is provided in the next few sections. The system and its infrastructure will use the latest stable versions of Apache, MySQL, PHP, Varnish, and Drupal. At the time of writing the latest stable versions are: Apache 2.42, MySQL 5.5., Varnish 3.0.2, Drupal 7.14, PHP 5.3 and Apache Solr 3.5.

Collaboration & CMS: Drupal

The Drupal CMS is a free and open source CMS and social collaboration toolset widely used in education community. Wireless Generation has implemented large scale Drupal solution for similar purposes in NYC ARIS Connect and our Drupal based solution has been selected by NYSED to support New York State-wide Education Data Portal CMSS process which is similar to the Digital Library in content scope for 300,000 educators.

We and our partners have customized this platform to support both horizontal and vertical scaling methodologies and the platform can be further clustered for high availability. Additionally, it can be easily deployed to a Cloud and used in SaaS solutions such as the one that the CONSORTIUM is offering to its member states.

Solr

The Solr search engine will be integrated with the multiple data sources, including the content management system and content repository, allowing users to search for content and data across the system using keyword references. Search requests will be presented on a results page that aggregates


all the hits from the various sources which users can sort, filter for further refinement, or explore the result links.

Vanish Cache

Varnish Cache is an accelerator for web applications. Varnish includes a configuration language, Varnish Configuration Language (VCL), which allows for the creation of policies to further tune performance to specific business needs. For the mostly read content to serve the overall CONSORTIUM community, we will implement Varnish to help the scalability of the site.

Caching: Memcached

Memcached is a general-purpose distributed memory caching system. It is used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source (such as a database or API) must be read from. Memcached runs on UNIX variants and is distributed under a permissive free software license.

Memcached's APIs provide a giant hash table distributed across multiple machines. When the table is full, subsequent inserts force older data to be purged in least recently used (LRU) order. Applications using Memcached typically layer requests and additions into RAM before falling back on a slower backing store, such as a database or files on disk.

Memcached is used by various parts of Wireless Generation’s Software as a Service (SaaS) infrastructures and we will bring this expertise to bear in the work for the CONSORTIUM.

MySQL

The proposed database for the solution backend is MySQL, which is a popular Structured Query Language (SQL) based and therefore relational database frequently used in web applications. MySQL is available in a dual open source and commercial license. It also has a native stored procedure programming capability. In the years following Oracle’s acquisition of Sun Microsystems (and MySQL which was a wholly owned subsidiary), numerous open source derivations, such as MariaDB and Percona have come to be. These provide different scalability options and this variety of database engines allows Wireless Generation to make effective architectural tradeoffs, while programmatically having the same programing interface and hence an ability to reuse existing code.

3.2.1 Functional Requirements

As part of the overall development, deployment, and operation process, we are prepared to implement our agile software development and configuration process in the area of functional requirement. This will include but not be limited to, the collection of detailed functional and technical requirements from CONSORTIUM members and provision of expert technical consultation to supplement CONSORTIUM expertise. These technical requirements will be used to drive design, configuration, and development activities.


As we execute our agile development processes, we will develop and make accessible to the CONSORTIUM all user stories and functional prototypes (including High Fidelity Prototype) that collectively represent the functional requirements for all required components. NOTE: Because this is an agile development process these stories can be refined and extended through the process. These refinements and extensions will also be available to the CONSORTIUM. The full, final set of all user stories is expected to be a comprehensive list of requirements that ensures each component within our fully integrated Digital Library platform supports the needs of the CONSORTIUM. More specifically, the requirement collaboration process will also define:

1. the suggested makeup of participants at the requirements-gathering sessions; 2. the information to be provided to participants; 3. the information to collect from participants, including types of questions to ask participants

and recommended formats; 4. the approaches to analyzing the data collected from participants to best inform

identification and refinement of requirements (including quantitative and qualitative methods); and

5. the sharing of sample resources as a starting point for the development of mock-ups by Wireless Generation for requirements gathering.

Following collaboration process, we will provide feedback on the results of the requirements-gathering effort so that the conclusions drawn are supported by evidence from the data-gathering and are consistent with existing research on score reports. In addition, we will focus on providing conclusions that address the needs of distinct audiences (teachers, students, parents, education administrators, and the general public), while complying with relevant professional quality standards. Critical in this review is that conclusions reached effectively leverage the Theory of Action of Smarter Balanced, structured to improve teaching and learning through a meaningful CONSORTIUM Assessment Digital Library system.

3.2.2 Technical Systems Requirements

We will create the following technical requirement document to address areas including, but not limited to, the:

• Technical Architecture including server hardware and platform software requirements • Multiplicity of environments, including:

o Development o Staging o QA o Disaster Recovery

• Physical Security • Virtual Security • Training • Production Operations


o Networking Requirements and Diagrams o Database, Data Storage, and Archiving Requirements and Approach o Systems Management and Monitoring Requirements o Middleware and Integration Software Requirements o Security Requirements and Approach for Applications, Data, and End-User Access o Interoperability protocols and interfaces o Performance capacity, including:

number of users concurrent users number of transactions to be handled anticipated response times for each component peak usage maximum and minimum number of concurrent users for each component

o data integrity o load balancing o firewalls o failover approach o redundancy and recovery o reliability o availability

• Security: o Component-to-component o User Authentication and Authorization o Item-Level Security o Student Data Security o Data at Rest

Fortunately, we have all of the detail and expertise for such infrastructure, since we run precisely this type of fully integrated platform in New York City (ARIS), DC Public Schools, and New York State (EDP CMS) as well as our own mCLASS® SaaS platform. Certainly, there will necessarily be some differences between these instances of the platform and the one we deliver to the CONSORTIUM uniquely optimized to its needs. But in no case, will we have to start from scratch. We know this domain, the particular type of solution, and we know how to operate it at scale and speed. No one is better equipped to help the CONSORTIUM provide the same as it rolls out these critical CMS and collaborative services to member states. At the “hand-off” of this solution, we will provide the CONSORTIUM with all the “run books” that we use to manage this integrated data platform.

3.2.3 Application Design

We will design the initial Digital Library system mock-ups of navigation and flow (“wireframes”), including static and dynamic screens consistent with the CONSORTIUM Technology Systems Architecture and consistent with the requirements defined in the test specifications and categories. As always, our


UX design will incorporate research and best practices, specifically noting how these considerations are impacting design decisions.

A Wireless Generation interdisciplinary team of experts will review initial mock-ups, for consistency with the Smarter Balanced specifications, gathered requirements, and expectations from the Theory of Action. In addition, we will provide feedback based upon current research and best practices. Each design will go through a review process that involves content, usability and navigation elements. These reviews will include consistency with professional quality standards, responsiveness to audience needs, accessibility issues and support for students with disabilities, English language learners, as well as the parents and educators of these students.

Using the results gathered, we will develop a set of application, technical-design, and systems- integration design deliverables that will be used to drive and support the programming and/or configuration of the software solution.

These design deliverables include, but are not limited to:

• network and infrastructure diagrams • configuration specifications (e.g., scheduling, database, ETL) • file layouts and specifications • A security design document that describes the overall approach to security for applications

and components developed under this contract. The document will include, but not be limited to,

o application security o integration security with other systems o denial-of-service attacks o database security o encryption o data integrity o prevention and identification of testing irregularities (e.g., unusual student actions)

• Test cases and test scripts to support unit testing, performance testing, UAT, and systems-

integration testing

3.2.4 Application Development

Wireless Generation will provide application software that meets the functional and technical systems requirements. As such, all code developed will use a mature systems life cycle and development process, have good readability, maintainability, and traceability.

We understand that the CONSORTIUM will consider the two (2) types of software solutions, listed below and we should propose one or the other, or a combination of the two.


1. A commercial off-the-shelf (COTS) product configured and customized to meet CONSORTIUM requirements, or

2. A custom-built application.

We are proposing a combination of these two models using open source components and customized Wireless Generation COTS. NOTE: Even though the code delivered for this project will contain customized Wireless Generation COTS product, the end solution will fulfill the requirement for long- term open source development and support the unique needs of the CONSORTIUM and its member states.

3.2.5 Quality Assurance and Testing

We will deliver a detailed test plan, indicating which processes and/or testing phases will occur during key milestones in the project, and that this plan may be modified or updated at later stages of the project. In addition, we will develop detailed test cases and test scripts to verify and validate system application components meet requirements and design specifications. As is typical in every one of our engagements, as system applications and components are developed, we will ensure that comprehensive testing to validate functionality and performance is performed. Our continuous build and test capabilities, automated regression capabilities, and domain expertise will all support these efforts.

In general, the testing activities will include, but not be limited to:

• unit testing • system testing • performance testing • systems-integration testing • UAT

We will oversee the prioritized resolution of defects found as a result of testing activities, and perform retesting to ensure that defect-resolution activities are successful, and do not introduce further defects.

We apply rigorous automated and manual testing strategies for our solutions because we understand that neither type of testing alone provides a truly acceptable outcome. Manual testing, although slower, can look for fine-grained visual problems in the interface as well as issues that prevent users for getting the value that they really might expect. These types of issues can easily be overlooked with automated testing, which is faster but very binary. The machine either ran the test and passed it or ran the test and failed it. Automation misses these finer shades of gray.

Manual testing alone, however, is not capable of performing a full “regression” of the system to ensure that after software changes were made none of the previously available functionality was negatively affected. This type of work can largely be reduced to a bunch of binary, pass or fail, tests of the previous capabilities that can be programmed once and replayed each time updates have been made.


3.2.5.1 Test Types

There are many types of software tests that we think apply in its best-practice development processes. The most common of them are taken in turn below to give the CONSORTIUM some perspective of how complete our quality efforts are required to be. We observe that there are many different ways these types of tests can be categorized and in many cases different names that can be applied, so we will focus on the most common test types and names.

Unit Tests

These are the most granular tests, written by the application developers around each block of code they contribute to the application. We apply best practices including code-coverage tools to ensure that we keep our unit test coverage as high as practicable at every stage of our development. In general, unit tests pose an assertion against the code and verify that this is indeed true as the code operates. For example, in one area of code we could assert that “2+2=4” and then verify that the code does actually return this result for these inputs. We cannot test for everything at the unit testing level, so other types of tests such as automated and manual functional tests are employed. Good examples of things that cannot be unit tested are complex interactions between multiple components in a system in particular when one of those components is external, e.g. a remote data source from a CONSORTIUM state system.

It is important to note that the set of unit tests is not required to be static after initial code release in our development process. As defects are discovered in the test environments, more unit tests are added to ensure they do not reoccur. This reinforces our testing suite overall, but in particular for regression testing of existing functionality after future upgrades and changes.

Functional Tests

Once a component is delivered, the build process on the test server integrates the newly released component into the current test release. (We employ continuous code integration as a best practice.) This allows QA to test each component or feature with other parts of the system as quickly as possible and ensures that new components both function as expected on their own and do not cause failures elsewhere. At this stage in the process, QA creates and applies both manual and automated functional tests to validate previously deployed functionality as well as newly deployed functionality and to ensure that the software is stable. Test engineers document all successfully completed tests and log any errors in our bug tracking toolsets for resolution by developers.

Functional tests go beyond simple assertions to test much more complex build-ups of inputs, observations of the system in progress, and review of complex outputs. However, functional tests are still largely localized to particular functional areas of the code, e.g. the data import functionality or the report template application capabilities. It is at this functional testing level that end user behaviors and use cases begin to become visible and be validated.


Continuous Integration and Regression Tests

Regression testing ensures that code that has been released into test and production environments and verified as working continues to work. Regression testing runs through existing test cases that have been executed and confirmed. We work to automate as much of the regression testing suite as possible so that it can be run in a shorter period of time much more frequently over the entire system.

Continuous integration is a best practice that we employ as well. Here we get newly completed components into test validation against the rest of the system as quickly as possible. With our continuous integration, immediately after a new component is code-reviewed and released it is built and deployed to a test environment where a thorough series of automated tests is run. This helps ensure that every time any code change is made both local and more global tests are run to catch errors early and resolve them as cost-effectively as possible.

There are two main varieties of regression testing that we apply, selecting each for the circumstances where they are most appropriate. These varieties are:

1. A subset of all the automated tests (both functional and unit tests) are executed on the continuous

integration-build environment. Each new build of the code base must pass all tests before being promoted to an integrated testing environment. QA Automation Engineers design the automated functional regression tests. These sets of tests are typically designed to only take 15 minutes or so to run. Thus, they do not slow down the workday too much.

2. The complete suite of test cases, both manual and automated, is used to verify the quality of a

complete product before its release. Automated tests can run outside working hours and take several hours. A complete regression test suite is run when some significant upgrade to the infrastructure of released code has taken place (e.g., a major version number increase in a database or web server framework.) Here the manual testing serves several primary purposes:

a. Provides a "hands-on" look at each user interface.

b. Tests new functionality while a complete set of automated tests is still being developed.

c. Finds subtle visual defects that are hard to find with automated tests.

d. Enables “exploratory testing” that more effectively mimics real user actions, including trying things that are "wrong" or contrary to the specified behavior. (Of course, Wireless Generation does build “negative” unit and functional tests and applies them both manually and automatically where certain negative uses of the system are expected. Many times, however, it is this “exploratory” manual testing by our skilled QA engineers that identifies these negative system uses we then protect ourselves against moving forward.)

Smoke Tests

Smoke tests, also called sanity tests, are usually run on production and UAT (staging) servers when an event has taken place that involves no change to the code, often as simple as restarting a server. These


can be viewed as a subset of the complete suite of manual and automated regression tests run on the Continuous Integration environment.

The rationale for using smoke tests instead of running the full regression suite is that the cost of a full regression can be foregone because we know that nothing within the system itself has been changed. This knowledge of “no change” does, of course, need to extend to the IT operational infrastructure. Any alteration of IP addresses, server configurations, etc. would also frequently push us to escalate from a smoke test to a full regression.

Performance and Load Tests

Performance testing establishes the expected performance of the system (parameters such as response times and latency) under various conditions as a benchmark to conduct future load testing against. Load testing validates the capacity models for the system. Here we validate projections of system performance under set numbers of users and certain sizes of data sets and workload mixtures.

These processes are far more complicated than they initially sound, and we would suggest that our expertise here may be critical to the CONSORTIUM’s overall success with this project. Just to provide a feel for the issues. We test the experience of a single user with large volumes of data and the performance of the entire system with many concurrent users, but that is not sufficient as these two types of measurement and performance are intertwined. A single user who has a usage pattern that accesses all the students in a school district may not notice any slowdown in application response until 100 other users are online simultaneously, perhaps with much smaller resource uses. We will not be able to present all the permutations our engineers and processes have learned to evaluate and manage over time here in this RFP. However, a few sample load-testing scenarios that we carefully blend together for our fully integrated reporting platform are:

• Multiple concurrent users, concentrating on average and peak numbers of users projected

for the system • Single users with the largest anticipated amounts of data in their results views • Data-input sources and data-output sinks not related to a standard reporting user • The experience of users as constrained by their own internet bandwidth and other client-

side issues. In some cases, our application developers can anticipate and limit problematic behavior caused by failures and limitations on the client side.

Certainly this magic all has to be quantified so that we know when we are improving and by how much. Just a few sample performance and load test metrics include:

• Page size (in bytes) • Response time (time from start of request to end of response) • Latency • DB requests per second • Resource usage/utilization on various types of servers


• End-to-end transaction times for user transactions which can only be obtained and optimized with very sophisticated logging that Wireless Generation has built into its platforms. Much of the complexity here is that these sessions involve many “button clicks” on the web pages and many actions at the server to fulfill all of these requests

• Usage/workload models and distributions of requests along with performance metrics

UAT testing

User Acceptance Testing (UAT) is the final step in the Quality Assurance process prior to production release. UAT is run on a mirror of the production environment, usually called staging (pre-production). UAT is driven by the users of the system, either QA engineers, the clients of the software, beta testers drawn from eventual user populations or all three.

UAT will be supplemented by automated "smoke tests” and other automated tests as described above but manual verification of intended behavior for actual users of the system around the tasks they need to accomplish is an invaluable extra step.

Although there is a formal User Acceptance Testing (UAT) period at the end, our agile software development allows UAT to begin much sooner. In fact, we often have user acceptance testing of all interim releases building up to the final code delivery. This both reduces the amount of time spent in final UAT and ensures that user issues and refinement needs are caught earlier on where they are less expensive to correct.

3.2.5.2 Test Automation Breakdown

Wireless Generation has developed a lot of expertise over time and continually puts a lot of focus on automated testing. This is because we see it as key to saving total time in development and increasing overall customer satisfaction. Since we have spent a lot of time discussing automated testing, we want to provide a quick overview of the major areas of test automation in our arsenal, including:

• Web application test automation • Database input and output • Service-layer testing (including third-party developer APIs) • Test ("Fixture") data generation

Each of these major areas involved in automated testing will be further explained below.

Web Application Test Automation

Here Wireless Generation creates code in Python and Java to simulate a user clicking through a series of web actions, entering data, and verifying that each page displays as expected. The emphasis is on data accuracy, response time, and error messages. Visual display testing can also be automated but manual verification is simpler and more reliable. The test code can be run (as described above under Automated Regression Testing) in two ways:


1. By a continuous-integration-build server launching a core subset of the entire automated test suite each time it rebuilds the test environment. (Here the application team members receive an email listing successes and failures for immediate feedback.)

2. As a more complete automated test suite manually run at a time when no one is waiting for it to

complete before continuing work. (This is typically done each night in the overnight hours when less development is taking place.)

The web application test code files are run by a framework service (WebDriver is our current standard) that is called from the build server. The framework instantiates a web browser on the test server, executes the tests, returns success and failure messages, and cleans up after itself. Since the automation is itself code, it is subject to some of the same processes as application code and therefore can leverage other best practices. For example, it is stored in the code repository and is subject to code review. The aims of code review are to improve critical code virtues such as accuracy, efficiency, encapsulation, and reuse. This saves time by leveraging and extending on existing knowledge and best practices.

Data Store and API Input, Output, and Data Transformation

A lot of data transformation takes place in this fully integrated reporting platform. Some of this transformation takes place via the data input/output Application Program Interfaces (APIs) and some in the data stores themselves. We typically deploy “before and after” testing of each of these data flows and transformations. Most often this testing is performed with nearly complete automation. There are two basic types of processing that are typically done for this testing:

1. Direct data extraction/collection which is then compared to a "gold copy" of the data which has

been verified as accurate. This ensures the data that was not expected to change comes in/out of the API and/or is inserted/stored into the database in the same way one day as the prior.

2. For data that we expect to change or be transformed, we often perform “reverse-

transformations” of the data and compare the results against the new input coming from the source or transformation. So, if we are transforming 1, 2, 3… to Jan, Feb, March… then we would reverse that transformation to ensure that what we expect to be coming into (or being produced within) the system actually is.

Certainly there is a lot more to all the rigorous data integrity checking both statically and dynamically that we perform, but this section gives an understanding of the issues and our best practices here.

Service-Layer Testing

For services which deliver data back and forth to other applications and often have no end-user interface, a high degree of test automation is needed. Here again we develop test suites in Java and Python to apply to these problems. A general flow of how this service-layer testing progresses is:

• Test programs/scripts invoke the service


o Test programs/scripts retrieve test data from XML files or comparably stored data structures and pass it into the service, AND/OR

o Test programs/scripts retrieve data from the service (often converting it to a known XML or other well defined data structure) and stores it

• Test scripts validate either/both returned data, success messages, and failure codes • Depending on what the service is expected to accept/provide, the semantics of meaning of

the services, evaluates the experienced behavior against what is expected

A very nice property of these sorts of automated tests is that they can typically also be run within a continuous integration suite on a test-environment build server.

Test ("Fixture") Data Generation

Simulations of real user data, usually called "fixture" data, can oftentimes be automatically generated according to certain rules (e.g. "Give us test result sets with every possible combination of four strands having individual scores on a scale of 0–10"). When we cannot auto-generate fixture data, it can be created manually by application domain experts. There are many ways to use fixture data:

1. To populate a test database with large amounts of data in a range that fulfills all test case

conditions

2. To begin application testing in parallel with database development

3. To validate APIs and other external integrations

4. To facilitate reuse of testing data across unit, functional, and other test suites

5. To drive load testing processes to desired levels 3.2.5.3 Leveraging Wireless Generation Test Expertise

For this project, Wireless Generation is proposing to provide, within the cost of the proposal, a full battery of tests at all levels (unit, functional, regression) and for all relevant technologies (web, data store, API, load). This test suite will also include manual testing, test automation, and test/fixture data generation/creation according to Wireless Generation best practices.

As described above, by blending manual testing with automation, our testing is able to evaluate:

• user experience in navigating the system and accessing and generating reports • clarity and completeness of reports that are produced • translated report text • Feedback gathered from a broad range of stakeholders (chiefs, governors, state

administrators, content experts, IHE representatives, classroom teachers, students, and national organization memberships).


This testing, of course, is beyond the more technical, binary elements of does a function actually return the right value in a specific situation. This begs the question of how do we know when functionality passes these “gray area” tests? Wireless Generation has a long history of answering this question with the agile requirements that are carefully collected, managed, and tested against in our development cycles. We collectively refer to this QA as “Usability Testing”.

Usability Testing

The initial definition of what is “usable” is defined by the initial sets of user stories. For example, the requirements may state that all report types available to a particular user under his/her login credentials are to be made available from a single screen in alphabetical order within a standard report design template for the platform. QA of this initial set of requirements then looks for the right reports to be displayed in the right order and rendered according to the specified design template. This testing is over and beyond all of the more technical considerations of working/failing. Because Wireless Generation does develop within an agile methodology, of course, refinements to these requirements and therefore to the definition of “usable” can occur at each checkpoint. Certainly, costs and timelines would rage out of control if Wireless Generation, ETS, and the CONSORTIUM did not work collaboratively together to get the right requirements up front, ensure that they are not over-engineered, and ensure that the overall level of revisions is managed.

The benefit, then, to the CONSORTIUM of using Wireless Generation for this project and fully leveraging our broad and deep range of expertise on what is most usable and what is psychometrically optimal is clear. As indicated here, Wireless Generation has tremendous experience designing, building, and deploying a wide range of reports at scale and speed for customers through our data platform projects including ARIS and mCLASS®. ETS, correspondingly, has broad and deep experience in reporting from the psychometric side and is uniquely qualified to align our reports — right out of the gate — with the CONSORTIUM’s Theory of Action. This ensures that all parties involved will minimize the time spent in rework around all of these very complex usability issues. (And this efficiency is over and above that gained by leveraging the Wireless Generation fully integrated data platform from a technology, scalability, performance, configurability… perspective.)

As requested here, Wireless Generation will work collaboratively with the CONSORTIUM in applying our best practices for large-scale Digital Library system development and UAT and refining the deliverable to align with CONSORTIUM needs. In this large-scale UAT process, Wireless Generation will:

• Solicit review, analysis, and feedback from a broad range of stakeholders. • Collect feedback from a representative sample of at least 200 teachers, educators,

education organization, and other stakeholders on the reports from large-scale Field Test development.

Wireless Generation propose that we stage a UAT environment earlier in the process and allow broader user review at various stages of the development.


3.2.6 Data Transfer and Input

ID Deliverables Artifacts

3.2.6.1 Wireless Generation will identify the formatting and metadata requirements for the content to be imported or converted into Digital Library based on the final design and metadata tagging specifications of the relevant Digital Library components. Wireless Generation will then complete a gap analysis to determine:

• Content that needs retagging or additional tagging • Content that has metadata that can be automatically

converted by an algorithm and / or “cross walk” • Content that requires reformatting, splitting, or indexing.

(For example a curriculum module is composed of many individual objects and may need to be organized and indexed properly to meet the user experience design requirements of the content management system.)

Content Gap Analysis

3.2.6.2 Wireless Generation will integrate live contents from external systems

We will execute the tagging, conversion, reformatting, splitting, indexing and other content integration work as identified in the above analysis and load that content into the Digital Library. Prior to commencing the execution, we will provide CONSORTIUM with a detailed work plan including:

• Review and sign-off processes including CONSORTIUM approvals

• A description of the approach for each set of content and specifications for any automated processing, tagging, or conversion of content

• A quality assurance and testing methodology and approach, to be executed by the vendor prior to sign-off on the final production deliverables

Content Integration

Wireless Generation will provide tools and resources to tag with meta-data, map to standards, load, and curate learning objects currently being acquired, owned, licensed, or used by the states.

3.2.7 Application Deployment

Wireless Generation will install all necessary hardware, software, and applications required to deploy Wireless Generation’s fully integrated Digital Library system for both the testing and production platforms. Following the contract period, the CONSORTIUM will take possession of the finished and


tested solution and deploy it for production on its own hardware and software infrastructure within its own facilities. To support the CONSORTIUM in this handoff, Wireless Generation will provide deployment plans and related documents along with system documentation.

3.2.8 Systems Integration, Quality Control, and Defect Resolution

3.2.8.1 System Integration

Wireless Generation will perform integration activities that meet the systems integration requirements and design specifications. The set of systems integration efforts are:

• Integration of all Digital Library components to fulfill the overall CONSORTIUM training and

help system process, for example, between Digital Library CMS subsystem to Digital Library search components, or between Digital Library SSO component and Digital Library Role Based Access Control component.

• Integration of the Digital Library system with other CONSORTIUM platforms so that resources can be referred to and used in other CONSORTIUM applications (for example, exemplar resources)

• Integration of the Digital Library with external digital learning resource repositories such as those procured by the States.

• optionally, Integration of Digital Library component with learningregistry.org for content tagging and aggregation

• Development of interoperability standards and services to facilitate import and export of data to and from state systems.

3.2.8.2 Quality Control

Most of the integration specification will be designed, reviewed, tested and agreed on themselves first. After the specification or standards have been confirmed, Wireless Generation will work with integration partners to define the quality criteria, test harness and testing procedures to validate the interfaces.

Please note: Integration activities in “external” systems might be performed by the other contractors developing those systems. However, Wireless Generation understands that it is to provide reasonable support to these other contractors including, but not limited to,

• providing technical consulting (e.g., advising on Digital Library components or integration points)

• coordinating testing activities • coordinating implementation or troubleshooting efforts


Responsibilities and levels of support will be mutually agreed upon by Wireless Generation, the CONSORTIUM, and other contractors after determining systems integration requirements. For integration efforts requiring code development, files, database scripts, and/or configuration settings (e.g., ETL settings), these will be developed using a mature systems life cycle and development process to ensure good readability (easily read and understood by other programmers), maintainability, and traceability.

3.2.8.3 Defect Resolution

Any integration defects will be first reported to the involved parties to validate against the integration specifications. If the defect cannot be covered by the integration specification, or the root cause cannot be easily determined, the teams will go through integration triage process for the defect to be first traced and then resolved.

3.2.9 Application Maintenance and Support

Wireless Generation will provide staff for support of the system as needed during the full contractual period.

The scope of these services will include:

• application management; • help-desk support; • systems enhancements; • systems maintenance; • adaptive and preventive maintenance; • performance maintenance; and • documentation updates.

3.2.10 Application Hosting

We will provide secure application hosting during the full contractual period to enable seamless access by all end-users. This hosting environment will provide integration with other CONSORTIUM technology components as described in the IT Systems Architecture.

In the contracting phase, we will describe:

• the physical facility • security • data protection and recovery plan • equipment used to host the web-based application • backup site

All data stored in the Digital Library will be owned by the CONSORTIUM or its member states, and as such, upon request from an authorized CONSORTIUM representative, Wireless Generation will provide


content/data from the Digital Library system to the CONSORTIUM and its member states at no additional cost beyond the cost of the contract through the data access capabilities proposed in the system. Wireless Generation understands that it may not use data or provide data to any other persons or entities, unless explicitly authorized by an authorized CONSORTIUM representative.

3.2.11 Knowledge Transfer, Transition, and Turnover

In preparation for contract completion and as part of the continuous improvement plan, Wireless Generation will work with the CONSORTIUM to document a list of requirements for hosting and maintenance activities required to successfully turn over and transition the applications developed/enhanced under this request to the CONSORTIUM or its designee. As part of these requirements for transition activities, we will document any technical training on the administration, maintenance, and enhancement of the application that may be required. This technical training may include, but not be limited to:

• application configuration • operational management • database administration • integration management • content conversion and data migrations • software upgrades

This handover/transition process will also provide a detailed technical manual and accompanying systems documentation.

3.2.12 Our Development Process

While each Wireless Generation development engagement is customized based on the state or district’s requirements, we always follow a similar development process to ensure on-time and on-budget delivery. The process is best described through an example of one of our largest systems development projects — the Achievement Reporting and Innovation System (ARIS), which we built in 2008-09 for the New York City Department of Education (NYCDOE).

Discovery Phase:

Prior to starting development on ARIS, we ran a discovery phase where we facilitated multiple working sessions with our New York City partners in order to devise a scope and release plan for the core functionality of the system. The sessions allowed genuine collaboration between Wireless Generation technical staff and New York City stakeholders and decision makers to jointly identify, prioritize and problem solve areas critical to the success of the project. We foresee the same type of outcome from a Discovery phase with the CONSORTIUM.

The following diagram represents the methodology we use for Discovery and Design.


The Product Design Process diagram shows the movement from gathering requirements to detailed design. This process can be broken down into four stages: Gathering, Synthesis, Ideation and Detailed Design.

• The Gathering portion of the process aims to uncover data related to the project. This stage

typically consists of stakeholder and user interviews, observations, and competitive research. It is meant to be a collection exercise — not an editing exercise.

• The Synthesis stage aims to make sense of the data that was collected in the Gathering stage. We would work to develop models of users — focusing on the sorts of goals and activities they engage in related to the project — and draw out scenarios of how the product would impact their lives. The models (which CONSORTIUM stakeholders validate) provides firm footing on which to base the design decisions that result in the ideation phase.

• The Ideation stage transforms the models developed in Synthesis into user stories that illustrate the functionality to be developed in easily manageable units. A detailed scope document summarizing in concrete terms the functionality to be created and integration to be done is also created. We would ask the CONSORTIUM to approve this document.

• The Detailed Design stage, which includes the design of specific wireframes and screens and of detailed data flows, would then be done at the start of each release. For similar projects, we have found that our discovery and design phase can take a total of four to eight weeks depending on the solution complexity.

The following table shows the breakdown of roles and responsibilities during this process.


Development: Design and Build/Installation and Configuration Phases:

After the Discovery phase, we will go into the Development phase using the Agile with SCRUM methodology, aligned with an iterative portfolio planning process which allows for planning to begin during Discovery. This phase will consist of multiple software releases.

This project consists of two types of software:

1. Custom 2. Off-the-shelf (with OSPI-specific configurations)

The high-level work plan describes the release schedule for both the custom and off-the-shelf portions of the system. The following diagram shows the general process for how the custom and off-the-shelf portions will be developed/configured and released in parallel.

1. Requirements from Discovery – The output of the Discovery activities are the input to the Planning Process

2. High Level Requirements – Product Management, representing the CONSORTIUM, creates high

level requirements and adds them to the product backlog as an Epic story


3. Portfolio Estimation – The Product Line's leadership considers the product functionality and the Engineering Director oversees the estimation process. This may require investigating solutions, or asking Product Management for more details, before getting to a reliable estimate

4. Executable – When the Engineering Director is comfortable with the estimate, they sign off and

mark the work as Executable

5. Scheduled – The Delivery Manager works with Product Management to understand the desired release time frame, works with the Director of Engineering to create a solid Capacity Plan, and reviews dependencies. The Delivery Manager schedules the functionality for a product demonstration milestone or release

6. Portfolio Plan – Once product initiatives are scheduled, there is a Portfolio Plan for what the

team will be working on

Note that the stories have not been broken down to the point where they can be successfully completed in a sprint, that will happen during Release Planning

The Portfolio Plan represents product initiatives which are high level commitments, which

the Product Manager can communicate to Sales and customers

7. Rolling Planning – These steps are repeated on an ongoing basis, throughout the year, to continually evaluate and add and adapt functionality on the Plan

Rolling Planning allows Wireless Generation, Inc. and OSPI to work collaboratively to continually optimize the Plan to maximize the business value delivered by the team, while maintaining the ability to project completion dates for critical functionality.

1. Detailed Requirements – Once the initiative is scheduled, Product Management works on

developing a vision and detailed requirements (though still not a full functional specification)

2. Release Planning – The whole team (all roles, Dev, QA, Design, UX, PM, working with the Product Owner) takes the detailed requirements and breaks down the initiative into User Stories small enough to finish in a sprint, discusses the requirements, and estimates the new stories

3. Release Plan Backlog – The User Stories for the initiative are now the backlog for the Release

Plan

The User Stories are small enough to be completed to the teams Definition of Done in a 2 week sprint

The new User Stories with the new estimates replace the high level Epic stories and

estimates which were in the Portfolio Plan – when there is a difference between the


original estimate and the new team estimate that is escalated back to the Product Line Leadership team for resolution

Not all Product Initiatives will be ready for Release Planning at the same time, they can be

phased in based on the milestones for the plan

Detailed User Stories from a Release Planning session can co-exist on the same Portfolio Plan with the High Level estimates from Portfolio Planning

i.e. The Product Line above has Release Planned Initiative A and B, but Initiative C is not

ready for Release Planning

The Release Planning process allows Wireless Generation, Inc. to collaborate closely with the CONSORTIUM throughout the software development process, creating more detailed plans as detailed requirements are developed. The Release Planning process leverages the collective knowledge of the team to create more detailed and accurate estimates as those requirements are available in increasing detail. The process also greatly increases the probability of building the right functionality through creating a shared understanding of user goals and functionality with the team members who actually implement those requirements.

1. Backlog Prioritization – Team Leadership, led by the Product Owner, prioritizes the backlog The priority is a balance of the highest value features and technical requirements like work

sequencing and technical debt


The Product Owner and Delivery Manager will work closely with the CONSORTIUM to prioritize the backlog in accordance with CONSORTIUM requirements

2. Functional Specifications – The Product Owner manages the process of preparing a User Story for the team to work on, also known as the "Story Ready Process" It is important that the Team has the information and artifacts needed to successfully

complete a User Story in a sprint The Team should work with the Product Owner to define the required inputs, which may

include: Functional Specifications Domain Rules Design and artwork

3. Sprint Planning Meeting Sprint Planning, like Release Planning, is a point where the team builds a deeper common

understanding of the work for each User Story The Product Owner presents the highest priority proposed work (Technical Leads present

the technical stories) to the team, shows any additional documentation or designs, and sets the Acceptance Criteria for the User Story

The team plans out the work to accomplish those User Stories, breaking the work into tasks and estimating in hours. The team then decides how much of the work to commit to for the sprint

4. In the Sprint The team updates tasks and remaining hours on the Sprint Board, tracking the burndown of

remaining hours Teams must scrum daily to review sprint tasks completed, escalate blockers, and otherwise

keep the work on track to meet the sprint commitment 5. Sprint Review and Retrospective

At the end of each sprint, teams must: Hold a Sprint Review, including live demonstrations of working software and a summary

of work completed and missed in the Sprint Hold a Sprint Retrospective to review their Scrum process, identifying areas that are

working well and those that need improvement. Hold a Sprint Planning meeting to determine their commitment for the coming sprint Update their Release Plan including recalculation of velocity and any reprioritization of

user stories 6. Sprint Demonstrations

At the end of each sprint, teams must give live demonstrations of completed software Every other sprint, maximum of 4 weeks apart, the team will hold a “Shared Review”

with CONSORTIUM stakeholders (it is possible to include the CONSORTIUM in every sprint review demonstration)


Quality Assurance Phase:

Wireless Generation has a highly experienced QA team — one of the strongest in the K–12 industry we believe. This same team is responsible for QA of the mCLASS portfolio of products: technically sophisticated handheld-and-web based products used in 200,000 classrooms across the US. The software is frequently enhanced and yet extremely stable.

Commitment to Quality starts with developer coding practices. The company’s development methodology, delivering working software on a regular basis throughout the development process, starts the Quality Assurance early, and continues on through the release process described below. As described in the Quality Assurance section, Developers create and execute unit tests while QA creates test cases, which are run through as each piece of functionality is created. Frequent deliveries of working software also allows stakeholder review of functionality implementation and user acceptance to start early in the process, greatly facilitating collaboration on functionality.

The QA team will develop detailed test plans to ensure thorough testing of every piece of functionality which will be linked back to a master traceability matrix. Our team consists of QA individuals who are experts in database testing as well as individuals who have years of experience with testing front-end applications performing both manual and automatic (e.g., Selenium) tests on the software. Where applicable, QA creates and maintains an automated testing suite which runs in a continuous integration environment. Automated testing enhances the ability to catch code dependencies where development for new functionality may break existing functionality, and reduces the turnaround time between coding and bug fixing.

Our QA team is integrated into the overall team from the beginning of the project, providing the client with robust software. Our aim is for the client acceptance team to find very few bugs. Before each release, a full gamut of test cases is performed on the software by our QA team. In order for each piece of functionality to be considered “done” (at the end of a sprint), it requires that a QA professional signs off on the code. After Code Complete at the end of each release (when developers stop creating new functionality), the QA team takes a full two weeks to do final checks and validation of the code’s quality.

The beta version of the platform is released to a small sub-section of the client’s user population (to be determined between Wireless Generation and OSPI). The configuration, then, would be approved by the client. After the approval of the beta release, the platform is ready for final production and release to the general user population. During all phases of the release process, problems are reported and managed through a software system designed to document, track, and communicate problem resolution. Fixes are then added to the platform to address the identified problem.

Releases:

Wireless Generation has a strong control process for releasing software, with a strong and effective emphasis on dependency management and quality controls. This process starts with the core requirement of having well defined testing environments where software is integrated and tested prior to release to production.


Once a product is in Production, the release process proceeds through four stages and environments:

1) Future QA – Development teams can start the process of integrating their code with the main branches here, and start testing with real data.

2) Current QA – Code is moved here when the release is a month out, and the QA teams start integration testing and regression with all the other new code for the upcoming release.

3) Staging (Pre-Production) – This environment closely mirrors the Production environment, and has two major periods and purposes:

a. Staging Week 1 – The release engineer runs the release as it will be performed on Production, testing the act of releasing itself. QA then tests to ensure that the release includes all intended features and bug fixes, etc., and necessary fixes are added to the release. During this period teams must obtain approval from the Change Control Committee before adding fixes or new features to Staging.

b. Staging Week 2 – During the second week on staging, the teams regress the applications again as the release is now stable and ready for production. During this period the Change Control Committee no longer approves new functionality for the release, and closes scrutinizes requests to add fixes in order to balance risk to the release. Where a formal User Acceptance Test process exists, user testing can be performed during this period, and the duration on Staging can be increased to accommodate such testing.

4) Production – Releases to Production are performed after hours to avoid customer impacts. The Release Engineer orchestrates the release, running the necessary scripts, and when complete QA tests to ensure that the release completed properly prior to opening the system back up to customers.

The process for Hotfixes, issues too urgent to wait for the next monthly release, is similar to the above process but will have a compressed timeframe. The Change Control Committee must approve all requested fixes prior to a release to Staging, and the committee strongly considers the perceived benefits of performing a hotfix against the risks posed by the release.

The Wireless Generation Project Manager will work with the CONSORTIUM to plan out the process and timing for releases to ensure minimum impact on CONSORTIUM Users after the initial Beta and Production releases.


3.3 Resource Tagging and Uploading System

3.3.1 Resource Tagging System

The resource tagging process is considered as an integral part of the CMS subsystem of the Digital Library. In addition to serve the internal content management workflow, the tagging components would support the following two needs: 1) to connect assessment reporting categories with resources to improve teaching and learning and 2) to search resources based on content/grade/topic categories including, but not limited to formative assessment practices, strategies, and tools.

The Digital Library tagging components will leverage 3 key sets of standards:

• LRMI meta-data standard; • Assessment to CCSS (Common Core State Standards) meta-data standards currently being

defined by SIF and IMS Global for PARCC and Smarter Balanced; and • CCSS taxonomy hosted by the NGA Center and CCSSO.

The tagging system will adopt both the taxonomy structure for authoring/displaying and the meta-data structure to internally store and map resources. The following diagram illustrates this process.


Recommend Learning Resource

Learning Object Federated

Search

CCSS Taxonomy Link

6 Search and Recommendation 5

1

Other Tag

CCSS Taxonomy

2

Tag Term Usage

Tagging

User Behavior

Ontology DB 4

Taxonomy 3

Learning Taxonomy Cache and Tag

Related Paradata Management Paradata

Object Metadata

Social Tagging

LRMI Tagged Learning Content

LearningRegistry Node

In addition to allowing the tags to help with site navigation, search, filtering and moderated groups, we plan to make the tags an integral part of the content recommendation. A conceptual process is illustrated below.


Following the development strategy definition, we will use an agile development method involving the Digital Library Review Board, Work Groups, State Leads, State Leadership Teams, and State Networks of Educators to incorporate their feedback on the list of resource attributes and the proposed set of required tags for resources in the Digital Library.

3.3.2 Resource Uploading and Removal System

The resource uploading and removal process is also considered as an integral part of the CMS subsystem of the Digital Library. This component manages the change management and data conversion activities of the CMS process. Similar to the “normal” CMS process but with extended support for batch uploading and removal, the system will provide appropriate editorial review for each resource and the overall permission process. Wireless Generation will implement an efficient process for an authorized administrator to edit, update, and remove resources. We will develop the steps for editorial review and approval for upload and removal. These procedures will form the basis of the required processes for each resource.

Following the strategic development strategy, we will use an agile development method involving the Digital Library Review Board, Work Groups, State Leads, State Leadership Teams, and State Networks of Educators to incorporate their feedback on the Resource Uploading and Removal System.


3.4 Quality Criteria Policies

3.4.1 Assemble and Moderate a National Panel of Experts

The CONSORTIUM requires the assembly of a National Advisory Panel of no fewer than eleven and up to twenty experts of formative assessment practices, adult on-line professional learning, and quality instruction incorporating universal design for a range of diverse learners. The panel will develop and recommend policies to establish quality criteria to be applied to all resources recommended for the Digital Library.

We believe that, as long as the panel reflects an appropriate level of diversity in experience and expertise, the optimal number of panelists is approximately 12. This allows for healthy idea exchanges, while keeping discussions and decision points manageable. We recommend a list of the following experts who represent the Smarter Balance criteria, who are leaders in the area of special education, gifted education, and ELL, and who provide a diverse background/experience in urban and rural education. We also believe that a focus in literacy, ELA and math is necessary due to the foundation of the CCSS. In addition, since the CCSS also has priorities in college and career readiness and literacy across the disciplines, we believe that representatives from science and social studies would also offer a valuable perspective as well. Lastly, our list has representation from the various states in the CONSORTIUM.

We are putting forth recommendations for more than the recommended 12 panelists with the knowledge that due to scheduling or other commitments, a number of the experts recommended may not be able to commit to the panel. The experts who constitute the panel will have their expenses paid, as well as receive an honorarium for their participation.

• Dr. Lynne Anderson-Inman – Learning with Tech (University of Oregon) • Dr. Deborah Loewenberg-Ball – Chair MCEE (University of Michigan) • Barbara Bowman – Co-founder, Erikson Institute, Early Childhood expert • Dr. Greg Cizek – Educational Measurement and Assessment (UNC Chapel Hill) • Charlotte Danielson – Teacher Evaluation, adult on-line learning • Dr. Lisa D. Delpit – Center for Urban Educational Excellence (Florida International University) • Dr. Lynn Fuchs – Special Education/Reading (Vanderbilt) • Dr. Roland Good – DMG, DIBELS author (University of Oregon) • Dr. Margo Gottlieb – Director, Assessment & Evaluation (Illinois Resource Center), Lead

Developer, World-Class Instructional Design and Assessment (WIDA)Consortium • Dr. Thomas Heir – Former United States Director of Special Education (Harvard) • Dr. John Hill – National Rural Education Association (Purdue University) • Dr. John Hintze – Curriculum-based measurement/assessment (University of Massachusetts) • Dr. Ruth Kaminski – DMG, DIBELS author (University of Oregon) • Dr. Sandra Kaplan – Gifted Education in Urban Areas (USC)


• Dr. Henry Kepner – Past President NCTM (University of Wisconsin) • Salman Khan – online learning, Khan Academies • Dr. Katherine McKnight – Adolescent Literacy, Technology Integration (NCTE) • Jay McTighe – Understanding by Design • Dr. Alan Schoenfeld – Math Education (UC Berkeley) • Lydia Stack – Center for Applied Linguistics, English Language Learner (ELL) author, expert • Dr. Carol Ann Tomlinson – Differentiation (University of Virginia) • Dr. Grant Wiggins – Understanding by Design • Dr. Steven Zemelman – Best Practices for Instruction, Director Chicago Schools Alliance

We have included brief biographies of the proposed panel members in section 3.4.1.1. We have already received commitments from several proposed panelists to serve on the Panel if we are awarded the contract. However, in order to comply with the 200 page limit of the technical proposal, we are not including the actual resumes or curricula vitae from those experts. We will work with all experts, and their respective employers or staff, to provide their resumes upon request.

The National Advisory Panel meetings will be held between January and March 2013, with the first one occurring within 30 business days of the contract execution as required in the RFP. We will hold three face-to-face meetings, each 2 days in length (for a total of 6 days), and will provide all the technical and administrative support for each meeting.

For each of the three 2-day meetings of the National Advisory Panel, Wireless Generation will arrange for the invitations, dates, times, venue and agendas. We have extensive experience in preparing for and leading large events and workgroup meetings like this, and will bring that experience to bear in both preparing the content and logistics for the meetings. We will work with the Formative Work Group, State Leads, State Leadership Teams, and Smarter Balanced Directors to construct agendas that will elicit the experience of our diverse Panelists in constructing policies for quality criteria. We will also prepare background slides to lead the discussion, and templates to lead brainstorming and sharing activities. We will schedule our staff to take notes and synthesize the discussion at each large and small meeting that takes place during the two days, so we can immediately reflect back drafts of policies to the Panelists. Finally, we will take care of all logistics to ensure the days flow smoothly, providing maximum time for collaborating. This includes selecting and setting up a venue, A/V equipment, and catering during the day, as well as planning the meetings in areas that are convenient to the participants.

The goal for the first meeting, to be held in January 2013, will be to develop the first draft of proposed policies for the Digital Library. Wireless Generation will provide all the technical and administrative support in order for the National Advisory Panel to reach that goal. Specifically, our project management team will convene the meeting and lead a brainstorm meeting with the panelists, doing breakout groups and small-team work to bring ideas to the fore. We will then facilitate a prioritization process to synthesize the proposed policies. After the 2-day session, our team will circulate to the Panelists the draft of policies completed by the group.


At the second meeting, held in February 2013, the Formative Work Group, State Leads and State Leadership Team and Smarter Balanced Directors will reconvene with the National Panel of Experts to provide feedback on the first set of proposed policies. We will encourage face-to-face participation, but realize that some members may not be able to join, and will therefore remote participants will be able to utilize WebEx to remotely attend the conference.

In March 2013, the third meeting will reconvene, face-to-face and via WebEx, the National Advisor Panel to finalize the proposed policies and to make final recommendations to the Smarter Balance Directors.

Through our extensive experience in working with assessment authors, curriculum experts, and external panels, we learned that a backward design approach is the best way to ensure the policies recommended by the National Advisory Panel will first outline and then define the quality indicators for the proposed policies. After those indicators are defined, we will work with the Panel to ensure those indicators will establish quality criteria to be applied to all resources recommended for the Digital Library.

We will facilitate the development of norms to clarify expectations and help the group work together more effectively. We will use the Nominal Group Technique protocol, which supports consensus- building, to moderate the National Panel and facilitate the development and refinement of the recommended policies between contract execution and April 2013.

All stakeholders will receive an on-line survey prior to the second meeting. The survey will allow Work Groups, State Leads, State Leadership Teams, and State Network of Educators to vote on proposed policies and provide feedback. This will allow the bulk of time during the second meeting to be spent discussing policies that were not unanimously approved and reviewing the comments/concerns raised. The National Panel of Experts will use the feedback provided by the survey and the subsequent conversations to finalize the draft policies. We will implement the Review Cycle described in Exhibit O to provide the final policies to the Governing States for a vote to be completed by May 2013.

The policies developed by the National Panel of Experts and approved by the Governing States will address all four areas as outlined in the RFP:

• Quality indicators of formative assessment processes and tools – These indicators will be

used in determining whether the Assessment Literary professional learning materials, process and tools provide the necessary feedback loops for the resources and structures required in the RFP and by the CONSORTIUM.

• Quality Indicators of professional learning – Our approach to professional learning is closely aligned to the Learning Forward Standards for Professional Learning. These standards will be used to ensure the quality criteria serve as a platform to develop leadership capacity, foster comfort with data sharing and usage, build learning communities focused on continuous improvement, and implementation support to foster long-term change.

• Quality indicators of effective instruction – These indicators will be used to guide the development of the Exemplar Instructional Modules, which will be made available to


educators in all grade bands for ELA/Literacy and Math, as well as determining which currently available resources are approved for inclusion in the Digital Library.

• Quality indicators of the effective use of evidence – The Formative Work Group, State Leads, SLTs, and SNEs will use these indicators to evaluate whether professional learning materials in the Digital Library and Exemplar Instructional Modules are effective in their use of evidence.

3.4.1.1 Proposed National Advisory Panel – Summary Resumes

Dr. Lynne Anderson-Inman is Director of the Center for Advanced Technology in Education (CATE), the Center for Electronic Studying (CES), the Oregon Writing Project at the UO, and the National Center for Supported eText (NCSeT), all in the College of Education at the University of Oregon. Anderson-Inman is an internationally recognized expert on the use of technology to improve reading, writing, and studying, with special emphasis on strategies to enhance the academic success of struggling learners. She pioneered the concept of “computer-supported studying,” conducting research and developing materials to increase students’ academic achievement by using the computer as a study tool. She has directed numerous federally funded projects, and is a widely published author on effective uses of technology, and a frequent speaker at national and international conferences. She earned a BA and MS from the University of Wisconsin and a Ph.D. from the University of Oregon.

Dr. Deborah Loewenberg Ball currently serves as dean of the University of Michigan School of Education, where she is also the William H. Payne Collegiate Professor and an Arthur F. Thurnan Professor. Ball is an expert on the professional training of teachers. Her program of research focuses on mathematics instruction, and on interventions designed to improve its quality and effectiveness. Ball has authored or co-authored over 150 publications and has lectured and made numerous major presentations around the world. Her website is used extensively by scholars, policymakers, and practitioners. Ball's research has been recognized with several awards and honors, and she has served on several national and international commissions and panels focused on policy initiatives and the improvement of education, including the National Mathematics Advisory Panel and the National Board for Education Sciences. She is an elected member of the National Academy of Education. She earned a Ph.D. from Michigan State University.

Barbara Bowman is one of the founding members of the Erikson Institute, and served as president of the institute from 1994 to 2001. She is an authority on early education, a national advocate for improved and expanded training for practitioners who teach and care for young children, and a pioneer in building knowledge and understanding of the issues of access and equity for minority children. She was chief early childhood education officer for the Chicago Public Schools and a former consultant to Secretary of Education Arne Duncan. She is past president of the National Association for the Education of Young Children (NAEYC) and has served on numerous boards, including the High Scope Educational Foundation, the Institute for Psychoanalysis, Business People in the Public Interest, the Great Books Foundation, the Chicago Public Library Foundation, and the National Board for Professional Teaching Standards. Among the honors Ms. Bowman has received are the Voices for Illinois' Children Start Early Award; Chicago Association for the Education of Young Children Outstanding Service to Children Award;


Harold W. McGraw Jr. Prize in Education; and the National Black Child Development Institute Leadership Award. She has a BA from Sarah Lawrence College and an MA from the University of Chicago.

Dr. Gregory J. Cizek teaches courses in applied psychometrics, statistics, program evaluation, and research methods at the University of North Carolina–Chapel Hill. His background in the field of educational assessment includes 5 years as a manager of licensure and certification testing programs for American College Testing (ACT), and 15 years of teaching experience at the college level. He is the author of over 200 books, chapters, articles, conference papers, and reports. His books include Handbook of Educational Policy (Academic Press, 1998); Cheating on Tests: How to Do It, Detect It, and Prevent It (Lawrence Erlbaum, 1999); Setting Performance Standards: Concepts, Methods, and Perspectives (Lawrence Erlbaum, 2001); and Detecting and Preventing Classroom Cheating (Corwin Press, 2003). Cizek has served as an elected member and vice president of a local school board in Ohio, and he currently works with several states, organizations, and the U.S. Department of Education on technical and policy issues related to large-scale standards-based testing programs for students in grades K–12. He began his career as an elementary school teacher in Michigan, where he taught second and fourth grades. He received his Ph.D. from Michigan State University.

Charlotte Danielson is an internationally-recognized expert in the area of teacher effectiveness, specializing in the design of teacher evaluation systems that, while ensuring teacher quality, also promote professional learning. Based in Princeton, NJ, she advises State Education Departments and National Ministries and Departments of Education, both in the United States and overseas. She is in demand as a keynote speaker at national and international conferences, and as a policy consultant to legislatures and administrative bodies. Danielson's many publications range from defining good teaching (Enhancing Professional Practice: a framework for teaching, 2007), to organizing schools for student success (Enhancing Student Achievement: a framework for school improvement, 2002), to teacher leadership (Teacher Leadership that Strengthens the Profession, 2006), to professional conversations (Talk about Teaching! Conducting Professional Conversations, 2009), to numerous practical instruments and training programs (both on-site and online) to assist practitioners in implementing her ideas. She holds a BA in history from Cornell University, and advanced degrees (in philosophy, economics, and educational administration) from Oxford and Rutgers Universities.

Dr. Lisa D. Delpit is an Eminent Scholar and Executive Director of the Center for Urban Educational Excellence at Florida International University in Miami. She has served on the Commission for Research in Black Education (CORIBE) and received the MacArthur Genius Fellowship and the award for Outstanding Contribution to Education from the Harvard Graduate School of Education. Her general areas of focus are in elementary education and language/literacy development, but Delpit is passionate about teaching culturally-relevant approaches to educating students and issues related to race, such as the educational access granted to minority groups. She is the author of Other People’s Children: Cultural Conflict in the Classroom Updated Edition (2006) and Multiplication is for White People: Raising Expectations for Other People’s Children (2012). She earned a BA from Antioch College and a MA and Ed.D. from Harvard University.


Dr. Lynn Fuchs is the Nicholas Hobbs Professor of Special Education and Human Development at Vanderbilt University, where she also co-directs the Kennedy Center Reading Clinic. She has conducted research on assessment methods for enhancing instructional planning and on instructional methods for improving reading and math outcomes for students with learning disabilities. Fuchs has published more than 200 studies in peer-review journals and sits on the editorial boards of 10 journals. Her awards include the Council for Exceptional Children's Career Research Award; Vanderbilt University's Joe B. Wyatt Distinguished University Professor; Vanderbilt's Earl Sutherland Award for Research Accomplishments; the American Education Research Association's Distinguished Researcher Award from the Special Education Research SIG; the 2001 Article of the Year Award for best article in the 2000 volume year in School Psychology Review; the 2000 Council for Exceptional Children/Division of Learning Disabilities Samuel A. Kirk Award for the exemplary practice article from the 1998 volume of Learning Disabilities Research and Practice; the 2000 Alumni Distinguished Faculty Scholar Award, awarded by the Peabody Alumni Board of Vanderbilt University; the 1998 American Educational Research Association's Palmer O. Johnson Award for the outstanding article appearing in an AERA-sponsored journal for the 1997 volume year; the 1998: Mayor's Educator of the Year Award (Nashville, TN); the 1997 Learned Article Award from the Educational Press Association; and the 1996 School Psychology Quarterly/American Psychological Association Division 16 Fellows Award for Best Articles. She earned a Ph.D. from the University of Minnesota.

Dr. Roland Good is an Associate Professor of School Psychology at the University of Oregon and co- author of DIBELS. Good completed his undergraduate degree in Elementary and Special Education, and he worked as a teacher in elementary general education and special education classrooms. He earned his doctorate from Pennsylvania State University in School Psychology and served as a school psychologist. In recognition of his contributions to the field of education, Penn State awarded Good its Excellence in Education Award in 2005. For the past 20 years, Good has led the program of research and development culminating in the Dynamic Indicators of Basic Early Literacy Skills (DIBELS). Good provides DIBELS professional development training to educators and administrators throughout the United States. He has also served on the editorial boards for School Psychology Review, School Psychology Quarterly, and the Journal of Special Education and has presented more than 100 papers at national conferences.

Dr. Margo Gottlieb is a nationally recognized specialist in the design of assessments, and the development of English language proficiency standards for English language learners (ELLs) in Pre-K–12. She began her career as an English as a second language (ESL) and bilingual teacher and has spent the past two decades providing technical assistance to governments, states, school districts, publishers, universities, and professional organizations. Gottlieb has also served on numerous national task forces, expert panels, and technical working groups and was a Fulbright Senior Specialist in Chile. As Lead Developer for World-Class Instructional Design and Assessment (WIDA), Gottlieb has spearheaded the design and development of the Pre-K–12 WIDA English Language Proficiency Standards and has authored the accompanying guides. In addition, Teachers of English to Speakers of Other Languages (TESOL) invited her to chair the committee on the national Pre-K–12 English Language Proficiency Standards. Gottlieb has a Ph.D. in Public Policy Analysis, Evaluation Research and Program Design, an

http://www.wida.us/standards/eld.aspx


MA in Applied Linguistics, and a BA in the teaching of Spanish. She also serves as Director of Assessment and Evaluation for the Illinois Resource Center and publishes extensively, including assessment instruments, handbooks, manuals, and articles. Her latest books are Assessing English Language Learners: Bridges from Language Proficiency to Academic Achievement (Corwin, 2006) and Assessment and Accountability in Language Education Programs: A Guide for Administrators and Teachers (with Diep Nguyen; Caslon, 2007).

Dr. Thomas Hehir is a Professor of Practice in Learning Differences at the Harvard Graduate School of Education where he teaches courses on educating students with disabilities and federal education policy. Hehir served as director of the U.S. Department of Education's Office of Special Education Programs from 1993 to 1999. As director, he was responsible for federal leadership in implementing the Individuals with Disabilities Education Act (IDEA). In 1990, he was associate superintendent for the Chicago Public Schools, where he was responsible for special education services and student support services. In this role, he implemented major changes in the special education service delivery system, which enabled Chicago to reach significantly higher levels of compliance with the IDEA and resulted in the eventual removal of the U.S. Department of Education's Office for Civil Rights as overseer. Hehir served in a variety of positions in the Boston Public Schools from 1978 to 1987, including that of director of special education from 1983 to 1987. He earned an Ed.D. from Harvard University.

Dr. John Hill is a professor of Educational Studies and the Executive Director of the National Rural Education Association (NREA). The NREA is a Purdue University-based group that pursues educational equity for rural school districts, helps fund research supporting that goal, and represents rural schools at the national level. Hill advocates for rural districts because they face unique challenges related to staffing, resource allocation, and funding, which makes it difficult for them to meet state standards. He has a BS from Manchester College, an MS from Indiana University, and a Ph.D. from Indiana University.

Dr. John Hintze is a Professor and Director of the University of Massachusetts School Psychology Program. Prior to entering university training he was a practicing school psychologist in New York and New Jersey for 10 years. Hintze is well known for his work in curriculum-based measurement, assessment, response to intervention, and data-based decision making. He is currently a senior consultant for the National Center on Response to Intervention, an Office of Special Education funded project sponsored by the U.S. Department of Education. Hintze currently serves on the editorial boards of more than 10 journals in school psychology and special education. His current research is in curriculum-based measurement, response to intervention, data-based decision making, research design and data analysis, and program evaluation. He earned a BA from New York at Cortland, a MA from Alfred, and Ph.D. from Lehigh.

Dr. Ruth Kaminski is Director of Research and Development for Dynamic Measurement Group and co- author of DIBELS. Kaminski’s academic background includes degrees in Speech Pathology, Early Intervention, and School Psychology. For the past 20 years she has conducted research on assessment and preventative interventions for pre-school and early elementary-age children. Kaminski has extensive experience providing consultation to Head Start agencies and public schools throughout Oregon and the United States. In addition, she has over 10 years’ experience as a classroom teacher and


speech/language clinician with pre-school-age children. In 2007, Kaminski was selected as the recipient of Pennsylvania State University's Excellence in Education Award, the highest honor bestowed on an alumnus of the College of Education. Kaminski is a participating faculty member in the School Psychology Program at the University of Oregon, where she earned a Ph.D.

Dr. Sandra Kaplan is a professor of Clinical Education at the University of Southern California, Rossier School of Education, where she specializes in gifted education in urban areas and differentiated curriculum for high-achievers. She works with Gifted and Talented Education (GATE) and is the past president of both the National Association for Gifted Children (NAGC) and the California Association for the Gifted (CAG). She developed Kaplan’s Grid to support effective teaching of gifted students via differentiation. This model helps educators construct curricular units by showing them how to modify the core to meet the needs of inner-city, gifted learners. Kaplan also developed an on-line professional development/certificate program: Differentiated Curriculum for Gifted Students through USC Rossier, as well as an on-line bank of standards-based lessons to help teachers learn about differentiated curriculum by giving them a set of field-tested lessons to teach from. She earned an Ed.D. from the University of California.

Dr. Henry Kepner is a Professor in the Department of Curriculum and Instruction at the University of Milwaukee in Wisconsin. He takes his expertise in mathematics education directly to school districts and classrooms, both locally and nationally. His research interests in mathematics education, teacher education, and the mathematical knowledge of teachers have resulted in numerous publications, funded projects, professional development programs, and speaking engagements. Kepner was President (2008–2010) of the National Council of Teachers of Mathematics. Kepner also served 5 years as program officer at the National Science Foundation in Washington, D.C. He was a founding member and first president of the Association of Mathematics Teacher Educators. He has served as president of the National Council of Supervisors of Mathematics, the Wisconsin Mathematics Council, and the Milwaukee Educational Computing Association. He served on the boards of directors of the National Council of Teachers of Mathematics and the School Science and Mathematics Association. He also taught middle- and high-school mathematics for 12 years in Milwaukee and Iowa City. Kepner earned his BS and MS degree in mathematics and Ph.D. in mathematics education at the University of Iowa.

Salman Khan is the founder of Khan Academy, a free, online educational institution that he launched in 2006. Khan has extensive knowledge in math and science including: an MBA from Harvard Business School, an MA in electrical engineering and computer science, a BS in electrical engineering and computer science, and a BS in mathematics from MIT. This knowledge, coupled with his desire to empower people of all ages to learn, prompted him to develop over 3,000 tutorial videos focused on math and science. Backing from the Bill and Melinda Gates foundation and Google have helped him expand his initial scope and build in additional features such as follow-up assessments for students based on their skill level and performance and a variety of teacher reports.

Dr. Katherine McKnight is an author, educator and consultant. Her career in education began as a high- school English teacher in the Chicago Public School system more than 20 years ago. Today, she serves as a professor of secondary education at National Louis University and as an on-site professional


development consultant for the National Council of Teachers of English (NCTE). McKnight is passionate about creating curricula that engage all students in the regular education classroom. She regularly publishes in professional journals and is the author of many books including The Teacher’s Big Book of Graphic Organizers, Grades 5–12 (recipient of the 2013 Teachers’ Choice Award). She received her B.A. degree from George Washington University, her M.Ed. from Northeastern Illinois University, and her Ph.D. from the University of Illinois at Chicago.

Jay McTighe was director of the Maryland Assessment Consortium, a state collaboration of school districts working together to develop and share formative performance assessments. Prior to that, McTighe was involved with school improvement projects at the Maryland State Department of Education where he helped lead Maryland’s standards-based reforms, including the development of performance-based statewide assessments. He also directed the development of the Instructional Framework, a multimedia database on teaching. In addition to his work at the state level, Jay has experience at the district level in Prince George’s County, Maryland, as a classroom teacher, resource specialist, and program coordinator. He also directed a state residential enrichment program for gifted and talented students. McTighe has co-authored 12 books, including the best-selling Understanding by Design® series with Grant Wiggins (see below). He has written more than 30 articles and book chapters, and has published in leading journals, including Educational Leadership (ASCD) and The Developer (National Staff Development Council). He earned his undergraduate degree from the College of William and Mary, earned a Master’s degree from the University of Maryland, and completed post-graduate studies at Johns Hopkins University. He was selected to participate in the Educational Policy Fellowship Program through the Institute for Educational Leadership in Washington, D.C., and served as a member of the National Assessment Forum, a coalition of education and civil rights organizations advocating reforms in national, state, and local assessment policies and practices.

Dr. Alan Schoenfeld is a professor of education and of mathematics at UC Berkeley. A fellow of the American Association for the Advancement of Science, American Educational Research Association and a laureate of the education honor society Kappa Delta Pi, he has served as president of the American Educational Research Association and as vice president of the National Academy of Education. He was given the Senior Scholar Award by AERA’s Special Interest Group for Research in Mathematics Education in 2008. Schoenfeld was lead author for grades 9–12 of the National Council of Teachers of Mathematics’ Principles and Standards for School Mathematics. He was one of the founding editors of Research in Collegiate Mathematics Education, and has served as associate editor of Cognition and Instruction. He has written, edited or co-edited 21 books and more than 150 articles on thinking and learning. He has an ongoing interest in the development of productive mechanisms for systemic change and for deepening the connections between educational research and practice. After earning a BA in mathematics from Queens College, Schoenfeld received a MS and Ph.D. in mathematics from Stanford University.

Lydia Stack is an internationally-known teacher, educator, and author. She currently presents SIOP workshops for the Center for Applied Linguistics and is a founding member of the Understanding Language Initiative on Language, Literacy, and Learning in the Content Areas at Stanford University. The


goal of this project is to enrich academic content and language instruction for English Language Learners (ELLs) in grades K–12 by making explicit the language and literacy skills that are required to meet the Common Core State Standards (CCSS) and Next Generation Science Standards. Previously she worked in the San Francisco Unified School District where she taught elementary and secondary English as a Second Language (ESL) teacher. In addition, she taught Second Language Acquisition classes at Stanford University and San Francisco State University. In 1991–1992 Stack was President of TESOL (Teachers of English to Speakers of Other Languages), an international association of teachers concerned with English language teaching worldwide. Stack has co-authored several textbooks and programs for English Learners including American Themes — a literature anthology for ACCESS students for the Office of English Language Programs (In production). On Our Way to English, K–5 — an ESL program for elementary students, 2010; The McDougal-Littell Literature and Language Arts Program, 2007; ESL Guide to McDougal-Littell Social Studies Program, 2004; On Our Way to English, Rigby, 2003; Visions (Books 1– 3), Heinle, 2003; Voices in Literature (Bronze, Silver and Gold), Heinle, 1996; Making Connections (Books 1–3); Heinle, 1995; and WordWays (Game boards, Cubes, and Cards), Alta Book Publishers, 1979.

Dr. Carol Ann Tomlinson is a career educator. Her expertise includes 21 years as a public school teacher, and 12 years as a program administrator of special services for struggling and advanced learners. She was Virginia's Teacher of the Year in 1974. More recently, she has been a faculty member at the University of Virginia's Curry School of Education, where she is currently William Clay Parrish Jr. Professor and Chair of Educational Leadership, Foundations, and Policy. Tomlinson is also co-director of the University's Institutes on Academic Diversity. She was named Outstanding Professor at Curry School of Education in 2004 and received an All University Teaching Award in 2008. A reviewer for eight journals, Tomlinson is also author of over 200 articles, chapters, books, and other professional development materials. For ASCD, she has authored several books including How to Differentiate Instruction in Mixed-ability Classrooms and The Differentiated Classroom: Responding to the Needs of all Learners and professional inquiry kit on differentiation. Recently, she co-authored a book with Jay McTighe titled Integrating Differentiated Instruction and Understanding by Design: Connecting Content and Kids and with Kay Brimijoin and Lane Narvaez co-authored The Differentiated School: Making Revolutionary Change for Teaching and Learning. For Corwin Press, she is co-author of The Parallel Curriculum Model: A Design to Develop High Potential and Challenge High Ability Learners. She has a BA from the University of South Carolina, and an M.Ed. and Ed.D. from the University of Virginia

Dr. Grant Wiggins is the President of Authentic Education in New Jersey. Grant consults with schools, districts, state and national education departments on a variety of reform matters. He is best known for being the co-author, with Jay McTighe, of Understanding by Design®, the award-winning and highly successful program and set of materials on curriculum design used all over the world; and of Schooling by Design. He is also a co-author for Pearson Publishing on more than a dozen textbook programs in which UbD™ is infused. His work has been supported by the Pew Charitable Trusts, the Geraldine R. Dodge Foundation, and the National Science Foundation. Most recently, Grant has worked on some of the most influential reform initiatives in the world, including Ted Sizer's Coalition of Essential Schools, the International Baccalaureate Program, the Advanced Placement Program; state reform initiatives in


New Jersey, New York, and Delaware; and national reforms in China, the Philippines, and Thailand. He earned his Ed.D. from Harvard University and his B. A. from St. John's College in Annapolis.

Dr. Steven Zemelman is the director of the Chicago School Alliance, which is a network of schools dedicated to the ideas of shared professional learning, distributed leadership, practice-based inquiry, and sustainability. He had a hand in several other Chicago-based initiatives such as the Illinois Writing Project and the Center for City Schools at National-Louis University, which he directed for eight years. Zemelman is an educational consultant dedicated to supporting school change in the areas of literacy education, school improvement, teacher voice, instructional-leadership teams, student/peer mediation, and curriculum development and design. In addition to this important work with schools, he has authored a series of books about best practices for instruction including: Bringing Standards to Life in American Classrooms (2012), Today’s Standards for Teaching and Learning in America’s Schools (2005), and New Standards for Teaching and Learning in America’s Schools (1993). He has a Ph.D. from Brandeis University.

3.5 Sustainable Leadership System

Sustainability Built In From the Beginning

Sound leadership systems are important for all projects, but in complex multi- stakeholder environments they are vital. The system must conform to established governance policies, have a clear organizational structure, establish roles and responsibilities, facilitate timely consensus- based decision-making, and support communication flows to the appropriate stakeholders. Developing and then enshrining these structures, processes, and policies will provide the framework; but to motivate participants, engender a sense of purpose, and to ensure sustainability, SLT and SNE need strong leadership and the capacity to evolve to meet new and changing challenges. Educators have something of a head start in this arena – they are typically good communicators and consensus builders. But our approach and plan will enable and build on these skills to lead the project to a successful conclusion.

Sustainability is not something that can be added near the end of a project. We have integrated sustainability into our approach, systems, and processes. To ensure sustainability we used a defined model, and establish operational and phase-end measures to determine if SLT and SNE are making progress toward sustainability and take corrective action based on performance measures.


Sustainability requires that SLT and SNE meet current needs and obligations, but also that they do so without compromising meeting future needs or obligations in the following four dimensions:

Financial –The system must be able to be funded at a level that ensures there sufficient resources for stakeholders to undertake their tasks for the duration of the project.

Social and Human–In similar situations, stakeholders move through stages characterized by initial confusion, interest and participation, and then decline as needs are left unmet or opinions are not adequately addressed. The challenge is to maintain stakeholder interest, as the project moves through various program and stakeholder transitions.

Environment– The system must be able to respond to policy and political transitions, varying stakeholder needs and maturity levels, and be flexible enough to change to meet stakeholder needs.

Processes– Complex and time consuming processes can yield significant results, but over time and faced with competing priorities, stakeholders tend to not complete complex or time consuming tasks, or complete them at a cursory level. The key is to design processes that are simple, scalable, and then document them.

Approaches – Challenges and Opportunities

Developing, implementing, and sustaining a part-time consensus-based organization of more than 2,700 educators across 27 states is a challenge. Notwithstanding this, we see significant opportunity for the CONSORTIUM and each state to contribute to and draw upon the host of aligned resources that the system will provide. Our approach is to develop policies, processes, and procedures that will enable SLT and SNE to produce results, but do so in harmony with existing structures, communication channels, and organizational cultures. Below we identify some of the issues and challenges for SLT, SNE, and other stakeholders.

Distributed Working Groups

Having teams spread out over twenty-seven states, and also noting that some of states are larger than many countries — special attention needs to be paid to remote leadership and management training and productivity systems. To meet this challenge we’ll provide guidance on organization structure and policy and we will advise SLT and SNE how to keep work on track and how to get the best out of people through remote means. We address this in more detail in the professional development section of the proposal.

Decision-Making by Consensus

All levels of the CONSORTIUM need to plan, problem solve, and decide by consensus. Consensus does a great deal to build teams trust, facilities communication and involve participants, but it requires that leaders at all levels are active and seek opinion, record discussions. Importantly team leaders need to plan that all decisions will take longer to make than in many other decision-making approaches. Also, leaders will need to actively manage participants — tamp down enthusiasm from


some, facilitate more involvement from others, and keep the momentum. We have guidance for facilitators and participants, decision-support tools, and our staff will support to enable this process during implementation. We list these tools later in our approach.

Managing Differences

Twenty-seven states will have twenty- seven different needs and approaches. The CONSORTIUM will encourage and in some cases mandate standardization, but with local needs and consensus-based decision-making, at the CONSORTIUM, SLT, and SNE level there will be differences. The Challenge is to define operational boundaries and welcome differences as part of the diversity of the CONSORTIUM. We assist the CONSORTIUM define boundaries or parameters in our supporting documents encourage diversity and emphasize this in our professional development training.

Maintaining Quality

Unlike some other multi-state education projects which are meeting and discussion focused, and deliverables are discussion papers and frameworks, the Digital Library success will in large part defined by the quality of its deliverables. We are confident that with training and SLT oversight, SNE members will be able to develop quality resources, and provide a level of quality assurance. In Addition, we will provide policy guidance and assist establishing the Digital Library Review Board.

Facilitating Quality Resource Tagging

PCG’s knowledge in Common Core State Standards, and PCG and Wireless Generation’s prior collaboration on content management systems, ensures that PCG will provide sound guidance and advice on tagging, uploading, and then on item vetting. We will assist the CONSORTIUM to stand up the Digital Library Review Board, and provide some organization and communication frameworks to develop capacity to generate an internal review capability. We propose to undertake a limited amount of this work as “tiebreak’s” for system tagging and as a guidance tool for the Digital Library Review Board and by providing a broader quality review ability if required for items loaded into the system by SLT and SNE. In addition we will work with the systems teams to ensure that the appropriate control measures and key businesses process we develop are reflected in the Digital Library system.

3.5.1 Recruiting State Leadership Teams and State Networks of Educators

Developing meaningful and productive networks takes effort on behalf of all stakeholders, requires high levels of diversity, but also requires participants who are informed, motivated, and who are skilled communicators. Setting recruiting policies and procedures in place to identify, screen, recruit and engage stakeholders is important, but the key to good participants is good messaging from the CONSORTIUM through to all stakeholders. The CONSORTIUM and SLT need to frame participation as an opportunity, something that is restricted to leaders and high achievers, which will provide insights into


best practice assessment approaches. In this section we will discuss our overarching approach, and then per the RFP we will outline two options to recruiting and compensate participating educators.

3.5.1.1 Layered Engagement Approach Meets Project Needs

Our project team will work with the CONSORTIUM on a number of levels to ensure that the right participants are recruited and retained. We will:

• Work with CONSORTIUM and relevant project staff to develop a positive and motivating

message about membership of SNE through a variety of formal and informal communication channels. We encourage staff to leverage their existing personal and professional networks to reach the broadest population possible.

• Establish communications with each SLT and provide an overview of our recruiting and compensation, and provide detailed step-by-step guidance on recruiting. Initially this will be CONSORTIUM SLT WebEx and thereafter we will tailor our communication and involvement to assist each SLT.

• Provide simple web-based tools to assist the CONSORTIUM, State Leads, and SLT undertake a thorough recruiting effort in a timely manner. Beyond this, we will provide seasoned a staff that understands recruiting in these types of projects and know at which stage in the recruiting process to provide support.

3.5.1.2 Best Practice Recruiting

Common to the two recruiting and compensation options, we list key criteria that either system must meet to be effective and efficient:

• Use existing structures where possible. We understand that the CONSORTIUM has some state-level organizational structures in place, as do some SLT that would enable a more timely recruiting cycle. We would look to leverage these systems, rather than create unsustainable parallel systems.

• Collaboratively establish recruiting policies. Candidate profiles, application policies and procedures, review criteria and procedures need to be mutually established and approved early so that the detailed step-by-step process flows are established early in the project. The CONSORTIUM has provided a useful outline, and we would look to provide additional definitions and criteria to enhance recruitment.

• Provide a Participant “bench”. Our experience from similar projects is that the attrition level in the first year can be as high as 15 to 20 percent and thereafter stabilize at approximately 5 percent. It is useful to retain records on a smaller number of approved, though not appointed participants so that open positions can be filled quickly and in an equitable manner.

Two Recruiting and Compensation Options

Recruiting and compensation for the most part are linear processes, and differ due to the degree of decentralization and centralization, levels of CONSORTIUM control and oversight, and the level of


contractor involvement. In both cases we envision that our role is significant during planning and startup, reduces during execution, and ceases after a knowledge transfer and procedural transfer. We will meet the 60 day timeline to standup these systems. For the purposes of comparison we list outline recruiting and compensation in separate tables below

Recruiting Options

Recruiting Stage Option One – Decentralized Option Two – Centralized

Develop participant profiles

Each SLT develop a participant profile and forwarded to CONSORTIUM. We provide SLT templates.

We work with the CONSORTIUM to develop profiles and forwarded outline profiles to SLT for use.

Confirm participant profiles as policy document

CONSORTIUM reviews profiles, and conducts edit and approve cycle. Sends approved document back to SLT

SLT edit profile and return to CONSORTIUM for approval.

Advertise/ notify for prospective participants

SLT develops communication and calls for participants via existing state channels.

CONSORTIUM provides outline communication and provides this to SLT for distribution via existing channels

Accept and compile applications

SLT receives and compiles applications via email.

CONSORTIUM, via Wireless Generation and PCG provides a web-based recruiting tool that compiles resumes.

Screen candidates: domain knowledge, personal attributes, and inform participants that were unsuccessful

SLT staff conducts a manual sort and exclude candidates that don’t meet agreed upon profile. Notify unsuccessful candidates manually via email

SLT use software to conduct a sort and exclude participants that do not meet profile agreed upon profile. System notifies unsuccessful candidates.

Rank candidates, and notify candidates if they were “Accepted” or are “Wait Listed”.

SLT Staff rank candidates, and then conduct diversity balance.

Manually Inform by email all participants by email if they are either “Accepted” or “Wait- Listed”.

SLT staff use CONSORTIUM software to rank candidates and conduct diversity balance.

SLT use System to inform participants of they are “Accepted” or “Wait-Listed”.


Recruiting Stage Option One – Decentralized Option Two – Centralized

SLT Chair write to supervisor of “Accepted” and “Wait- Listed” candidates to inform them of time and responsibilities and requesting support

SLT manually sends letters via email to all “Accepted” and “Wait- Listed” candidates.

CONSORTIUM sends a template to SLT who in turn sends the letter to supervisors.

Gather/confirm pertinent information from candidate

SLT confirms key personal data via email and sends that to the Implementation team for account rights/ creation.

SLT extracts data from system to create user accounts.

Replacing Participants SLT reviews resumes and looks for replacement participants and contacts them.

SLT uses system to find “Wait Listed” participant and offers them a position in an SNE.

Compensation

Compensation Stage Option One – Decentralized Option Two – Centralized

Notify participants of participation and work tracking and payment processes

SLT develop their own system to track participation .This would include some data transfer from Wireless Generation. Payments would be made centrally.

CONSORTIUM, via Wireless Generation and PCG provide an automated participation and work tracking system that issues stipends payments in line with CONSORTIUM policies. Policy would be that SLT approve payments and they would be made in batches every month or six weeks.

Track participation SLT determine systems and tracks work accordingly. Request reports from Wireless Generation.

Wireless Generation and PCG provide system that track participation.

Use participation to develop stipend payments

SLT use CONSORTIUTM policies to develop payment figures.

CONSORTIUM, via Wireless Generation and PCG provided system calculates payment in line with CONSORTIUM policy. Noted exceptions could be sent to SLT for manual review and approval.

Batch approve stipend payments

CONSORTIUM/ SLT approve payments

CONSORTIUM/ SLT approve payments


Compensation Stage Option One – Decentralized Option Two – Centralized

Pay stipends SLT make arrangements for payments.

CONSORTIUM make payments via Wireless Generation and PCG provided system

Resolve issues Participants resolve payment issues with SLT.

Participants resolve payments issued with Wireless Generation and PCG.

3.5.2 Robust Governance System

” The best ideas in education will never come from me or anyone else in Washington, D.C. They’re always going to come from a local level. We want to empower local

educators to have a chance to make a difference. ” Arne Duncan, 2010

The CONSORTIUM’ vision and strategy are clear, as is the Secretary’s for Education — there is a need for local control and educators at all levels want the chance to make a difference. When we reviewed the RFP’s scope of work, and based upon our combined experience to date, we developed two governance models that would conform to this strategy, could be implemented rapidly, and are sustainable. We envision that, while authority and decision-making remain similar for both models, organizational, communication, and decision-making structures will change as the project matures. Organizational re- design is something that should occur and can do so with significant changes to governance and lines of authority. We envision our role to develop frameworks and provide an initial level of advice, and then gradually work ourselves out of the system.

3.5.2.1 State Level Teams – The Key to Success

The success of the governance system hinges on the ability of SLT s’ ability to harness their SNE, while balancing top down guidance from State Leads and the CONSORTIUM. The SLT has a broad span of control or around 100 participants in each SNE, so managing participants will take some time, as will performance measurement and monitoring which we will facilitate by distributing system reports. Importantly SLT will need to vet additional resources for inclusion in the Digital Library, and as discussed previously will be supported by Wireless Generation and PCG in this task.

The RFP Lists the SLT as consisting of between 5 to 8 members, and we think that the number should be closer to 8 to account for the depth and breadth of domain knowledge required, the coordination time with SNE members, the need to proactively monitor quality and conduct gap analyses, and the need for sustainability beyond 2014.

3.5.2.2 Digital Library Review Board– The Key to Quality

In the previous section we introduced this topic, but we will consult widely and assist the CONORTIUM developing the structures and provide process guidance to provide a reliable, sustainable, and focused


quality system. We envision that the board will have wide membership and that they will report both to the CONSORTIUM and to their respective State Leads and SLT.

3.5.2.3 Matrix Governance System – Supports Multiple Stakeholders and

Concurrent Work, But Increases Complexity

Given the current CONSORTUIM structure, number of and authority of Stakeholders, and need for concurrent work — a matrix model best supports these needs. Matrix governance models allow for tasks to be isolated and undertaken with reference to other teams, but without multiple reporting layers to slow decision-making. A weakness of the model, is that leaders at all levels have to pay close attention to coordination measures to ensure that the same work is not being undertaken by other groups and that decisions that influence other groups are passed throughout the system. We are confident with the controls and structures we outline in the next section provide sufficient structure and guidance to make matrix governance work. The diagram below delineates the system by “layers”, which are either one of; organizational levels, knowledge domains, or major task groupings. Hierarchy within layers is shown by vertical alignment, and lastly we color code accountable and responsible parties.


Note: Not all lines of authority, control, and communication are shown in the outline system diagram

Key: CONSORT = CONSORTIUM

CONSORT PMO = CONSORTIUM assigned Project Manager and Staff Green color – Final Approval Yellow color – Accountable for deliverables Blue color – Responsible for decision making


Advisory Teams

CONSORT PMO

3.5.2.4 Hierarchical Governance Systems – Simple and Sustainable, But

Lacks Inclusiveness

Hierarchal models are well understood because they are commonplace in government, and lines of authority, control and communication correspond with the organizational structure, but these systems tend to stifle participation as stakeholder’s ideas are passed from one layer to the next which typically implies a level of approval. The model we are proposing is essentially a series of models starting from the top of the CONSORTIUM and ending with SNE members in their respective states. The diagram the below shows the relationship is more detail

Hierarchical Leadership System.

CONSORT

Formative Assessment

Practices

Professional Learning

Work Group

WireGen/ PCG IT

State Leads

Expert Panel State Lds.

Teams (SLT)

State Net. Educators

(SNE)

SNE Leads 3.5.2.5 Composition – Adding Clarity by Defining Roles for All Stakeholders

Both governance models show relationships for each stakeholder group, and as we outline in the next section as part of implementation guidance, we list summary roles for each group. Where possible we attempt to restrict roles to specific layers for the major of participants, so that participants can focus on a smaller number of primary tasks and not become overly concerned about upstream and downstream issues that they may not be able to influence.


Clear Task Focused Roles and Responsibilities – RACI

Leaders can better control their organizations, delegate tasks, and monitor work when individual roles and responsibilities are clear for major work tasks. Too often roles and responsibilities are outlined in policy, but specified to the level of detail that supports actionable guidance for stakeholders. We will work with stakeholders at all levels to develop Responsible Accountable Consulate ad Informed (RACI) for major tasks and ongoing business processes so that decision-making, handoffs, and timelines are clarified.

Decision-Making and Authority

Authority for both organizational structures is the ability to make final decision for a particular task or activity. For instance; signing-off on a project deliverable, approving assessment items for upload, authorizing stipend payment to participants are examples of the link between delegated authority and decision-making. Our approach, and which parallels the use of the RACI matrix is that while there can be many contributors, robust conversations can occur, and works can be distributed and shared — for clarity there needs to be a single accountable approval point.

3.5.3 Ongoing Governance Plan Implementation

Developing, implementing and sustaining a governance model is a challenge in a multi-stakeholder environment. What can be taken for granted in less dispersed environments needs to be formalized in dispersed environments, especially so when staff have full-time responsibilities to their districts and state entities. Key staff must communicate their vision, but provide actionable near term objectives to keep participants focused, and remove obstacles to success along the way. Another challenge is track and report on progresses and success, and to do that in way that is meaningful without being burdensome to participants. Lastly the organization must evolve to meet changing needs and increased levels of participant self-direction.

3.5.3.1 Written Guidance Is Essential

Implementing and then supporting an agreed upon governance system takes management support at all levels, a staff learning period where roles and responsibilities are learned, and lastly and most importantly a clear definition of policy, procedures, enabling tools. Our approach in similar projects has been to develop a the governance model, and then provide guidance and support to the CONSORTIUM to develop a single policy and procedure manual that invariably becomes known colloquially as the “handbook” or “guidebook”. The benefit of this is that the manual becomes the single authoritative reference and also:

• Provides an overview of supporting processes • Demonstrates the relationships between processes • Provide sufficient detail to support governance operations

The manual that we envision for this project would likely include topics including:


• Governance Management Structure State Leads Digital Library Review Board Role and composition SLT Role and composition SNE Communication Processes RACI Charts (Responsible, Accountable, Consulted, Informed) Quality measures Performance Measures and Monitoring

• Strategy End-state vision 0 to 2 year – objectives 3 to 5 year – goals Milestones and Deliverables

• Resources Allocation and Control Workload Management Process Work Prioritization Process Item upload and tagging Item quality review Item approval Process

• Policies Participation approval and payments Expenditures and reimbursements Local policies – State specific

3.5.3.2 Removing Obstacles to Increase Efficiency – Bottlenecks, Wait

Times, and Handovers

Governance models evolve and what was appropriate control and guidance at project initiation may not be appropriate over time. For example, as SLT mature and staff better understand issues, or as the interactions between SLT and SNE become more relationship based rather than organizationally based some procedures may be relaxed. We want SLT and SNE to take responsibility and become involved, and this becomes difficult where participants feel they are following processes and are delayed for no apparent reason. Conversely, we want the Digital Library Review Board to have a feedback loop to SLT and SNE to control quality and to provide feeback and reduce errors. CONSORTIUM and SLT leadership should review processes, identify bottlenecks, wait tomes or excessive handovers and take corrective action and reflect this in the manual.


3.5.3.3 Communication – Actionable, Less Frequent, and Less Formal is

More

We have developed a communication strategy, and listed the key tenets of this in our sample communication plan in Attachment B. Our approach is similar to best practices approaches, but with three notable exceptions :

Actionable

We will advise SLT and SNE how to develop actionable communication that targets the appropriate stakeholders, describes the situation or need, and then has a ‘call to action’. We want all stakeholders, especially at the local level to know when they are required to do something, be it comply to a request, or complete a task. Simple steps like prefacing email titles with “Action Required” or “For Information” allow participants to make the decision to read the correspondence or disregards it.

Less Frequent

High email volumes, ongoing conference calls or webinars, and constant notifications from collaborative workspaces do little to increase clarity. We recommend that information correspondence be brief and be restricted to a single communication once per week. For action tasking correspondence, unless needs are time sensitive we recommend that tasks or action items are bundled to reduce messaging.

Less Formal

We see a role for less formal communication between interested parties and encourage the use of social media where appropriate. Informal communication allows peers to maintain relationships, share ideas, and discuss issues that they may not be willing to do in official forums. Our recent experience is some participants will use social media, and others may not, but it is still useful tool and can expand the network of interest within informal educator groups.

3.5.3.4 Additional Tools and Techniques

Below we list other tools and techniques that have worked well on similar projects, and are examples of the advice we will provide to SLT and SNE.

Relevant Frequently Asked Questions (FAQ) Reduces Ambiguity.

There will be some state specific or network specific questions, but there are typically a number of questions that are applicable across all SLT and SNE. The Digital Library Review Board and SLT should compile and then distribute these for review.

Work Transparency Improves Performance

We propose that as part of the communication plan that we use data from the library to highlight work undertaken by respective SLT. We find that this fosters a level of competition among high achievers and provides some social pressure to under achievers and tends to raise performance. In addition, it is a use ‘barometer’ of work undertaken and grounds discussions in actual performance measures.


Awards Motivate Innovative and Productive Teams

A tiered recognition system administrated by the CONSORTIUM would be a valuable and cost-effective tool that rewards innovative and productive individuals and teams. Teams and individuals want to be recognized for their hard work, innovative ideas, and leadership. We think that a collectively designed award system would be useful to during initiation but is also a useful recruiting, publicity, and sustainability tool.

National Reach – Wireless Generation and PCG

There will be occasions during project initiation that Wireless Generation and PCG may need to provide support to states and between both firms, we have offices throughout the United States that we can utilize to support local events. The map that follows shows where we are located relative to the CONSORTIUM states. Our experience is that in some cases a rapid intervention on our behalf can resolve problems early before they become larger and more difficult to resolve.


3.6 Smarter Balanced Formative Assessment Practices and Professional Learning Resources

For more than a decade, Wireless Generation has been responsible for fundamental, system-wide change in states, districts, and schools. Since the release of our first software products in early 2002, we have been working with educators and families at all levels to drive the adoption of formative assessment and the use of data to personalize instruction. As technology tools and software have evolved, our commitment to supporting the human infrastructure that puts these tools into practice at scale has remained one of our core strengths. We have guided state departments of education in rolling out high-priority early literacy and mathematics initiatives, providing support in their communications and implementation planning. We have led state- and district-wide training programs — including direct teacher and administrator training as well as Training of Trainer (TOT) and Master Trainer (MT) programs that allow other staff to turn-key our professional learning experiences. We have worked closely with state departments of education to bring to life their Race to the Top (RTTT) initiatives, providing intensive coaching and professional development in every school in Delaware and Rhode Island, and have served the lowest-achieving schools in large urban districts across the country. This variety of professional learning experiences is possible only because of the number and quality of our trainers and coaches. Our team includes over 100 full- and part-time coaches located across the country. All former educators with an average of almost 20 years of experience, they immediately connect with participants and forge long-lasting relationships to support educators of all backgrounds.

In addition to our scalable delivery capabilities, we develop materials and resources to lead both face-to- face and online or mobile-based professional learning experiences. Our team of 10 instructional designers and content developers includes former educators and coaches who understand both cutting- edge instructional design and what is practical for any given group and technology.

We have extensive experience with iterative design, and use a similar process with the instructional design of our professional learning materials as with our software. This iterative design method is called the “Twin Peaks” process. It enables us to lead structured brainstorming exercises in which we expand our thinking by gathering the most cutting-edge ideas in the marketplace, and then synthesize and narrow the scope to what is realistic for a given goal, timeline, and specific PD experience or asset.


This process will expand the perspectives and options across all of the design questions, followed by a winnowing process in which key ideas can be transformed into a coherent vision for each deliverable. This vision will begin to answer the questions defined earlier, yet will also generate new questions. As a result, the design process will be inherently iterative, and prototypes will be living documents.

For the requirements outlined in the RFP, we will leverage our extensive experience to develop 144 Assessment Literacy professional learning modules that address how to build, administer, score, evaluate, interpret, and use the data from formative, interim, and summative assessments. Each grade band, as well as English Language Learner and Special Education teachers, will be able to access examples that are specific to their standards and will receive templates and tools that help them implement these practices and protocols immediately. We will also provide at least 50 exemplar instructional modules, again focused on each grade band for ELA/Literacy and mathematics. These modules will show exemplar applications of the CCSS, providing resources to help teachers incorporate skills into their practice. In addition, we will produce online learning modules that walk all stakeholders, including parents/families and students, through navigation and interpretation of the relevant data on the Smarter Balanced score reports. We are uniquely positioned to provide this level of support, as our company is building the score reports to release next school year. We will work closely with the CONSORTIUM to plan and deliver a high-quality Training of Trainers (TOT) program for the State Leadership Teams, enabling them to use all available resources to train the State Networks of Educators, who will in turn disseminate the materials/resources to each state’s professional learning networks. Lastly, we have relationships with other organizations that could enable us to sequence the above material into certificate-bearing courses.

3.6.1 Professional Learning on Assessment Literacy

Wireless Generation has a history of providing high-quality products and services that enable educators to access, understand, and use a variety of assessment materials to inform effective teaching strategies and classroom best practices. In developing Assessment Literacy content as outlined in this RFP, we will uphold our standard for high-quality professional development as we guide educators to a clear understanding of what the Smarter Balanced Consortium framework means to their practice.

Perhaps our greatest strength as a company is the ability to leverage a multitude of data, helping teachers take actionable steps that lead to increased student learning. However, the old adage of “garbage in, garbage out” holds true in this case: any data point is only as good as the assessment that produced it; teachers can only make meaningful decisions about classroom instruction if the data they’re using comes from an a measure that truly assesses student learning. Without a clear understanding of the assessment, teachers cannot know how to effectively use the data to improve instruction. This assessment literacy — understanding assessment design, what an assessment can and cannot tell you, and how to use the information gathered from the assessment appropriately — is essential to not only make smarter decisions, but fight the nationwide problem of “data-use malpractice.” No matter how statistically significant or reliable the CONSORTIUM assessments become,


we must have assessment-literate leaders at the state, district, school, and classroom levels in order to maximize the effectiveness of the assessments.

Each area of Assessment Literacy — formative, interim, and summative — will be developed as a set of robust stand-alone learning tutorials that meet the Smarter Balanced Quality Criteria. This format allows maximum flexibility so educators can truly focus their learning by topic. It also prevents educators from rushing through the professional learning, and ensures the tutorials contain all the critical details necessary to truly understand the nuances of each type of assessment.

Tutorials will be hands-on and interactive, and will be built to work for both individual educators (e.g., posing individual reflection questions throughout) and Professional Learning Communities (PLCs) to encourage collaboration and a social networking type of teaching community (e.g., suggesting guided discussion questions for the group). For example, an individual teacher might be asked, “During your most recent formative assessment what modifications did your students receive to meet their needs?” while PLCs might explore a question such as, “When you built your last common formative assessment, how did you ensure the assessment met the needs of your ELL and/or SWD students?” We believe that the inclusion of the PLC model for learning, a key standard of the Learning Forward Professional Learning Standards, helps to not only foster collaboration, but supports collective responsibility.

The key objectives of each assessment type, outlined in items (a) through (h) below, will be distributed into six modules within the tutorials: Introduction, Building, Administering, Scoring, Evaluating, and Data Use, to support not just learning about Smarter Balanced Formative, Interim, and Summative assessments, but effective classroom and school-based assessments. We believe that embedding each of the modules into the tutorials will ensure that the learning is not separate from the day-to-day responsibilities of educators. The two sections that will be separate from the three main tutorials are the Digital Library (g) and Glossary (h), which will be inclusive of terms and resources from all three tutorials.

While we believe it is important to provide educators a learning experience that relates closely to their classroom responsibilities, our experience shows that it is equally important that professional development match their content and grade bands for maximum effectiveness. The tutorials will follow the same learning progressions, but the embedded video examples, reflection questions, and support documents will be tailored to ELA and Math grade bands K–2, 3–5, 6–8, and 9–12.

Each module in the Assessment Literacy tutorials will contain interactive content that will take between 5 and 15 minutes to complete. Content will have a combination of embedded videos, voice overs, quizzes, reflection questions, and text for teachers to interact with. Our intent is to keep modules shorter and more specific, so that individuals or groups of teachers can customize their learning sequence to fit their own needs. However, while teacher choice is an important part of our plan, there is also a choice of two sequences that teachers can choose (or be directed) to follow. The first sequence will contain three distinct progressions: before the school year, throughout the school year, and post school year. This sequence will map the sessions in a way that teachers get the right information in a


just-in-time cycle. The second sequence would be organized by assessment type: formative, interim and summative as outlined in section 6.1 of the RFP.

The specific content for the Assessment Literacy tutorials is outlined in items (a) through (h) below. We will create a minimum of eight versions of each of the sessions: Math and ELA modules for K–2, 3–5, 6– 8, and 9–12 to ensure that teachers are given an experience that will match their content and grade band. The only exceptions to this are The Digital Library (g) and Glossary (h) as they are more universal than the other areas.

Each module will have printable guides and rubrics that teachers and PLCs can use to evaluate their own lesson plans, instruction, and assessments. These documents will support reflective practices, and help teacher’s self-identify areas for improvement. Additionally, each module will go through the same procedure for approval as outlined in Exhibit O.

3.6.1.1 (a) Technical Materials

One of the first sets of tools educators will need is the technical materials that guide the types of assessments and data that are available to CONSORTIUM members. Specifically, teachers must be assured that don’t need to be psychometricians or statisticians to understand content and item specifications. They will, however, need to be exposed to some of the basics so they can be knowledgeable about what various data sources are telling them and what to do about it. Our teacher- friendly online modules and materials will not only introduce these technical materials, but will emphasize ways in which they can inform classroom instruction and assessments, thereby making the materials relevant and meaningful. We will pay close attention to how the materials relate to the planning, delivery, and assessment of classroom instruction and ensure that they are delivered by August 2013.

Educators will learn about evidence-centered design by exploring the Content Specifications through the use of activities to enhance their understanding of the claims and assessment targets for their grade levels and subject areas.

Item and Task Specifications modules will develop educators’ understanding of the item-writing process and further enhance their understanding of evidence-centered designed items and tasks. Our online materials will explain terminology in teacher-friendly language and will clarify the connections between instructional processes and assessment outcomes.

Educators will also learn the important role test blueprints play in ensuring a focused and balanced classroom assessment plan. Teachers will learn to construct a test blueprint through a backward-looking design approach that focuses on four key questions:

1. Which standards will be measured? 2. What method of questioning (multiple choice, constructed response, tasks, or projects) best

uncovers this information based on the realities of their school and classroom? 3. What will success look like? 4. How much evidence will suffice in providing the data necessary to make the desired decision(s)?


These four questions are the groundwork to assembling a test blueprint that supports the creation of a focused assessment plan that will uncover the insights into student learning based on standards.

We will also provide training modules regarding Smarter Balanced technical materials related to the use of the personal needs profile (PNP) and accommodations and accessibility tools. We believe that this learning should not happen in isolation, but should be made through real connections to students’ current work. Through the PNP course, teachers will work through designing a developmentally appropriate PNP for one of their current students. Educators will learn about elements of Universal Design and bias to better understand how items/tasks function and then apply this knowledge to review and create items for their own classrooms that are accessible to all students. Educators will learn about measurement error and how the processes used by CONSORTIUM help to minimize error and its impact on data. Educators will gain an understanding of the specialized supports available that ensure whether the knowledge and skills of each student meets standards-based criteria.

3.6.1.2 (b) Balance of Summative, Interim, and Formative Assessments

We understand the importance of collecting a range of evidence of student learning in an efficient, systematic fashion that includes formative, interim, and summative components which must be consistent with the purpose statements developed by the Test Design Work Group. Each of these components combines to form a rigorous, integrated Balanced Assessment System capable of assessing easy-to-complex skills and tasks, up to and including college and career readiness. The assessments included in a Balanced Assessment System will be:

• Reliable – producing consistently stable results across multiple administrations of the

assessment • Valid – measuring their intended targets • Flexible – evolving with the needs of the student and the standards addressed • Technology-enhanced – adopting best practices for computer-based assessment, including

rigorous assessment, scoring and reporting

With the above considerations in mind, we recognize that each aspect of a Balanced Assessment System contributes unique qualities to the system. When approaching formative assessment, we will build into sessions explicit examples of the Five Attributes of Classroom Formative Assessment, as defined by the Council of Chief State School Officers:

• Learning progressions that articulate the sub-goals of the ultimate learning goal; • Learning goals and criteria for success that are clearly communicated to students; • Descriptive feedback that is evidence based and linked to intended instructional outcomes

and criteria for success; • Self- and peer assessment are prioritized to provide students an opportunity to think meta-

cognitively about their learning; • A collaborative culture is established between students and the teacher.


These attributes should always be used by educators in the classroom setting to monitor in-the-moment learning in order to adjust classroom learning experiences. We will be deliberate — through our videos and student/ teacher work exemplars — in making clear the distinction between classroom formative assessment (exit tickets, informal questing) and common formative assessment (a standards progress- monitoring test). We understand the value of the less-formal classroom formative assessment while appreciating the worth of the collaboration that occurs among educators during the common formative assessment process.

As an intermediate step between common formative and summative assessments, interim assessments are administered at prescribed times. Interim assessments will be used instructionally; the data educators glean from these optional assessments will allow them to adjust both their curriculum and instruction. Rather than being a single objective (classroom formative) or a restricted sampling of objectives (common formative), interim assessments determine student progress towards mastery on a somewhat limited set of objectives. This leads to summative assessment. Equally important, it will highlight that data from these interim assessments are not necessarily intended to be used for teacher evaluation, as that would change the purpose of the assessment.

Teachers will learn that summative assessments are the set of tools and measures that serve as an assessment of learning. These assessments are administered at the end of a learning cycle, such as a unit, quarter, or school year. They can be low-stakes or high-stakes based on the context in which they are given. Because they are used for evaluation and accountability purposes, summative assessments should align with curricular outcomes and reflect prior formative and interim assessments.

Users will have the ability to learn about the similarities and differences of the three assessments types side by side, or may choose to learn about one of the assessments free of comparison at a time. The side-by-side comparison of key facts and features of the assessment types will fit the learning needs of some teachers while some will need to focus on one assessment type at a time. We will provide real- world examples of what each assessment type could look like so that teachers are able to see a multitude of assessments.

3.6.1.3 (c) Assessment for ELLs, SWDs, and Other Student Groups.

We have extensive experience helping educators use assessment data to improve achievement for all students. Over the past 6 years, we have built a proven framework, called Taking Action with Data (TADA), which outlines the skills educators need to make data useful in the classroom. Through a six- phase curriculum, we lead teachers and leaders through easy-to-use strategies for employing academic and non-academic data to inform instruction for all students, including ELLs and SWDs. For details about all phases of the TADA framework please see Attachment A.

We focus largely on formative assessment data, which is short-cycle and actionable. It allows educators to adjust instructional plans based on real-time data so teachers can differentiate instruction for struggling students. Taking action with formative assessment data, educators can make rapid adjustments to best meet the identified needs of students. This is especially critical for traditionally


under-represented student groups, so that all the adults working with each student are informed of specific areas of need in advance of a summative assessment.

The Smarter Balanced summative assessment will provide educators with data on ELLs, SWDs and other traditionally under-represented student groups in comparison to the grade-level standards. Summative assessment data can also measure growth, an important component for certain student subgroups, including ELLS and SWDs, who, although may not meet the grade level expectations of the summative assessment, are likely to show significant growth over time as compared to interim assessments or summative assessments across consecutive years. Analyzing trends in data over time is a central skill to the Wireless Generation framework for Taking Action with Data and vital in the effort to close the achievement gap.

Through the online modules, teachers will learn the best practices for working with low, high, and average SWD students and ELLs with basic, intermediate, proficient, or advanced English language proficiency levels in a variety of contexts and grade levels. Throughout the module, key questions that teachers should ask themselves, such as: What background knowledge does my ELL student need in this lesson? or What vocabulary should I pre-teach? and useful resources, such as lists of Spanish/English cognates and ideas for modifying formative assessments, will help teachers create lessons and select instructional support for ELL and SWD students. In addition, ELL and SWD best practice routines and strategies will be highlighted and collected into a reference guide for ease of teacher use.

3.6.1.4 (d) Visuals and Graphics

To extend and enhance the user experience of the Digital Library, we will embed links in visuals and graphics that will take users directly to the key term, website, videos, animations, or supplementary documents. In addition, the linked information will describe the relationship of the linked visual or graphic to the Smarter Balanced Assessment System and then correlate the use of that type of assessment with specific opportunities for improved teaching and learning. Teachers will have an opportunity to rate the reference and add comments about how they modified or altered the assessment. This experience will center on the area of the system the user is in, while the linked material will appear in a new window to ensure that users can always return to their place in the module. We will work with our graphics department to enhance existing CONSORTIUM images and graphics while creating new images to make a visually enticing experience.

3.6.1.5 (e) Evidence Centered Design

Through the new CCSS standards, it becomes increasingly clear that traditional approaches to collecting information about student mastery may not provide accurate evidence about students’ abilities, particularly if students struggle to justify their answers. Rather than presuming a student can or cannot justify answers through the quality explanation, the premise behind Evidence-Centered Design (ECD) is that students should have multiple opportunities to demonstrate acquisition of knowledge through a series of performance tasks. Critical to ECD are students’ ability to demonstrate evidence based on the eight key claims asserted by the CONSORTIUM: four in English/Language Arts, and four in Mathematics.


Teachers will see video examples of ECD throughout the Assessment Literacy modules, providing ample opportunity to observe and practice the ECD techniques. For example, one video may include a student asked to perform mathematical operations to demonstrate his or her thought process. In a traditional assessment, a student who struggles with reading or constructing written explanations due to a disability or a lack of English language proficiency may appear to have not mastered this standard, even if he/she has actually mastered the mathematics standard. However, ECD provides the student with opportunities to show proficiency in another way than initially assessed, so that teachers can see that the student actually has mastered the math standard, but may need help in writing. Essentially, ECD provides students with a variety of conduits through which they may prove mastery of the standard and its related claims. In addition to classroom/student videos, interviews with teachers who have already begun to use ECD strategies in assessment and everyday classroom activities will be included. Teachers will walk away with a list of guiding questions to reflect upon when integrating ECD into their classroom.

3.6.1.6 (f) Reporting System

Wireless Generation has a long history of helping teachers and instructional leaders interpret results from assessment data and plan for instruction. We have built online tools that enable teachers to group students based on interim assessment data and create unique lesson plans for each group according to their specific needs and learning goals. Some of our tools, such as Burst:Reading, even do this for teachers automatically, using algorithms that analyze data and “prescribe” instruction for struggling students. We have also developed several professional development workshops — complete with templates and tools to guide teachers’ practice — to lead these practices in schools. Our experience in this arena is critical — Smarter Balanced assessments will include some measures of artificial intelligence (e.g., the grading of writing assessments), and we have created professional learning materials that establish trust for algorithmic assessment processes, eliciting buy-in from teachers about how the results can impact their instructional practices. Finally, we have built a robust data coaching practice, in which we work with small groups of educators to build their skills in using data to drive instruction. Since March of 2011, we have been providing data coaching services to the teachers in each school in the state of Delaware.

We have extensive experience in empowering teachers to facilitate frequent, low-stakes data conversations with teachers, students, and parents that incorporate student awareness of personal goal-setting, achievement targets, and monitoring of personal progress. We will adapt content from these sessions to prepare teachers to employ reflective listening and questioning with positive intent regarding curriculum/instruction data, achievement data, and how to share reports with students and families.

In addition to our experience with data in general, we are in the unique position to provide real-time support on interpretation of the score reports for the CONSORTIUM interim and summative assessments. As the selected provider (per CONSORTIUM RFP # 15) of the score reports, we will work closely with our development teams to ensure that we have early access to the report designs, building them into training materials even prior to release. We will also be able to provide input and feedback into the report designs as we learn educators’ needs and struggles. This cycle will become invaluable in


making sure we can quickly produce high quality modules that can begin to guide effective assessment data use before students take interim and summative assessments.

While these modules will focus on interim and summative assessments, teachers will find valuable information in the modules that will relate to their own formative assessment practices. Each module will walk users through a low-, average-, and high-achieving student score report and provide suggested next steps to take with the student, the student’s family, and other teachers. Additionally, each report module will be paired with suggested protocols for sharing reports with students and families.

3.6.1.7 (g) Digital Library

As the Digital Library is populated with both professional learning and curriculum resources, Wireless Generation will tap into its rich history in the area of building capacity among educators in order to make the Digital Library more advantageous for all. The Digital Library modules will lead users through the system in a systematic way that not only focuses on learning the system, but also leads the user to resources based on the standard they put into the system, and the grade level. The modules will start with sample student data to reinforce skills and protocols learned in other modules up to selecting a resource that meet the user’s needs. We will also build guiding questions into the module that will help users evaluate whether the resource fully meets their needs, or if they should continue searching.

Administrators will also learn how to use data to determine educators’ needs, and will use the Digital Library to populate educator professional learning plans. If their school’s data shows that a certain teacher needs support in teaching algebra to Grade 8 ELL students, the principal will be able to recommend that the teacher access the grade 6–8 online Math tutorials, as well as related curricular resources. Administrators will also learn how to follow up on their PD recommendations to make their coaching more impactful through focused observation and leading through questioning to name a few.

3.6.1.8 (h) Assessment Glossary

In all CONSORTIUM states, educators are working hard to develop rigorous applications of the CCSS, including the new language requirements associated with these common standards. As part of the high- quality educator resources available in the Digital Library, we will provide an Assessment Glossary that supports educators in developing a common understanding of the terms and language associated with Assessment Literacy. The Assessment Glossary will contain a comprehensive list of key terms and definitions that develop educators’ understanding of and ability to use assessment terminology to improve teaching and learning. The Assessment Glossary will ensure that educators have easy access to a comprehensive set of terms related to assessment literacy. It will be created with context-sensitive help, so that users can pull up definitions in the context of the work they are doing at any given point.

3.6.2 Exemplar Instructional Materials

Wireless Generation has a comprehensive background in designing and creating instructional modules using a formative assessment approach that promotes the use of data to drive instruction. We know the


primary importance of moving past putting systems in place and getting to actually generate change at the classroom level. We believe that these exemplar instructional modules will lead educators to a clear understanding of what the Smarter Balanced Consortium framework means to their practice and also to what the Common Core State Standards require teachers and student to do to reach mastery. We understand the best methods to influence the thinking and behavior of teachers and administrators to help them improve student performance and we have specific, technical and education experience with the Common Core.

Our experience with creating systems of instruction and the corresponding PD in states (RI, DE) and in large urban districts (Indianapolis Public Schools and Detroit Public Schools), as well as throughout the country using our mCLASS products has shown significant instructional results. We believe that we can incorporate the same data driven strategies to maximize understanding and use of The Smarter Balanced Assessment Consortium (CONSORTIUM) framework to drive student achievement.

Multiple video and text examples of the instructional shifts for both ELA and Math will be embedded in the Exemplar Instructional Modules. Specifically, the Exemplar Instructional Modules for all grade bands will be built to accomplish two goals: to model the instructional shift (i.e., writing activities, cross- content area examples, and testing deep understanding) and to explicitly teach educators how to facilitate these shifts.

For ELA/Literacy, this means a focus on:

• Balance of Informational and Literary Text • Literary in the Content Areas • Increased complexity of text • Text-Based questions and answers • Academic Vocabulary • Writing using evidence

And for Mathematics, it includes:

• Focus • Coherence • Fluency • Duel Intensity • Deep Understanding • Mathematical Practices across various standards • Writing using evidence

As required in the RFP, we will develop at least 50 modules that meet the CONSORTIUM’s technology standards. While much of the content could simply be presented once, we feel that each user should have a tailored experience that directly relates to the work they are doing in their grade and/or content. Equally as important to making the module relevant to the work at hand for the user, the modules must


be engaging. We envision modules that are highly interactive, with multiple places to click for deeper exploration, video vantage points from different stakeholder groups, short focused-reading excerpts, and printable takeaways for users that still rely on printed resources, to name a few. While the amount of information embedded into each module will be extensive, descriptors will be short and explicit as to what the user will learn from each module so that users can explore learning opportunities from differing grades and contents to meet their professional learning needs.

3.6.2.1 (a) Matrix for Instructional Modules for Grade Bands

By February 2013, we will develop a matrix for the instructional modules for each of four grade bands (K–2, 3–5, 6–8 and 9–12) in both ELA and Math and for the corresponding classroom context. This matrix will be available for review by work groups and by all states and for approval by the Formative Work Group and Director of Professional Learning. With our extensive experience in PD, we will create, as part of each module, the professional learning materials for educators on how to use the module to improve classroom instruction of the CCSS for all learners in diverse settings.

3.6.2.2 (b) Prototype for the Exemplar Instructional Modules

By March 2013, we will develop a prototype for the Exemplar Instructional Modules and professional learning materials, and will get feedback on the prototype using the guidelines in Exhibit O. The review and feedback cycle by work groups and all states will define the guidelines for the prototype. We have extensive experience with iterative design, and use a similar process with the instructional design of our professional learning materials as with our software. We have experienced, firsthand, tens of thousands educators’ journey with the CCSS through our work in Delaware, Rhode Island, Detroit, Indianapolis, and thousands of other districts throughout the country. This on-the-ground experience will prove invaluable when developing clear, concise, and user friendly modules layered onto the current realities that each user group faces. While we will fully follow the guidelines in Exhibit O, we believe it is more efficient for groups to respond to tangible ideas as the basis for feedback and direction.

3.6.2.3 (c) Module Delivery Schedule and Dates

The actual Common Core State Standards in ELA are broken into two grade bands — K–5 and 6–12. The Math Common Core State Standards are by grade levels K–8 and high school (9–12). However, in order to best address the key instructional shifts that are a part of the CCSS in both ELA and math, we will break the instructional modules into the following grade bands: K–2, 3–5, 6–8 and 9–12.

The first group of 12 modules will be delivered by August, 2013 and will consist of four modules (two Math and two ELA) for grade bands K–2 and 3–5 and two modules (one Math and one ELA) for grade bands 6–8 and 9–12. These modules will follow the same format regardless of the grade and content and will include the corresponding learning materials for educators to be able to use the module to improve classroom instruction of the CCSS for all learners in a diverse setting. The first set of modules for grades K–5 will focus on two standards in both math and ELA. In grades 6–12, the modules will focus


on one of the standards for ELA and one for math. The modules will follow the prototype approved by the Formative Work Group and Director of Professional Learning.

For the February, 2014 delivery date, 20 additional modules will be delivered and will consist of four modules (two Math and two ELA) for grade bands K–2 and 3–5 and six modules (three Math and three ELA) for grades 6–8 and 9–12. These modules will follow the approved prototype and will focus on ELA and math standards not covered in the August 2013 delivery date modules.

For the final delivery date (June 2014), all the remaining standards will be covered in ELA and Math for K–12. Included as part of both the ELA and Math standards will be modules focused on writing and reading across the curriculum, with particular focus on informational reading and writing, and evidence- based writing.

Grade Band

Content Modules by Delivery Date Module

Delivery Total August 2013 February 2014 June 2014

K–2 Math 2 2 3 7

ELA 2 2 3 7

3–5 Math 2 2 2 6

ELA 2 2 2 6

6–8 Math 1 3 2 6

ELA 1 3 2 6

9–12 Math 1 3 2 6

ELA 1 3 2 6

TOTAL 12 20 18 50

3.6.2.4 (d) Monitor, Adjust, and Use Feedback to Improve Modules

During the rollout of the first two sets of modules in August 2013 and February 2014, we will create, manage and monitor a database that will track comments by teachers about each of the instructional modules and also the improvements made to the Digital Library. By the June 2014 final delivery date, Wireless Generation will have made adjustments and improvements to the modules, based on the feedback. As part of each module’s professional learning materials, a link will be provided to take educators to an online site where they can submit comments throughout the rollout time span. Existing as standard operating procedure at Wireless Generation is an opportunity to provide feedback on products and service. We value the educators as the consumers and know that as they use the modules and become more familiar with the formative assessment process and the CCSS, their feedback will improve the final modules.


3.6.2.5 (e) Tagging of Instructional Modules

As part of the system-wide tagging for the Digital Library and interim and summative assessments, each instructional module will also be tagged using the same system.

3.6.2.6 (f) Format and Structure of Exemplar Instructional Modules and

Teacher Resources

Each exemplar Instructional Module will use a similar format and structure. Wireless Generation believes strongly that continuous school and district improvement depends on teachers, principals and school-based support staff using data in an inquiry-based way. We will use our cycle of inquiry model and “assess, plan, teach” to structure the full cycle of instruction using formative assessment practices.

Inquiry happens in both short and long-planning cycles depending on role and purpose. In the classroom, teachers engage in rapid cycles of inquiry when they use the results of formative assessment data to inform lesson plans and instructional strategies on the results of formative assessment. Grade- level and departmental teams use the process to study the efficacy of common lesson plans and instructional strategies in weekly, monthly and annual cycles of inquiry. School-based support staff members use the process to monitor the implementation of an intervention program and the entry and exit of students in that program. School leadership teams and administrators use the process to develop school-wide priorities, such as a focus on academic vocabulary, plan interventions across classrooms, and assess the efficacy of interventions, programs and services. The framework strengthens district administrators’ ability to monitor school performance, guide educators through the evaluation process, and conduct program evaluation. While the process appears simple, it is designed to transform mindsets and encourage a total re-imagination of how teaching and learning occurs throughout a system.

Instructional Modules

To provide a structure that will work for each of the grade bands for both ELA and Math; for a unified approach in terms of the balance between video and text examples; and so that the professional learning materials can be used in the most standardized way across grade levels and content; each instructional module will:

• Start by identifying the learning targets for the lesson, which align with the CCSS in ELA or

Math. • Begin lesson with a formative pre-assessment to determine the level of knowledge students

have about the learning target. (ASSESS) • Use formative assessment data/information to plan the lesson so that the teacher knows

individual students’ needs related to the learning targets. (PLAN) • Provide examples of instruction that can teach the learning target and also meet the needs

of diverse learners.(TEACH) • Conclude the lesson with a formative post- assessment whose data will determine the plan

for the next lesson (ASSESS)


• Contain embedded links to the Digital Library resources, Assessment Literacy modules and glossary, and additional resources as available.

Each module will show video clips on:

1. Classroom instruction of the learning targets 2. Formative assessment (pre and post instruction) of the learning targets 3. Teacher reflection following the lesson 4. Student reflection following the lesson — from diverse student groups

The length of each video will be between 1–3 minutes, with the final length of each module no longer than 15–25 minutes.

Teacher Resource Guide

Teachers will be provided with a resource guide in an online format that corresponds to the instructional module with the following components:

1) Graphic/visual and text version model of the full cycle of instruction using formative assessment practices

2) Student work samples (by grade band) from diverse student groups 3) Graphic/visual and text version of the key instructional shifts in K–12 math and ELA 4) Corresponding informational text lists and links for each grade level to cross curricular materials 5) Corresponding graphic/visual and text version of the writing required in each grade level 6) Evidence gathering tools and techniques (by content and grade band) that will provide teachers

will the necessary information to meet the learning targets, including pre-assessments (prior to lesson), post-assessments(following the lesson)

7) Curriculum resources that align to the increased rigor of the CCSS, focus on CCR 8) Links to resources on CCSS, formative assessments, appropriate accommodations for CCSS

content and meeting needs of diverse learners.

3.6.3 Using Smarter Balanced Score Reports and Aligned Educator Resources

It is critical that the districts, schools, and families in each CONSORTIUM state understand and know how to use the resources available to them as part of the Digital Library. The initial communication and training they receive sets the stage for their level of engagement; if they are engaged early and easily, and if they understand “what’s in it for them,” they will be more likely to incorporate the system and its resources into their daily practice. Further, if they can access materials on how to search the system and find resources to help them interpret the CONSORTIUM reports, they will be able to use the Digital Library immediately to address the learning needs highlighted in the reports.

We will work with the State Leadership Teams and State Networks of Educators to create a robust Communications Plan that will prepare teachers, administrators, parents, and students for the implementation of the Digital Library in the summer of 2014. We will develop prototypes of the messaging and communications vehicles by April, 2014, and will have training materials ready by June. This will facilitate a smooth rollout of the system prior to the 2014–2015 school year.


3.6.3.1 (a) Building Capacity for SLTs and SNEs

Even before the instructional resources and professional learning modules are available, all stakeholders need to receive communications about what to expect and when. Then when the Digital Library is launched in the summer of 2014, the State Leadership Teams (SLTs) and State Networks of Educators (SNEs) — with our support — will help teachers, administrators, parents, and students understand the wealth of information available to improve student learning. We will work closely with the SLTs and SNEs to build a comprehensive Communications Plan that lists for each stakeholder group:

• Their knowledge/experience base with assessment literacy and CCSS instruction; • Message(s) they need to receive about the Digital Library and resources; • Most effective medium for each message (e.g., email, phone call, memo, presentation); • Person/organization responsible for drafting each communication; • Person/organization responsible for delivering each communication; and • Identify local partner organizations or groups to assist with message delivery • Timeline for each communication (e.g., dates, frequency of messages).

Please see Attachment B for a sample Communications Plan.

It is important that stakeholders receive communications in advance of the summer rollout, during the summer, and into the 2014-15 school year. It is also essential that they receive messaging through different media. The Communications Plan — mindful of the fact that the spring and summer before the 2014-15 CCSS implementation will be incredibly busy — will contain several different messages and communication vehicles for each stakeholder group to ensure that many attempts are made to effectively communicate about the myriad resources available through the Digital Library.

For example, teachers may receive an email in the spring after their end-of-year summative assessment letting them know that there will be reports available over the summer that will help them plan for next year’s instruction. Just before leaving school, teachers may receive an electronic registration for summer training and PD on reports, and at the training session, they may receive a one-page flyer with more information about all the resources available in the Digital Library. Throughout the year, they will receive a quarterly newsletter from the SLTs giving updates about new content in the Digital Library and reinforcing messages relayed at the training sessions. Similarly, the team may create a website that state departments can link to their (and their districts’) home pages to help parents understand the new resources that will be available for summer and fall. Parents may then receive a memo with their back- to-school information — drafted jointly by our team and the SLTs/SNEs and sent from the school principal — exposing them to the rich resources available. Finally, we could provide a script for a short demo of the Digital Library during Fall Parent Nights at each school.

We will work with the SLTs and SNEs to develop The Communications Plan collaboratively, seeking input from the Functional Team along the way. Specifically, we will develop the Communications Plan and prototypes of all messages through the summer launch, by April 2014.


3.6.3.2 (b) Training Materials for Interpreting Assessment Data

Wireless Generation spent the last decade training teachers, administrators, educators, and parents how to access and interpret the data in interim and formative assessment reports. Each of our mCLASS assessment and intervention products — K–6 literacy and Math, K–12 interim assessment, Pre-K readiness — comes with a 1- or 2-day training session about how to use the assessment tool and access and interpret its reports. We have also built a year-long series of professional development workshops that take coaches and teachers through four 2-day sessions that explore, in a hands-on format, all the data available through these assessments, and create specific lesson plans for each group of students based on their individual learning needs as highlighted in the reports. Finally, our customized, job- embedded services, such as data coaching and Professional Learning Community (PLC) facilitation help teachers look at all the data available to them — both academic and non-academic — and make concrete plans about how to improve student learning in their classes.

All of our professional learning experiences require participants to access and use authentic data in real- time. This is a core belief for us, and a non-negotiable in all of our professional learning experiences. It provides ample guided practice time for educators or parents, so that they “learn by doing,” and so that they accomplish real work during the session, rather than having to recreate the experience after they leave the session.

We will bring our beliefs and experience to bear in designing and developing the training for the CONSORTIUM reports. We will create separate online modules for each stakeholder group, including teachers, administrators, educators, parents, and students. We recognize that each group has a unique interest in the same data sources, and it is our responsibility to provide vastly different contexts, examples, and formats to best serve each group. For example, teachers need easy access to item-level detail across every student in their class, so they can determine trends and plan instruction for various groups and individuals. School leaders need to see aggregate data about what types of instructions and programs work, so they can make decisions about curriculum and instructional approaches. Parents and families want to understand whether their children are progressing normally through the information they need to learn to succeed in school and life. There is no way that a standard-reports training module could satisfy these different stakeholders — we need to differentiate and provide learning resources for each. For example, the professional learning modules for teachers might explore the details of both Learning Map and matrix reporting, providing context-sensitive help for every element of the report and templates for teachers to group students and develop lesson plans with resources from the Digital Library. On the other hand, parent modules might be very personalized, walking a parent through his or her child’s own student report, and sharing ideas about how he/she can help reinforce learning at home.

Regardless of the formats, each group will have access to three sections of modules that focus on:

• Reports overview and navigation (“What?”) • Interpreting reports and analyzing data (“So What?”) • Instructional Next Steps (“Now What?”), which will include using search tools and content-

tagging system to access resources to address learning needs


All three modules will contain interactive components such as video, self-assessments, reflection exercises, and planning tools. We will work with the Formative Work Group, Director of Professional Learning, and the Reporting System Contractor to map out the objectives, agendas, and learning objects for each module, and to ensure that we cover each topic with the appropriate examples and hands-on activities to engage learners deeply in the content. It is important that the videos highlight not only assessment and instructional experts, but two other “real world” contexts: (1) classroom instruction that shows evidence of data-driven instruction; and (2) stakeholders from CONSORTIUM states themselves, explaining their adoption process of all types of assessments to drive instruction. This lends credibility to the modules, and helps other users identify with what this new instruction could look like in their world.

T he “What? ” Modul es

It is critical that each stakeholder group sees the connection of interim assessments to summative assessments, and we will focus on that during the Reports module. Showing the deliberate connection of the data in each report will support the notion of how powerful interim assessment can be as a checkpoint before summative assessment. Each stakeholder group will have videos and side-by-side comparisons that walk through not only the meaning of the reports, but also potential next steps. For example, teachers will have examples of how to take the reports and incorporate the information into a Professional Learning Community (PLC) discussion, and families will have suggested next steps for interacting with their child and teacher.

T he “So What? ” Modul es

These modules will take each stakeholder group deeper into the data on each report, helping them uncover patterns of success and need in their classes, schools, or districts. We will also help them begin to determine possible root causes of the patterns they identified, so they can determine why students perform and progress as they do. Even for parents, these modules will help them uncover the reasons for their child’s progress or struggle.

T he “N ow What? ” Modul es

These modules will move the user from understanding the data and root causes to making concrete instructional plans for each student or group of students. We will provide templates and protocols for planning, and will lead each stakeholder group through an ongoing cycle of inquiry to test new strategies, evaluate their effectiveness, and adjust as necessary to meet student needs. The modules will include ideas for families about how to reinforce learning at home.


Experience and Coordination with Reporting System Contractor

As the selected contractor for CONSORTIUM RFP 15 reporting system, we are expertly positioned to collaborate with the teams listed in the RFP as contributors to the interim- and summative-reports training. We understand the goal-oriented design of the Wireless Generation data and reporting platform, and know how to train educators to use the tools that guide them to a thorough understanding of what their data means and helps them develop and execute sound action plans grounded in that data. We will work closely with the Wireless Generation product- development team to understand the design of each report being built as part of CONSORTIUM RFP 15. Our regular process is to participate in Agile “scrum” meetings each week with the development team, so that we can develop training materials as the reports are being developed. We will use this process with the team creating interim and summative reports for the CONSORTIUM, to ensure an efficient build process for the training materials.

We have extensive experience building training materials for both Learning Map reports and Matrix reports. We have trained thousands of teachers on mCLASS:Beacon, explaining how each of the CCSS standards are reflected on the Learning Map, and how to access the results from each standard by clicking on that hexagon. We also have experience training on Now What? Tools for

Learning Map Score Report Now What? Tool Report

all of our products, helping teachers practice with the goal-oriented design of these reports.


3.6.4 Develop Training Materials and Training for State Leadership Teams

The CONSORTIUM has envisioned a multi-level Training of Trainers (TOT) approach to disseminate the educator resources built as part of the Digital Library. This approach is efficient, requiring only about 250 people on State Leadership Teams (SLTs) to be trained directly. It then assumes that SLTs train the State Networks of Education (SNEs), who can go forth and train state-wide professional learning networks. These state-wide networks are responsible, in turn, for providing training as necessary to individual stakeholders within the districts and schools in each state. To avoid dilution of the training messages, we recommend that groups of no more than 25 SLTs are trained together to ensure enough time for practice and planning. We also recommend that Wireless Generation coaches be available during the “turn-key” process to help support SNEs as necessary.

3.6.4.1 (a) Developing the Professional Learning Experience

Developing engaging, enriching, and thought-provoking professional development for SLT’s is an essential aspect to the work requested in this RFP. While the learning modules and resources may be of the highest quality, without full SLT support and understanding of the resources, educators may never realize the benefits of this system. In order for us to create this type of learning experience for SLTs, we will follow the review and feedback cycle as stated in exhibit O to ensure not only the quality of the experience, but also the long term support and usability of materials by member states. To greatest extent possible, we will work to build materials and experiences that can be adapted to meet unique member state needs, while ensuring core CONSORTIUM ideas and philosophies are consistent.

3.6.4.2 (b) Format and Objectives of Regional Meetings

To provide sufficient training opportunities for the SLTs, we propose conducting 2-day face-to-face sessions in five regional locations for each set of educator resources as outlined in the RFP:

(1) Inventory of currently available resources;

(2) Professional Learning on Assessment Literacy;

(3)Exemplar Instructional Modules for ELA: K–2, 3–5, 6–8, 9–12;

(4) Exemplar Instructional Modules for Math: K–2, 3–5, 6–8, 9–12;

(5) Score Reports and Aligned Resources.

Because a maximum of 50 participants will attend each location, we recommend two groups of up to 25 people be trained simultaneously for each of the five topics listed above. We will therefore provide a total of ten 2-day sessions for each topic during the 2013–2014 school year. A sample matrix of regional locations follows, although we recognize that this list may shift as we learn more about the size of each State Leadership Team.


Regional Location

States Included # of Sessions

Topic Timing

Northwest ID, MT, OR, SD, WA 2 x 2-day Inventory of Available Resources August 2013 West CA, HI, NV 2 x 2-day Inventory of Available Resources August 2013 Midwest IA, KS, MI, MO, WI 2 x 2-day Inventory of Available Resources August 2013 South NC, SC, WV 2 x 2-day Inventory of Available Resources August 2013 Northeast CT, DE, NH, ME, VT 2 x 2-day Inventory of Available Resources August 2013

Northwest ID, MT, OR, SD, WA 2 x 2-day Exemplar Instructional Modules – Part 1 September 2013 West CA, HI, NV 2 x 2-day Exemplar Instructional Modules – Part 1 September 2013 Midwest IA, KS, MI, MO, WI 2 x 2-day Exemplar Instructional Modules – Part 1 September 2013 South NC, SC, WV 2 x 2-day Exemplar Instructional Modules – Part 1 September 2013 Northeast CT, DE, NH, ME, VT 2 x 2-day Exemplar Instructional Modules – Part 1 September 2013

Northwest ID, MT, OR, SD, WA 2 x 2-day Assessment Literacy December 2013 West CA, HI, NV 2 x 2-day Assessment Literacy December 2013 Midwest IA, KS, MI, MO, WI 2 x 2-day Assessment Literacy December 2013 South NC, SC, WV 2 x 2-day Assessment Literacy December 2013 Northeast CT, DE, NH, ME, VT 2 x 2-day Assessment Literacy December 2013

Northwest ID, MT, OR, SD, WA 2 x 2-day Exemplar Instructional Modules – Part 2 February 2014 West CA, HI, NV 2 x 2-day Exemplar Instructional Modules – Part 2 February 2014 Midwest IA, KS, MI, MO, WI 2 x 2-day Exemplar Instructional Modules – Part 2 February 2014 South NC, SC, WV 2 x 2-day Exemplar Instructional Modules – Part 2 February 2014 Northeast CT, DE, NH, ME, VT 2 x 2-day Exemplar Instructional Modules – Part 2 February 2014

Northwest ID, MT, OR, SD, WA 2 x 2-day Score Reports June 2014 West CA, HI, NV 2 x 2-day Score Reports June 2014 Midwest IA, KS, MI, MO, WI 2 x 2-day Score Reports June 2014 South NC, SC, WV 2 x 2-day Score Reports June 2014 Northeast CT, DE, NH, ME, VT 2 x 2-day Score Reports June 2014

Total All CONSORTIUM

States 50 x 2- day

All Topics Aug 2013–June 2014

Each regional location will be located somewhere with easy airport access and ample hotel space to accommodate up to 50 SLTs. We will be responsible for finding the facilities for each session, but will work with the CONSORTIUM team and participating districts in or around the regional location to determine low-cost alternatives (e.g., schools, universities) for facilities. We will also provide lunch and snacks for each session as part of the cost of facilities.

Regardless of topic, each 2-day session will follow a similar format to allow for optimal learning and practice. We will customize our standard Training of Trainers (TOT) format to fit the needs of the SLTs, but generally expect each session to follow the same format:

• Introductions and Warm-up Activity • SLT share (SLTs share what went well and what needs improvement from their work with

previous sessions)


• Overview of topic(s) to be covered • Demonstration of Digital Library and relevant online materials/resources • Individual and group practice using Digital Library and relevant online materials/resources • Development of plans and presentations for SNEs • Teach-backs (SLTs practice giving the SNE presentations) • Wrap-up • Evaluations

In our experience, this provides participants with enough hands-on practice to understand the system, while focusing on action planning specific to his/her own state. Each of the sessions will have portions available on-line to provide additional focused practice for SLT members needing extra practice with the technology. The SLT share portion is our way of pre-assessing for facilitators to modify sessions to meet SLT real world needs, and for SLTs to hear what their peers succeed and struggle with, which fosters the collegial relationships needed to fundamentally make this work across all states. The teach-back concept enables our coaches to provide feedback on each SLT’s presentation, using a rubric that SLTs will then be able to take and use as a resource as they prepare for their rollout to SNEs. We will be able to give SLTs the opportunity to respond to real-world questions they may get during their presentations around such things as Bring Your Own Technology (BYOT) and other potentially common questions. To insure a broad range of use across multiple devices, and so that SLTs can use state specific tech tools, it is the goal that this platform be built to meet those needs and for that capacity.

3.6.4.3 (c) Training Materials

One of the most important parts of this face-to-face training is providing each SLT with the time and process to identify the priorities for its own State Network of Educators, based on their specific CCSS implementation schedule. Another critical element is providing the time for SLTs to get independent feedback from their peers around the country, about how their state is implementing the standards, assessments, and professional learning tools.

After an introduction of the Digital Library navigation and tagging system, as well as exploratory exercises with the relevant professional learning materials, we will spend significant time actually planning for the SNE and eventual state and district rollout of each set of materials. We will develop graphic organizers/resources to help the SLTs conduct a needs assessment of all the stakeholders who will be trained in their state, including, but not limited to the following:

• SNEs • Statewide Professional Learning Networks • District administrators • Building-level leadership • Teachers • Parents • Students • Community groups/organizations


We will expose participants to the many options that exist for training, including online and face-to-face professional learning and resource materials. Throughout the process and training, we will continually monitor the resources and professional learning of stakeholders through feedback and make adjustments as needed based on specific SLT needs and CCSS implementation schedules.

3.6.4.4 (d) Develop tools for SLTs to use to self-evaluate their own

understanding and ability to train the SNEs…

Training school, district, and state leadership teams as part of a Training of Trainers (TOT) model is at the core of our typical implementation plan. Through a TOT model, the SLTs will learn how to train their SNE counterparts on the Digital Library and elements of assessment literacy, instruction, and reports. In addition to the specific training in each topic, SLTs will also get a professional development learning rubric that helps them assess their skills before and after the intensive 2-day training sessions. The rubric gives participants a chance to provide honest self-assessment and reflection for areas that are weaknesses or where additional training is needed to ensure program success. During the 2-day sessions, we will be able to provide reinforcement activities for these development areas, to make the SLT members as comfortable as possible prior to working with the SNEs. SNEs will also be able to use the rubric for both the pre and post assessment to evaluate their own implementation and readiness to disseminate the resources to state-wide professional learning networks.

3.6.4.5 (e) Evaluation Tools to Document the Effectiveness of Regional

Meetings.

Just as we build an inquiry model into the work teachers do with data and instruction, we believe that we must use the same inquiry model to develop the most-effective-possible professional development sessions. We will adapt our existing end-of-session evaluations to fit with each of the five topics, and will require each participant to complete the online survey before leaving the 2-day session. The data from all five regional locations will be compiled, analyzed, and shared with the Functional Team members as part of the iterative process. We will then incorporate feedback into another iteration of the professional learning materials for the SLTs to use with their SNE, and will provide opportunity for feedback from the Functional Team members per Exhibit O. After another round of training, we will again compile, analyze, and share the feedback, incorporating changes into the materials to ensure SLTs have updated materials for their training sessions.


3.7 Currently Available Resources

Highly effective educational professionals are resourceful by nature; however, some of their effectiveness is impeded by spending countless hours searching for professional development articles, research, lesson plans, or best practices and routines. This time spent searching, instead of planning and learning, can lead to an educator’s lack of growth, a sense of frustration, feelings of lack of support, and potentially even a reason to leave the profession.

The Common Core State Standards (CCSS) will impact the educational landscape in countless positive ways. In order for educators to effectively and appropriately implement the CCSS, professional learning must be deliberate, intentional, and easily accessible. By developing and curating a robust set of high- quality resources in formative assessment practice, curriculum/instruction, and professional learning, the Digital Library will enable all educators across America to access the information they need to truly implement these new, rigorous standards. This will lead to improved teacher effectiveness on instruction of the CCSS. For the resources to be high-quality and to radically change instruction in the ways necessary to properly teach the new standards, they need to be vetted and compiled into an easily searchable and user-friendly format within the Digital Library. We believe that this resource library can support educators by reducing the amount of time spent searching for resources, and allowing them to spend more time on their own professional development. The more informed, supported and skilled our educators are, the more likely that long-term, systemic gains will be experienced by their students.

3.7.1 Comprehensive Inventory of Resources Currently Available

3.7.1.1 (a) Curating Supplemental Resources

Wireless Generation is well positioned to create a comprehensive inventory of existing royalty-free formative assessment, instructional, and curriculum resources. Our experienced trainers, and many of our staff, are former teachers and school leaders with an average of close to two decades of experience. They are located all over the country, including in almost all CONSORTIUM states, and have taught in classrooms throughout those states. We will leverage their first-hand experience as they provide suggestions for resources culled from lists they've compiled and used themselves with success. For examples of some of the resources available for use without permission, please see Section 3.7.1.5. In addition to these resources, other robust resources will be pulled from expert authors, practices, and organizations that meet the Smarter Balanced Quality Criteria Policies, as developed by the National Panel of Experts.

The Supplemental Resource Library will be built in two ways to offer users a choice in search functionality. Users can simply search by keywords or resources tagged to an objective for an experience similar to “Google-ing,” or they can choose advanced search engine optimization functionality techniques to search by criteria referenced in section 7.1a of the RFP, such as: author, grade level,


research base, content area, instructional focus, formative assessment focus, professional learning focus, or appropriate user. Search results will include a variety of resource types, such as videos, webinars, articles, tools, and lesson plans in commonly accepted file types (e.g. Microsoft Office files, .PDF, .MOV) to ensure educators are able to use the majority of the files, even considering school and district firewalls and security policies.

Our professional learning materials and instructional resources outlined in Section 6 will also link to select resources in the Supplemental Resource Library. For example, a graphic organizer that helps a department chair plan for a Math department data analysis meeting may be available as a link from the Grade 9-12 Mathematics Formative Assessment module. At the same time, the Building Formative Assessments module may have links at the end of the module to the Math section of the Digital Library resources.

3.7.1.2 (b) Process for State Leads and State Leadership Teams to review

Supplemental Resources

To generate buy-in for the Digital Library from teachers across all 27 consortium states, we will ensure that a thoughtful process is developed to ensure collaboration of all consortium members. Our method will create an inclusive process for the State Leads and State Leadership Teams to evaluate the Supplemental Resources and to review the inventory and request additions or deletions. We will also suggest the usage of article comment features, and a rating system by users to help determine the usefulness of each resource or to provide suggestion/alternations to the resource. We will create this process directly with each stakeholder group to determine their needs, ideal time commitment for the process, and desirability to interact with other member states before content is added or removed. We will also vet all resources for appropriateness. Wireless Generation will pilot the process with each state before submitting a finalized process in May 2013.

3.7.1.3 (c) Updating Supplemental Resources

We anticipate that educators in every CONSORTIUM state will be constantly uploading new supplemental resources to the system. Every six months, we will spend four weeks reviewing and vetting the submissions, and updating the Supplemental Resource Library. During this window, our staff—all former educators, many from member states—will read through the many examples that have been submitted, using quality criteria to decide what can be “approved” by the CONSORTIUM for inclusion. As links can change or move, we will also verify that current resources are still active and safe. After the four-week window has closed, we will utilize the developed permissions process, outlined in Section 3.7.1.4(d) below, to ensure the submitted content meets the specified guidelines. After clearing the process, the content will be posted to the Supplemental Resource Library, followed by communication to all member states that the files have been uploaded. We will invite SLTs and SNEs to join our team during these windows, using these opportunities to build their capacity to continue this vetting and approval process after the engagement is finished.


3.7.1.4 (d) Requesting and Receiving Permission to use the Resources

Each file that is posted to the Supplemental Resource Library will go through a rigorous checks and balances process to ensure first, the legality of posting the content, and secondly, the quality of the content. The process for determining the legality, and further securing the permission rights, will be developed by January 2013 to ensure that the delivery of content to states in May 2013 as references in section 7.1b of the RFP. We expect to set a process that enables permission to be secured for each resource for at least five years. We will work with our legal team to ensure that all verbiage is clear and consistent with legal standards and research best practices from around the technology industry in order to select an approach that best meets the needs of the RFP.

Once permission is granted, we will work with our technical team to tag and upload the resource to the Digital Library.


3.7.1.5 List of Available Resources

Wireless Generation has a wealth of complimentary resources in formative assessment practices and professional learning that can develop the Digital Library for the State Network of Educators for their use in providing recommended supplemental resources. Below is a list of just some of the resources available for use without permission.

Topic Author,

date Open source location Research

base Resource

type Description Audience

What are Formative Assessments and Why Should We Use Them?

Judith Dodge, 2009

http://www.scholastic.com/teach ers/article/what-are-formative- assessments-and-why-should-we- use-them

National Forum on Assessment

Professional development article

Provides background on what formative assessments are and provides some examples, strategies, types, instructions, differentiating and using RTI.

General education teachers, grades 3-8

Learning from Formative Assessment

Jennifer Atkison

http://www.scholastic.com/teach ers/article/learning-formative- assessment

University of Pittsburgh's Institute for Learning

Professional development articles

Provides ideas on setting clear expectations and holding students accountable for learning.

General education teachers, all grades

Formative Assessment: Helping Students Grow

Deb Aronson 2007

http://www.ncte.org/magazine/ar chives/126802

University of South Carolina and the Center for Inquiry

Professional development article

Provides examples of schools that have successful formative assessments.

General education teachers, all grades; principals and school leaders

Tools to achieve CCSS Various authors and dates

http://www.achievethecore.org/ Common Core State Standards

Sample lesson plans, rubrics, and instructional

Provides sample lesson plan prototype in a variety of subject fields for teachers in various

General education teachers, all grades

http://www.scholastic.com/teachers/article/what-are-formative-assessments-and-why-should-we-use-them






http://www.scholastic.com/teachers/article/learning-formative-assessment




http://www.ncte.org/magazine/archives/126802

http://www.ncte.org/magazine/archives/126802

http://www.achievethecore.org/


Topic Author, date

Open source location Research base

Resource type

Description Audience

materials, articles, teacher blogs

grades for formative assessment tasks.

Sample CCSS lesson plans

Various authors and dates

http://www.achievethecore.org/s earch/formative%20assessments

Common Core State Standards

Lesson plans, activities

Provides examples of schools that have successful formative assessments, in both reading and writing.

Grade 2 and 7 ELA teachers

The Smarter Balanced Assessment System


http://www.smarterbalanced.org/ k-12-education/teachers/

Smarter Balanced Consortium

Instructional materials, articles, videos

Sample lesson plans, rubrics, videos, webinars, and instructional materials for a variety of formative assessment tasks.

General education teachers, all grades; principals and district leaders; parents

No Child Left Behind and the assessment of English Language Learners

Kristina Robertson, 2008

http://www.colorincolorado.org/a rticle/22763/

No Child Left Behind, TESOL


Sample lesson plans, rubrics, videos, links, and instructional materials for a variety of formative assessment tasks designed specifically for English language learners (ELLs) Sample lesson plans, articles, rubrics, videos, links, tools, and instructional materials for a variety of formative assessment tasks

ELL Teachers at all grade levels; parents

http://www.achievethecore.org/search/formative%20assessments

http://www.achievethecore.org/search/formative%20assessments

http://www.smarterbalanced.org/k-12-education/teachers/

http://www.smarterbalanced.org/k-12-education/teachers/

http://www.colorincolorado.org/article/22763/

http://www.colorincolorado.org/article/22763/


Topic Author, date


Resource type


designed specifically for English language learners (ELLs); using formative assessment to guide language instruction.

World class instructional design and assessment

Various http://wida.us/searchResults.aspx ?cx=0001878867407992537742:bj kids4qwcy&cof=FORID:10&q=for mative%20assessment

WIDA, University of Wisconsin System, TESOL


Sample lesson plans, articles, rubrics, videos, links, tools, and instructional materials for a variety of formative assessment tasks designed specifically for English language learners (ELLs); using formative assessment to guide language instruction.


Advancing excellence in English teaching


http://www.tesol.org/SearchResul ts?cx=009209657917998192142% 3Adzwo2sspoue&cof=FORID%3A1 0&ie=UTF- 8&q=formative+assessment&siteu rl=www.tesol.org%2FGSearch.htm %3Fsrch%3Dformative%2520asses sment&ref=www.tesol.org%2F&ss =

Teachers of English to Speakers of Other Languages (TESOL)

Instructional materials, articles, presentation s

Rebooting the Formative Assessment Process for English Language Learners (ELLs).


Guidelines for formative assessment


http://www.fairtest.org/search/n ode/formative%20assessment

FairTest, Smarter Balanced Assessment Consortium

Instructional materials, professional development articles

Articles on the value of and guidelines for formative assessments.

Teachers at all grade levels; parents

http://wida.us/searchResults.aspx?cx=0001878867407992537742%3Abjkids4qwcy&cof=FORID%3A10&q=formative%20assessment






http://www.tesol.org/SearchResults?cx=009209657917998192142%3Adzwo2sspoue&cof=FORID%3A10&ie=UTF-8&q=formative%2Bassessment&siteurl=www.tesol.org%2FGSearch.htm%3Fsrch%3Dformative%2520assessment&ref=www.tesol.org%2F&ss











http://www.fairtest.org/search/node/formative%20assessment




Topic Author, date


Resource type


Implementing with fidelity


www2.ed.gov/programs/racetoth etop/communities/hunt-institute- briefing-packet.pdf

James B Hunt– Institute for Educational Leadership


Articles which explain the importance of implementing CCSS with fidelity.

Teachers at all grade levels; principals; district level personnel

The CCSS home-school connection


http://school.familyeducation.co m/educational-issues/parents- and- school/34360.html?detoured=1

Various Parenting articles, tools

Articles which explain the importance of CCSS and that provide a variety for tools for parents to use by subject area; home- schooling links.

Families

CCSS Research and alliances


http://www.relcentral.org/researc h-alliances/ra1-state-longitudinal- data-systems-and-teacher- preparation/

Regional Educational Laboratory Central

Research, articles, alliances

Research articles on CCSS, alliances, such as the Native American Educational Research Alliance, Teacher Preparation and Common Core Standards Alliance.

Teachers; administr ators

Implementing CCSS with students with disabilities


http://www.cec.sped.org/AM/Te mplate.cfm?Section=CEC_Today1 &TEMPLATE=/CM/ContentDisplay. cfm&CONTENTID=15269

Council for Exceptional Children


Articles include those which articulately explain CCSS and the challenges of implementing it with students who have special needs.

Teachers; special education case managers

http://school.familyeducation.com/educational-issues/parents-and-school/34360.html?detoured=1






http://www.relcentral.org/research-alliances/ra1-state-longitudinal-data-systems-and-teacher-preparation/






http://www.cec.sped.org/AM/Te


Topic Author, date


Resource type


Alternative assessment for students with disabilities


http://www.cehd.umn.edu/nceo/ National Center for Educational Outcomes

Professional development articles, tools

Contains tools and articles with examples of alternative assessments for special education students.

Teachers; families; special education case managers

Students with disabilities and the CCSS


http://wceruw.org/search Results.php?cx=00187886740799 2537742%3Apejfgvyyeaq&cof=FO RID%3A10&ie=UTF- 8&q=formative+assessment&sa.x= 0&sa.y=0&sa=Search

Wisconsin Center for Educational Research

Professional development articles, tools

Contains tools and articles with examples of alternative assessments for special education students.

Teachers; families; special education case managers

What Works Clearinghouse


http://ies.ed.gov/ncee/wwc/topic. aspx?sid=2

Various Professional development articles, tools, links

Contains tools, articles, and links to help teachers prepare students in the area of college and career readiness.

Teachers

Understanding CCSS Various authors and dates

http://www.leadandlearn.com/res ource- center/search?f%5B0%5D=type%3 Awebinar

The Leadership and Learning Center

Webinars Contains many complimentary webinars on CCSS topics: Avoiding Pitfalls of the CCSS, Meeting Literacy Demands of CCSS, From Drowning in Data to Evidence of Impact, Implementing CCSS Math Practices, etc.

Teachers, principals

College and Career Readiness


http://sat.collegeboard.org/regist er/when-to-take-sat

Various Research reports, sample tests

Contains research reports and data on students, state by state,

Teachers, principals, families

http://www.cehd.umn.edu/nceo/

http://wceruw.org/searchResults.php?cx=001878867407992537742%3Apejfgvyyeaq&cof=FORID%3A10&ie=UTF-8&q=formative%2Bassessment&sa.x=0&sa.y=0&sa=Search







http://ies.ed.gov/ncee/wwc/topic.aspx?sid=2

http://ies.ed.gov/ncee/wwc/topic.aspx?sid=2

http://www.leadandlearn.com/resource-center/search?f%5B0%5D=type%3Awebinar





http://sat.collegeboard.org/register/when-to-take-sat

http://sat.collegeboard.org/register/when-to-take-sat


Topic Author, date


Resource type


who have taken the ACT. Makes connections to the data and meeting CCSS. Sample test questions also given.

Financial literacy Various authors and dates

http://themint.org/ Various Tools, lesson plans, activities

Contains resources for students, teachers, and families to facilitate college and career readiness with financial literacy.

Teachers, principals, families

http://themint.org/


3.7.2 Identification and Recommendation of

Supplemental Resources

One of the most important elements of the Digital Library is that it is a “living and breathing” system— one that keeps getting smarter as more educators use it to contribute best practices and resources to help other educators. This crowd sourcing aspect is one way to measure the true adoption of the system, and is a leading indicator of how apt teachers, administrators, parents, and students will be to take ownership of their learning in the CCSS world. Therefore, in addition to our work to seed the Library with high-quality resources, it is our responsibility to ensure that SNEs and educators trained by the SNEs will also identify, create and recommend supplemental resources in assessment literacy, classroom instruction, and formative assessment that will be complimentary to the Smarter Balanced resources.

3.7.2.1 (a) Process for SNEs to Identify Supplemental Resources

We will work with the SNEs to develop processes and protocols to identify supplemental resources to be added to the Digital Library. Because the SNE represents each state in the Smarter Balanced CONSORTIUM and has been recruited and selected by the SLTs and State Leads, their involvement in the process and protocols to identify and recommend additional resources for the Digital Library using their content and grade level expertise will encourage diverse and relevant supplemental resources. We will provide processes and goals to help them become true champions, generating new resources and inspiring other educators to do the same.

3.7.2.2 (b) Help Desk Services for SNEs during Resource Additions

Beginning in January 2014 and continuing throughout the timeframe of the contract, we will provide Help Desk phone and email support services to the SNEs while they work through the process to identify, recommend, upload, and tag supplemental resources that will be added to the Digital Library. Educators will be able to call the Wireless Generation toll-free support line, and will be transferred to a CONSORTIUM Digital Library expert, who can provide not only support with the technical upload process, but support for their pedagogical questions concerning potential supplemental resources.

We offer a professionally staffed Helpdesk Operation, our systems are professionally designed and maintained, and we are fully equipped and able to provide the support required by the District for this project. Since January 2008, we have handled more than 225,000 customer calls, and logged just over 200,000 CRM Support cases. The average hold time for those calls has been an industry gold standard of less than one minute- averaging just 44 seconds, including peak times with an equally impressive first- response resolution rate of more than 93%. For the most recent peak time, from September through October 2012, we handled more than 40,000 calls, in an average of 26 seconds of wait time and actual talk time of around 10 minutes.


Our Helpdesk Support will be available via toll-free phone, email or live chat support provided by expert resources with product, application and technical skill sufficient to effectively support end users, for the hours of 7:00AM to 7:00PM EST weekdays, excepting major holidays when school is not in session.

We can provide the CONSORTIUM with regular reports on Support contact activity, including the following data: call volume; trends and patterns within call volume; types and origins of contacts; types of problems reported; number of contacts resolved; average time to resolution; and other reports standard to the operation of a professional contact center.

We design initial and ongoing resource training in anticipation of issues that drive customer satisfaction and performance that guarantees it. As a result, customer requests for change are infrequent. Nonetheless, when such requests arise, we are prepared to take positive steps to resolve issues. In the past year, guided in part by customer requests, we have added online chat and an eCommerce solution. Additionally we have provided additional staff for online training via Webex or individual consultation beyond the basic Helpdesk contact.

The Help Desk will provide guidance and support to SNEs to ensure that the processes and protocols that will be developed are followed for supplemental resources. We will provide service-level agreements that stipulate response time, so educators can receive timely responses to all their questions.

3.7.2.3 (c) Moderated Collaboration for SNEs

To best support the SNE collaboration of grade-level and/or content-area groups in the identification, recommendation, approval, and upload process of the supplemental resources, we will provide moderation of the process prior to the final submission into the Digital Library. We will work with the SNEs to review resources for a four-week period every six months for submission of supplemental resources. This inventory of supplemental resources will be uploaded after it has been vetted by the processes and protocols identified in collaboration with the SNE and Wireless Generation.

3.7.2.4 (d) Monthly Status Reports

As required in the RFP, we will provide monthly statistics to the State Leadership Teams (starting January 2014 and continuing through the life of the contract), analyzing the number of resources recommended and approved, by state, to the Digital Library. Utilizing log records and system metrics, it will be possible to track number and type of uploaded resources, participants and dates of usage.

3.8 Records of Decision Making

Wireless Generation understands that it is responsible for facilitating all in-person and online/teleconference meetings necessary to complete the scope of work under this request. This includes providing meeting minutes and documenting all decisions made during contract activities (e.g., reviews, decisions, conference calls). These records will be presented to work groups in a timely manner for review and confirmation of content.


Wireless Generation will maintain version control (e.g., time/date stamps) of documents processed, and all documents will be made available in a secure collaboration site. At the completion of the contract, all documentation will be turned over to the CONSORTIUM.

Wireless Generation is expert in effective communication with its clients and understands that this is also required in this case. Wireless Generation will communicate to CONSORTIUM membership and larger external stakeholders regarding the progress of deliverables.

3.9 Contractor Meetings

Wireless Generation agrees that it will be responsible for its own travel costs associated with the twice- annual, CONSORTIUM-wide WorkGroup/Contractor Collaboration Conferences, convened to support project-specific and across-consortium contractor engagements. We also understand that we will be expected to propose any additional meeting(s) necessary to support the work.

We understand that any funds from a travel allocation not spent on travel will revert back to OSPI upon conclusion of the contract unless amended in writing by both parties.

Wireless Generation will work closely with stakeholders on the design of the requirements-gathering sessions as well as the design of the reports, and Wireless Generation will also support Smarter Balanced by participating in the following meetings:

Kick-off Meeting

Wireless Generation will send participants to attend a two-day Kick- off Meeting between Smarter Balanced and Wireless Generation.

Semi-annual Contractor Meeting

Wireless Generation will send at least three participants to the first of the Semi- annual Contractor collaboration Meetings. However, Wireless Generation may also send to these meetings additional Subject Matter Experts (SME) who can best support the contract efforts.

Virtual Meeting

Wireless Generation will support CONSORTIUM members or other Workgroup/Contractors involved with this implementation participating remotely (via Webex, or through PolyCom video conferencing facilities) in regularly scheduled (end of iteration retrospective meetings), or milestone (code complete, QA signoff, release planning, etc.) meetings. In addition, Wireless Generation will support these virtual meetings on an ad hoc basis for issues or status discussions throughout the implementation as needed.


3.10 Project Schedule In performing a detailed analysis of the information presented in the RFP, Wireless Generation has prepared a summary project schedule below.

Phases and Milestones 2012 2013 2014 Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

Methodology Phase Initiation Phase Discovery /Vision Phase Analyze/Refine Phase Agree/Plan Phase Document/Execute Phase Document/Execute Phase

Development Phase Discovery/Requirements Phase Specifications/Architecture Phase Prototyping and Feedback Phase Scaling/Finalizing Phase Quality Assurance Phase First Release of Digital Library Milestone

Professional Development Components National Panel of Experts Development of Assessment Literacy Modules Development Of Exemplar Instructional Modules Development of Score Reports Development of State Leadership Training Materials

State Leadership Training Inventory Communications Plan Legend: Implementation of Milestone Optional Training Through Contract Completion


4.0 Deliverable Matrix

For each item in the Deliverable Matrix, Wireless Generation has noted the Proposal section in which we discuss the creation of the deliverable and fully describe the deliverables to be developed, the RFP requirement it supports, the parties involved in its development and, in conjunction with Proposal Section 3.10, the projected dates each will be delivered. Exhibit O of the RFP outlines the review and approval process to be utilized for each. As is noted in the RFP, this Deliverable Matrix will be finalized during contract negotiations.


EXHIBIT J Deliverables Matrix – Preliminary

ID

Deliverable

Description of content to be included

RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

Assign a sequential numerical value to each item listed for ease of reference.

Deliverable Title Provide a brief description of the Deliverable content. Referen ce the RFP require ment(s) that the Deliver able is intende d to fulfill.

Provide an estimated completion date.

List, in sequence, those Smarter Balance team entities that need to review this deliverable (to be completed with work group Co- chairs and Contract Manager).

List, in sequence, communications events related to this deliverable.

The CONSORTIUM should review the sections indicated below for our details regarding compliance with the requested deliverables.

1.0 Formative Resources & Professional Development Processes & Systems Establish the systems, process, and protocols to ensure the successful development and implementation of the project scope of work.

1.1 Comprehensive Development Strategy

Develop and document a comprehensive development strategy for the Digital Library and the educator resources that ensures Smarter Balanced stakeholders have access to critical professional learning resources in time for the 2013-14 school year. Publish the approved document on the Smarter Balanced public website.

C.2.(f ).1

TBD See Exhibit O See Exhibit O 3.1

1.2 Component Plans & Processes

Formal documentation of component plans and processes to execute the scope of work of the project. Processes will be developed in conjunction with key stakeholders following an iterative approach. Plans include, but are not limited to the following:

• Resource Tagging System (3.1) • Editorial Review Process (Upload/Removal) (3.2) • Quality Criteria Policies (4.1) • Recruitment Plan (5.1) • Governance Process (5.2) • Continuous Improvement (5.3) • Exemplar Instructional Materials Plan (6.2) • Digital Library Launch (6.3a) • Training Processes (6.4) • Inventory of Currently Available Resources (7.1) • Educator-Submitted Supplemental Resources (7.2)

Varie s

Varies See Exhibit O See Exhibit O 3.2.1; 3.3.1; 3.3.2; 3.4.1; 3.5.1; 3.5.2; 3.6.2; 3.6.3.1; 3.6.4; 3.7.1; 3.7.2; 3.7.2;


ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

• Communication Plan (2.1) • Collaboration Processes (2, 2.2)

All component plans are subject to approval prior to work start.

1.3 Sustainable Leadership System

Develop and deliver sustainable governance system that provides for editorial review and selection of digital library materials and which includes representatives of the State Leads, State Leadership Teams, and State Networks of Educators. The leadership system must be flexible and adaptable for long-term capacity building across CONSORTIUM states, while implementing the participation policies developed to ensure an equitable distribution of participants.

C.2.(f ).5

TBD See Exhibit O See Exhibit O Section 3.5.1 to Section 3.5.3

2.0 Quality Criteria Policies Establish expert panel & guide for the development of Quality Criteria Polices for Digital Library Resources.

2.1 National Panel of Experts Establish National Advisory Panel of 12 experts on formative assessment practices, adult on-line professional learning, and quality instruction incorporating universal design for a range of diverse learners. The panel will develop and recommend policies to establish quality criteria to be applied to all resources recommended for the Digital Library. The policies will build on national standards in professional learning and current research in effective formative assessment practices

C.2.(f ).4.1

TBD See Exhibit O See Exhibit O 3.4.1

2.2 Quality Criteria Policies Convene the National Advisory Panel within 30 days after contract execution and hold at least three face-to-face, 2-day meetings of the National Advisory Panel to develop Quality Criteria Policies for digital library resources. At the second meeting, the Formative Work Group, State Leads and State Leadership Teams will attend face-to-face meetings and by WebEx to provide feedback to the panel on the first

C.2.(f ).4.1

May 2013

See Exhibit O See Exhibit O 3.4.1


ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

draft of the proposed policies. Facilitate the development and refinement of the recommended policies through Governing States approval process by May 2013.

3.0 Sustainable Leadership System Form & maintain leadership and contributor teams to ensure the sustainability of the Digital Library.

3.1 State Leadership Teams Establish State Leadership Teams. The process should include, but not be limited to the following: • Identifying candidates to fill specific content expertise • Contract management and payment • Involving stakeholders who represent diverse groups of learners • Filling vacancies • Developing plans to maintain and continuously improve

leadership, facilitation, and governance structures post-2014

C.2.(f ).5.1

Within 60 days of contract executio n

See Exhibit O See Exhibit O Section 3.5.1

3.2 State Networks of Educators

Establish State Networks of Educators. The process should include, but not be limited to the following: • Identifying candidates to fill specific content expertise • Contract management and payment • Involving stakeholders who represent diverse groups of learners • Filling vacancies • Developing plans to maintain and continuously improve

leadership, facilitation, and governance structures post-2014

C.2.(f ).5.1

Within 60 days of contract executio n

See Exhibit O See Exhibit O Section 3.5.1

3.3 Digital Library Review Board

Establish Digital Library Review Board made up of representatives from State Leads, Work Groups, and State Leadership Teams to systematically monitor quality, conduct ongoing gap analyses, and enhance and broaden the scope and depth of resources beyond the grant’s end in 2014.

C.2.(f ).5.2

TBD See Exhibit O See Exhibit O Section 3.5.2 and Section 3.5.3

4.0 Formative Resources & Professional Development Develop the resources to populate the Digital Library and equip State Leadership Teams to train State Networks of Educators in Formative assessment practices.

4.1 Exemplar Instructional Modules

Develop and deliver comprehensive learning modules that meet the Smarter Balanced quality criteria, and which include lesson plans, templates, curriculum resources, evidence collection tools, video clips of classroom instruction and teacher analysis, descriptive feedback strategies, and follow-up planning materials. Modules will be developed across grades K through 12 in English language arts/literacy and mathematics in three phases:

· Prototypes: Four prototypes are developed for approval that will

C.2.(f ).6.2

May 2013– June 2014



ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

cover K-2, 3-5, 6-8, 9-12 for ELA or Math no later than May 2013 · Phase 1: Ten to twelve (10–12) modules representative of the grade spans (K–2, 3–5, 6–8, 9–12) and content areas (ELA/Literacy, Mathematics) by August 2013 · Phase 2: Fifteen to twenty (15–20) additional modules representative of the grade spans (K–2, 3–5, 6–8, 9–12) and content areas (ELA/Literacy, Mathematics) to be delivered February 2014 · Phase 3: The remainder no later than June 2014

Additionally, the contractor will be responsible for implementing the tagging system for the instructional modules consistent with the Digital Library navigation system and interim and summative score report tagging.

4.2 Inventory of Current Resources

Research, compile, and deliver comprehensive inventory of formative assessment, instructional, and curriculum resources that meet the Smarter Balanced Quality Criteria Policies as well as the technology, tagging, and upload requirements of the Digital Library.

a. Once the inventory is approved, the CONTRACTOR will provide it

to the State Network of Educators for their use as they identify and recommend supplemental resources in Assessment Literacy and to complement the Exemplar Instructional Modules. The inventory will be updated every 6 months during the project.

b. The CONTRACTOR will develop a process to request and receive permission to use the resources for at least 5 years and to upload resources identified through the inventory with the required technology and tagging.

C.2.(f ).7

May 2013


4.3 Vendor-Developed Training Products 4.3.1 Professional

Learning on Assessment Literacy

The Assessment Literacy training products are online training materials that may be used as a unit or as stand-alone sections in a flexible way to build the knowledge and skills of educators to effectively use data from summative and interim assessments and formative evidence collected in classroom instruction to improve teaching and learning. The training materials will be developed in a model aligned with the Learning Forward Standards for Professional Learning and Smarter Balanced Quality Criteria and completed in stages between August 2013 and December 2013 to provide resources for educators during the 2013-14 school year.

C.2.(f ).6.1

May 2013 - Decembe r 2013



ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

Prototypes of all deliverables will be developed for review by Work Groups and All States and then approval by the Formative Work Group and the Director of Professional Learning before drafts of the deliverables are developed for feedback and review cycles (See Exhibit O). The Professional Learning materials will include training for educators on key priorities of the Smarter Balanced Assessment System as follows:

Prototypes: May 2013 · Six prototypes are developed across the areas of Formative, Interim, and Summative assessments for approval that will cover one of each of the following; Introduction, Building, Administering, Scoring, Evaluating, and Data Use for by May 2013

Phase 1: August 2013 · Teacher-friendly online materials to support a clear understanding of Smarter Balanced technical materials (e.g., Content Specifications, Item Specifications, and Test Blueprints) due August 2013

Phase 2: December 2013 · 144 modules developed to cover both ELA and Math for K-2, 3-5, 6- 8, 9-12 describing of the balance and integration of summative, interim, and formative assessments into a Balanced Assessment System consistent with the purpose statements developed by the Test Design Work Group in Smarter Balanced Contract 09 · Visuals and graphics with embedded links to provide information about a particular type of assessment and how it relates to the Smarter Balanced Assessment System and improved teaching and learning · Description of Evidence-Centered Design and the importance of evidence collection in developing quality assessments · Understanding of how the results from the reporting system for Smarter Balanced interim and summative assessments will help teachers plan for instruction to help students identify and attain learning goals · Understanding of how to use resources in the Digital Library to support student learning goals and educator professional learning


ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

plans · Assessment Glossary See C.2.(f).2.6.1 for a comprehensive list of requirements.

4.3.2 Smarter Balanced Score Reports & Educator Resources

Develop and deliver training materials for administrators, teachers, educators, parents, and students to understand how to interpret Smarter Balanced interim and summative reports. Include how to search for resources using the search tools and content tagging system in the Digital Library to address learning needs highlighted by the interim score reports.

Training materials will be based on an approved prototype and have undergone at least one (1) iterative feedback cycle from the State Leadership Teams and the State Networks of Educators. (See Exhibit O)

C.2.(f ).6.3

June 2014


4.3.3 Digital Library Resources

Develop and deliver training materials on how to use the professional learning materials that accompany the release of the Digital Library application. These training materials provide guidance on how to incorporate the digital library resources into a professional learning plan with facilitator tools and templates. The tools will include a variety of options, such as webinars, videos, tutorials, PowerPoint presentations, and a how-to guide for use of the Smarter Balanced- developed and inventory of currently available resources on the Digital Library.

c. Develop a prototype of the training materials and training

delivery process (i.e., proposed format for the training materials, online tools, and web-based help services, and training model) that will be used in 4 to 5 regional face-to-face meetings of 30 to 50 participants for the roll-out of the following educator resources. Once the prototype is reviewed in a feedback cycle with Work Groups and State Leads and approved by the Formative Work Group and Director of Professional Learning, develop specific plans for each training using the review and feedback cycle in Exhibit O.

d. Training materials will include, but will not be limited to the following: • Needs assessment for all stakeholders to identify priorities

for customized training • Entry points for each intended audience (teachers,

C.2.(f ).6.4

August 2013, February 2014, June 2014



ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

administrators, students, and parents) • A variety of options, such as webinars, videos, PowerPoint

presentations, and tutorials • Frequently asked questions and answers for using the Digital

Library and the resources • Using the feedback response tool • Showing the connections in the assessment system and

supporting components via technology tools e. Develop tools for State Leadership Teams to use to self-evaluate

their own understanding and ability to train the State Networks of Educators to use the resources and disseminate the resources to state-wide professional learning networks.

f. Develop evaluation tools to document the effectiveness of the regional meetings and use the feedback from participants to continuously improve the training of State Leadership Teams.

4.4 Educator-Submitted Supplemental Resources

As Smarter Balanced resources are developed by the CONTRACTOR, the State Networks of Educators and educators trained by the SNEs will identify and recommend complementary supplemental resources in assessment literacy, classroom instruction, and formative assessment. Manage this process to ensure a minimum of five Digital Library updates are delivered.

C.2.(f ).7.2

January 2014– End of Contract


4.5 Formative Training for State Leadership Teams

Formative trainings serve two purposes: the first is to demonstrate to state leadership teams how to use vendor-developed training materials and navigate the Digital Library. The second is to equip State Leadership Teams to, in turn, train State Networks of Educators on formative practices and use of developed resources.

Provide trainings to demonstrate to State Leadership Teams in regional face-to-face meetings how to use the professional learning materials that accompany the release of the Digital Library application, assessment literacy training, exemplar instructional modules, and summative and interim score reports.

At these training meetings, which take place between August 2013 and June 2014, the CONTRACTOR uses the training materials and training approach developed to accompany each Smarter Balanced–developed resource to train the State Leadership Teams to use the resources on the Digital Library application and to understand how to train the State

C.2.(f ).6.4

August 2013– June 2014



ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

Networks of Educators and disseminate the training and resources throughout the state-wide professional learning networks on the following:

a. The purpose and appropriate use of the resources in professional

learning to improve classroom instruction b. How to apply an understanding of the resources and Smarter

Balanced Quality Criteria Policies to identify and recommend additional resources for upload to the Digital Library

A total of five trainings will be held:

1. August 2013: inventory of resources currently available approved

for use by educators (See C.2.(f).2.7.1) and teacher-friendly materials to support clear understanding of CONSORTIUM technical materials (See C.2.(f).2.6.1),

2. December 2013: Professional Learning on Assessment Literacy (See C.2.(f).2.6.1),

3. September 2013: Exemplar Instructional Modules released in iterative stages (See C.2.(f).2.6.2) (1 of 2)

4. February 2014: Exemplar Instructional Modules released in iterative stages (See C.2.(f).2.6.2) (2 of 2)

5. June 2014: Use Score Reports and Aligned Resources on the Digital Library (See C.2.(f).2.6.3),

5.0 Digital Library Application Develop and deliver FAPPL Digital Library Application.

5.1 Functional Requirements Conduct a thorough requirements definition process to gather, develop, and deliver detailed functional requirements documents (e.g., use cases, requirements matrices) for all required Components. The requirements listed in Exhibit M can be considered an initial set of requirements that the CONTRACTOR is expected to build upon and refine. The outcome is expected to be a comprehensive list of requirements that will ensure that the application supports the needs of the CONSORTIUM. As the requirements are completed, the CONTRACTOR will manage the approval process for review and sign- off.

C.2.(f ).2.1


5.2 Technical Systems Requirements

Conduct a thorough requirements definition process to gather, develop, and deliver detailed technical systems requirements

C.2.(f ).2.2



ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

documentation for all required Components. The requirements listed in Exhibit M can be considered an initial set of requirements that the CONTRACTOR is expected to build upon and refine. Technical systems requirements for the application must adhere to the IT Systems Architecture guidelines. The requirements should address areas such as, but not limited to the following: • Technical Architecture Definition:

o Server Hardware and Software Requirements for multiple environments

o Development o Staging o Quality Assurance o Disaster Recovery o Physical Security o Virtual Security o Training o Production o Networking Requirements and Diagrams

• Database, Data Storage, and Archiving Requirements and Approach

• Systems Management and Monitoring Requirements • Middleware and Integration Software Requirements • Security Requirements and Approach for Applications, Data, and

End-User Access • Performance capacity (number of users, concurrent users, number

of transactions to be handled, anticipated response times for each component, peak usage, maximum and minimum number of concurrent users for each component, data integrity, load balancing, firewalls, failover approach, redundancy and recovery, reliability, and availability)

• Security: o Component-to-component o User Authentication and Authorization o Educator-level Security o Student-level Security o Parent-level Security o Data at Rest


ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

5.3 Application Design Develop detailed application design and technical specifications deliverables that will be used to drive and support the programming and/or configuration of the software solution.

C.2.(f ).2.3


5.4 Application Development The CONTRACTOR shall provide either (a) a commercial off-the-shelf (COTS) product configured and customized to meet the CONSORTIUM’s requirements or (b) a custom-built application that meets the complete set of functional and technical systems requirements, including, but not limited to software product, development tools, support tools, data migration software, integration software, and installation software. The code shall have good readability (easily read and understood by other programmers), maintainability (easily modified and maintained), and traceability (code elements should correspond to a design element).

C.2.(f ).2.4


5.5 Quality Assurance & Testing

As system components and training materials are developed, the CONTRACTOR will be required to conduct comprehensive testing to validate functionality and performance, including test plan and results for unit testing, system testing, user acceptance testing, quality assurance testing, performance testing, and operations testing.

C.2.(f ).2.5


5.6 Application Deployment The CONTRACTOR will install all necessary hardware, software, and applications through multiple deployments according to the timelines determined in conjunction with the CONSORTIUM and the Major Milestones.

C.2.(f ).2.7


5.7 Data Transfer & Input Ensure transfer and storage of data (qualitative and quantitative) to support the collection of formative evidence and educator professional learning portfolios and integrate with other systems as specified in the Systems Architecture.

C.2.(f ).2.6


5.8 Systems Integration, Quality Control, & Defect Resolution

Ensure the Digital Library is developed consistent with the Smarter Balanced IT Systems Architecture requirements and all other related technical documentation, and collaborate with other Smarter Balanced IT VENDORs to ensure the Digital Library is a fully integrated part of the overall Smarter Balanced IT solution. This may include contributing to systems integration requirements gathering and design activities, quality assurance testing, and defect resolution.

C.2.(f ).2.8


5.9 Application Hosting The CONTRACTOR shall provide secure application hosting allowing for seamless access by all end users. The hosting environment shall integrate with other Smarter Balanced technology components, regardless of where those components may reside. The CONTRACTOR

C.2.(f ).2.9



ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

shall specifically comment on the physical facility, its security, its data protection and recovery plan, and the equipment used to host the web-based application, as well as the backup site.

5.10 Knowledge Transfer, Transition, & Turnover

Implement continuous improvement plan for the Digital Library application post–September 2014, including but not limited to a complete list of requirements for hosting and maintenance, a subscription system with user permissions and access menus, moderated collaboration groups, and resources differentiated by user access to the Test Delivery System through a single user sign-on.

C.2.(f ).2.11

TBD See Exhibit O See Exhibit O 3.2.11; 3.5.3

6.0 Maintenance and Support Services Ensure the successful implementation and continuous improvement of all processes & materials developed.

6.1 Formative Resources & Professional Development Processes & Systems

Manage the implementation of all component plans and systems developed.

Varie s

Continuo us

N/A See Exhibit O 3.6.2

6.2 Formative Resources & Professional Development

Continuously monitor user comments and recommended improvements/adaptations to the Digital Library. Update earlier modules using user feedback before final delivery in June 2014.

Varie s

Continuo us

N/A See Exhibit O 3.6.2; 3.6.2.4

6.3 Collaboration & Help Desk

Provide moderated collaboration & Help Desk support for the State Networks of Educators to support each other in grade-level and/or content-area groups in the identification, recommendation, approval, and upload process, and during the training dissemination process.

Varie s

Continuo us

N/A See Exhibit O 2.13, 3.2.11, 3.7.2, 5.3,

6.4 Digital Library Application Maintenance & Support

Provide staff enhancements and production support of the system as needed, in addition to the ongoing development activities for the remaining deliverables. The scope should include, but not be limited to application management, Digital Library help desk support, systems enhancements, systems maintenance, adaptive and preventive maintenance, performance maintenance, and documentation of updates provided during the contract.

C.2.(f ).2.10


6.5 Reporting 6.5.1 Bi-Annual

Implementation Report

Evaluate and report on the implementation of the comprehensive development strategy every six months throughout the project. Include in the evaluation report the following areas: • Evidence of progress toward meeting the goals of the Digital

Library

C.2.(f ).1.3

Bi- Annually

TBD See Exhibit O 3.1


ID

Deliverable


RFP Ref.

Completi on Date

Reviewers

Communications Plan

Proposal Section

• Perception data related to the measurable benefits to educators in 2013-14

• Documentation that the indicators of a successful implementation are present in participating states or documentation of causes and adjustments made to address indicators of an unsuccessful implementation

• Accomplishment of key milestones and timelines

6.5.2 Digital Library Usage Statistics

Document the number of resources recommended and approved by each participant in the State Network of Educators by state. Report monthly statistics to the State Leadership Teams starting in January 2014 through the end of the contract.

C.2.(f ).7.2

Monthly, starting January 2014

N/A See Exhibit O 3.7.2.4

6.5.3 Monthly Q&A Reports

Provide a Frequently Asked Questions and Answers report to the Formative Work Group and the State Leadership Teams on a monthly basis.

C.3.2. 1

Monthly N/A See Exhibit O 3.5.3

7.0 Project Management


5.0 Outcomes and Performance Measurement

In this section, we describe the impacts/outcomes that our solution will achieve as a result of these efforts, including how these outcomes may be monitored, measured and reported to the CONSORTIUM.

Impact/Outcome Measurement Method of Reporting Increased Student Achievement Summative and interim

assessment scores Accessed from Smarter Balanced Score Reports

Increased usage of Professional Learning Materials and Curricular Resources

Number of times stakeholders (teachers, administrators, educators, parents, students) access resources from the Digital Library

Log Analysis

Increased Report Utilization Score Reports accessed per user over an interval of time

Access Logs

Improved Teacher Effectiveness in Implementing the CCSS

Increased involvement of all stakeholders in student learning

More organizations and individuals contributing to the Digital Library

Increased scores in the relevant sections of each state’s teacher evaluation system Attendance at training sessions (e.g., state-/district-led training, parent nights); number of times stakeholders access resources Number of resources submitted for inclusion in the Digital Library

Self-reported by each state or district Self-reported by each state or district; log analysis Upload Logs


6.0 Risks

The VENDOR should define risks it identifies as being significant to the success of the project. Include in proposal effective monitoring and mitigation plans to manage these risks, including reporting of risks to the agency’s and CONSORTIUM’s designated contract manager.

For any risk that could jeopardize the Beta test, Field Test, or operational timelines, VENDOR should include a proposed contingency plan, including triggers (deliverables or milestones missed) that would be used to determine whether to execute the contingency plan.

Risk Title Probability Impact Mitigation Plan Contingency Plan

Technological infrastructure in schools is not able to support access of resources

Medium Streaming video and other features are blocked because of bandwidth, firewalls, or other technical issues in schools/districts

Work with CONSORTIUM to communicate technical requirements with all stakeholders very early in project lifecycle

Templates and other non-streaming materials are uploaded to be shared locally

Stakeholders are unavailable for training due to contract, schedule, or other issues

Medium to High State training program does not have desired impact for implementing the resources and processes for professional learning

Work with states/district leaders to set expectations well in advance of PD planning for 2013-14 to ensure days are reserved for training

If days are not available, support CONSORTIUM in creating job-embedded PD options (e.g., Webinars, asynchronous tutorials)

Implementation fidelity is sub- optimal

Medium to High Lower usage of online professional learning materials and resources, leading to less effective implementation of CCSS

Additional communication to stakeholders at district and school level

Support CONSORTIUM in creating additional trainings/support on learning materials and resources

Assessments or Score Reports are delayed in their release

Low to Medium Training materials are unable to be completed in time

Delay training until reports are finalized

Conduct training on schedule and add addenda as necessary to be sent to trained participants

technical proposal - office of superintendent of public instruction

Documents