open source innovations for better assessment brandt redd cto ncsa 22 june 2015
TRANSCRIPT
![Page 1: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/1.jpg)
Open Source Innovationsfor Better Assessment
Brandt ReddCTO
NCSA22 June 2015
![Page 2: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/2.jpg)
2
Smarter Balanced at a Glance
• 23 Members: 21 States, US Virgin Islands, Bureau of Indian Education
• Represent more than 14 million students.
• Formative, Interim, and Summative assessments aligned to the Common Core State Standards for English Language Arts andMathematics.
![Page 3: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/3.jpg)
3
Achievements in the Last 12 Months
• New home at UCLA
• Released open source assessment software (Assessment Production, Test Delivery, Scoring, Reporting, Digital Library)
• Summative tests delivered to 6.5 million students in 18 states.
• Millions of interim tests delivered.
• More than 570,000 registered teachers on the digital library.
• Nearly 200 colleges accept Smarter Balanced assessment for placement purposes with more on the way.
![Page 4: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/4.jpg)
4
Devices Used in Spring 2015 Tests
Windows47%
ChromeBook33%
MacOS12%
iPad8%
Others1%
Fraction of 38 million test sessions using each device type.
5.6% Windows 838.5% Windows 7 0.1% Windows Vista 2.3% Windows XP(percent of all devices)
(Preliminary Results as of 22 May 2015)
![Page 5: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/5.jpg)
5
Innovative Features
• Constructed Responses: Graphing, Equations
• Accessibility Features: Text-To-Speech, Refreshable Braille, Tactile Graphics, more
• Computer Adaptive Tests (CAT)
![Page 6: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/6.jpg)
6
SmarterApp.orgSpecifications
• Core Standards (Format and API)• Item Format• Item Metadata• Content and Test Packaging• Test Results• Reporting API• Data Extracts• More
![Page 7: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/7.jpg)
7
Interoperability Standards and Specifications used by SmarterApp
• Common Core State Standards
• SmarterApp Assessment Item Format
• Common Education Data Standards (CEDS)– Smarter Balanced Test Results Loaded into “Align”
• IMS Content Packaging
• SAML and OAuth (Single sign-on)
• Smarter Balanced Profile of IMS QTI/APIP*
• LRMI and Learning Registry*
• Accessible Rendered Item format (ARI)**Planned
![Page 8: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/8.jpg)
Interoperability Challenge
• Each member state chooses its service provider to administer, score, and report the Smarter Balanced tests.
8
![Page 9: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/9.jpg)
9
Technical Challenges
• Deploying open source across vendors• Packaging assessments for multiple
vendors• Collecting data in multiple formats from
multiple states and vendors• Efficient assessment production system• Sustaining pace of innovation
![Page 10: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/10.jpg)
10
Open Source Tools for theAssessment Lifecycle
TestBlueprintStandards
ItemSpecifica-
tions
ItemAuthoring
ItemBanking
TestAuthoring
TestPackage
TestSpecifica-
tions
TestDelivery
ItemScoring
TestScoring
TestResults
HandScoring
ReportingIndividual
& AggregateReports
DataWarehouse
AssessmentDesign
AssessmentProduction
AssessmentDelivery
Reporting& Analytics
![Page 11: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/11.jpg)
11
Assessment Production
• Current Status– Contracted out for Assessment Production Support in
2014-2015 and 2015-2016– Open Source Item Authoring, Item Bank, and Test
Authoring tools on SmarterApp.org
• Roadmap– Integrated assessment authoring system based on
software production systems– Software RCS for Item Pool– Open Source and Multivendor Item Authoring– Open Source Test Authoring and CAT Simulation
![Page 12: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/12.jpg)
12
Assessment Delivery
• Current Status– Summative tests delivered to more than 6.5 Million Students in
Spring 2015.– 18 States and 5 Test Delivery Vendors– 3 States used the Open Source Test Delivery System (TDS)
• Enhancements for 2015-2016– Increased robustness, convenience, and capacity in the TDS– Universal Secure Browsers– Accessibility Updates
• Enhancements for 2016-2017– Horizontal server scaling– New Accessible Rendered Item (ARI) format– Summative Handscoring– Graphing Calculator– Accessibility Updates
![Page 13: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/13.jpg)
13
Reporting & Data Warehouse
• Current Status– Used by 4 States in 2014-2015– Some hosted by Smarter balanced, others hosted by
vendors
• Roadmap– Updated “Test Results Transmission” format– Public Reporting– Clearer and more customizable Individual Student
Reports– Links to the Digital Library
![Page 14: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/14.jpg)
14
Digital Library of Formative Materials
• Current Status– More than 2,200 curated resources– More than 575,000 registered teachers– Community features – content submission,
sharing of experiences, feedback
• Roadmap– Better integration with the Learning Registry– Improved Search– Local Community Groups– More Content– More Community Features
![Page 15: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/15.jpg)
15
Accessible Rendered Item Format
• “Like a PDF for Assessment Items”
• Facilitates innovative new item types.
• Complimentary to QTI/APIP
• Leverages JavaScript for accessibility and scoring
• Supports multiple item authoring systems – each specializing in certain item types
• Rendering modes:– Response Capture– Inline feedback (for formative use)– Response rendering (for scoring and review)
• xAPI and Caliper compatibility
![Page 16: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/16.jpg)
16
Opportunities
• For Assessment Technology Vendors– Watch for Smarter Balanced RFPs– Incorporate SmarterApp technology into your
solutions for interoperability and cost savings– Deploy technology for Smarter Balanced states
• For Assessment Developers– Leverage production and delivery tools– Consistent delivery platform across subject areas
• For States and Districts– Consistent secure browser technology across all
assessments– Build state and local assessments on the same
technology platform as Smarter Balanced
![Page 17: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/17.jpg)
17
Q&A
• Resources– http://www.smarterbalanced.org– http://www.smarterapp.org
• Specifications including test results
![Page 18: Open Source Innovations for Better Assessment Brandt Redd CTO NCSA 22 June 2015](https://reader035.vdocument.in/reader035/viewer/2022070407/56649e395503460f94b2b0f0/html5/thumbnails/18.jpg)
Sample Individual Student Report
18