asg cost model presentation new jersey department of education october 21, 2009
DESCRIPTION
October 21, 2009 Assessment Solutions Group Description/Structure The ASG Assessment Cost Model is a variable input, metric based calculation output model Specific assessment program variables are input to the model and applied against cost factors to derive assessment cost Several hundred variables are contained in the model Numerous detailed input variables are assembled for each of the functions and elements involved in constructing an assessment (test development, production and manufacturing, logistics, editing and scoring, reporting, psychometrics, program management, quality assurance, information technology, etc.) These variables are input into the software and matched to metric based calculations/formulas and/or databases to convert the assessment program variables into dollar costsTRANSCRIPT
ASG Cost Model Presentation
New Jersey Department of Education
October 21, 2009
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 2
Overview Model description/Structure Key inputs and outputs Databases Reports Uses Questions
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 3
Description/Structure The ASG Assessment Cost Model is a variable
input, metric based calculation output model Specific assessment program variables are input to the
model and applied against cost factors to derive assessment cost
Several hundred variables are contained in the model
Numerous detailed input variables are assembled for each of the functions and elements involved in constructing an assessment (test development, production and manufacturing, logistics, editing and scoring, reporting, psychometrics, program management, quality assurance, information technology, etc.)
These variables are input into the software and matched to metric based calculations/formulas and/or databases to convert the assessment program variables into dollar costs
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 4
Description/Structure (cont.)
The cost for the entire assessment is built from the ground up, element by element, function by function
The model also contains “efficient cost metrics” for each key cost area and function so that a state’s assessment program cost data can be compared to those of an “efficient” assessment This allows states to see where their
assessment program is cost effective and where there are opportunities to improve costs
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 5
Description/Structure (cont.)
Proprietary functions and calculations have been developed based on our years of experience in the industry
Model outputs have been tested against real data
The model is currently built for ASG use only. We are in the process of developing a web based application that could be shared with states.
Cost Model-Main Menu
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 6
Administration
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 7
Administrative Center
Key Functions:
- Update Rate Tables- Define Security Settings- Create Custom Reports
Assessment Map
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 8
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 9
Key Inputs and Outputs Numerous assessment component inputs
are available for each key area, function etc. Assessment cost can be developed using
detailed data or higher level, summary data Cost outputs by function, area, cost type,
etc. are generated Key metrics are presented to allow
comparison to model efficient cost data for that function/area/component
Canned and custom reports are available at the summary and detailed level
Assessment Cost Model
Key Inputs
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 10
This is a sample of the key inputs.There are over 200 different inputs that are captured in the assessment model.
Data Input Screens
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 11
Data Input Screens - cont’d
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 12
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 13
Databases/TablesThe Model Contains the Following 70+ Tables to Manage Rates & Data Capture Rate Tables for Each Cost Component (examples):
Design/Composition Rates Printing Rates Freight/Shipping Rates Labor Rates – by State, by Occupation Rates for Delivery of On-Line Assessments Open Ended Scoring Reader Rates Item Acquisition/Permission Rates
Key Databases - Labor
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 14
Key Databases - Travel
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 15
Travel Table - Includes Airfare, Lodging and Meal per diem rates for all 50 States and the District of Columbia
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 16
Reports Data presented in report and graphical formats Canned reports include:
Assessment Cost Summary Assessment Cost by Function Detail Assessment Functional Cost Fixed and Variable Cost Data Assessment Price Composition (direct, indirect and margin) Key Metrics Summary & Functional Area Metrics Summary Assessment Costs vs. “efficient costs” Assessment Metrics vs. “efficient metrics” Other reports are currently in development
Reports can also be easily customized per customer request
Assessment Cost Model
Reporting
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 17
Reporting – cont’d
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 18
Costs by Functional Area & Cost per StudentYear 1 (2012) cost for each functional area and cost per
student
October 21, 2009Assessment Solutions Group
www.assessmentgroup.org Confidential
19
Total Cost Cost per Student
Content Development 1,651,325$ 3.82$ Composition/Pre Press 378,957$ 0.88$ Printing 1,759,796$ 4.07$ Warehouse/Distribution 287,404$ 0.66$ Freight Costs 1,013,851$ 2.34$ Receiving/Scanning/Machine Scoring 863,741$ 2.00$ Open Ended Scoring 4,696,106$ 10.85$ Score Reporting 295,374$ 0.68$ Quality Assurance 144,690$ 0.33$ Psychometrics 184,152$ 0.43$ Technology Support 493,564$ 1.14$ Program Management 766,309$ 1.77$ Total 12,535,268$ 28.96$
13.2%3.0%
14.0%
2.3%
8.1%
6.9%
37.5%
2.4%1.2%
1.5%
3.9%6.1%
Cost per Student Breakdown
Content Development Composition/Pre Press PrintingWarehouse/Distribution Freight Costs Receiving/Scanning/Machine ScoringOpen Ended Scoring Score Reporting Quality AssurancePsychometrics Technology Support Program Management
State Assessment Program Cost Data
Costs for the program by functional area for 2012 - 2014
October 21, 2009Assessment Solutions Group
www.assessmentgroup.org Confidential
20
2012 2013 2014 TotalTotal Cost by FunctionContent Development 1,651,325$ 1,456,422$ 1,514,679$ 4,622,426$ Composition/Pre Press 378,957 394,115 409,880 1,182,951$ Printing 1,759,796 1,830,188 1,903,395 5,493,379$ Warehouse/Distribution 1,301,255 1,353,305 1,407,437 4,061,997$ Receiving/Scanning/Machine Scoring 863,741 898,290 934,222 2,696,253$ Open Ended Scoring 4,696,106 4,883,950 5,079,308 14,659,365$ Score Reporting 295,374 307,189 319,477 922,040$ Program Management 766,309 796,961 828,840 2,392,110$ Quality Assurance 144,690 118,282 123,013 385,985$ Psychometrics 184,152 191,518 199,178 574,848$ Technology Support 493,564 427,753 444,863 1,366,179$ Grand Total 12,535,268$ 12,657,974$ 13,164,292$ 38,357,534$
Cost per StudentCost per student for the state assessment program for the years
2012-2014
October 21, 2009Assessment Solutions Group
www.assessmentgroup.org Confidential
21
2012 2013 2014 Avg CostTotal Cost per Student by FunctionContent Development 3.82$ 3.37$ 3.50$ 3.56$ Composition/Pre Press 0.88$ 0.91$ 0.95$ 0.91$ Printing 4.07$ 4.23$ 4.40$ 4.23$ Warehouse/Distribution 3.01$ 3.13$ 3.25$ 3.13$ Receiving/Scanning/Machine Scoring 2.00$ 2.08$ 2.16$ 2.08$ Open Ended Scoring 10.85$ 11.28$ 11.74$ 11.29$ Score Reporting 0.68$ 0.71$ 0.74$ 0.71$ Program Management 1.77$ 1.84$ 1.92$ 1.84$ Quality Assurance 0.33$ 0.27$ 0.28$ 0.30$ Psychometrics 0.43$ 0.44$ 0.46$ 0.44$ Technology Support 1.14$ 0.99$ 1.03$ 1.05$ Grand Total 28.96$ 29.25$ 30.42$ 29.54$
Cost by Content AreaCost by content area for the years 2012 - 2014
October 21, 2009Assessment Solutions Group
www.assessmentgroup.org Confidential
22
$3,612,433
$3,170,401 $1,315,057
$1,249,855
$2,223,060
$964,463 2012 Cost by Content Area
Reading Math Science Social Studies Writing Editing & Mech
$3,649,747
$3,190,451 $1,322,107
$1,254,384
$2,271,776
$969,509 2013 Cost by Content Area
Reading Math Science Social Studies Writing Editing & Mech
$3,795,737
$3,318,069 $1,374,991
$1,304,559
$2,362,647
$1,008,290 2014 Cost by Content Area
Reading Math Science Social Studies Writing Editing & Mech
$11,057,916
$9,678,920 $4,012,155
$3,808,797
$6,857,483
$2,942,262 Total Cost by Content Area
Reading Math Science Social Studies Writing Editing & Mech
Cost by GradeCosts for the state assessment program by grade for 2012 - 2014. Note: grades 4 and 8 are tested more often; grade 3 uses consumable test books
October 21, 2009Assessment Solutions Group
www.assessmentgroup.org Confidential
23
2012 2013 2014 TotalTotal Cost per GradeGrade 3 1,290,123$ 1,311,013$ 1,363,453$ 3,964,589$ Grade 4 2,085,552$ 2,103,288$ 2,187,420$ 6,376,260$ Grade 5 1,379,098$ 1,388,449$ 1,443,987$ 4,211,534$ Grade 6 1,388,489$ 1,405,356$ 1,461,570$ 4,255,415$ Grade 7 1,363,475$ 1,372,070$ 1,426,953$ 4,162,498$ Grade 8 2,183,805$ 2,205,645$ 2,293,870$ 6,683,320$ Grade 9 -$ -$ -$ -$ Grade 10 956,381$ 971,182$ 1,010,029$ 2,937,592$ Grade 11 1,202,516$ 1,205,976$ 1,254,216$ 3,662,708$ Grade 12 685,829$ 694,995$ 722,795$ 2,103,618$ Grand Total 12,535,268$ 12,657,974$ 13,164,292$ 38,357,534$
Cost by Content Area & Grade
Costs for the state assessment program by content area & grade for 2012 – 2014
October 21, 2009Assessment Solutions Group
www.assessmentgroup.org Confidential
24
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Grade 12 Total
2012Reading 679,265$ 489,427$ 495,275$ 505,919$ 474,068$ 505,998$ -$ 462,480$ -$ -$ 3,612,433$ Math 610,858 416,430 453,523 426,915 442,961 421,675 - - 398,039 - 3,170,401 Science - 458,430 - - 446,445 - - - 410,182 - 1,315,057 Social Studies - - 430,300 - - 425,260 - - 394,295 - 1,249,855 Writing - 394,353 - 455,656 - 478,598 - 493,901 - 400,552 2,223,060 Editing & Mech - 326,913 - - - 352,274 - - - 285,277 964,463 Grand Total 1,290,123$ 2,085,552$ 1,379,098$ 1,388,489$ 1,363,475$ 2,183,805$ -$ 956,381$ 1,202,516$ 685,829$ 12,535,268$
2013Reading 691,078$ 493,646$ 499,815$ 510,884$ 477,717$ 510,924$ -$ 465,683$ -$ -$ 3,649,747$ Math 619,934 417,730 456,393 428,720 445,365 423,227 - - 399,081 - 3,190,451 Science - 461,409 - - 448,988 - - - 411,709 - 1,322,107 Social Studies - - 432,241 - - 426,956 - - 395,187 - 1,254,384 Writing - 401,909 - 465,751 - 489,567 - 505,499 - 409,051 2,271,776 Editing & Mech - 328,594 - - - 354,970 - - - 285,945 969,509 Grand Total 1,311,013 2,103,288 1,388,449 1,405,356 1,372,070 2,205,645 - 971,182 1,205,976 694,995 12,657,974
2014Reading 718,721$ 513,392$ 519,808$ 531,320$ 496,825$ 531,361$ -$ 484,310$ -$ -$ 3,795,737$ Math 644,732 434,439 474,649 445,869 463,180 440,156 - - 415,044 - 3,318,069 Science - 479,865 - - 466,948 - - - 428,178 - 1,374,991 Social Studies - - 449,530 - - 444,035 - - 410,994 - 1,304,559 Writing - 417,985 - 484,381 - 509,150 - 525,719 - 425,413 2,362,647 Editing & Mech - 341,738 - - - 369,169 - - - 297,382 1,008,290 Grand Total 1,363,453 2,187,420 1,443,987 1,461,570 1,426,953 2,293,870 - 1,010,029 1,254,216 722,795 13,164,292
Total Costs and Key Metrics by YearTotal program costs and key metrics by year for 2012 - 2014
October 21, 2009Assessment Solutions Group
www.assessmentgroup.org Confidential
25
Functional Area Cost Item Unit of Measure Total Cost Total Units Cost per Unit Total Cost Total Units Cost per Unit Total Cost Total Units Cost per UnitContent Development Item Acquisition Items Acquired 4,524$ 158 28.56$ 4,705$ 158 29.70$ 4,893$ 158 30.89$
Permission Costs Permissions 135,386$ 65 2,089.29$ 140,801$ 65 2,172.86$ 146,433$ 65 2,259.77$ Item Development Total Items Developed 325,697$ 158 2,056.17$ 245,897$ 158 1,552.38$ 255,732$ 158 1,614.47$ Art Costs Art Pieces 14,295$ 128 111.75$ 14,866$ 128 116.22$ 15,461$ 128 120.86$ Item Banking NA 17,335$ NA NA 18,028$ NA NA 18,750$ NA NADevelopment Travel NA 654,733$ NA NA 512,795$ NA NA 533,307$ NA NANRT Licensing NCOunt 499,356$ 390,800 1.28$ 519,330$ 390,800 1.33$ 540,103$ 390,800 1.38$
Composition/Pre-Press Total Composition Costs Pages Composed 378,957$ 6,700 56.56$ 394,115$ 6,700 58.82$ 409,880$ 6,700 61.18$
Printing Total Printing Costs Bound Units 1,759,796$ 3,203,741 0.55$ 1,830,188$ 3,203,741 0.57$ 1,903,395$ 3,203,741 0.59$
Warehouse/Distribution Total Warehouse Costs Pounds Shipped 287,404$ 383,502 0.75$ 298,900$ 383,502 0.78$ 310,856$ 383,502 0.81$ Freight Out Cost Pounds Shipped 427,331$ 383,502 1.11$ 444,424$ 383,502 1.16$ 462,201$ 383,502 1.21$ Scoring Freight In Pounds Returned 586,520$ 350,106 1.68$ 609,981$ 350,106 1.74$ 634,380$ 350,106 1.81$
Receiving/Order Entry Documents 38,778$ 1,356,100 0.03$ 40,329$ 1,356,100 0.03$ 41,942$ 1,356,100 0.03$ Document Prep/Staging Documents 468,487$ 1,356,100 0.35$ 487,227$ 1,356,100 0.36$ 506,716$ 1,356,100 0.37$ Image Scanning Sheets Scanned 95,804$ 6,512,200 0.01$ 99,636$ 6,512,200 0.02$ 103,622$ 6,512,200 0.02$ Document Editing N-Count 260,671$ 432,800 0.60$ 271,098$ 432,800 0.63$ 281,942$ 432,800 0.65$
Open Ended Scoring Total Cost Scores 4,696,106$ 6,511,586 0.72$ 4,883,950$ 6,511,586 0.75$ 5,079,308$ 6,511,586 0.78$
Score Reporting Total Cost Pages Printed 295,374$ 1,731,200 0.17$ 307,189$ 1,731,200 0.18$ 319,477$ 1,731,200 0.18$
All Areas Total Program Cost N-Count 12,535,268$ 432,800 28.96$ 12,657,974$ 432,800 29.25$ 13,164,292$ 432,800 30.42$
2014
Receiving/Scanning/Machine Scoring
2012 2013
Summary of Impact of Options/Changes in Costs
October 21, 2009 26
Chart shows cost savings or increases related to various options/changes to the program for 2012 - 2014
Reporting – cont’d
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 27
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 28
Uses The model can be used in several ways to
improve assessment cost and efficiency: Developing a new RFP
ASG cost sheets key in discerning if vendors understand your program and are proposing apples to apples solutions
Making tradeoffs between various assessment components Develop the best assessment for the available dollars
Review of an existing program to determine areas of inefficiency and opportunity
Assessment costs tend to be too high because of inefficiencies in the program design
Working with the assessment vendor to reduce/eliminate efficiencies and lower costs is a win/win for both the vendor and customer
Determine the impact of joining a state consortia Devising scope changes
October 21, 2009Assessment Solutions Group www.assessmentgroup.org 29
Questions ASG would be happy to answer any
questions or provide a state specific demo of the model on request.
Contact information: Barry Topol - 210-859-9920
[email protected] Pat Hennon - 210-376-7654
[email protected] John Olson - 210-859-5593
[email protected] Ed Roeber - 517-614-4877