![Page 1: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/1.jpg)
Innovative Item Types: How to Evaluate the Bang for the Buck
Brian Bontempo, Ph.D.Microsoft Corporation
![Page 2: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/2.jpg)
Continuum of Innovation
![Page 3: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/3.jpg)
Complexity of Innovation
Non-Text StimuliPolytomous ScoringMultiple Selected ResponseSimple Constructed ResponseCase Based ItemsInteractivity
Complex Constructed ResponseAutomate Judgment ScoringInteractivity
![Page 4: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/4.jpg)
Shoot the moon
If you fail to shoot the moon in the game of Hearts, the consequences are monumental.
![Page 5: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/5.jpg)
Goal
Provide a framework whereby an organization can evaluate a testing program and make sound decisions about the development and implementation of Innovative Item Types.
![Page 6: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/6.jpg)
Synthesis
Innovative Item TypesEvaluationCognitive and Performance Assessment
![Page 7: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/7.jpg)
Process of Evaluation
Involve StakeholdersEstablish the goal in using IITAscertain IIT mobilityIdentify possible directionsGauge the impact of the directions on the StakeholdersCollect & analyze some dataShare your results
![Page 8: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/8.jpg)
Involve Stakeholders
Item Developers – Minimize Development EffortManagers – Minimize Development CostPsychometricians – Maximize Test Predictive CapabilitiesTest Delivery Vendor – Minimize Test Publishing EffortsTest-Takers – Minimize Test Administration Effort Test-Users – Maximize Test Utility
![Page 9: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/9.jpg)
Why Implement IIT?
Improve the quantity of information gatheredImprove the quality of information gatheredImprove the efficiency of gathering informationImprove the efficiency of processing information Improve the fidelity of the information
![Page 10: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/10.jpg)
Why Implement IIT?
Gather a different type of information
![Page 11: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/11.jpg)
Domain
Job Task
Difficulty Content
Innovative Item
Traditional Item
![Page 12: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/12.jpg)
Performance without Knowledge can be incomplete
Knowledge
Performance
![Page 13: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/13.jpg)
Ascertain IIT Mobility
Do you know how information is obtained by professionals on the job?Do you have confidence in your scoring rubric development process?Do you have enough access to SMEs to serve as scoring judges?Does your profession have a clearly defined, widely accepted conceptual model for practice?
![Page 14: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/14.jpg)
Ascertain IIT Mobility –Stakeholders Perspectives
How quickly do tests need to be developed?How skilled are the item developers?How flexible is the test administration time?How skilled are your test-takers with the interface?What are your test delivery limitations?
What level of consistency is required of the test-delivery system?Test delivery vendor specifications
![Page 15: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/15.jpg)
Identify Possible Directions
Add Non-Text StimuliAdd Tables, Charts, Audio/Video ClipsAccess reference materialAccess internet help
Expand Item Scoring RubricDichotomous Polytomous
Expand Number of Responses / ItemSingle Multiple Response Option
Add New Item Response FormatsSelected Constructed Response
![Page 16: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/16.jpg)
More Solutions
Increase Task ComplexityFill-in-the-Blank Essay
Change Item Scoring MethodPredefined Rubric Judgment Based Review Automated Judgments
Increase InteractivityCase-Based Items Branching Cases Simulation
![Page 17: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/17.jpg)
Gauge the Impact of the Solutions on the Stakeholders
How much extra time to develop items?How much extra cost to develop items?How much is reliability changed?How operationally reliable are the items?How much extra time to answer the items?How much extra fidelity is achieved?
![Page 18: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/18.jpg)
Did you think of this?
CheatingGamingConfounding variables
Reading comprehensionUser computer literacyConsistency of administrative experience
![Page 19: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/19.jpg)
Collect Pilot Data
ID – Development TimeM – Development CostPsy – Item Performance Characteristics by InnovationTDV – System ReliabilityTT – Customer Satisfaction with Item TypeTU – Validity Study - Critical Incidents
![Page 20: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/20.jpg)
Share Results
Bring your results to ATP!
![Page 21: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/21.jpg)
Bang for the Buck
Innovations should provide deliverables along the wayInnovation deliverables should improve the quality and development of later innovationsAssess the added value with each innovation
![Page 22: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/22.jpg)
Bibliography
Parshall, C. (In press) Innovative Item Types. Computer Based Testing: Building the Foundations for Future Assessments. NJ: Lawrence Earlbaum Associates. Drasgow, F. and Olson-Buchanan, J.B. (1999). Innovations in Computerized Assessment. NJ: Lawrence EarlbaumAssociates.Haladyna, T.M. (1994) Developing and Validating Multiple-Choice Items. NJ: Lawrence Earlbaum Associates.Osterlind, S. J. (1998) Constructing Test Items: Multiple Choice, Constructed-Response, Performance and Other Formats. Boston : Kluwer Academic Publishers.
![Page 23: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/23.jpg)
History of Innovative Assessment
Performance AssessmentPortfolio
![Page 24: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/24.jpg)
What’s unique about today’s innovative item types?
InteractivityAutomated Scoring
![Page 25: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/25.jpg)
Issues – Item Developers
Time to DevelopEase of Content CreationEase of Item ConstructionItem Accuracy
![Page 26: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/26.jpg)
Issues – Test Taker
Validity – Do the test tasks seem appropriate?Fidelity – Are the test tasks real?Cheating – Are other people cheating?
![Page 27: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/27.jpg)
Issues – Managers
Cost of DevelopmentImpact on Test VolumesCustomer Satisfaction
![Page 28: Innovative Item Types: How to Evaluate the Bang for the Buck](https://reader036.vdocument.in/reader036/viewer/2022071602/613d6837736caf36b75cf72e/html5/thumbnails/28.jpg)
Issues – Psychometricians
Performance of ItemsAmount of Information Item ProvidesTest QualityCongruence between item type and cognitive taskDependency – Case, Item, Task, or StepWeighting – Cases, Items, Tasks, or StepsPrecision and Accuracy of Scoring Rubric