power up your learning with assessments
TRANSCRIPT
WelcomeHow Assessments “Power-up” Your Learning Solution
Rose LoisiGlobal Solution OwnerSAP Education
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 3
Welcome and Agenda
Market Drivers for Assessments and the Future of Testing“Your learning solution today, and your learning solution tomorrow withassessment & testing functionality”
Why Validate Learning and Assessments to AchieveContinuous Learning Outcomes“Power up your learning solution with assessments & certification > deliveringtraining, defensibility, authoring, reporting capabilities”
Real Customer Case studies will be shared around how trustable and traceableassessments can be achieved
Closing Remarks and Q&A
Putting Assessments in Perspective. Thoughts and Considerations for aContinuous Learning Strategy
Rose LoisiGlobal Solution OwnerSAP Education
Che OsborneVice President of StrategyQuestionmark
Rose LoisiChe Osborne
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 4
What’s Driving the Market for Assessment Management
Measure and Enhance Learning� Measure knowledge, skills and attitudes –
before, during and after learning� Correct misconceptions and increase
knowledge retention
Regulatory Compliance� Ensure knowledge of essential laws,
regulations and health/safety guidelines� Demonstrate compliance and reduce risk
Qualification and Certification� Deliver reliable, defensible tests to certify
employees� Ensure sound personnel decisions
Sales Channel Verification� Ensure competence and product knowledge
of sales professionals� Increase revenues and customer
satisfaction
Technical Channel Verification� Ensure competence and expertise of
technical and service professionals� Reduce costs and improve customer
satisfaction
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 5
46%more likely to be first tomarket
Innovation Productivity Time to Market
Quality Profitability
37%greater employeeproductivity
34%better response tocustomer needs
26%greater ability to deliver“quality products”
17%more likely to be marketshare leaders
Skills for the Future
58%more prepared to meetfuture demand
Companies Who EmbraceLearning Outperform Their Competition
Bersin & Associates, 2012
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 6
Fitting Assessments within the Learning Roadmap
Save opportunity costsSAP Virtual Live Classroom TrainingScheduled Live Online
SAP Online TrainingAnytime, anywhere (eLearning, eAcademy)
Enable your workforceon demandon any SAP topicSAP Learning HubSubscription-based access to alleLearning content
Validate LearnerKnowledge, validate, tracequality and integrate tolearningIndividual certificationMulti-level and role-based exams thrutesting , assessments and exams
Accelerate end userperformance & effectivenessSAP Knowledge AccelerationOff-the-shelf end user enablementcontent
Create your own learningmaterialSAP Workforce Performance BuilderCustomized learning content
Measure solution adoptionand workforce performancelevelSAP User Experience Mgmt by KNOAVerify end user performance and knowledgeacquisition
Manage and track enterpriselearning activitiesSAP Enterprise Learning / Learning Solution/ xLSOTarget, schedule and track training
Get live hands-on trainingSAP Classroom TrainingScheduled Live Classroom
Introduction to SAP AssessmentManagement by Questionmark
Che OsborneVice [email protected] June 2015
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 8Internal
Agenda
• Brief Overview of Questionmark Corporation• Questionmark in Aviation
• Assessment Solutions enabled by Questionmark• Collaborative Authoring• Blended Delivery options including:
• Secure Browser• Mobile Devices• Embedding Assessments in Blogs, Wikis, Portals and more
• Reports and Assessment Result Analytics
• Observational Assessments
• Translation Management for Multilingual Assessments
• Integrating Questionmark with other systems
• Why Questionmark OnDemand
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 9Internal
About Questionmark
Background� Founded in 1988� Offices in US, UK, Canada; Distribution
partners in 27 countries� Questionmark provides technologies and
services that enable organizations tomeasure knowledge, skills, and attitudessecurely for certification, regulatorycompliance and improved learningoutcomes.
Customers� 1,000 current enterprise customers
worldwide� Corporate and Government (Learning/HR
function); Higher Education; AwardingBodies and Test Publishers
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 10Internal
What can we learn from assessments?
Did theylike it?
Did theylearn it?
Can theydo it?
Can weprove it?
Trustable results are essential to answering these questions
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 11Internal
Questionmark: A complete range of Assessment types
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 12Internal
SAP Assessment Management
AnalyzeAbility to evaluate results
and inform stakeholderswith timely, meaningfulreports
DeliverBlended delivery to
support a broad range ofdevices and securityneeds
SAP Assessment Management, cloud editionSAP Assessment Management, on-premise edition
Create� Collaborative, secure
authoring of surveys,quizzes, tests, and exams
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 13
How assessment authoring is managed
Questions organized by topicsMultiple authors able to collaborate21 different types of questions
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 14Internal
Wizards to create 21 item typesv *Drag and Drop
v *Essay
v *Explanation
v Fill in Blanksv *Hotspot
v *Likert
v Adobe Flash
v Adobe Captivate
v *Matching
v Knowledge Matrix
v Survey Matrixv *Multiple Choicev *Multiple Responsev *Numericv File Uploadv *Pull-down Listv Rankingv Select a Blankv *Text Matchv *True/Falsev *Yes/No
§ Enable authors to create more interesting and probing questions
§ Reveals more information during an assessment in a shorter time
§ Not offered by simpler assessment tools in LMS’s
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 15Internal
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Evaluation
Synthesis
Analysis
Application
Comprehension
Knowledge
Questions betterif use application/ comprehension
Not justknowledge
Blooms Taxonomy
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 16Internal
Multilingual Assessments
Slide 16
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 17Internal
Right to Left Delivery
Slide17
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 19Internal
Link. Launch.Embed.
Secure BrowserWeb Browser Questionmark to Go
Integration Windows / CDTest Centers
Printing andScanning
Mobile Devices
Social Networking PlatformsEmbedded in Wikis,
Blogs, and Web pages
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 20Internal
Responsive Design
Slide20
2. Device drivers used to format assessment for device/browser
Part
icip
ant
Dev
ices
4. On-the-fly auto-sizing of content and controls
1. Senses device/Browser 3. Sends assessment to browser
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 21Internal
Formative QuizzesCourse Evaluations
DiagnosticTests
LowStakes Tests
MediumStakesExams
HighStakesExams
Leve
lsof
Mon
itorin
gLe
vels
ofM
onito
ring
Delivery Stakes and Devices
Stakes of Assessment
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 22Internal
Questionmark SecureOne Setting – Multiple Platforms
Slide 22
Windows
Mac
iPad
Printing
Screen capture
Task switching
Control/system keys
Right-click / context menu options
View source
Accidentally exiting exam
Prevents….
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 23Internal
Analysis
How are our Assessments doing? How are our Participants doing?
Did the learninginvestment impact
performance?Where are the skills andknowledge gaps in my
organization?
Are my questions andtests valid and reliable?
What did participantsthink of the learning
experience?
What did participantslearn from the training or
course?
How do I coachparticipants to improve
performance?
How did one group oftest takers compare to
another?
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 24Internal
Item Analysis: Are the questions reliable? Defensible?
Item details and statistics
Breakdown of item responsesby choice
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 25Internal
Coaching Report: Individualized Reporting for Participantsand other Stakeholders
Configurablequestion-by-
questionreporting
Compareparticipants
to othersor to
benchmarks
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 27Internal
Business drivers - key reasons our customers useassessments
Measure and Enhance Learning� Measure knowledge, skills and attitudes –
before, during and after learning� Correct misconceptions and increase
knowledge retention
Regulatory Compliance� Ensure knowledge of essential laws,
regulations and health/safety guidelines� Demonstrate compliance and reduce risk
Qualification and Certification� Deliver reliable, defensible tests to certify
employees� Ensure sound personnel decisions
Sales Channel Verification� Ensure competence and product knowledge
of sales professionals� Increase revenues and customer
satisfaction
Technical Channel Verification� Ensure competence and expertise of
technical and service professionals� Reduce costs and improve customer
satisfaction
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 28Internal
IndustryFinancial Services / Banking
NeedsDelivers 300,000+ assessments annually for risk management and to meetannual compliance requirements
DriversNeed to centralize and standards item creation, item banking and assessmentauthoringAbility to conduct item analysis to ensure reliability and defensibility ofassessment content
Example: Regulatory compliance – Global Accountancy
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 29Internal
Management Report ExampleA) Number of people 1,000B) Time for training if no testing (2 hours each) 2,000 hours
C) Time to take test (20 mins / test) 333 hoursD) 25% pass test and skip trainingE) 75% do compliance training 1,500 hoursF) Hours of training avoided (B – C - E) 177 hoursG) Average cost / hour $100H) Money saved (F * G) $17,700
Proving ROI: Saving money by “testing out”
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 30Internal
Example: Sales Channel Verification - Retail
WhySalesEnsure essential product knowledge/differentiator� Increased revenue� Increased customer satisfaction
TechnicalReduce time to solve problems� Increased customer satisfaction� Reduced costs
Verify KSAs inChannel to
Market
Directemployees
Sales
Technical
Channelpartners
Sales
Technical
Who
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 31Internal
IndustryRetail / Consumer Products
NeedsSales Channel Verification� Test knowledge of 300,000 retail sales reps
DriversScalability – compressed time-frame of product knowledge testing requiredability to deliver 40,000 assessments per hourRapid authoring to meet short time-frame product releasesBenefitsIncreased revenuesIncreased customer satisfaction
Major Electronics Retailer
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 32Internal
Example: Learning and Development – Oil and Gas
Industry� Oil & Gas
Needs� Uniform assessment mechanism across the
organization� ensure standards are maintained� track staff progress over time� measure training effectiveness� provide training and assessment to global
workforce� Tie test results to career advancement
Key Drivers� Ability to deploy Internet-based training
essential for global company with 126,000employees in 85 different countries� Multiple tests for each stage of employee
development� Central repository for assessment results� Measurements aligned with global
standards for employee performance
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 33Internal
Example: Aviation industry – Multiple business drivers
Slide33
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 34Internal
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessment for Ground Crew
Slide34
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 35Internal
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessment for Flight Crew
Slide35
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 36Internal
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessment for Front Desk
Slide36
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 37Internal
SAP Assessment Management by Questionmark
SAP Assessment Management byQuestionmark provides the ability to:
• Create collaborative, secureauthoring of surveys, quizzes,tests, and exams
• Deliver blended delivery to supporta broad range of devices andsecurity needs
• Analyze results and informstakeholders with timely,meaningful, and defensible reports
QuizzesTest
ExamsSurveys
QuizzesTest
ExamsSurveys
QuizzesTest
ExamsSurveys
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 38Internal
What can we learn from assessments?
Did theylike it?
Did theylearn it?
Can theydo it?
Can weprove it?
Trustable results are essential to answering these questions
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 39Internal
Value of Assessments in the Workplace
Life, Limb and Livelihoodon the line
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 40Internal
How assessments mitigate risk and demonstratecompliance
Regulators encourage them
Demonstrate commitment
Find problems early
Document understanding after training
Increase knowledge / reduce forgetting
Reduce cost & time for compliance training
Reduce human error
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 41Internal
What are some of the other solutions available from Questionmark?
Slide41
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 42Internal
Mobile Delivery of Assessments
Deliver to thedevice in your
hand
Deliver to thedevice in your
handObservationalassessmentsObservationalassessments
Mobile testcenters
Mobile testcenters
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 43Internal
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Mobile AppsAndroid
iPhone, iPadand iPod touch
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 44Internal
Scan a QR Code With Your Mobile Device to Start anAssessment!
Benefits• Ease of access for
increased response rates• Demographic data easily
captured• Better data accuracy• Easier analysis of survey
results by demographic
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 45Internal
Observational Assessment
Certification Process• Observational assessments can beused as critical components of thecertification process – in some casesare required by regulatory authorities.
Medical and Dental Schoolexams:• Medical and dental schools makeextensive use of observationalassessments such as the ObjectiveStructured Clinical Examination (OSCE).
Equipment operation:• Observational assessment can be usedto record and rate operators' skills,abilities, safety practices and adherenceto required procedures
Example Applications
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 46Internal
Embedding Assessments in Wikis, blogs, portals
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 47Internal
Benefits of Embedding in Blogs, Portals, Wiki’s
For Learners For Organizations
• Focus the learners’ attention
• Practice memory recall
• Identify what was learned
• Provide feedback to assist thelearner to:
• Correct misconceptions• Instill confidence• Point to relevant material
• Embrace 70+20+10 withconfidence
• Easily use assessmentsthroughout the learning process
• Provide seamless userexperience
• Centralized management ofassessment content and results
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 48Internal
Launch andTrack
• AICC• SCORM
API / DataIntegration
• QMWISe• OData• xAPI• LTI• HR-XML
Connectorsfor…
• SAP• Blackboard• Moodle• SharePoint
Embedded /iFrame / Portals
• Wikis• Blogs• Portals• SharePoint
Can I integrate Questionmark with various otherEnterprise Systems?
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 49Internal
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Custom Reportsand Dashboards
MicrosoftOffice BI Tools
Slide49
Results Warehouse
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 50Internal
Integrating with SuccessFactors Learning – AICC/SCORM
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 51Internal
SAP LSO and SFL Customers Using Questionmark
Sanlam
• Industry• Financial Services
• Needs• Demonstrate its
compliance withgovernmentregulations andprove employeecompetence
• Key drivers• Reporting to
documentcompliance
• Blended delivery• Integrate with SAP
Human Resourcesand EnterpriseLearning
Nebraska PublicPower District
• Industry• Energy and Utilities
• Needs• Health and safety;• Meet nuclear
regulatoryrequirements
• Key Drivers• Centralize
management ofitems and exams
• Collaborative,authoring for SMEsand instructors
• Monitored(proctored) exams
• Integrate with LSO
BNSF
• Industry• Transportation
• Needs• Testing 35,000+
employees in inHazmat, RailroadSecurity, etc.
• Certify 20,000transportationemployees
• Key drivers:• Robust authoring
features• Accessibility• Security and limit
retakes• Integration with
LSO
IntermountainHealthcare
• Industry• Healthcare
• Needs• Testing and
assessment for30,000 employeesat 21 hospitals and162 clinics forregulatorycompliance, HIPAAtraining
• Key drivers:• Accommodate 80+
authors• Online delivery to
multiple locations• Timed assessments• Integrate with
SuccessFactors
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 52Internal
Ways to Learn More
Questionmark Blog!• http://blog.questionmark.com
Questionmark Training Course• http://www.questionmark.com/training
White Papers• http://www.questionmark.com/whitepapers
Manuals, Knowledge Base and Best Practices Guides• http://www.questionmark.com/perception/help
Web Seminars• www.questionmark.com/seminars
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 53Internal
Assessments through the Learning Process
Security of Questionmark’s OnDemand Service
Alignment, Impact and Measurement with the A-model
Embedded Assessments: Building KnowledgeChecks, Surveys and Other Assessments intoLearning Materials
How Questionmark Aids ISO 10667 Compliance
Recommended White Papers
https://www.questionmark.com/whitepapers
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 54Internal
Ways to Evaluate
Slide54
Try Questionmark OnDemand• 30 Day Trial – Software as a Service• https://www.questionmark.com/go/try430days
Try installable version ofQuestionmark• Download – 30 day trial• https://www.questionmark.com/download
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 55Internal
Questionmark OnDemandScalable, SaaS-based Testing and Assessment
Solution
SSAE 16 AuditedData Center
State-of-the-artdata centers
SaaS /Rapid Deployment
24/7 Monitoring Global 24/7 supportavailable 99.9+% Uptime
Scalable to meethigh-volume, peakdelivery needs
Always have thelatest version
Turnkey service - nohardware to buy andmaintain
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 56Internal
Keep in Touch
Slide 56 blog.questionmark.com
www.questionmark.com/newsletter
twitter.com/questionmark
linkedin.com Look for Questionmark under “Groups” and join
youtube.com/questionmarkvideo
flickr.com/questionmarkpictures
facebook.com Become a “fan” for news, updates and more
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 57Internal
• Confidently use data to connect Learning Outcomes to Strategy
• Tools (and learning materials) available for SMEs• Help harvesting content and produce valid/reliable tests
• Question Banks• Easily organize thousands of questions
• Responsive Design• Work with the devices and browsers that your users use
• Multi-lingual user interfaces• Use worldwide
• Translation Management• Easily manage multi-lingual assessments
• Joined up data model• Helpful report for user• Assessments are proven to be defensible
Why Questionmark?
© 2014 SAP SE or an SAP affiliate company. All rights reserved. 59
Powering-up Your Assessments and LearningMaking it a Strategic Priority
SAP Insider Article