june 19, 2012

June 19, 2012

Evolving Your PBT ProgramThe Citrix Story

Kpayah Tamba, Citrix

© 2012 Citrix | Confidential – Do Not Distribute

About Citrix


Multiple-choice Items

Scenario-based Items

Simulations

Combination of Live Labs, Simulations, Board Defense, and Scenario-based Items

At Citrix, we continue to raise the bar

Real World Skills

Live Labs


Citrix Certification Program

CCIA

CCEE

CCAA

CCA

Top 3 cert tiers consist of exams with performance based items


Design Approaches at each Tier

Board Defense, Live Lab, and Simulation

• CCIA

Simulation and Scenario-based

• CCEE

Simulation and Scenario-based

• CCAA

Mostly Scenario-based, with one exam containing simulation

• CCA


Citrix Evolution

1997

2003

2006

2012

Multiple-choice based exams

First PBT based exam

75% of cert tracks now

includes PBT exams

First simulation

based exam

How we evolved our program


• Standard multiple-choice: single response and multiple response items

Phase I: MCQ Approach


Phase I: MCQ Cont’d

• Prosᵒ Easier to developᵒ Cheaper to developᵒ Easier to gradeᵒ Easier to deliver

• Consᵒ Highly vulnerable to item theftᵒ Not the best item type for assessing

performance based tasks


Phase II: Live Lab Approach

• Testing using the real products in a virtual environment


Phase II: Live Lab Cont’d

• Prosᵒ Test takers embraced this new form of

testingᵒ Citrix was able to measure hands-on

skillsᵒ The value (face validity) of our

certification program increased

• Consᵒ This new form of testing was not

scalableᵒ Candidates had to travel long distances

to take the examᵒ The design and delivery mechanisms

limited the number of exams that could be delivered at any given point

ᵒ The manual scoring resulted in long wait times for results


Phase III: Simulation Approach

• Testing using simulated environments that look and function almost like the real products


Phase III: Simulation Items Cont’d

• Prosᵒ Simulations are scalableᵒ Simulations allow for the measurement

of hands-on skillsᵒ Scoring is automated

• Consᵒ Simulations are costly to buildᵒ Simulations take a considerable

amount of time to design and developᵒ Our simulations are closed-ended;

therefore they imposed limitations on our test takers


Phase IV: Board Defense and Live Lab Items Approaches

• In 2012, Citrix rolled out a six-week intensive training program geared towards certifying our top level audience

• At the end of six weeks of training (eight hours a day, five days a week), candidates are required to complete a rigorous 14-hour hands-on practicum in one sitting, where they build a virtualization solution consisting of Citrix products

• Candidates must also defend the solution they build before a panel of SME judges


Phase IV: Board Defense and Live Lab Items Cont’d

• Prosᵒ The use of both live lab and simulation

items allow for Citrix to continue to measure hands-on skills at a high level, while using two different design approaches based on our need to scale

ᵒ The use of the board defense approach allows us to determine whether test takers know the “why” behind the “how”

• Consᵒ The cost to develop simulation items is

still a problemᵒ Simulations take a considerable

amount of time to design and developᵒ Live labs are difficult to scaleᵒ Live labs are difficult to scoreᵒ A uniform board defense experience is

also difficult to maintain

So what’s next for us at Citrix?


Next Steps

• Increase our PBT footprint

• Fully automate the scoring of our live lab items

• Complete the development of a grading tool for our board assessment

• Localize our live lab items

• Localize our board assessment exam

Conclusion


Conclusion

• As you embark on your next steps to expand your PBT program, ask yourself the following questions:ᵒ How will this design approach improve my program?ᵒ How do I start?

• Do I have the budget?• Do I have the skills, and if not, can I hire or contract the right talents?

ᵒ What is my end goal?ᵒ How do I get there?


Q & A

Work better. Live better.


Group Activity• The design approach document provides:

ᵒ Examples of design approachesᵒ Characteristics of those design approachesᵒ When to use themᵒ When not to use them

• Exercise Objective:ᵒ This exercise will help us identify solutions to potential issues that one could encounter

when using a design approachᵒ We will break up into four groupsᵒ Each group will focus on an approach

• Group A – Live Environment• Group B – Board Assessments/Oral Exams• Group C – Simulating Software Using Software• Group D – Simulating Hardware Using Software


Group Activity – Cont’d

• In each group, for the assigned design approach, identify solutions to the following issues:ᵒ Skills required to develop item typeᵒ Delivery challengesᵒ Psychometric challengesᵒ Other challenges


Wrap Up

• We will regroup

• A representative from each group will share with us how his or her group plans to address the challenges associated with their assigned design approach

june 19, 2012

Documents

citrix confidential

citrix program

citrix designs

performance based items

scenariobased itemsat

simulation items

live lab items

board assessment items