olf1206 402 livingston

For More Measurement at More Levels--Engage the Community: A

Success Story at Microsoft Ludovic Fourrage, Microsoft Corporation & Laura L. Livingston, Resolute Corporation

Produced by

December 7 & 8, 2006

402

Designing & Developing Online Assessments & Evaluations

Designing & Developing Online Assessments and Evaluations


Page 1Session 402 – For More Measurement at More Levels--Engage the Community: A Success Story at Microsoft, Laura L. Livingston, Resolute Corporation & LudovicFourrage, Microsoft Corporation

Session #402

ObjectivesIdentify business, program and learning goals to guide your evaluation effortsUse self-reporting in surveys to gather data useful at Levels, I, II, and III and for ROI Capitalize on opportunities to involved the field in building a “culture” of community involvement in gathering dataClose the loop on evaluation data through reporting and actionUse data as a digital dashboard to adjust the program

Session #402

Polling Slide #1How is your elearning program funded?




Session #402

Polling Slide #2What is the primary way you currently prove the worth of your elearning program to your management?

Session #402

Our SituationAudience:•Technical Pre-Sales Specialists Worldwide •On the road much of the time•Priorities: customers and quota

Existing Training Programs were: •Predominately Instructor-Led and didn’t scale•Released too late in product life cycle•Not role-specific, plain vanilla

Resources: Modest funds and small staff

Management Support: Minimal




Session #402

Our Charter

Create low-cost learning solutions that:•Integrate effectively into the audience’s work environment•Focus on the audience’s specific content needs•Get content to the audience before products release•Are flexible, agile, and easy to support and grow

Session #40212/3/2006 6

Our Solution−

Virtual classes delivered worldwide• Asynchronous• Emerging content• Virtual classroom on Share Point• Volunteer Instructors facilitate classes• Student collaboration• 60 to 70 classes per year One hour Live Meeting webcasts• Technical Experts from Product Groups• Emerging content• 225 per yearMentored Self-StudyPodcasts (coming soon)




Session #402

Demo

Session #402

Our StrategyPartner with the field•Grow a great relationship•Trust them to know what they need •Give them what they ask for•Focus on “just in time” and “just enough”

Evaluate and respond•Evaluate at multiple levels •Publish and respond visibly to findings

Grow program support from the bottom up




Session #402

Build evaluation into everything we do

Foster a culture of ongoing dialog with the our constituents

Get the data we need to fix or improve the program, quicklyProvide ongoing “evidence” to the field that we hear and act on their input

Provide the data to managers and execs who need to support and fund the program

Our Evaluation Strategy

Session #402

Questions

About Overall Strategy




Session #402

How We Gather Data

Identify goals (business, program, learning)Design metric(s) to measure each goal:•At different levels (Levels I-III)•With multiple audiences•Through audience self-reporting

Thread each metric through multiple surveys for multiple audiences

Session #402

Example

Post Class SurveySix Month SurveyManager SurveyInstructor SurveysFocus Group

Academy classes help improve Customer satisfaction

Rating of 7.5 or higher

Improve Customer Satisfaction

MeasurementMetricGoal

Business Goals




Session #402

Example Questions

As a result of taking this class I think the satisfaction level of my customers will increase. (Pre, Post, 6-Month Student Surveys and Instructor Surveys)

I have the knowledge and skills of (Product) that I need to do my job. (Pre, Post, 6-Month Student Surveys)

If you have a success story to share, what is the amount of revenue you think was generated as a result of taking this class?(6-Month Student Survey)

Session #402

How We Gather Data

Administer surveys at multiple points in time Use focus groups to gather additional data on observed trends and issues




Session #402

Surveys

Pre-ClassPost-Class6-MonthDevelopment Instructor

Delivery Instructor Manager

WithdrawalNon-CompletionAcademy Live Post-Session

Academy Live PresenterSelf-Study

Session #402

Survey Tool




Session #402

Other Data SourcesFor data on consumption/reach:•Microsoft personnel databases to track reach by job roles •Content repository for number of hits on each offering (ex: facilitated class materials, Academy Live audio files)

Pilot data for new programs:•Surveys•Call-downs to participants

Student bios on the virtual classroom•Customer issues being faced

Session #402

Other Data Sources

We stay in touch with our constituents•“Ear to the ground”•We invite people to send feedback whenever and however•We collect all thanks, kudos, success stories, ideas, suggestions, recommendations

We ACT on input if supported by other dataWe respond to all input with THANKS!




Session #402

How We Measure Learning

Level I – Reaction•Post Class/Post Session Surveys•Six Month Survey•Withdrawal/Non-Complete Surveys

Level II – Learning•End-of-Class Problem (Job Deliverable)

Level III – Transfer•Six Month Survey•Manager Survey•Focus Groups

Some ROI through six month survey

Session #402

QuestionsAbout Gathering Data




Session #402

Polling Slide #3What do you do with eval data you gather?

Session #402

How We Analyze DataTrack and compare responses from all audiences

Observe trending over time

Mine data on field input regularly

Note successes and areas needing improvement•Follow up with focus group and/or Advisory Board

Academy ClassesImprove Customer Sat

Students

Managers

Instructors




Session #402

Polling Slide #4How often do you report evaluation data to your business stakeholders?

Session #402

How We Report DataAnnual Summary ReportBiannual Reports and Presentations•Field Managers•Academy Program Management•Business Groups•Advisory Board

Our WebsiteMonthly communiqués to the field and management




Session #402

Reporting Impact

"After the class I was able to discuss roaming profile improvements in Windows Vista with my customer. It was the right discussion at the right time. They are now seriously looking at accelerating their Vista deployment by a year because of the TCO savings they can achieve."

"I had three customer face to face meetings after the class. Two of the customers were convinced and generated about 80K SQL in sales. I’ve also had more impact in marketing events with Partners. I must say, SQL 303 is a great course."

Session #402

QuestionsAbout Analyzing and Reporting




Session #402

How We Use The Data

Generate support and good will of the field and management (our internal customers)

Ensure continual funding and support growth

Make program adjustments and enhancements

Promote the culture of ongoing assessment and response

Session #402

Success IndicatorsFunding tripled over 3 years•Management sees the Academy as a success story for Microsoft

Scaling this year to more internal audiences and to partnersProgram satisfaction data from all constituentsHappy repeat customers•86% of worldwide audience consumes multiple offerings per year

New customers •Over 30% of students are not from the target audience




Session #402

Our Recommendations

Build evaluation into everything you doPlan what data you need to collect to prove the worth of your program−up frontCollect business and program data along with the learning dataDon’t just collect data, use it:•Build support for your program•Monitor and adjust your program

Let your constituents know that you hear them and take action on their concerns

Session #402

Questions

olf1206 402 livingston

Documents