insights from customer needs and satisfaction at ucf's feeds
TRANSCRIPT
Insights from customer needs and satisfaction at UCF’s FEEDS
programResults from the Voice of the Customer component of a Six Sigma methodology
project.
Roberto Champney1 Alain Gaumier2 Naomi Morris1
1University of Central Florida 2Optimum Consulting, Inc.
Objectives
• Project Background and Description
• Six-Sigma methodology overview
• Voice of the Customer (VOC)
• Results and Insights From VOC
• Recommendations
Project Background
• FEEDS- Florida Engineering Education Delivery System.
• Created in 1982 by the Florida State Legislature
• The system has evolved from a tape system in 1982 to a CD and web-based education option in its current form.
• The University of Central Florida (UCF) is one of 12 universities in the state that participates in the program, and holds the largest enrollment, 38% (2002-2003).
Project Background
• ASQ (American Society For Quality) Community Good Works Project.
– Improve communities through the use of quality tools and technologies.
– Provides skills transfer to non-traditional quality practitioners.
• Project Goals1. Document the fall 2003 version of the FEEDS
system.
2. Identify Priority Process Improvements.
DefineDefine
ImproveImprove
ControlControl
MeasureMeasure
AnalyzeAnalyze
Six-Sigma methodology overview
DMAIC
Classical View of Performance Practical Meaning
of “99% Good”:20,000 lost articles of
mail per hour15 minutes of unsafe
drinking water each day5,000 incorrect surgical
operations per week2 short or long landings
at most major airports each day
200,000 wrong drug prescriptions each year
No electricity for almost 7 hours each month
3 Sigma
4 Sigma
6 Sigma
93.32%
99.38%
99.99999%
HistoricalStandard
CurrentStandard
NewStandard
CapabilityLong-Term
YieldStandard
Voice of the Customer (VOC)
• Technically there are numerous ways to enhance the system, yet there has been little attempt to focus this enhancement to represent the needs of its users. – Identify customers and their needs.
– Collect and analyze reactive data.
– Consider proactive approaches.
– Convert collected data into customer needs.
– Sort out the most important attributes “ctq”.
– Obtain specifications for the critical-to-quality characteristics.
• Conducted a student focus group.
• Conducted several interviews with FEEDS students to:– Capture major needs categories and issues.– Further understand FEEDS users.
• Developed a survey concerning the satisfaction and needs of FEEDS student users.
• End of Semester FEEDS Student Online Survey.– Set adequate performance levels.– Correlations to identify key drivers.– Align opportunities for improvement with FEEDS UCF priorities.
Voice of the Customer (VOC)
The Surveys
• Two surveys over a two semester period
• Interview comments were used to create the survey questions:– 30 and 39 questions
– Some demographics.
• gender, study program, program level.
– Three types of questions (5-point likert scale and open ended).
• Satisfaction with current attributes of the system.
• Usefulness and Importance of particular attributes of the system (hypothetical and current).
• Comments on particular issues with the system.
The Survey (Sample Questions)
• Overall how satisfied are you with the FEEDS system?
• Overall how effective are FEEDS courses compared to live courses?
• How satisfied are you with the quality of the video stream (vision and audio) on your FEEDS courses?
– Audio
– Image
• How satisfied are you with the communication with FEEDS professors?
The Survey (Validating Sample)
Percent (%)
FALL '03 SPRING '04
Aerospace Eng 3.83 4.73
Environmental Eng 3.24 4.05
Industrial Eng 17.70 20.95
Mechanical Eng 9.14 4.05
Eng Technology 18.29 21.62
Civil Eng 8.85 8.78
Computer Eng 10.32 12.84
Computer Science 7.96 6.08
Electrical Eng 12.09 13.51
Other 8.55 3.38
Total 100.00 100.00
•Response Rate: 338 (1st survey) and (149 2nd Survey)
•Mass email, and site banner.
•Percentages represent actual FEEDS student body distribution
Insights From VOC
Survey Questions
Satisfaction Effectiveness
Fall '03 Spng '04 Fall '03 Spng '04
FEEDS Effectiveness 0.690 0.539
FEEDS Satisfaction 0.690 0.539
Communication With Professor / Satisfaction 0.649 0.444 0.522 0.320
Image Quality Satisfaction 0.511 0.381 0.365 0.366
Audio Quality Satisfaction 0.439 0.434 0.330 0.353
Response Time From Professor -0.251 -0.166 -0.126 -0.080
Response Time From Professor / Satisfaction 0.334 0.187
Notes Before Class satisfaction 0.206 0.012
How Consistent is Delivery of Course Material 0.258 0.247
Correlations are significant at the 0.01 level (2-tailed).
Insights From VOC Overall Satisfaction Comments69% and 54% of respondents gave comments
• First Survey
•30 % positive comments
•41 % negative comments
• Second Survey
• 39% positive comments
• 19% negative comments
Note: one respondent may have contributed to more than one category of comments.
- Other11%
No Comment29%
Negative Remarks
4%
Positive Remarks
6%
- Professor Issues12%
+ Convenience24%
- A/V negative14%
No Comment42%
Convenience21%Positive
Remarks3%
Other3%
A/V positive2%
A/V negative4%
Professor Issues -4%
WebCT +3%
Professor Issues +3%FEEDS Site -
3%
NegativeRemarks
3%
Stream Availability +3%
Class Materials +3%
Assignment Confirmation -
3%
Insights From VOC
• On Spring ’04 UCF FEEDS upgraded their stream format.
Fall '03 Spring '04
Mean SD Mean SD
FEEDS Effectiveness* 3.80 1.11 4.01 1.03
FEEDS Satisfaction 3.97 1.05 4.14 0.85
Image Quality 3.45 1.26 3.61 1.15
Sound Quality 3.63 1.17 3.80 1.13
How Consistent is Delivery of Course Material 3.39 1.16
Notes before class satisfaction 3.49 1.18
Communication with professor, satisfaction 3.65 1.19 3.65 1.18
Response time from professor, satisfaction 3.58 1.32
Insights From VOC
Difference
Mean
Statistic Std. Error
SPRING '04 Video Audio -0.70 0.11
Video Image -0.86 0.11
Assignment Confirmation -0.98 0.14
Course Material Consistency -0.07 0.06
Notes Before Class -1.04 0.13
Availability of Stream -0.09 0.11
• Prioritize future efforts using differences between ‘received’ and ‘desired’
• See SERVQUAL survey method.
Delivery of Service Quality," Journal of Marketing, April 1988, pp. 35-48. Parasuraman, Zeithaml and Berry, "SERVQUAL: A Multiple-Item Scale for Measuring Customer Perceptions of Service Quality," Journal of Retailing, Spring 1988, pp. 12-40
Insights From VOC• How often are you able to attend a live class?
– 29% to 39% always (4 & 5 scores)
• For what reasons are you not able to ALWAYS attend a live class?
Count
Personal Schedule 81
Distance 54
Convenience 44
Other 40
Class Schedule 23
Work 20
Count
19 4 0 7 30
15 3 1 7 26
15 4 8 3 30
32 6 7 16 61
81 17 16 33 147
1-25%
26-50%
51-75%
76-100%
CourseworkCompleted
Total
SPRING '041-25% 26-50% 51-75% 76-100%
Coursework via FEEDS
Total
Insights From VOC
• Please indicate all the mediums with which you communicate with FEEDS professors? (Check the two most frequent)
Percent
Email 91.2
Person 49.3
WebCt 39.9
Phone 31.8
Other 2.0
Insights From VOC• How long does it takes on average to get a
response from your professor?• How satisfied are you with how long it takes to
get a response from your professor? – 55 % satisfied (4&5 score)
Insights From VOC
Where do you regularly watch the FEEDS stream or CD? (Check the two most frequent)
– Home 126– Work 47– Library 17– Computer Lab 18– Other 8 (Beach, Starbucks, Outdoors)
Notes access preference Percent
Download 67.5
Email 29.4
Buy Bookstore 2.6
Other 0.5
In the event that you could have access to the class notes before the class session, what would be the best way to gain access to them?
Insights From VOC• How often have you experienced missing material
from a FEEDS stream?
• In the event that you have experienced missing material from a FEEDS stream, what part was missing?
Yes52%
No48%
1-Never to 5-Always
-3-15%
-5-1%
-4-8%
-1-44%
-2-32%
Start16.3
Middle8.8
End31.3
None43.6
VOC Instrument Recommendations
• ~5 interviews will identify most of the problems.
• Don’t get carried away with questions, you’ll have to make sense of them later.
• Everything is important or useful. – Be careful with these types of questions. – Ex. Teacher assistant meetings:
• Usefulness score: 4.07• Attend score: 2.79
VOC Instrument Recommendations
• Open ended comments are most valuable, but very time consuming.– Use at least two persons for categorization.
• Provide enough space for comments.– 255 characters is not enough
• Use online surveys, don’t send attachment surveys
Insight Summary• Identify Student Strata
– 100% remote students.• Miss student interaction.• Miss out on after class talk.• Value recording of ‘pre-class’ and ‘break’ periods
– Convenience students• Register for FEEDS but attend live classes. • Use FEEDS as backup or study aid.
– Gender differences• Females more than males value the interaction with other
students and professors (per comments).
– Study Program differences• Teacher Assistant meetings, and problem solving groups.
Insight Summary
• User satisfaction and system effectiveness are correlated.– Effectiveness is driving satisfaction.
• Communication with professors is critical to satisfaction and effectiveness.
• ~1/3 of students regularly attend live classes• Email and in-person communication is used
mostly for communication between FEEDS students and professors.
Insight Summary
• Provide access to class notes before class, when possible.
• Consider scanning professor hand notes if possible.
• Chalk/White Board is not appropriate for this medium.
• Live students should see the same image as Remote students.– They’ll become the ‘directors’ of the class production.
Insight Summary
• Standardize equipment and training. – Encourage (enforce) professor training.
• Consider standardizing method of delivery and procedures (i.e. all use WebCT, all email assignments).– Use one location for files, content, or materials
access. – Need commitment from professors.
Insight Summary
• Computer demos are particularly difficult to follow (unable to read buttons and options).
• Most of students’ problems with the ‘stream’ are with accessing it due to technical difficulties.
• Provide content online, consider avoiding mailings (they consume resources).
Insight Summary
• Other online services or university sections confused as part of FEEDS (i.e. WebCT).
• Apply VOC methodology with other stakeholders– Professors, Administrators, etc.
Questions ?