is sig meeting, april 8, 2008 understanding technology project risks and predicting project...
TRANSCRIPT
IS SIG Meeting, April 8, 2008
Understanding Technology Project Risks and Predicting Project
Performance
Dr. Andrew GeminoSimon Fraser University
IS SIG Meeting, April 8, 2008 2
Agenda
• Introduction• Part 1: IT Project Performance• Part 2: Elements of Performance
• Discussion: How does your company evaluate performance
• Part 3: Predicting IT Project Performance• Part 4: PM Practice and Performance
IS SIG Meeting, April 8, 2008 3
Motivation for My Research
• Application of Information Technology plays an important role in Canada’s capacity to innovate• The Info. And Comm. Technology Sector
• Contributes $61 bil to Canadian GDP (in 1997 constant dollars)• Comprises 5.8% of Canadian GDP• Employs over 560,000 Canadians• Responsible for 38% of all private sector R&Dhttp://strategis.ic.gc.ca/epic/site/ict-tic.nsf/en/h_it06143e.html
• Our focus is to advance our ability to understand and influence IT project performance.
IS SIG Meeting, April 8, 2008 4
The Team
• Blaize Horner Reich, Simon Fraser University• Chris Sauer, Oxford University, UK• Andrew Gemino, Simon Fraser University
• Website: www.PMPerspectives.org
• Funding: • Social Sciences and Humanities Research Council
(SSHRC) • Initiative for the New Economy (2003-2006)• Research Grant funding (2007-2010)• INE Communication Grant
• Natural Sciences and Engineering Research Council (NSERC)
• Research Grant Funding
IS SIG Meeting, April 8, 2008 5
Some Recent Output
• Sauer, C., Gemino, A, and Reich, B.H. “Managing Projects for Success: The Impact of Size and Volatility on IT Project Performance”, Communications of the ACM, 60:11, Nov. 2007, pp. 79-84.
• Gemino, A., Horner-Reich, B. and Sauer, C. "A Temporal Model for IT Project Management", Journal of MIS, Winter 2007–8, Vol. 24, No. 3, pp. 9–44 (awaiting publication).
• Gemino, A., Reich, B. and Sauer, C. "Beyond Chaos: Examining IT Project Performance" Proceedings of the 2nd Annual International Research Workshop for the Special Interest Group for Information Technology Project Management (SIGITProjMgmt), Montreal, Canada, Dec. 8, 2007.
• Reich, B.H., and Sauer, C. “Myths About Information Technology Project Performance”, Proceedings of the Administrative Sciences Association of Canada (ASAC) 2007, Ottawa, Canada, June 2-5, 2007.
• Gemino, A., Reich, B.H. “Estimating Risk in Information Technology Projects”, Proceeding if the Americas Conference on Information Systems, Keystone, Colorado, Aug. 9-12, 2007.
5
IS SIG Meeting, April 8, 2008 6
PART 1: IT Project Performance
IS SIG Meeting, April 8, 2008 7
The Standish Report
• The Chaos Report (1994)• Many have heard the numbers
• On time, budget and scope – 16%• Challenged 53%• Abandoned 31%
• Average Performance• Cost overruns 189%• Time overruns 222%• % of Original Specs 61%
• http://www.standishgroup.com/sample_research/chaos_1994_1.php
IS SIG Meeting, April 8, 2008 8
Standish Surveys Show Improvement
0%
10%
20%
30%
40%
50%
60%
Succeeded Failed Challenged
1994
1996
1998
2000
2002
2004
2006
(complied from press releases from www.standishgroup.com )(complied from press releases from www.standishgroup.com )
IS SIG Meeting, April 8, 2008 99
Some Questions about Standish Figures
• “Most such academic papers and guru reports cite the same source for their crisis concern—a study published by the Standish Group more than a decade ago”
Robert Glass, CACM, 2006
• Reasons to doubt the Standish figures• Inherent bias due to perspective on “failure” • Tough test for “success” – all targets at 100%• Sampling method is very unclear
• what projects are in the sample?
IS SIG Meeting, April 8, 2008 10
Survey Research
Study 1: Computer Weekly UK
• Web based survey of• Readers and PM’s
• Asked IT project managers about their last project (completed or abandoned)
• 421 full responses• Average 17 years industry
experience• Average 9 years as project
manager
Study 2: PMI Chapters in Ohio
• Web based survey• 3 PMI chapters in Ohio
• Asked IT project managers about their last project (completed or abandoned)
• 194 full responses• Average age, 43• Average years experience,
15• Average PM training: 34
days
10
IS SIG Meeting, April 8, 2008 11
What we found
• 65% of projects in the UK and 66% US samples had good performance• Delivered within a reasonab;le contingency (approx
7%) of ALL targets. • Compare with 63% challenged or abandoned in
Standish
• What’s the difference? • Experienced Project Managers• Data collection
• We asked for actual variances from original goal
• Cluster Analysis• Didn’t use hard line (99% is not challenged)• “let the data speak for themselves”
11
IS SIG Meeting, April 8, 2008 12
IT Project Types: UK Study
PerformanceVariance
Type 1:
Abandoned
n=39
Type 2:
BudgetChallenged
n=22
Type 3:
ScheduleChallenge
dn=76
Type 4:
Good
n=250
Type 5:
Stars
n=31
Performance Variances (Actual as % of Originally Planned) – 100%
Schedule N/A +18% +46% +2% +1%
Budget N/A +40% +16% +7% -24%
Scope N/A -12% -16% -7% +15%
IS SIG Meeting, April 8, 2008 13
UK - Project Characteristics
Size Characteristi
c
Type 1:Abandon
Type 2:Budget
Challeng
Type 3:ScheduleChalleng
Type 4:Good
Type 5:Star
Budget(Median, in ₤
000’s)₤ 1,000 ₤ 625 ₤ 500 ₤ 450 ₤ 2,000
Effort(Average
Person Months)
798 557 212 89 170
Duration (Average
Elapsed Time in Months)
17.4 20.0 13.0 11.2 15.3
Team Size(Effort/
Duration) 35.7 17.7 12.9 7.3 9.8
IS SIG Meeting, April 8, 2008 14
IT Project Types: US Study
Performance Variance
Type 1:Abandoned
Projectsn=16
Type 2:
Challengedn=13
Type 3:Schedule
Challengedn=36
Type 4:
Goodn=87
Type 5:
Starsn=32
Performance Variances (Actual as % of Originally Planned – 100%)
Schedule N/A +41% +107% +4% +1%
Budget N/A +25% +43% +3% -12%
Scope N/A -12% -4% -9% +3%
IS SIG Meeting, April 8, 2008 15
US - Project Characteristics
Size Characteristi
c
Type 1:Abandon
Type 2:Budget
Challeng
Type 3:ScheduleChalleng
Type 4:Good
Type 5:Star
Budget(Median, in
US $ 000’s)$2,000 $700 $1,042 $670 $600
Effort(Average
Person Months)
N/A 126.4 163.2 255.1 66.5
Duration (Average
Elapsed Time in Months)
N/A 16.7 16.5 15.8 9.5
Team Size(Effort/
Duration) N/A 6 11.0 16.4 6.0
IS SIG Meeting, April 8, 2008 16
Surprises:Some IT Projects Exceed
Expectations1. The IT Performance story is not all bad
1. 2/3rds of projects are performing well2. Some IT projects exceed expectations
1. 7% of UK2. 17% of US projects
• These projects that exceed expectations are not mentioned in Standish Group Reports • We found them because we did not constrain
“success” in the data collection
16
IS SIG Meeting, April 8, 200817
IT Project Performance
The Impact of Size and Volatility
IS SIG Meeting, April 8, 2008 18
Risk Associated with Project Size
1(a) Risk of Underperformance due to Effort
25%31%
36% 36% 38%
50%
77%
100%
0%
33%
67%
100%
25 or less 25-50 50-100 100-200 200-500 500-1000 1000+ 2400+
Effort (person months)
IS SIG Meeting, April 8, 2008 19
Risk Associated with Project Size
1(b) Risk of Underperformance due to Duration
24%21%
34%
25%
36%
48%
0%
33%
66%
< 3 months 3-6 months 6-9 months 9-12 months 12-18months
18+ Months
Duration (months)
IS SIG Meeting, April 8, 2008 20
Risk Associated with Project Size
1(c) Risk of Underperformance due to Team Size
28%
38%
26%
33%
58%
0%
33%
67%
<5 FTE 5-10 FTE 10-15 FTE 15-20 FTE >20 FTE
Team Size (Effort / Duration)
IS SIG Meeting, April 8, 2008 21
Risk Associated with Volatility
2(a) Risk of Underperformance due to Governance Volatility
22%33%
60%53%
74%82%
0%
33%
67%
100%
0 changes 1 change 2 changes 3 changes 4 changes 5+changes
Governance Volatility (changes in PM or Sponsor)
IS SIG Meeting, April 8, 2008 22
Risk Associated with Volatility
2(b) Risk of Underperformance Due to Project Target Volatility
11%
25%
34%
57%
0%
33%
67%
< 3 changes 3-6 changes 6-9 changes 9+ changes
Project Target Volatility (changes to project targets)
IS SIG Meeting, April 8, 2008 23
Recommendations from UK Study
What level of performance can organizations expect?
Guideline From experienced project managers, 2 out of 3 projects should be implemented within small variances of budget, scope and schedule.
Rationale Two thirds of our sample were better-performers. On average these projects come within 7% of all targets.
IS SIG Meeting, April 8, 2008 24
Recommendations from UK Study
What levels of risk are normal for IT projects?
Guideline For any project, the baseline risk of underperforming is 25%.
Rationale Even the smallest and shortest projects in the sample had on average a 25% chance of underperforming.
Guideline To keep projects below a 50% risk of underperforming, limit them to -less than 500 person months of effort.-under 18 months duration and -team size less than 20
Rationale Size is an issue because large teams increase coordination costs; long projects increase the risk of external change and lose momentum.
The best single measure of risk from size has proved to be effort. However, risk cannot be managed by reducing one dimension of size at the expense of others.
IS SIG Meeting, April 8, 2008 25
Recommendations from UK Study
What can organizations do to reduce IT project risk?
Guideline Develop and retain or hire experienced project managers
Rationale Experienced project managers achieved better results than have been previously reported.
Guideline Keep projects short and manageable without being obsessively minimalist
Rationale We found little difference in risk between 3 and 12 month projects.
Guideline Plan and budget for continuity in leadership during the project. This may involve incentives for the project manager or understudies for key roles.
Rationale Changes in key personnel occur for personal, performance and strategic reasons. A PM change costs the project 15% of targets, so resources put towards avoiding this risk are a good investment.
Guideline Change targets only for strategic reasons (e.g. changes to increase business value of the project)
Rationale In the aggregate, changes in targets are disruptive and reduce performance.
IS SIG Meeting, April 8, 2008 26
PART 2: Group Discussion Elements of Performance: Let’s hear
from you –
How does your company measure Project Performance?
IS SIG Meeting, April 8, 2008 27
Are Budget and Schedule variances good measures of IT Project
Performance?
27
http://www.codinghorror.com/blog/images/software_engineering_explained.gif
IS SIG Meeting, April 8, 2008 28
How should performance be measured?
• Two basic outcomes in project• Process Outcomes
• On time• On budget• On specs
• Product Outcomes• Value delivered• Benefits• Quality
PM’s are often measured here
But are expected to deliver here
IS SIG Meeting, April 8, 2008 29
Question
• Does your company consider both product and process when considering evaluating performance?
• Should this be done? And if so, how?
IS SIG Meeting, April 8, 2008 30
PART 3: Predicting IT Project Performance
What factors determine IT Project Performance?
IS SIG Meeting, April 8, 2008 31
What Affects Performance
• Risk Factors and Resources• Initial (A-priori) Factors
• Knowledge resources Team. PM, Sponsor, Clients
• Structural Factors Technical complexity, budget, duration, effort
• Emergent Factors• Organizational Resources
Top Management Support, User participation• Volatility
Governance changes, Target Changes, Environment changes
IS SIG Meeting, April 8, 2008 32
What Affects Performance
• Project Management Practice• Expertise Coordination
• Who are knowledge leaders• How can they be accessed
• Horizontal Communication• Across team and clients
• PM Methods and Tools• Traditional PM
IS SIG Meeting, April 8, 2008 33
US Study: Results
Knowledge Resources
Structural Risk
(Size & Tech Complexity)
OrganizationalSupport
Rsources
Project Management
Practices
VolatilityRIsk
ProjectProcess
Performance
ProjectProduct
Performance
0.292**
0.290**
0.502**
0.298**
0.359**
R2 = 0.361
R2 = 0.269
R2 = 0.076
R2 = 0.219
R2 = 0.385
-0.321**
0.396**
0.154*
-0.209*
A-Priori Emergent Performance
IS SIG Meeting, April 8, 2008 34
Results
• With this model we can explain• Approximately 40% of the variance in Process
performance• Approximately 22% of the variance in Product
performance
• We can also show the strength of the relationships between risk and resource factors associated with IT Projects. • Strength and direction is given by the number
associated with each line.
IS SIG Meeting, April 8, 2008 35
Results
• Two Broad Forces Acting on Projects 1. Forces of Evil
• Structural risk is strongly related to volatility and volatility is negatively related to process performance
• Large projects often have a bumpy ride and are challenged in regards to being on time and budget.
35
Structural Risk
(Size & Tech Complexity)
VolatilityRIsk
ProjectProcess
Performance
0.502**
R2 = 0.361
R2 = 0.385
-0.321**
IS SIG Meeting, April 8, 2008 36
Results
Knowledge Resources
OrganizationalSupport
Rsources
Project Management
Practices
ProjectProcess
Performance
ProjectProduct
Performance
0.292**
0.290**
0.298**
0.359**
R2 = 0.269
R2 = 0.076
R2 = 0.219
R2 = 0.385
0.396**
0.154*
2. Forces of Good• Increased knowledge is
related to higher organizational support and higher level of PM Practices.
• Increased PM Practice is related to better process and PRODUCT performance
IS SIG Meeting, April 8, 2008 37
Summary
• IT Project performance is a critical issue in Canada’s ability to innovate.
• We have a good idea of the factors that affect
process performance. More work needs to be done. • It is important to consider initial factors as well as
emergent factors when considering performance.
• We do not yet have a good understanding of how these factors affect product performance• More work required
IS SIG Meeting, April 8, 2008 38
Questions?
If you are interested in more information please visit: www.PMPerspectives.org