acosm2003 benchmarking lessons v1.0 printversion. · pdf file“benchmarking is the...
TRANSCRIPT
By : Ewa WasylkowskiSenior Consultant
Total Metrics
Benchmarking –Lessons You Don’t Want to
Learn the Hard Way
ACOSM 2003Sydney, Australia
Definitions“Benchmarking is the process of continuouslycomparing and measuring an organisation withbusiness leaders anywhere in the world to gaininformation which will help the organisation takeaction to improve its performance.”
American Productivity and Quality Centre Benchmarking Management Guide
Cambridge, Mass., 1993, Productivity Press
“Benchmarking is the process of determining ametric baseline for an organisational or functionalunit for the purpose of comparison”
Total Metrics – Metrics Glossary of Terms
Benchmarking Process Model
1. Determine BenchmarkPurpose
2.Determine BenchmarkMethod & Frequency
7. Filter, Map & ValidateData
12. Monitor Progress
PLAN
3. Select Benchmarker
4.Identify QuantitativeMeasures & QualitativeInformation
5. Identify Project Set
6. IdentifyBenchmarking Peers
8. Normalise AgainstPeer Data
9. Derive BenchmarkPerformance Indicators
10.Determine Gap
11. Benchmark Report -Recommendations
DO CHECK
13. Review Result
14 Resolve Disputes
15. RecalibrateBenchmarks
ACT
1. Determine BenchmarkPurpose
Benchmarking PurposesEstablish market position amongpeersDemonstrate competitiveness &continuous improvement in pricing &service levelsIdentify ‘Best Practice’Identify process improvementopportunitiesSet competitive range for metricsbaselineDecision making re outsourcing
Benchmarking MethodYear-on-Year– On-going annual reporting of
performance measures– Assesses trends - rate of improvement– Internal Benchmarking– Require 2-4 years data before suitable
base for comparison is availablePoint in Time– Assesses performance against peers– External Benchmarking– Must ensure benchmark period is
representative of past and/or futureperiods
IT Benchmarking AreaApplications DevelopmentMaintenance & SupportNetworkData CentreHelp DeskDesktop/LAN ServerCharge Back Practices
Benchmarking Lessons -Planning
There are no ISO (or other) standardsfor IT benchmarking.
Benchmarking is an imprecise tool as itis not possible to find directlycomparable:
OrganisationsContractsBenchmark Periods
The method and underlyingassumptions must be transparent andauditable.
Criteria for SelectingBenchmarker
Relevant Experience– Within Region eg Australia, Asia/Pacific– Within Industry Sector– With Organisations of similar size
Nature and Extent of BenchmarkDatabase for selected metrics– Attributes & Measures Collected– Segmentation by Industry– Australian/regional data– Projects by project type– Currency of data– Data integrity and validation– Comparable Peers !!!!!!
Criteria for SelectingBenchmarker (cont.)Methodology Proposed– Extent to which a customised
solution is possible (if required)– Sampling techniques (if proposed)– Sample findings, reports &
conclusionsLogistics– Required resource commitments– Ability to meet project timetable
Cost
Metrics/PerformanceIndicators
Applications Development– Productivity
• Hours per Function Point delivered• Function Points delivered per Full Time
Equivalent (FTE)– Cost Effectiveness
• $ Cost per Function Point delivered
Maintenance & Support– Productivity
• Function Points maintained per FTE– Cost Effectiveness
• $ Cost per 1000 FPs maintained.
Benchmarking Lessons– Planning cont.
Effective benchmarking begins atthe contract stage.Capabilities and costs ofbenchmarking organisations varywidely.It is naïve to expect a single resultthat factors in all the complexitiesof the systems development &business environment.
Identify Project SetBenchmark period determinescandidate projects.Sample set vs entire populationFor sample set:– Sampling techniques
• Systematic• Stratified• Cluster sampling
– Sample set sizeCategorise projects by type
A & D Project/Activity TypesDevelopment
New Custom Built
EnhancementCustom Built
Quarterly Release
Package Customisation
Pre-DevelopmentEstimates
Feasibility Studies
Business Cases
Cancelled, Withdrawn,On HoldPackages ‘Vanilla’Non Development
PM / Consulting only
DB Conversion & Tuning
Project Deployment/Impl.
Function PointGenerating
Non - FunctionPoint Generating
Productivity Metrics
Hrs per FP delivered
FPs delivered per FTE
$’s per FP delivered
Profile Metrics
% Effort Expended
% Cost Expended
(by category)
M & S Project/Activity TypesMaintenance & Support
Production Support
Software/Hardware Upgrades
Capacity Planning &Management
Corrective Maintenance
Perfective Maintenance
Decommissioning
Production Control &Scheduling
Application Security/AccessControl
Daily Operations Support
Support of ProductionEnvironment
Table Updates
Help Desk Support
Non - FunctionPoint Generating
M & S Metrics
Software Portfolio Size
FP’s maintained perFTE
$ Cost per 1000 FP’sMaintained
Identify Project Set
Start Benchmark Period
End Benchmark Period
Clearly “In” Projects
Effort & Cost Invoiced
RepresentativeBenchmark Year ?
Must establish whetherbenchmark period project set istypical of past and indicative offutureChanges in proportion of effortexpended may arise from– Planned changes in project type
mix– Altered demands of project types– Altered productivity & efficiency
Identify BenchmarkingPeers
Selecting benchmarking peers &Benchmarker are synonymous.– A valid benchmark must be
based upon comparable dataBe pro-active in selectingbenchmarking peers.– Profile your organisation and/or
contract.Establish representative set ofpeers
Benchmarking Lessons– Planning cont.
Inclusion/exclusion rules forprojects that span benchmarkperiod boundaries must be clearlydefined and equitable.
Data Filtering, Mapping & Validation
Measure STANDARDS MEASUREMENT METHOD
DATA VALIDATION
EFFORT
Project Chart of Accounts Project Phases Project Activities FTE Definition
Time Recording MethodResource Levels Uncollected Work EffortNormalised Effort Re-work Effort
Internal Validation Benchmark Validation ?
COST
Project Cost Model Direct Labour Cost only?
Actual vs Invoiced $ Spend Calculation
Financial Audit Benchmark Validation ?
FUNCTIONAL SIZE
FSM Standard Adjusted/Unadjusted FPs
Measured Result Default Counts Derived From PDR Approximated “Back-fired”
Internal Review External Audit Benchmark Validation ?
Data NormalisationFinancial Factors– Cost of Living– CPI Adjustments– Exchange Rates
Industry Sector and SizeOperating Environment– Differing Awards– Regulatory Constraints
Scope of ServicesTechnology PlatformsData Currency
Data Normalisationcont.
Service LevelsApplication Type & businessenvironmentProject MixSoftware Delivery Methods– Custom built– COTS
Regional MarketsOther contract unique
Benchmarking Lessons– Analysis
Benchmarking primarily involvesdata mapping not data collection.Data submitted must be uniformlyvalidated to ensure comparability.Effort & Cost data must becorrectly mapped to FPs delivered.The data normalisation methodmust be transparent andauditable.
Impact FactorsIdentify factors that impactbenchmark results which arebeyond the control of supplier:– Industry Sector– Technology Environment– Project Type Mix– Rework due to requirements
instability– Non-standard systems software– Other
Review ResultsReview by benchmarker beforeissuing draft report.Report presentationClient & Supplier ReviewUpdate based upon feedbackApprovalFinal Report
Resolve DisputesDetail dispute resolutionmechanism in contract.– Report review feedback– Internal dispute resolution
• Tie-breaker mechanism
– Arbitration - Independent person• Formal notice in writing• Benchmarker must co-operate• Benchmarker right of response
– Legal resolution
Benchmarking Lessons– Review
At outset agree process by which reportis issued, reviewed & approved.
In contract include provision forappointment of ‘Independent Person”.
A “dispute” may be simply the failureof two reasonable parties to reachagreement upon a possible range ofoutcomes.
Recalibrate BenchmarksBenchmarker to specifycompetitive rangeIdentification of “New Metrics”Adjustment of Contract Metricsto New MetricsRemoval of work from contractscope
Benchmarking Lessons– Act
Remember the primary purpose ofbenchmarking is process improvement.
Agree next benchmark period andbenchmarking process improvements.
Benchmarking is not a replacement forregular performance reporting.
Summary
Benchmarking should strengthenrather than divide the relationshipbetween client and supplier.
All parties should understand andacknowledge the limitations of thebenchmarking process.
The true benefits of benchmarking liein the discovery of processimprovement opportunities.
Thank Youand
Good Luck with yourBenchmarking !
Total Metrics Pty LtdSuite 1 / 667 Burke RoadCamberwell Victoria 3124
Phone (03) 9882 7611Fax (03) 9882 7633Email [email protected]