cots integration estimation-erp.pdf
TRANSCRIPT
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Headquarters U.S. Air Force
COTS Integration Estimation: Enterprise Resource Planning Systems
1
Dr. Wilson Rosa
James Bilbro (Cirrus/JBCI)
USC CSSE Annual Research Review 2012
March 6, 2012
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Outline
ERP Overview
Data Collection and Analysis
Estimating Models
The Way Forward
2
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Headquarters U.S. Air Force
3
ERP Overview
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
What is ERP?
4
Enterprise Resource Planning (ERP) systems are typically
Commercial-Off-The-Shelf (COTS) software applications
that integrate an organization’s core business functions
around a unified data base.
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
How complex is an ERP
Implementation?
5
While an ERP system can fall under the category of
Major Automated Information System (MAIS) it is NOT
simply just another IT system!
Treating an ERP system as though it were just
another IT system is guaranteed to result in significant
cost and schedule overruns!
Implementing an ERP system involves a complex mixture of
activities that are impacted by:
business processes
organizational makeup and culture
hardware, and software development and modification
And much, much more ---
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
How complex is an ERP Implementation?
6
23
A Spectrum of COTS-Based Systems
Multiple products from multiple suppliers integrated to
collectively provide system functionality
• probably more flexible in supporting end user/business processes
• project maintained• integration, engineering focus
products/parts are “black boxes”COTS, NDI, legacy
COTS-Intensive
SystemsCOTS-Solution
Systems
One substantial product (suite) tailored to provide significant
system functionality
• generic solutions; tightly coupled to end user/business processes
• vendor maintained • tailoring, parameterization focus
Slide Presented at SERC COTS/NDI T&E Workshop, 15 January 1999 by
Software Engineering Institute (SEI)
ERPs are
COTS-Solution
Systems….
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
How is ERP implemented?
Business processes are automated via an integrated
COTS software application:
7
Integration is typically
done by a 3rd Party Vendor
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
What is the problem?
8
Exceeds Schedule 61.10%
Exceeds Budget 74.10%
2010 Results of Industry Survey of 185 Implementers
Industry as well as DoD has significant cost and
schedule problems in implementing ERP systems!
All Major DoD ERP Programs
Have Exceeded Original Cost
and Schedule Estimates by
more than 30%!
As of Dec. 2009, DoD had
invested $5.8B in their ERP
programs!
DoD Industry-wide
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Reasons for Cost Growth
AFCAA Root Cause Analysis
1. Estimation: A lack of understanding of the new technology
and business environment led to the use of inadequate cost
models and dubious estimating methods
2. Schedule: limited budgets have forced decision makers to
extend the period of performance of “Level of Effort” related
tasks – Civilian, Contractor, and Military FTEs
3. Engineering: Inexperience with Oracle/SAP Customization
has led to underestimation of requirements. Difficulty
changing business processes to match ERP processes
4. Quantity: war-fighter need has led some program offices to
reassess user and implementation requirements
9
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
System Developer
38%
Deployment Hardware and
Software7%
Program Office
9%
Site Activation, User Training,
Data Migration18%
Interim Logistics Support
28%
Impact of Schedule Overruns? Extends “Standing Army” Costs!!!
10
Impact of “Standing Army” Costs (Major DoD ERPs)
~55% of Total Development
~1-10$M / Month Overrun
LEGEND
“Standing Army” Costs
Final Project Product
Standing Army -- Level of Effort activities… not itself a work item directly associated with accomplishing the final project
product, service or result, but rather one that supports such work, its duration is based on the duration of the discrete work
activity it is supporting
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Headquarters U.S. Air Force
Data Collection and Analysis
11
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Instrumentation and Dataset
Dataset: Collected from 22
projects government-wide
including 19 programs
from DoD
Instrument: A modified
version of the Software
Resources Data Report
(SRDR) questionnaire* New fields were added to collect
data on infrastructure -- operational
sites, users, software licenses,
servers, tailored objects, CPUs, etc.
12
*Defense Cost and Resource Center (DCARC)
A.
1. System/Element Name (version/release): 2. Report As Of:
3. Authorizing Vehicle (MOU, contract/amendment, etc.): 4. Reporting Event:
Submission # _1____
(Supersedes # __N/A__, if applicable)
5. Name of Development Organization: 8. Lead Evaluator of CMM Certification:
5a. Address of Development Organization: 9. Affiliation to Development Organization:
5b. POC Name:
POC Phone Number:
POC Email:
10. Precedents (list up to ten similar systems completed by the same organization or team):
11. Comments on Part A responses:
B.
13. Peak staff (maximum team size in FTE) that worked on and charged to this project:
14. Percent personnel that was: Highly experienced in domain:X% Nominally experienced:X% Entry level, no experience:X%
15. Comments on Part B responses:
C.
24. Comments on Part C responses:
DD Form 2630-3 Page 1 of 4
22. Legacy System Phase-Out: Number of external legacy systems (i.e., not under project control) expected to be shutdown
whenever the ERP system becomes fully operational. Not planned for migration.
23. Legacy System Migration: Number of external legacy systems (i.e., not under project control) expected to migrate into the
new ERP system.
17. ERP Modules: Standard business modules required by the customer. These standard modules are captured in the ERP
Tool (e.g., SAP, ORACLE 11i, PeopleSoft). Examples of these include Accounts Receivable, Accounts Payable, General
Ledger, etc. Please also provide the list of modules used by the product.
18. Business Processes: Number of processes required by the customer. Processes fall under Business Modules --
System's Process Flow Diagram Level 2. The following example shows three processes required for the Accounts Receivable
business module:
19. Business Sub-processes: Number of sub-processes required by the customer. Sub-processes fall under business
processes -- System's Process Flow Diagram Level 3. The following example shows three sub-processes required for the
________ business process:
20. Functional Requirements: Number of software functional requirements needed by the customer. Functional
Requirements fall under business sub-processes -- System's Process Flow Diagram Level 4. This information is also found
in the System Requirement Specification (SRS). The following example shows a functional requirement:
21. Legacy System Interfaces: Number of external legacy systems (i.e., not under project control) expected to interface with
the delivered ERP system
12. Project Functional Description:
Project RequirementsProvide Actuals
at Final Delivery
16. Business Modules: Number of business modules required by the customer. This information is found in the System's
Process Flow Diagram Level 1 (PFD Level 1). Examples of these include Accounts Receivable, Accounts Payable, General
Ledger, etc. Please also provide the list of business areas required by the product.
Description of Actual Development Organization
6. Certified CMM Level
(or equivalent):
7. Certification Date:
Product and Team Description
ERP Resources Data Report: Final Developer Report
Page 1: Report Context, Product Description and Project Requirements
Report Context
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Cost Drivers (Functional)
Name Appropriateness Definition
RICE+ Inception or
Elaboration Phase
(Milestone B)
Sum of Reports (R), Conversions (C), Interfaces (I), and Extensions
(E+). Those objects within the ERP Software requiring customization,
as these are not compliant with user requirements
Reports (R): Number of formatted and organized presentations of data
Interfaces (I): Number of boundaries across which two independent
systems meet communicate with each other.
Conversion (C): Number of processes that transfer or copy data from an
existing system to load production systems.
Extension (E+): Number of programs that are in addition to an existing
standard program. Extensions are additional programming functionality.
Sometimes referred to as “Enhancements
Functional
Requirements
Early
Inception Phase
(Milestone A)
Functional requirements are defined as a detailed breakdown of business solution
outcomes in terms of the intended functional behaviors of the software
application.
System Interfaces Product Start
(Pre-Milestone A)
Number of external legacy systems (i.e., not under project control) expected to
interface with the delivered ERP system.
Migrated Systems Migration of a legacy or existing application to a new operating environment. It
could also mean to replace a legacy system, moving the data from the legacy
application to the new application while preserving data integrity.
13
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Cost Drivers (Non-Functional)
14
Infrastructure: Servers, CPUs, etc.
Operating Locations
Business: users, processes
Software: applications, licenses
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Cost Driver Selection:
Pair-wise Correlation Results:
RICE is strongly correlated to COTS Integration effort
Requirements is correlated to COTS Integration Effort
However, System Interfaces and Migrated Systems are poorly
correlated to COTS Integration Effort
Recommendation:
Use RICE as software size variable at Elaboration Phase
Use Requirements as software size variable at Inception Phase
15
EFFORT
EFFORT 1
Requirements 0.718
System Interfaces 0.296
Migrated Systems 0.471
RICE 0.918
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Distribution of Total ERP
Implementation Cost:
16
COTS Integration
Support21%
COTS Integration
Effort17%
Interim Logistics Support
28%
Site Activation, User Training,
Data Migration18%
Program Office9%
Deployment Hardware &
Software7%
Influenced by Non-Functional Cost Drivers
Influenced by Functional Cost Drivers
LEGEND
Only 17-38% of Total Cost is captured by Parametric Cost Models
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
COTS Integration Productivity Hours/Tailored Object (“RICE”)
Productivity influenced by Business Area (Finance vs. Supply Chain)
Productivity NOT influenced by ERP COTS Tool (SAP vs. Oracle)
0
200
400
600
800
1000
1200
1400
1600
1800
Supply Chain Finance
Ho
urs
/RIC
E
Business Area
Productivity Comparison by Business Area
17
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
COTS Maintenance Staff (FTEs): System Developer Only
18
43%
34%
21%
16%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Post Go-Live Year 1
Post Go-Live Year 2
Post Go-Live Year 3
Post Go-Live Year 4
Staf
fin
g Le
vel
Post Go-Live System Developer Staff (As Percent of Go-Live Year Staff)
Predicts the Annual Maintenance Staff as a Percent of Go-Live Year Staff
Sustaining Engineering
System Operations
Software Maintenance
Data collected from 5 programs
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Headquarters U.S. Air Force
19
Estimation Models
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
COTS Integration:
COTS Integration Effort
Glue Code Assessment Tailoring
20
Scope:
1. effort for assessing and tailoring COTS software applications or modules …
2. all necessary labor and material …for analyzing, designing/building/configuring,
and testing the required business objects [within the ERP COTS application] –
reports, interfaces, conversions, extensions,…, scripts, enhancements…
**MIL-STD-881 C Reference: K.4.2
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
COTS Integration Effort: As a Function of Software Requirements
21
UNIT
EQUATION
METRICS
R2 PRED
(25)
CV
(%) N
Hours 218.9 * Requirements 65% 23% 90.4 17
Application
Suitable for programs at “Product Start” or “Early Inception” Phase
Includes Financial and Supply Chain ERPs (150 – 5500 Requirements)
Limitation
Dataset only captures Government Sector ERPs, SAP/ORACLE/MOMENTUM Tools
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
COTS Integration Effort: As a Function of Tailored Objects (“RICE”)
22
UNIT
EQUATION
METRICS
R2 PRED
(25)
CV
(%) N
Hours 204.4*Report + 2021*Interface + 2713*Conversion + 496.8*Extension 82% 32% 47.8 20
Hours 922.2 * RICE 83% 58% 63.8 22
Application
Includes supply chain and Finance ERPs
Limitation
Dataset only captures Government Sector ERPs, SAP/ORACLE/MOMENTUM Tools
Note: Reports may include Forms, and Extensions may include Workflow and Bolt-ons
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
COTS Integration Schedule: As a Function of Size and FTEs
23
UNIT
EQUATION
METRICS
R2 PRED
(25) CV(%) N
Months
5.995*RICE0.4621 / FTE0.3332 79% 71% 32.0% 14
Months REQ0.6369 / FTE0.3844 98% 60% 43.2% 15
FTE = Full Time Equivalent Staff; RICE = Report, Interface, Conversion, Extension; REQ = number of software
requirements
Application
Dataset Range (15-1500 RICE; 150 – 5500 Requirements)
Limitation
Dataset captures System Developer and Government Staff
Dataset only captures Government Sector ERPs
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
COTS Integration Support:
COTS Integration Support
Systems Engineering
Program Management
Change Management
System Test & Evaluation
Training Development
24
**MIL-STD-881 C Reference: L.6
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Systems Engineering…Cost As a Function of COTS Integration Cost
25
WBS Element(s) UNIT
EQUATION
METRICS
R2
PRED
(25)
CV
(%) N
System Engineering
Program Management
Change Management
BY10$K 1.347 * COTS Integration Cost 82% 50 55.0 16
Application
Dollars in Thousands, Base Year 2010 (BY10$K)
COTS Integration Range: $2,000K (LOW), $200,000K (HIGH)
Limitation
Dataset only captures System Developer Cost
Dataset only captures Government Sector ERPs
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
System Test & Evaluation Effort As a Function of Software Requirements
26
WBS Element UNIT EQUATION (MEAN)
Descriptive Statistics
Median StdDev CV(%) Lower
Bound
Upper
Bound N
Development Test
& Evaluation Hours 137.5 * REQ 93.6 48.5 71.7 72.1 202.9 18
REQ = number of software requirements
Application
Dataset Range (150-5500 Requirements)
Only captures Development Test & Evaluation
Limitation
Dataset captures System Developer Staff
Dataset only captures Government Sector ERPs, Supply Chain and Finance
Excludes Operational Test & Evaluation
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
System Test & Evaluation Effort As a Function of Tailored Objects (“RICE”)
27
WBS Element
UNIT
EQUATION
METRICS
R2 PRED
(25)
CV
(%) N
Development Test &
Evaluation Hours 437.1 * RICE 72% 6% 77.9 18
RICE = sum of reports, interfaces, extensions, conversions
Application
Dataset Range (15-1500 Requirements)
Only captures Development Test & Evaluation
Limitation
Dataset captures System Developer Staff
Dataset only captures Government Sector ERPs
Excludes Operational Test & Evaluation
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
System Test & Evaluation Schedule As a Function of Size and FTEs
28
WBS Element UNIT EQUATION
METRICS
R2 PRED
(25)
CV
(%) N
Development Test &
Evaluation Months REQ0.6315 / FTE0.4945 96% 55 47.4 12
FTE = Full Time Equivalent Staff; REQ = number of software requirements
Application
Dataset Range (150-5500 Requirements)
Only captures Development Test & Evaluation
Limitation
Dataset captures System Developer Staff
Dataset only captures Government Sector ERPs
Excludes Operational Test & Evaluation
I n t e g r i t y - S e r v i c e - E x c e l l e n c e 29
The Way Forward
I n t e g r i t y - S e r v i c e - E x c e l l e n c e
Proposed Research Initiatives
Calibrate COCOTS to address
ERP Systems using AFCAA Data
Add research results into the
TruePlanning® framework
Add dataset to Data-Manager
and calibrate SLIM-Estimate
30
SLIM-Estimate™