Corrections Technology Association (CTA) 2015

June 1, 2015Daytona, FL

Launch Transformation Boosted by a Strategic Advisor

2

With You Today

Aaron GarnerExecutive DirectorResearch & Technology Indiana Department of Correction (IDOC)

Jack Umansky, PMPDirectorKPMG LLP345 Park Ave New York, NY. 10154518-951-8199 | JackUmansky@KPMG.com

Keven P. StarDirectorKPMG LLP500 Capitol Mall, Suite 2100916-690-6702 | Kstar@kpmg.com

3

Why Projects Need a QA, IV&V or Technical Advisor?

Source: http://onproductmanagement.net/wp-content/uploads/2010/08/treecomicbig.jpg.

4

IV&V – Return on Investment (ROI)

5

The Challenges of IT Projects – Problems Launching

“72% of all IT Projects are late, over budget, lack functionality, or are never

delivered as planned”

Meta Group

“Only 25 percent of telecom IT projects succeed, a rate that is worse than IT projects across all industries, where

28 percent succeed.”

Standish Group

“…the odds of a FORTUNE 500 software development project coming in on time

and on budget are unfavorable – only one in four succeed.”

Standish Group

The Project Management Profession Lives in Troubled Times…..

“75% of 1,450 firms surveyed exceeded their IT deadlines and more than 50% of

them substantially exceeded their budgets. Key reasons were poor planning,

inadequate studies on how IT projects relate to the firm’s needs and lack of

management support.”

KPMG SurveyWhy?

6

A large US state tried and failed twice to implement a new child support enforcement system at a taxpayer cost of $1 billion (USD). In addition, the State was sanctioned another $1 billion (USD) in federal penalties until it successfully implemented a system on its 3rd attempt.

Implementation of a new $55 million payroll system for a large urban school district resulted in $53 million in overpayments to teachers, months of non-payments, and additional costs of $35 million for repairs and delays.

A Southern state planned a $68 million construction project of two data centers located on the floodplain of a dam determined by the Army Corps of Engineers to be at risk of failure.

With the $100 million price tag already spent, a state’s enterprise financial management system project was indefinitely suspended after an independent assessment determined that the state failed to adhere to industry best practices for funding, planning, and implementation.

A $171 million five-year project to build and run a state’s Medicaid claims system is behind schedule and bogged down in litigation, with the state blaming the vendor for unacceptable work product and the vendor blaming the state for inefficient project management and lack of knowledge about the current system.

Nearly half of eight months’ worth of Medicaid fraud documents were destroyed and scores of prosecutions were compromised when the outsourcing vendor for one of the largest states failed to follow file backup procedures.

A large Northeastern city has yet to deploy a new billing system for its half-million water customers after spending over three years and $18 million, more than twice what it expected to pay.

State auditor (CA) recognizes need for independent assessments as $1Billion spent on failed projects.

Recent IT Project Failures at State and Local Municipalities

7

Strategic, Technical, and Business Transformation Advisory

  Phase 1 Strategic Planning

Business Process Analysis and Re-engineering

Requirements Definition

  Phase 2

Vendor and Software Selection

Assistance

  Phase 3

Project Management, Monitoring, and Quality Assurance

Risk Management

Independent Verification and Validation (IV&V)

8

Phase 1

Business Process Analysis and Re-engineering

Why Define Requirements?

9

Requirements Definition

Requirements describe what the system should do or what the system should allow a user to do, not how it will be done.

What are Requirements?

• A requirement is a “condition or capability needed by a user to solve a problem or achieve an objective.”

• Requirements are the functions we expect a system to be able to perform and at what level of performance the functions are desired.

• Requirements are specific, measurable, testable and attainable.

What Requirements are NOT.

• Requirements are not specifications regarding how a function will be implemented.

• Requirements do not describe how the system or user will do it.

10

Requirements Definition – Examples

The following are sample requirements:

• The system will allow users to search by Inmate/Offender ID

• The system must support a minimum of 5,000 concurrent users

The following are NOT requirements:

• The user will select search criteria using radio buttons and a drop down menu

• The system will display the alert message in red

11

Phase 2

Vendor and Software Selection Assistance

12

Vendor and Software Selection Assistance Objective

Objective: Assist government clients to prepare a solicitation document for a Systems Integrator (SI), guide , advise, and assist in the evaluation process, and support the negotiation and contracting of the SI for the implementation of a new system.

Approach: Pre-Selection Assistance

Initial Analysis

Risk Assessment

Oral Presentations

Final Analysis

Post Selection Assistance and Contract Negotiations assistance

Pre-Selection AssistanceDevelop a “long-list” of potential vendors

Initial Analysis

Utilize Proposal Assessment Matrix (PAM)

Item 1

Item 2

Item 3

Identify potential risks & questions for the vendor

Risk Assessment

Facilitate planning and develop questions for orals

Oral Presentations

Update Proposal Assessment Matrix

Final Analysis

Prepare Final Evaluation Report

Post Selection Assistance

13

Selection Assistance Services Background

Selection Assistance Services can assist clients with the evaluation and selection of systems integrators, solutions, products, or technologies as part of the projects that may result. Our SAS approach supports:

Design of the selection process

Determination of scoring criteria

Research relevant vendors

Drafting of requests for information/request for proposals based on specific selection criteria

Performing objective risk assessments of submitted proposals

Scoring vendor responses based on selection criteria

Support with vendor inquiries

Assist agency with oral presentations and software demonstrations

Assistance with the selection including collating vendor responses and facilitating scoring workshops

Assist agency with contract negotiations, drafting, review including risk assessments and recommendations

14

Phase 3

Project Management, Monitoring, and Quality Assurance

Risk Management Independent Verification and

Validation (IV&V)

15

Representative Tasks, Roles and Responsibilities

Approach that is tested and tailored specifically for the Corrections environment

Approach that includes proven templates and processes developed and successfully implemented for Corrections initiatives

Based on well-known project management and quality disciplines (e.g., IEEE, PMI’s PMBOK and CMMI)

Risk-based and forward-looking

Drives quality in the project processes and project deliverables

Risk, Issue and Action

Item Management

Requirements

Traceability and Scope

Management

Independent Project

Monitoring / Assessment

Ongoing Schedule

management, monitoring

and reporting

Deliverables Management

Your Trusted Advisor

16

Typical Tasks

Deliverable Review Management:

• Help facilitate SI deliverable preparation, submission and DOC review process

• Develop and maintain deliverable tracker

• Independently verify that deliverables and work products meet contractual and DOC expectations and professional standard requirements

• Review SI deliverables, provide actionable comments, and recommendations

Risk/Issue Management

• Identify, document, present risks and issues to DOC Management, provide actionable recommendations and assist with risk mitigation planning

• Assist with proactive risk mitigation

Review SI Workplan w/ Status Reports and report on/provide the following:• Tasks late and overdue• Tasks slipping or at risk for missing due dates• Critical Path analysis• Resource challenges• Identify DOC dependencies/responsibilities• Generate reports to help track project

schedule, estimates to complete, etc. • Raise concerns and recommendations as

appropriate

Action Item Management

• Maintain Action Item List and assist DOC in resolving action items related to the systems implementation,

• Establish regular tracking mechanism and document actions and decisions made.

17

Typical Tasks

Attend and participate in project meetings and assist with the following:• Provide deep subject matter knowledge and

advice to assist DOC• Provide integration across work streams

and DOC business process owners and SMEs

• Review meeting decisions and documented meeting minutes

• Provide recommendations as appropriate

Assist with Requirements traceability• Participate in meetings (JAD, conversion,

design sessions, and others) and review deliverables for adherence to requirements

• Review and assess requirements coverage, linkages to artifacts (functional designs, code, test cases, etc.)

• Perform periodic audits of requirements coverage at project milestones

• Provide ongoing advice and recommendations

Conduct Change Request (CR) analysis• Assist DOC with evaluation of SI CR

Impact Assessments• Review SI Impact Assessments for

appropriateness • Review Level of Effort estimation• Assist DOC with tracking of CR LOE

“debits” and “credits” as needed

18

Why Invest in Project Quality Assurance and/or IV&V?

Independent assessment of deliverables, activities, and processes

Subject matter expertise in individual disciplines

Unbiased technology and project recommendations

Insight into leading practices from other jurisdictions

Partnership to provide assessments and recommendations

The federal government supports and funds third-party QA/IV&V for federally funded projects (45 CFR Part 95) ((e.g., Federal OCSE for Child Support projects)

Jurisdictions require QA/IV&V for significant projects (e.g., Indiana, NYC, NASA, California)

Costs of Project Assurance (IV&V, QA, PMO, others) can include:

Fiscal Costs

Time and Resource Costs to coordinate with Assurance teams

Cultural Impacts

Sponsorship Costs

The Quality Assurance Value Proposition

A Titan study conducted at NASA determined:

— The use of an Independent Project Oversight can reduce the total ownership costs of the system

— Nearly two-thirds reduction in Defect Density with full lifecycle Independent Project Oversight A case study conducted by Digital, Finding Defects Earlier Yields Enormous Savings, indicated that each phase a

defect survives it costs approximately three times as much to fix that defect (http://www.digital.com/solutions/roi-cs2.php)

Another case study, conducted by NASA, A Case Study of IV&V Return on Investment (ROI), indicated that the ROI for IV&V on a software development project was between 1.25 and 1.82

For another project, the IV&V ROI was 11.8 (Estimating Direct Return on Investment of Independent Verification and Validation using COCOMO-II, J.B. Dabney, G. Barber, and D. Ohi)

State auditor recognizes need for independent assessments as $1Billion spent on failed projects. California Bureau State Audit Report - https://www.auditor.ca.gov/pdfs/reports/2014-602.pdf

The Titan Study argues the ROI figures understate the full value of an Independent Project Oversight as a result of the following factors: • Watchdog effect – The presence of a QA vendor makes the developer more conscientious and

less likely to cut corners• Improved maintainability – QA reviews improve the accuracy, readability, and general usability of

system documentation• Better understanding and response to risks – QA offers impartial evaluations and

recommendations as to how to proceed when there are difficult alternatives

19

20

What Quality Assurance is NOT!

QA is not a guaranty of project success – issues in or out of a project’s control may ultimately overtake other best efforts:

• Incorrect or insufficient staffing

• Funding issues

• Lack of timely or appropriate decision making

• Absence of appropriate tools

• Absence of senior management or key stakeholder support

QA is not “staff augmentation” or an open checkbook:

• QA does not perform a client’s job if the client happens to have insufficient staffing

• QA must perform contracted functions and not “stop-gap” or out-of-scope activities

QA assesses information and provides recommendations based on those assessments:

• The objective is not to find fault with staff; it is to help provide potential improvements in existing processes, alternatives that may not have been recognized or considered, and independent, unbiased views and suggestions

QA helps “ensure” quality – but does not provide absolute assurance

21

What is Independent Verification & Validation (IV&V)?

Verification – focused on whether the software and its associated products and processes:

- Conform to requirements

- Satisfy standards, practices, and conventions during life cycle processes

- Successfully complete each life cycle activity and satisfy all the criteria for initiating succeeding life cycle activities

Validation – focused on whether the software and its associated products and processes:

- Satisfy system requirements at the end of each life cycle activity

- Solve the right problem

- Satisfy intended use and user needs

Commonly accepted industry definition – IEEE 1012-2008

VerificationAre you building the system in the right way?

Simply put…

ValidationAre you building the right system?

22

IV&V versus QA

Key differences can be summarized in three basic areas:

Independence

-IV&V provider must be independent of the project team

-QA provider works at the direction of the project team

Risk Remediation

-IV&V cannot participate in the remediation of identified risks

-QA can advise and assist project management in risk remediation

Participation

-IV&V can be conducted as a periodic assessment or ongoing

-QA is generally involved ongoing throughout the project lifecycle

23

Similarities between IV&V and QA

Typical IV&V Scope that also applies to QA

• Inspect and provide feedback on project management and system development life cycle (SDLC) plans and practices against key quality attributes.

• Verify whether project practices conform to industry leading standards and practices

• Inspect software and system engineering artifacts (e.g., requirements documentation, system design, technical architecture, system development and configuration, testing, training, implementation, deployment and system maintenance).

• Provide artifact feedback regarding deficiencies (e.g., when project documentation does not comply with industry standards or other project-specific references.)

• Validate traceability of project requirements throughout the SDLC

• Provide verification of test results reports via an appropriate method

• Provide risk assessments and mitigation recommendations

24

Qualities of a Good IV&V and QA Function

A Qualified QA or IV&V Function Should: Be Independent and Objective Bring an Already Established Approach that is Focused on Risks and Controls Provide Reasonable and Actionable Recommendations for Identified Deficiencies Have Tool Sets Built Around a Structured, Repeatable Process Based on IEEE

Standard 1012-2004/2012 and the Project Management Body of Knowledge (PMBoK)

Have the Bench Strength to Deploy and Maintain Continuity for a Team Experienced with the Program/Agency or Domain, IV&V, Project Management, IEEE, and Relevant Technology Skills that Can Serve as a Trusted Advisor

Articulate an Approach that Builds in Opportunities for Collaboration Without Compromising Independence and Makes Efficient Use of State DOC and System Integrator’s Time

Focus on successful implementation of your system

• IV&V/QA does not get rewarded for the volume of findings they make

25

Recurring Themes

State Staff Resources are Insufficient or Not Appropriately Qualified To Manage Project

Project Organization and Lines of Authority are Blurred or dysfunctional Insufficient Quality Assurance and Quality Control Lack of Appropriate, Usable Project Plans Lack of Resource-Loaded Schedules and Use of Critical Path Analysis Lack of Project/QA Metrics or Use Thereof Incomplete or Poorly Maintained Requirements Management And Traceability Lack of Communications Between Project Teams and Stakeholders Inadequate Risk Management, Including Insufficient Risk Definition, And Lack of

Mitigation Strategies and Prioritization Poor Adherence to Project Management Plans, and Contract And Industry

Standards

26

Questions?