florida palm iv&v assessment report · 2016-06-27 · notes # of risks transitioned into issues...

Department of Financial Services

Date: 12/10/2015

Version 1.0

Monthly IV&V Assessment Report

(November 1 – 30, 2015)

Department of Financial Services Monthly IV&V Assessment Report

November 1-30, 2015

of 66 12/10/2015

Table of Contents

Executive Summary ................................................................................................................... 4

Project Status ......................................................................................................................... 4

Summary Project Health Assessment..................................................................................... 4

Approach and Review Activities ................................................................................................. 6

Review Structure .................................................................................................................... 6

Assessment Approach ............................................................................................................ 7

Detailed Assessment ................................................................................................................. 8

1. Project Execution - Mandated Criteria ................................................................................ 8

1.1 Scope Management and Change Control ..................................................................... 8

1.2 Schedule Management ................................................................................................11

1.3 Cost Management ........................................................................................................13

1.4 Risk Management ........................................................................................................15

1.5 Issue Management.......................................................................................................18

2. Project Governance ...........................................................................................................21

2.1 Governance .................................................................................................................21

2.2 Decision Management .................................................................................................25

3 Project Execution – Additional Criteria ................................................................................30

3.1 Performance Management ...........................................................................................30

3.2 Staffing Management ...................................................................................................32

3.3 Action Item Management .............................................................................................36

4 Procurement and Contracts ................................................................................................38

4.1 Procurement and Vendor Management ........................................................................38

5. Quality Management .........................................................................................................44

5.1 Quality Management ....................................................................................................44

5.2 Deliverable Management .............................................................................................48

5.3 Lessons Learned Management ....................................................................................50

6. Communications and Outreach .........................................................................................52

6.1 Communications Management .....................................................................................52

6.2 Stakeholder Management ............................................................................................54

6.3 Content Management ...................................................................................................57

6.4 Collaboration Management ..........................................................................................59

Appendix 1. Project Element Value Assessment and Mapping .................................................60

Appendix 2. Interviews and Meeting Summary List ...................................................................62


November 1-30, 2015

of 66 12/10/2015

Appendix 3. Project Documentation Reviewed ..........................................................................64


November 1-30, 2015

of 66 12/10/2015

Executive Summary To reduce project risk and increase the likelihood of the Florida PALM Project achieving the desired outcomes, Grant Thornton conducted an Independent Verification and Validation (IV&V) of the program to review performance and progress made on observations identified in the Initial IV&V Assessment Report. This report represents an assessment of the Project for the time period from November 1, 2015 to November 30, 2015. Throughout our assessment, Grant Thornton used a Red / Yellow / Green indicator to evaluate the current status of each area and metric. The table below contains an explanation of each status:

Status Image Description

Critical

This assessment category or area has significant challenges, or the concern identified will have an immediate negative impact if not addressed.

Concern/ Off Track

The assessment category or area has challenges, which if not addressed have a high likelihood of causing a negative impact to project outcomes.

On Track/ No Material Concern

The assessment category or area is on track without material issues.

Project Status The purpose of this IV&V assessment report is to evaluate the effectiveness of the Project’s execution and accuracy of the Project’s internal reporting, including status. During the month of November, the Project reported an overall red status due to the sustained delay in the release of project funding and the risk introduced by continuing to incur obligations without a clear understanding of when, or if, funding will be released. At issue is a disagreement between the Chief Financial Officer and the Legislature regarding the interpretation of proviso with respect to funded scope and multi-tiered governance structure. Execution of the pre-DDI activities, including the finalization of the standardized business process models, collection of agency input on the system requirements, development of the ITN, and work on the Data Management Plan remains on track. IV&V has reviewed the Project’s weekly and monthly status reports and is in agreement with the status as reported.

Summary Project Health Assessment Each of the Project Health Indicator categories were reviewed and prioritized based on multiple criteria, including the strength of correlation between the category and desired project outcomes. The relative value of each category is included in parentheses following the health category. Additional detail of the value assessment approach and results can be found in Appendix 1.

Project Health Indicator

Status / Trend

Current Status Justification

Project Execution - Mandated Criteria (High)

Current

Previous

The Project continues to be challenged with communications of significant Decisions and Change Requests. Key project stakeholders remain unclear on essential topics including scope and PMP processes.

R

Y

G

Y


November 1-30, 2015

of 66 12/10/2015


Status / Trend


Scope Management and Change Control

Schedule Management

Cost Management

Risk Management

Issue Management

Trend

The Project continues to remain on schedule, as defined in PCR 3, and has an appropriately detailed and baselined schedule for a Pre-DDI system implementation project.

The Project continues to be within budget, with the largest discrepancy being the timing of invoice payment v. current cash projections. As the budget release remains at an impasse, the Project will begin to incur unbudgeted interest costs on unpaid invoices for accepted deliverables.

The Project has a strong risk management program, proactively addressing the large majority before they become issues, which negatively impact the Project. The Project will need to continue to monitor progress to ensure that risks are assessed as they are identified.

The Project has a fully formed issue management program, which is addressing specific project issues. As additional root causes of current project risks and issues become apparent, it may be beneficial to actively manage these as issues.

Project Governance (High)

Current

Previous

Trend

The Project worked with AST and IV&V to suggest revisions to project governance specifically related to the responsibilities and composition of the Steering Committee. This was documented in an updated Governance Charter, which was provided to the Legislature, and was updated within the PMP.

There continues to be disagreement between the Legislature and the CFO related to proviso compliance of the current multi-tiered Governance structure.

Project Execution - Additional Criteria (Medium)

Current

Previous

Trend

Internal project execution elements are well defined and continue to be executed per the outlined plans.

Basic project performance metrics are in place, with significant opportunity for enhancement if more evaluative metrics can be tracked (e.g. budget to actual).

Procurement and Contracts (Medium)

Current

Previous

Trend

The project has and continues to follow a well-defined process for managing procurements and administering vendor contracts.

Quality Management (Medium)

Current

Previous

Trend

The Project accepted a Quality Management System (QMS) Strategy in November and is working to implement the strategy. The implementation of the QMS will standardize and formalize the current internal review processes and metrics in use by the project.

Communications and Outreach (Medium)

Current

Previous

The Project continued to actively engage the agencies in the requirements review process, and the participating agencies feel that their input is being incorporated.

Y

Y

Y

G

G

G

G

Y

Y

Y


November 1-30, 2015

of 66 12/10/2015


Status / Trend


Trend

The Project is in the process of updating the Communications Plan, and has improved interactions and information sharing with AST.

An updated stakeholder analysis and revision to planned communications is necessary to ensure that all stakeholders such as the Legislature are receiving proper and accurate information directly from the Project.

Recommendations

Additional observations and recommended actions to address each of the identified weaknesses are outlined within the detailed assessment sections of this report. Corrective action plans created by management to address initial recommendations are included as well.

Approach and Review Activities

Review Structure The Project review focused on the critical Project Management elements, which are listed below. For each project element, we include below the observations and recommendations from our initial IV&V assessment report, any actions planned by management as a result of the initial report, and current observations and recommendations for the assessment criteria element. 1. Project Execution - Mandated Criteria

1.1. Scope Management and Change Control 1.2. Schedule Management 1.3. Cost Management 1.4. Risk Management 1.5. Issue Management

2. Project Governance 2.1. Governance 2.2. Decision Management

3. Project Execution – Additional Criteria 3.1. Performance Management 3.2. Staffing Management 3.3. Action Item Management

4. Procurement and Contracts 4.1. Procurement and Vendor Management

5. Quality Management 5.1. Quality Management 5.2. Deliverable Management 5.3. Lessons Learned Management

6. Communications and Outreach 6.1. Communications Management 6.2. Stakeholder Management 6.3. Content Management 6.4. Collaboration Management

Y


November 1-30, 2015

of 66 12/10/2015

Assessment Approach The IV&V assessments approach includes independent research, stakeholder interviews, and reviewing and analyzing project files and documentation. In executing the IV&V and developing IV&V deliverables, the evaluation team applies recognized Project Management best practices and industry standards including:

Project Management Body of Knowledge (PMBOK®), 5th edition

ISO 31000:2009, Risk Management – Principles and Guidelines

Information Technology Infrastructure Library (ITIL) v3

Institute of Electrical & Electronics Engineers (IEEE) 1012-2012 standards (System and Software Verification and Validation)

IEEE 12207 standards (software development), in particular Standard 730 – SQAP (Software Quality Assurance Plans) section 12, “Supplier Control” and Standard 1058 –software Project Management Plans

American Society for Quality Body of Knowledge (QBOK)

The following figure presents the process Grant Thornton has taken to conduct the monthly assessment of the Project.

Figure 1 - Grant Thornton Assessment & Report Development Process


November 1-30, 2015

of 66 12/10/2015

Detailed Assessment 1. Project Execution - Mandated Criteria

1.1 Scope Management and Change Control Key Metrics

Item Observation Current Status

Previous Status

Notes

Number of Change Requests

10

Change requests continue to follow the documented Change Request process as outline per the Project Management Plan (PMP).

Process Observation Summary Change Requests continue to follow the documented Change Request process as outlined per the Project Management Plan (PMP). Regarding scope and change control concerns, the Florida PALM Project Team updated the Steering Committee Charter and corresponding sections of the PMP in conjunction with AST and IV&V. Once the new Steering Committee is convened, the change request log should be shared and any questions addressed.

G G


November 1-30, 2015

of 66 12/10/2015

Assessment Category Initial Observations and Findings

Initial Recommendations

Current Management Action Plan

Current Observations and Findings

Current Recommendations

Category Status

Determine that a structured process is in place for identifying, documenting, reviewing, and obtaining appropriate approval for any change to project scope. Verify that the correct Stakeholders are included in the decision making process for scope changes.

The project has a change control process which is outlined in the Project Management Plan. The process outlines the approach, data elements to be captured, and relevant stakeholders to include. The Project Steering Committee reviews any scope changes which have an impact on project schedule, budget, or the defined project outcomes. The Steering Committee currently includes representatives from DFS.

Include additional stakeholders in the scope decision process. (Additional detail is included in section 2, Governance below)

1) Develop Project Governance Charter with expanded membership and route for approval. 2) Update Project Charter and Project Management Plan to reference Project Governance Charter and route for approval.

The Project developed an updated Steering Committee Charter and updated the Governance section of the PMP in coordination with AST and IV&V to include Stakeholders from the FFMIS agencies, AST, and the Legislature. The Charter has been shared with the Legislature.

After the Governance changes are made and a new Steering Committee is convened, review the Project change request log with the new Steering Committee Members and address any questions.

Current

Previous

Trend

Verify that pending project changes and decisions are clearly communicated to impacted Stakeholders in the appropriate level of detail.

Initial change requests were communicated within DFS and to stakeholders via the vendor forums. Some key stakeholders such as AST and the Legislature have different understandings of the scope and impact of the change requests and were not included in targeted communication after the decision was made by the project.

Include additional stakeholders as appropriate in the decision communication process to ensure that stakeholders have a common understanding of the project changes and impacts.

1) Develop Project Governance Charter and route for approval. 2) Once approved, assemble Steering Committee with expanded membership with agendas to include review of previous and future decisions and project scope changes.

AST, the FFMIS agencies and the Legislature are included in the updated governance structure. Project changes including scope, schedule, and cost are documented in the Project Change Log on SharePoint, and are referenced in the weekly and monthly status reports.

Convene the updated Steering Committee as soon as possible and review any concerns about project scope with the participants.

Current

Previous

Trend

Y

G

R

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that any changes to the project scope (which manifest via modifications to the project work plan, work breakdown, contract or baseline schedule) adhere to the defined change control processes and the appropriate governance levels for change process (as defined in the Project Governance document).

The project does have a Scope and Change Management Plan as a part of the PMP (Appendix 8 - Scope and Change Management). There is an MS Visio Process Work Flow developed for the Change process as well as an accompanying MS Excel spreadsheet detailing associated data field. The Project Change Request (PCR) Log resides on SharePoint. Any changes affecting scope are reflected in the corresponding documentation whether it is the schedule, WBS, etc. The Steering Committee has reviewed all changes meeting the defined thresholds. The Project Director reviewed and approved all others.

None, the project has been following its documented process. As changes to governance are made, this will cascade through the project.

N/A The Project continues to follow the process outlined within the PMP.

No new recommendations.

Current

Previous

Trend

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that project decisions and scope changes do not negatively impact the desired results of the Project.

Decision 4 was logged by the project on June 1, 2015 to exclude payroll functionality from the initial ITN to reduce risk and complexity due to the potential People First transition. The decision to potentially postpone the payroll functionality has been widely interpreted by external parties as having removed it from the project’s scope.

Determine and document when and how the project will move forward with the upgrade of the current FLAIR payroll functionality if it will not be in the initial ITN. Clearly communicate the timing, and impacts to all stakeholders.

Record decision with due date.

An open decision was logged establishing an action item and timeframe for the Project to make a decision about how and when the FLAIR Payroll can be replaced. One dependency of this decision is the finalization of the People First support contract procurement.

Address the payroll implementation decision as planned when the appropriate information is available from DMS.

Current

Previous

Trend

1.2 Schedule Management Key Metrics


Previous Status

Notes

% Complete 73% Percent complete in the Master Project Schedule is based on the work effort identified in the tasks entered.

Number of Late Tasks

0 There are no late tasks on the Master Project Schedule.

Process Observation Summary The Florida PALM Project Team continues to follow the steps outlined in the PMP and all Work Tracks are planned through the entire Pre-DDI phase. During the month of November, the individual Track Schedules were combined into the existing Master Project Schedule to facilitate better visibility and reporting. The Master Project Schedule is baselined for all tasks for December. The Project continues to update tasks percentages daily and weekly and Work Tracks meet regularly to update the project schedule.

Y

G

G G

G G


November 1-30, 2015

of 66 12/10/2015






Category Status

Identify whether the Project is on schedule. If any schedule slippage exists, identify whether project leadership has assessed severity of the slippage, and identified mitigating actions to address.

The project is on schedule. The initial baseline was created on 8/21/2015 with baseline updates made on 9/17/2015 and 9/28/2015 based on approved project change requests

Continue to formalize the relationship between the individual track schedules and the master schedule.

N/A The Project combined the individual Track Schedules into a single Master Project Schedule for ease of reporting. The Project continues to be on schedule as reported.

No new recommendations. Current

Previous

Trend

Determine if project milestones have been comprehensively identified and documented (in the Master Project Schedule, Project Management Plan or other Project Management artifacts).

Project milestones are called out in the Master Project Schedule. The schedule has a section specifically dedicated to tracking milestones and deliverables, which is easily identified and can be tracked and modified as needed. The milestones are appropriate for the scope and approach to the project.

Continue established process as identified in the PMP.

N/A Project milestones continue to be called out in the Master Project Schedule.


Current

Previous

Trend

Determine if a process exists for comparing planned (baselined) versus actuals and addressing any deviations identified.

The FL PALM Project has a baseline and there are on-going activities for tracking any slippage or deviations per their defined process for maintaining the Master Project Schedule. The schedule is reviewed by the PMO and Track Leads on a weekly basis for open project tasks.

Continue established process as identified in the PMP.

N/A The Project continues to review the Project Schedule against baseline on a regular basis. This process is facilitated by the combination of the Track Schedules into the Master Project Schedule.


Current

Previous

Trend

G

G

G

G

G

G


November 1-30, 2015

of 66 12/10/2015

1.3 Cost Management Key Metrics


Previous Status

Notes

Cumulative Budget Cost Variance

2

The Project is within budget. The cost baseline was re-planned with change request 3 in September, and there is no variance.

Project Budget v. Actual Spend

10% The Project is within budget for the work expected to be performed to date.

Cost Performance Index (CPI)

1 N/A Per AST’s request, CPI is currently being tracked in the Master Project Schedule.

Process Observation Summary The Project Team has finalized cost management policies and procedures and the cost management section of the PMP been updated. Both documents have been reviewed for quality and approval. Cumulative cost impacts related to change orders are also being summarized for project reporting on a monthly and quarterly basis. Additionally, a process was agreed to with AST to begin tracking Schedule Performance Indicator (SPI) and Cost Performance Indicator (CPI) using the Project Master Schedule and proxy values for project cost based on schedule completion. In the future, the Project Team anticipates reporting on managed versus unmanaged cost, including schedule impacts, which will be reported on a monthly basis as part of the Performance Support Tools (PST) implementation. In the interim, the Project Management Office (PMO) will maintain the Master Project Schedule change and re-baseline to enable the CPI and SPI tracking using the proxy values until a separate PST is available.






Category Status

Determine whether the project budget appears sufficient to support the activities associated with the scope of the Project.

The project budget for the year appears sufficient based on the Spend Plan and the list of two-party agreements.

None N/A The project budget for the year appears sufficient based on the Spend Plan and the list of two-party agreements.


Current

Previous

Trend

G G

G G

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine the Project's compliance with the defined cost management techniques in the Project Management Plan (cost management plan component).

The cost management section of the PMP is high-level and outlines the major cost management elements. There does not appear to be any internal policy or procedure documentation formalizing tactically how each of the cost management elements will be performed and managed.

Formalize the current cost management approach through documentation to ensure the same can be followed on a repeatable basis

1) Finalize supporting Cost Management documentation as part of PMP Phase 2 implementation. 2) Route for Document for Approval. 3) Distribute approved/published documentation.

The Project Team finalized cost management policies and procedures and updated the PMP to reflect the updated procedures.

Provide communication as appropriate to the PMO and other project participants of the updated cost management policies and procedures.

Current

Previous

Trend

Verify that a project baseline has been established. Assess the projected costs approved and the actual costs of implementing and maintaining the completed project against the baseline.

The project schedule was recently baselined and the spend plan updated with that baseline. Previously, the planned spend was not compared to actual expenditures in project reports.

Develop a reporting method to compare the planned spend to actuals, including cumulative spend by project phase to evaluate trends. As part of this reporting method, clarify the planned timing of the funds if possible to reduce confusion from payment cycle delays (e.g. planning for when a deliverable is due, but an invoice won’t be paid until up to a month after deliverable approval.

1) Update spend plan to identify planned expenditures fiscal year to date. 2) Obtain clarification from IV&V regarding recommendation to show "planned timing of the funds if possible to reduce confusion from payment cycles"

The Project Schedule was baselined and the spend plan updated according to the baseline. The cost reporting was modified to include cumulative spend and the cumulative spend plan. The Project is on budget, and continues to manage internal and Contractor expenses as planned for the identified work products. Due to timing of deliverable approvals and actual invoice payment, the project’s actual expenditures tend to lag between 1 and 2 months behind the spend plan.

Continue to work with AST to determine different options for reporting planned v. actual spend so the Project is not penalized for the planned deliverable approval and payment cycle.

Current

Previous

Trend

Y

G

Y

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that monitoring is being done on progress against project costs and expended effort.

Currently, cost monitoring is performed by the PMO against the schedule and contractor deliverables. This achieves the desired effect of ensuring that the project is received the desired outcomes without adoption of a formal EVM approach or tool.

Continue to monitor, manage, and report on project spend against budget by major contractor deliverable in addition to project schedule, implementing an appropriate EVM approach when the tools are available.

N/A Cost monitoring continues to be performed by the PMO against the schedule and Contractor deliverables.


Previous

Trend

Verify that the right performance measures are being collected and managed for cost management.

Not all KPI’s identified in the PMP are being reported on a regular basis, specifically the cost variance and contractor payment variance are not included on regular reports.

Update the spend plan report to include the planned expenditures by date to enable time based cost comparisons.

1) Update spend plan to identify planned expenditures fiscal year to date. 2) Obtain clarification from IV&V regarding "Not all KPIs identified in the PMP are being reported upon on a regular basis".

The cost variance and Contractor payment variance are now being included on the Project Team’s regular reports. In addition, the spend plan has been updated to include year to date numbers for better reporting and tracking.

Continue to work with AST to determine different options for reporting planned v. actual spend so the project is not penalized for the planned deliverable approval and payment cycle.

Current

Previous

Trend

1.4 Risk Management Key Metrics


Previous Status

Notes

# of risks transitioned into issues

2 This metric includes risks, which could not be mitigated and have become project issues.

Process Observation Summary Risks on the Florida PALM Project continue to be tracked and managed using the MS SharePoint portal following the risk management processes as outlined per the PMP. The Florida PALM team will begin to risks turning into issues report on a monthly basis as a part of the Quality Management System (QMS) implementation.

G

G

Y

G

G G


November 1-30, 2015

of 66 12/10/2015

Assessment Category Initial Observations

and Findings Initial

Recommendations Current Management

Action Plan Current Observations and

Findings Current

Recommendations Category

Status

Verify that a project Risk Management Plan and supporting procedures are in place and complete.

There is a Risk Management Plan embedded into the Pre-DDI PMP and a Risk Management Process detailed in MS Visio with Meta Data (fields) captured in complementary MS Excel worksheet. All Risks are available via the FL PALM SharePoint site for all Team Tracks.

None N/A The Risk Management process is defined in the PMP.


Current

Previous

Trend

Verify that the Risk Management plan is being followed and is effectively identifying, tracking, and managing project risk in a standardized fashion.

The project team members are following the identified risk management plan. Training on the risk management plan, approach, and SharePoint risk tracking tool was provided to the project team on September 1.

None N/A The Risk Management plan from the PMP is being followed.


Current

Previous

Trend

Determine whether the Executive Sponsorship understands effective risk management planning and execution and are reinforcing the importance of sound risk management in their communications with project personnel.

The Project Director and Steering Committee members are consistent in their definition of risks and issues, and are pushing the project to identify and manage risks before they become active issues.

None. N/A Risk definitions continue to be consistently used and communicated by the project.


Previous

Trend

G

G

G

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine whether Project Team members possess a command of the defined risk management plan and processes.

Project team members and the Work Tracks are fully ‘on board’ with the risk management approach including holding a weekly RAIDL (Risk, Action Items, Decisions, and Lessons Learned) review meeting where new and open risks are reviewed.

None. N/A Project Team members continue to consistently follow the Risk Management processes.


Current

Previous

Trend

Verify that consistent criteria for evaluating and quantifying risk likelihood and severity levels have been defined.

Evaluation criteria are clearly defined in the PMP and corresponding training material.

None N/A The Criteria outlined in the PMP continue to be used.


Current

Previous

Trend

Verify that risks have been prioritized for mitigation.

Risks, when assessed, are entered into SharePoint with criteria to enable their prioritization. Each of the risks tracked in SharePoint has been prioritized and weighted.

None N/A The Project continues to review and prioritize risks, however a significant backlog of un-reviewed risks accumulated during November.

Ensure that regular RAIDL meetings address newly identified risks on a timely basis.

Current

Previous

Trend

G

G

G

G

G

Y


November 1-30, 2015

of 66 12/10/2015

1.5 Issue Management Key Metrics


Previous Status

Notes

Open Issues 2 The Project has two open issues related to release of budgeted funding.

Open Issues past Target Completion

2 This measures the progress made on issues and their aging past their original due date.

Issue Inclusion 2 Project issues are documented and analyzed using the Project’s SharePoint tool.

Issue Completeness 2 Current issues have the appropriate level of detail entered in the tracking tool for their status.

Process Observation Summary Issues on the Florida PALM Project are tracked and managed using a MS SharePoint portal. The Project actively manages project risks, and has had very few of these risks which have become issues. The project issue database is used for issues, which have an active impact on the project scope, schedule, or execution. Action items and questions are addressed by the Project Tracks, and are not tracked in the project issue log. As the project reviews and manages the current open issues, it may be appropriate to enter document additional ‘root cause’ issues, which have been caused the current project challenges.






Category Status

Verify that guidelines to determine which issues are subject to a formal evaluation process have been established and maintained.

Formal definition of issues (and distinction between issue and risk) is documented in the PMP and was documented and rolled out to the project team in the project PMP rollout training held on September 1st.

Make issue guidelines and training materials available via a link from the project SharePoint issues database if feasible.

N/A The Project continues to follow the issue process as outlined in the PMP.


Previous

Trend

Y G

Y G

G G

G G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify issues are collected and analyzed, and corrective actions have been created to address them.

Of the six issues documented, none have had the Corrective Actions field completed. Actions or steps taken have been documented in other field categories. The one issue entered after 9/1/2015 does not have a corrective plan identified.

Monitor that as issue records are created going forward, that all of the fields in SharePoint are completed as defined.

1) Update Issue #7 with corrective actions. 2) Verify procedures for issue management include review of corrective actions.

Issue #7 has been updated with appropriate corrective actions. The issue management process now includes a review of corrective actions entered, which are then updated when appropriate.

Continue to collect, analyze and document issues in the SharePoint issue log. Ensure all of the fields in SharePoint are completed per the field definitions.

Current

Previous

Trend

Verify the existence and institutionalization of an appropriate project issue tracking mechanism that documents issues as they arise, enables communication of issues to proper Stakeholders, documents a mitigation strategy as appropriate, and tracks the issue to closure.

The Issue Management Plan is embedded into the Pre-DDI PMP. Issues are available and tracked via the FL PALM SharePoint site Issue Log Dashboard, Summary and Preview. Appropriate and necessary fields are captured but are not being utilized to the fullest. Training for the entire program was performed on issue categorization and entry into the SharePoint tool during the initial PMP Rollout training on 9/1/2015.

None The Issues capture tool was updated in the middle of September. The IV&V team will continue to monitor use of the database by the project team.

N/A The Project continues to use the MS SharePoint issue management log.


Current

Previous

Trend

Y

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that corrective action has been taken on identified issues.

There is a field in the SharePoint issue record which was not utilized for the issues converted prior to 9/1/2015 when the issues tracking database was updated. The one issue entered after 9/1/2015 does not have a corrective plan identified.

Use the action plan field within the SharePoint issue log to track and manage the steps being taken to address the open project issues.

Refer to Issue 7 and action plan identified above.

The corrective action steps for the open issues have been updated.

Continue to ensure all of the fields in the SharePoint issues log are completed per the field definitions.

Current

Previous

Trend

Y

G


November 1-30, 2015

of 66 12/10/2015

2. Project Governance

2.1 Governance Key Metrics


Previous Status

Notes

N/A N/A N/A N/A The individual metrics associated with project governance processes will be tracked within those specific sections of the document.

Process Observation Summary The Project, working with AST and IV&V developed an updated Steering Committee Governance structure designed to include additional project participation, including AST and the FFMIS agencies. This was formalized in the Steering Committee Charter and updates to the PMP made during the month of November. There is disagreement and confusion with the Legislature over the specific wording of the Steering Committee charter, and the difference in governance required during a pre-DDI effort and during the implementation process.






Category Status

Evaluate effectiveness of project governance for a project of this size and complexity.

A multi-tiered (four tier) governance structure is in place for Florida PALM, consisting of Track Managers, a Project Director, Project Steering Committee (chaired by an Executive Sponsor), and the CFO. In general, governance processes are defined and consistently followed, with one exception as identified below. The effectiveness of governance was found to

A governance document should be drafted, and appended to the PMP, to include a description of a revised project governance structure. The initial focus of this governance document should be the Project Steering Committee. The consideration of additional tiers, including their composition, engagement, and timing should be determined by

1) Develop Project Governance Charter and route for approval. 2) Update Project Charter and Project Management Plan to reference Project Governance Charter and route for approval.

The updated Project Governance Charter addresses the concerns previously identified.


Current

Previous

Trend

Y

G


November 1-30, 2015

of 66 12/10/2015






Category Status

be adequate at the Track Manager, Project Director and Project Executive Sponsor (CFO) tiers, and weak at the Steering Committee tier. Project Director tier The governance at the Project Director tier was found to be effective. A Project Director was assigned with the appropriate complex information technology systems acquisition and implementation experience. The governance processes executed at the Project Director tier are generally well defined and consistently applied. There is a single exception relating to how Project Change Requests are differentiated from decisions. Decision management observations will be addressed in another section of this report. Project Steering Committee tier

the new Steering Committee. Project Steering Committee governance structure that is inclusive of those key stakeholders outside of DFS who will be materially involved in or impacted by the project. At a minimum this should include: DMS(People First), DMS (MFMP), Governor’s Office (LAS\PBS), and AST(Project Oversight), The Project Steering Committee governance document should also ensure responsibilities are clearly defined such that Steering Committee members have an equal voice, and members are enabled to provide attributable feedback in a transparent and timely manner, ensuring decisions are informed to the greatest extent possible, and represent the commitment of the entire Steering Committee.


November 1-30, 2015

of 66 12/10/2015






Category Status

The governance at the Project Steering Committee tier was found not to be effective in terms of membership, but effective in terms of clearly defined and consistently executed governance processes. The composition of the current Steering Committee includes key DFS stakeholders, but no external entities. Project Executive Sponsor tier The governance at the Project Executive Sponsor level was found to be effective. The Project Executive Sponsor tier governance processes are clearly defined and consistently applied, particularly those related to managing decisions, ensuring executive commitment, and maintaining focus on the achievement of planned outcomes. At the time of this assessment, consideration is being

Any changes to the current multi-tiered governance structure should be made through defined decision and change control processes, including the escalation of material decisions for the consideration of the reconstituted Steering Committee.


November 1-30, 2015

of 66 12/10/2015






Category Status

given for an additional tier of governance that will allow for public transparency, in the form of an Advisory Board or Advisory Council.

Assess executive Stakeholder buy-in, participation, support and commitment, and that open pathways of communication exist among all Stakeholders.

While there are clearly defined processes for stakeholder management, including processes for communications, engagement, resistance identification and management, these processes have not resulted in consistent understanding of the project across all key stakeholders. There are significant communication and participation gaps between the project and key stakeholders in the Legislature. Stakeholder management is generally considered a function of Organizational Change Management (OCM), and the lack of available qualified OCM resources to fill the open OCM Manager role on Florida PALM may be a

With the Legislature’s responsibility to both fund the project and ensure that the project is in compliance with proviso language upon which the funding is contingent, the Legislature must be engaged at the same level as other key stakeholders and take an active role in the communication.

1) Develop Project Governance Charter (with members of Legislature included) and route for approval. 2) Once approved, assemble Steering Committee with expanded membership with documented agendas and meeting minutes (including participation of Steering Committee Members).

The updated Project Governance Charter was created and forwarded to the Legislature for review and comment. All concerned parties need to speak with each other in plain language to address the root cause of the problem. Once positive and proactive dialog has occurred leading to a common understanding, formal letters coming to a mutual agreement between the CFO and the Legislature need to be signed.

Continue to escalate the communication issues around the governance structure between the Legislature and the Project, pushing for direct and specific input on concerns specifically related to the current phase of the project.

Current

Previous

Trend

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

contributing factor in the inconsistent application of stakeholder management processes.

Verify that lines of reporting and responsibility provide adequate technical and managerial oversight of the project.

Technical and managerial oversight gaps exist with the absence of key of entities with management responsibilities or owners of critical systems directly impacted by the implementation of Florida PALM of the project.

Adopt a revised Steering Committee composed of key stakeholders outside of DFS that will be materially involved in or impacted by the project. At a minimum this should include: DMS (People First), DMS (MFMP), Governor’s Office (LAS\PBS), and AST (Project Oversight).

1) Develop Project Governance Charter (with members of People First, MFMP, LAS/PBS, OPB and AST included) and route for approval. 2) Once approved, assemble Steering Committee with expanded membership.

The updated Project Governance Charter addresses the concerns previously identified.

Convene the updated Steering Committee as soon as practical. Current

Previous

Trend

2.2 Decision Management Key Metrics


Previous Status

Notes

% Open 8.4% N/A N/A Number of decisions that are in the following status: ("Developing" or "Approval Pending”)/ total number of decisions.

% Closed 20.5% N/A N/A Number of decisions that are in the following status: ("Closed") / total number of decisions.

Process Observation Summary The processes associated with decision management are clearly defined in terms of required information, approvals, and escalations. The Project made progress on improving the governance structure over the previous period, and is currently working on updates to the policies and procedures related to how decisions are identified and managed.

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that Stakeholders have bought-in to all changes, which impact project objectives, scope, cost, or schedule.

The processes associated with decision management are clearly defined in terms of required documentation, approvals, and escalations. The one weakness with the decision management process is the lack of formal criteria for the circumstances upon which a decision must be logged. Because all stakeholders have not participated in Steering Committee meetings to date, all stakeholders have not had the opportunity to understand the rationale behind, nor bought-in to all of the decisions and changes made by the current governance structure.

Criteria should be clarified for the circumstances upon which a decision should be logged. A potential standard for consideration would be the Florida Information Technology Project Management and Oversight Standards, Chapter 74-1, F.A.C. Decisions that modify either scope, schedule or cost, according to the following metrics:

a. Scope - Changes

that modify the

project scope as

documented in the

approved Project

Management Plan

b. Schedule -

Changes to major

deliverable due

dates or key

milestone dates

c. Cost - Variances of

greater than +/-

10% of project

budget within

spending plan

categories

1) Develop Project Governance Charter and route for approval. 2) Once approved, assemble Steering Committee with expanded membership with agendas to include review of previous and future decisions and project changes.

The Project Governance was updated, but because there continues to be disagreement between the Legislature and the CFO related to proviso compliance of the Governance structure, the updated Steering Committee has not met. Criteria clarifying the circumstances under which a decision should be logged are being developed by the Project, but have not been finalized.

Finalize and publish the criteria for documenting a formal project decision. Have the new Steering Committee Members review previous project decisions and address any questions as they are raised. Proactively review decisions, which have a high profile (such as the timing of the Payroll implementation).

Current

Previous

Trend

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Key decisions and change notifications should be revisited and reviewed under the new composition of the Steering Committee. Because of the volume of logged decisions, revisiting every decision would be impractical and unnecessary. New Steering Committee members should review the decision log, and should raise questions with any decision to the Steering Committee. However, those decisions which resulted in a change which would have been discussed with the Steering Committee (see above) should be proactively discussed on the Steering Committee.


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that the Project and Sponsors have reviewed and approved any project changes that may impact general scope or direction to help ensure alignment with the Project objectives.

The project and sponsors have reviewed and approved any project changes. This is inclusive of all changes, not solely those changes that impact scope or intended outcomes of the project

None. N/A The Project Steering Committee and Sponsors continue to review any project decisions as they are made and presented.

Have the new Steering Committee Members review previous project decisions and address any questions as they are raised. Proactively review decisions, which have a high profile (such as the timing of the Payroll implementation).

Current

Previous

Trend

Determine whether The Project and Sponsors have reviewed and approved any deviations from the project baseline budget prior to further action (e.g., contract modification).

The project management team has been exemplary in their commitment to executing the process for documenting the impacts of all changes. This is true not only in terms of impact to the schedule, but also in terms of impacts to deliverables, resources, and budgets.

None. N/A Project Management continues to document project decisions and share with the Steering Committee as appropriate.

Have the new Steering Committee Members review previous project decisions and address any questions as they are raised. Proactively review decisions, which have a high profile (such as the timing of the Payroll implementation).

Current

Previous

Trend

Determine whether the baseline (and any re-baselines) to the Project Schedule was vetted through and received approval from Project Sponsors.

Only certain tasks within the project schedule have been baselined. Any tasks associated with procurement specific activities, such as the release of the ITN, have not been baselined. The rationale for this approach is these tasks have dates that are both very fluid, and may change

For those tasks where procurement activities have not been baselined, and the project has only a date range for the completion of the tasks, the project should consider baselining those tasks according to a date at the most conservation side of the range.

Baseline all tasks in the Master Project Schedule.

Beginning in November, the Project combined the individual Track Schedules into the Master Project Schedule for ease of reporting. This combined project schedule was baselined.

Continue to monitor usage of the Master Project Schedule and adjust as it benefits reporting and visibility of the overall project status.

Current

Previous

Trend

G

G

G

G

Y

G


November 1-30, 2015

of 66 12/10/2015






Category Status

multiple times leading up to the actual dates, and have dependencies on external entities upon which the project has no direct control. For those tasks that have been baselined, all changes to the baseline are traceable through properly executed and approved Project Change Requests (PCRs)


November 1-30, 2015

of 66 12/10/2015

3 Project Execution – Additional Criteria

3.1 Performance Management Key Metrics


Previous Status

Notes

N/A N/A N/A N/A The individual performance metrics for the Project are identified and captured in each specific section (e.g. cost, schedule, etc.).

Process Observation Summary The Project continues to formalize how it reports on the performance metrics defined within the PMP. For example, in November, the cost reporting processes were formalized and the Project began reporting both CPI and SPI. The Project is working with the IV&V Vendor to develop an implementation plan for the Quality Management System (QMS), which will include refinements to the performance metrics and implementation of Project Support Tools (PST) to enable easier collection and reporting of project performance.






Category Status

Evaluate implementation of the performance management activities and metrics to determine if processes are effective and are being followed.

Performance management metrics are being manually tracked. Some are reported formally on a regular basis, and some are being reported when requested. As project support tools are identified and implemented, the project team plans to track additional metrics.

Continue to refine the metrics and create a formal process to document those metrics not being reported in a formal manner.

1) Approve IV&V and QMS Implementation Strategies. 2) Implement KPIs/KPMs according to timeline agreed to between the State and the IV&V Contractor.

The Project continues to track and report on their existing performance metrics as defined within the PMP and is adding additional metrics (such as SPI and CPI).

Continue to execute the QMS Implementation Strategy to update the gathering and reporting of project performance metrics and use of PSTs.

Current

Previous

Trend

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Evaluate if appropriate mechanisms are in place for project self-evaluation and process improvement.

The project outlined a Quality Management System for continuous process improvement within the PMP at a high level. Through engagement of the IV&V vendor, the project is in the process of developing and implementing the QMS. In addition, the project is actively tracking lessons learned.

Continue to formalize the documentation of performance metrics and establish a regular reflective process to facilitate ongoing improvement.

1) Approve IV&V and QMS Implementation Strategies. 2) Implement KPIs/KPMs according to timeline agreed to between State and IV&V Contractor.

The Project finalized the QMS Implementation Strategy, which includes specific evaluation and self-improvement steps. The Project, in conjunction with the IV&V Vendor, is working to implement specific QMS tools and processes according to the strategy.

Continue to formalize and implement project policies and procedures around performance management as outlined within the QMS Implementation Strategy and the QMS Implementation Plan as it is being formalized.

Current

Previous

Trend

Verify that selected measurements are being obtained and analyzed to manage the Project and support organization needs.

Key metrics are being captured and reported as part of the weekly and monthly reporting process. Some metrics, such as CPI and SPI are not being captured.

Continue to refine the capture and reporting of performance metrics across the project, using the existing tools where possible, and the PST when available.


The Project finalized the QMS Implementation Strategy, which includes specific evaluation and self-improvement steps. The Project, in conjunction with the IV&V Vendor, is working to implement specific QMS tools and processes according to the strategy.

Continue to formalize and implement project policies and procedures around performance management as outlined within the QMS Implementation Strategy and the QMS Implementation Plan as it is being formalized.

Current

Previous

Trend

Verify that the Project’s performance is periodically reviewed and aligned with current and anticipated needs, objectives, and requirements of the organization, customers, and end users as appropriate.

The project team is developing and evolving internal processes as they are needed. This process has not been formalized.

Establish a formal performance review process on a regular basis in addition to the lessons learned reviews.


The QMS Implementation Strategy outlines a regular process for evaluating the effectiveness and efficiency of project performance.

Continue to formalize and implement the QMS including its continuous review elements.

Current

Previous

Trend

Y

G

Y

G

Y

G


November 1-30, 2015

of 66 12/10/2015

3.2 Staffing Management Key Metrics


Previous Status

Notes

Percent of scheduled positions filled.

83% (20 of 24)

The Project is unable to fill open positions, particularly the OCM lead due to the issue with the budget amendment.

Turnover 9.5% (2 of 21 team members)

Percent of team members leaving the Project in the past 6 months. The two Project Team members who left went to positions in DFS divisions where they will continue to support the Project.

Process Observation Summary The PMO manages a formal staffing and onboarding process for state staff and the Contract Manager manages the process for Contractors. Clear onboarding procedures and support materials have been developed and are shared with new members of the team to provide a clear and consistent background on the Project and operating guidelines moving forward. The Project Team looks for the best possible candidate for each position, from both a skillset and personality fit. This has led to longer lead times to fill some positions, but has resulted in a stronger, more capable Project Team. The most recent attempts to hire an OCM Manager have been hindered by the delay in the release of the budgeted funding.






Category Status

Verify that project roles and associated responsibilities are clearly defined, documented, and communicated and verify that each Project Team member is mapped to a distinct project role.

The project has a clear organization structure with well-defined job descriptions for each position.

None. N/A The Project continues to have a clear organization structure with well-defined job descriptions for each position.


Current

Previous

Trend

Y G

G Y

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine whether the design and structure of the Project Management Team is sufficient to support the Project given the project's size, scope and complexity.

The structure and staffing of the PMO team is appropriate for the Pre-SSI effort. Two additional contractors have been added to support project management activities including risk and quality management.

None. N/A The structure and staffing of the PMO team continues to be appropriate for the Pre-DDI effort.


Previous

Trend

Determine whether the design of the organizational structure reflects clear lines of reporting, responsibility and accountability.

The project organization structure is clear and the team members are clear with how they report and support each other.

None. N/A The Project’s organization structure is clear and the team members understand their reporting relationships and how they support each other.


Current

Previous

Trend

Determine whether the Project operates in accordance with the organizational structure, wherein the defined/documented lines of reporting, responsibility and accountability are practiced/adhered to by Project Team members.

The project team follows the project structure with each track lead managing their team.

None. N/A The Project Team continues to follow the project structure with each track lead managing their team.


Current

Previous

Trend

Determine whether project resources appear appropriately qualified to perform their assigned duties (based on the defined project roles/responsibilities).

The current project resources appear to have the background experience and knowledge required for their positions.

None. N/A The current project resources appear to have the background experience and knowledge required for their positions.


Current

Previous

Trend

G

G

G

G

G

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine the project's compliance with the resource management techniques defined in the Project Management Plan.

The project team is following the staffing planning and onboarding processes outlined within the PMP. Each team member is provided standard onboarding training and compliance with these activities is monitored for each employee and contactor.

None. N/A The Project Team continues to follow the staffing planning and onboarding processes outlined within the PMP.


Current

Previous

Trend

Evaluate the stability of the Project Team and cross sharing of knowledge to buffer for vacations and possible turnover.

Within each track, there is coordination and sharing of project tasks. A contactor was hired to augment the PMO capabilities when the PMO manager left for maternity leave. Once the SDS tack fills its two open resource requests, there should be adequate redundancy.

Continue to actively monitor the project teams and continue to manage the hiring process to fill the open positions as quickly as the right candidates can be found.

1) Post role for SDS team members in People First with candidate interviews in December expected. 2) Log risk for budget release impacting recruitment - Risk #94, #113, #115, and #116.

Within each track, there continues to be coordination and sharing of project tasks. Once the SDS tack fills its open resource requests, there should be adequate redundancy. The Project posted a role for the open SDS positions with candidate interviews expected in December. The Project also logged risks from the ongoing budget amendment delay and the impact it could have on project staffing.

Continue to monitor the Project Team and continue to manage the hiring process to fill the open positions as quickly as the right candidates can be found.

Current

Previous

Trend

G

G

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine if adequate time and resources are assigned to each project phase to gauge whether the Project is sufficiently "resourced" (time, money and staffing) for future work.

The resource levels estimated within the project schedule appear adequate. The project has been able to complete tasks to date within the estimated time and has not had to move critical path activities due to project resource constraints. In areas with open positions, some tasks are showing resource constraints. One example is the vacant OCM Team Work Track Manager position which is currently being filled part time by the Project Director. To date, this has addressed the immediate need, but with a decrease in results.

The project should continue to follow its staffing plan and engage the additional resources as scheduled, and as possible. Note that the vacant OCM position mentioned has had four offers made, the most recent could not be filled because project funding has not been released. If the resources are not onboarded as planned, this could cause a decrease in effectiveness of the overall team.

1) OCM Manager hired by 1/25/2015 or 60 days prior to ITN SSI release. 2) Release RFQ for OCM Support Services on 12/30/2015, contingent upon Project Funding release. 3) Log risk for budget release impacting recruitment #115 and #116.

The resource levels estimated within the Project Schedule continue to appear adequate. The Project has been able to complete tasks to date within the estimated time and has not had to move critical path activities due to project resource constraints. The Project has made plans and added activities to the Project Schedule to address the open OCM needs. Additional delays in onboarding the OCM lead or the OCM Support Services contractor could have a negative impact on project execution.

The Project should continue to follow its staffing plan as best possible based on the reality of the delayed budget amendment and the withholding of funds.

Current

Previous

Trend

Y

Y


November 1-30, 2015

of 66 12/10/2015

3.3 Action Item Management Key Metrics


Previous Status

Notes

Overdue Action Items.

0 out of 9 Action Items, which are more than 1 day past their due date.

Average Action Item Aging (past due).

0 Days None at this time.

Percent of Items Closed less than 1 day after due date.

73%

This includes all Action Items with a due date after 4/8/2015 when the new Project Risk Manager started. Note that no action items closed after 6/30/2015 were late.

Process Observation Summary The Project has a strong discipline around identifying, assigning, tracking, and following up on action items. Each open Action Item is reviewed at each track’s regular (weekly or bi-weekly) RAIDL (Risk, Action Item, Decision, and Lessons Learned) meeting. At the bi-weekly PMO RAIDL meeting, the team reviews all of the open Action Items to validate their status.






Category Status

Verify that Action Items are clearly defined and differentiated from project tasks, issues, and risks.

Action Items are defined in the PMP and the project training material which was distributed with the initial training.

None N/A Action Items are defined in the PMP and the project training material, which was distributed with the initial training.


Current

Previous

Trend

G G

G G

G G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that Action Items are managed in a consistent manner across the Project and are at an appropriate level of detail.

The Action Items entered in SharePoint have a varying level of detail from early in the project, but become much more consistent after 4/8/2015 when the new project risk manager started.

Continue to have the regular PMO track review of open action items. This review ensures that items are consistent in their level of detail between project tracks.

N/A The Action Items recorded during the most recent period have consistent detail.

Continue to have the regular PMO track review of open action items. This review ensures that items are consistent in their level of detail between Project Tracks.

Current

Previous

Trend

Verify that open Action Items are regularly reviewed to ensure that they are being managed, and that if there is a change or larger impact, they are transferred to the Project Schedule where appropriate.

The project reviews open Action Items by track at each of the regular RAIDL meetings. In meetings where items are mentioned, the team is diligent about entering them into SharePoint.

Continue this regular meeting discipline and continue the active review of the Action Items to ensure they are added to the project schedule where appropriate.

N/A The Project continues to review open Action Items by track at each of the regular RAIDL meetings.

Continue this regular meeting discipline and continue the active review of the Action Items to ensure they are added to the Project Schedule where appropriate.

Current

Previous

Trend

Verify that metrics are in place and reviewed on a regular basis regarding the status and closure of Action Items.

The project currently tracks action items which are open and reviews on a regular basis. There are no separate metrics tracked.

Identify Action Item health metrics such as percent closed by due date and average aging, and add these to regular (monthly or quarterly) internal project review processes.

Include Action Item metrics as part of the QMS.

The Project currently tracks open action items and reviews them on a regular basis. There are no separate metrics tracked. Additional metrics will be reviewed as part of the implementation of the QMS.

Identify Action Item health metrics such as percent closed by due date and average aging, and add these to regular (monthly or quarterly) internal project review processes.

Current

Previous

Trend

G

G

G

G

Y

Y


November 1-30, 2015

of 66 12/10/2015

4 Procurement and Contracts

4.1 Procurement and Vendor Management Key Metrics


Previous Status

Notes

% Contracts on Schedule.

100% Current contracts are on schedule for deliverable or hourly service delivery.

% Contracts on Budget.

100% Current contracts are within planned budget.

Contract Execution on Schedule.

Yes Current planned procurements are on schedule.

Process Observation Summary Procurements and Vendor contracts are managed through the PMO. The Project has two designated procurement professionals who manage the procurements from the initial strategy, to contract execution, and through contract closeout. There have been no significant changes to the procurement processes during the reporting period.

G G

G G

G G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine if procurement has explicitly mapped the procurement objectives and strategy to the program objectives, and if such mapping accurately reflects the procurement strategy being followed.

The PMO is currently finalizing operationalizing the Procurement Management elements of the PMP providing the guiding principles while leveraging the DFS Contract Management Lifecycle Guide. As part of this process, a specific Procurement Strategy Document is created to guide each individual procurement. Thus far, Contract services have been procured through the IT and Management Consulting state term contracts.

Finalize content map and continue to adhere to the established and statutory competitive procurement protocols, rules and guidelines.

N/A The PMO continues to manage procurements following the PMP and execute each Work Track’s Procurement Strategy as outlined.

Finalize the mapping of project objectives to the procurement strategies and continue to adhere to the established and statutory competitive procurement protocols, rules and guidelines.

Current

Previous

Trend

Determine if solicitation documentation clearly articulates measurable project objectives, and clearly identifies what (if any), project objectives are the responsibility of the Contractor.

Solicitations clearly define the roles, responsibilities of the vendors and contractors, either through definition of fixed price deliverables, or the definition of roles.

None. N/A No new procurements were executed during the reporting period.


Current

Previous

Trend

G

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine if the solicitation documentation provides adequate detail on the project functional requirements, and if the solicitation process offers bidders, an adequate opportunity for dialogue on functional requirements so that questions, ambiguities and misunderstandings can be rectified prior to contract award.

The procurement process followed to date includes posed requirements, an allowance for a question and answer period, additional time for vendors to review the answers, and oral presentations if necessary. During contract finalization, the program negotiates the scope and needs which best fit PALM, tightly documenting all revisions and maintaining transparency in the process.

None. N/A The plans for the ITN include time for questions and dialog to enable bidders to gain a full understanding of the requirements. No other procurements were worked during the reporting period.


Current

Previous

Trend

Determine whether key project risks have been shared with potential Contractors. Determine whether significant drivers of risk (e.g., organization and business process change, limitations in number and skills of agency resources, number and complexity of legacy systems to be replaced, etc.) to the potential Contractors have been explained in such a way that bidders can address and mitigate them in their bid.

Known risks have been identified in previous bid documents and through the Q&A process. Once vendors are engaged by the project, they participate in the ongoing risk and issue review process for their related work tracks.

None. N/A The plans for the ITN include time for questions and dialog to enable bidders to gain a full understanding of the requirements. No other procurements were worked during the reporting period.


Current

Previous

Trend

G

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine whether documented evaluation processes exist and are being consistently followed.

The Procurement Team reviews the procurement strategy and documents evaluation criteria for each procurement based on best value and the necessary skillsets for the work to be performed. When formal independent responses are provided, references have been validated for each contract. The General Counsel’s office is actively involved; outside counsel is available when needed. The Final step is Sponsor review.

None. N/A The BPS and Procurement Teams are planning the evaluation process for the ITN. No other procurements were worked during the reporting period.


Current

Previous

Trend

Determine whether the Evaluation Team has received training and ongoing guidance necessary for them to evaluate bids in an effective and consistent manner.

Evaluation staff for bids is chosen based on individual knowledge of the functional and technical processes. As needed, coordination sessions have been held prior to evaluation of the bids.

None. N/A No procurements were worked during the reporting period.


Previous

Trend

G

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine if matters that pose a risk to project scope are promptly escalated to the appropriate parties, including Contracts.

As critical decisions and scope items were identified on the project, they have been documented and managed either as project changes or project decisions. After a change has been approved, the vendor implications on schedule and budget have been addressed.

None. N/A As critical decisions and scope items for the ITN are raised, they have been documented and managed either as project changes or as project decisions. No other procurements were worked during the reporting period.


Current

Previous

Trend

Determine if invoice review processes are established.

The project team established and is following a process whereby invoices are reviewed and approved by the Contract Manager, who verifies that the services and deliverables on the invoice have been received and appropriately approved.

Formally document the invoice review process.

1) Provide DFS Contract Management Life Cycle Guide (referencing Section 7.4). 2) Provide Reference Guide for State Expenditures published by the DFS Division of Accounting & Auditing.

The Project is following the DFS Contract Lifecycle Guide and the guidelines published by the DFS Division of Accounting & Auditing for invoice review


Current

Previous

Trend

G

G

Y

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine if the defined quality assurance and deliverable approval processes are being followed in the review and acceptance of all contract deliverables.

The approval process is clearly defined, and has been followed. The quality assurance processes are still under development. See additional detail in sections 5.1 and 5.2 below.

Complete the QMS Strategy which is in process in collaboration with the IV&V vendor and roll out the resulting policies, procedures and tools as quickly as possible.

1) Provide Florida PALM Project Deliverable Review form in support of quality assurance activities. 2) Provide DFS Contract Life Cycle Guide (referencing Section 7.4) in support of quality assurance activities. 3) Incorporate quality assurance process updates identified by the QMS.

Deliverable review and approval processes are being followed for all contract deliverables. The QMS Implementation Plan will include additional detail for the deliverable review and evaluation processes.

Complete the pilot and rollout of the QMS deliverable review processes as outlined in the QMS Implementation Strategy.

Current

Previous

Trend

Evaluate Contractor change request and defect tracking processes.

To date, contractors have put forward no material change requests. Deliverable defect tracking is inconsistently managed, and some vendors have had significant rework.

Establish a management and review process for large content deliverables to facilitate and support interim reviews and partial work product approval to prevent delays in reviewing and performing repetitive rework on large documents.

Design and implement deliverable evaluation metrics as part of the QMS Implementation Plan Rollout is consistent with the approved QMS Implementation Strategy.

To date, Contractors have put forward no material change requests. Deliverable defect tracking is managed for individual deliverables, with all defects addressed prior to deliverable approval. The QMS Implementation Strategy references specific deliverable performance metrics, which are under development as part of the QMS Rollout.

Complete the pilot and rollout of the QMS deliverable review processes as outlined in the QMS Implementation Strategy.

Current

Previous

Trend

Y

G

Y

Y


November 1-30, 2015

of 66 12/10/2015

5. Quality Management

5.1 Quality Management Key Metrics


Previous Status

Notes

Vendor Service Quality Scores.

N/A N/A N/A Not yet tracked but will be rolled out with QMS.

Deliverable Quality Scores.


Project Quality Improvement Index.


Process Observation Summary A high-level approach to Quality Management is outlined within the PMP. During November, the Project finalized the QMS Implementation Strategy and began work on the QMS Implementation Plan. The Project will perform more detailed planning for the QMS Rollout and begin developing QMS policies, procedures, and metrics in December.






Category Status

Evaluate and make recommendations on the Project’s Quality Assurance Plan, procedures and organization.

The project has a high level Quality Management Plan outlined in the PMP. It includes a high-level outline of the deliverable quality review process.

Complete the QMS Strategy which is in process and roll out the resulting policies, procedures and tools as quickly as possible.

1) Support discussions with IV&V Contractor to finalize QMS Implementation Strategy and Implementation Plan (Deliverable IV&V1). 2) Once approved, implement QMS Strategy.

The QMS Implementation Strategy has been finalized and the Project Team is working with the IV&V Team to develop and roll out the QMS.

Complete the QMS Implementation Plan and manage the progress of the QMS development against the plan.

Current

Previous

Trend

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that project self-evaluations are performed and that measures are continually taken to improve the process.

Self-evaluations are not being performed.

Create and implement evaluations as part of the QMS.


Self-evaluations were included in the approved QMS.

Continue to implement the QMS including the initial self-evaluation planned for winter 2016.

Current

Previous

Trend

Verify that QA has an appropriate level of independence.

The program has an internal QA/ Risk manager and has engaged an IV&V vendor with services including the development, rollout and monitoring of the project’s QMS.

None. N/A QA is monitored by the QA/Risk Manager and Grant Thornton as the IV&V vendor. During the review period, both maintained appropriate independence from development of project deliverable content.


Current

Previous

Trend

Y

G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that all process definitions and standards are complete, clear, up-to date, consistent in format, and easily available to project personnel.

Initial deliverable review and quality processes are outlined within the PMP. The QMS Process is still at the strategy level. When it is complete it will include: the following elements:

Deliverable Tracking

Deliverable Expectation Document (DED)

Deliverable Development

Quality Control Approval

Complete the QMS Strategy which is in process and roll out the resulting policies, procedures and tools as quickly as possible


The QMS Implementation Strategy was completed and approved. The Project is in the process of planning its execution which will include:

Deliverable Tracking

Deliverable Expectation Document (DED) Development

Deliverable Development

Quality Control

Deliverable Review and Approval

Continue to implement the QMS including the full deliverable lifecycle development and review process.

Current

Previous

Trend

Assess whether the roles and responsibilities are clearly defined and responsible personnel are identified related to quality assurance and deliverable approval processes.

Reviewer roles are defined in the PMP and have been identified prior to submission of each deliverable.

None. N/A Deliverable review responsibilities continue to be clearly defined and communicated to appropriate Project Team members.


Current

Previous

Trend

Verify that the QA organization monitors the fidelity of defined processes in all phases of the Project.

Processes are still being defined.




Continue to implement the QMS including the monitoring of project processes.

Current

Previous

Trend

Y

Y

G

G

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that the quality of products produced by the Project is monitored by formal reviews and sign-offs where appropriate.





Continue to implement the QMS including the monitoring of project processes and approvals.

Current

Previous

Trend

Verify that guidelines for deliverable and work product rework are clearly defined.





Continue to implement the QMS including processes for tracking defects and monitoring rework.

Current

Previous

Trend

Verify that relevant Stakeholders are made aware of the results of evaluations and quality trends in a timely manner.





Continue to implement the QMS including reporting of quality evaluations.

Current

Previous

Trend

Y

Y

Y

Y

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that process and product quality assurance activities are recorded in sufficient detail so that status and results are known.





Continue to implement the QMS including documentation and tracking of metrics, findings, and action plans.

Current

Previous

Trend

5.2 Deliverable Management Key Metrics


Previous Status

Notes

Schedule changes due to deliverable development or review.

2

There have been two schedule changes to enable more review and processing time for a deliverable after the majority of the work on the deliverable was complete.

Process Observation Summary The Project follows a structured deliverable review process. Key elements of that process include that the development and review steps for each deliverable are included in the Master Project Schedule, and that a Deliverable Expectations Document (DED) is created for each, defining the scope, content, and review process for the deliverable. A review committee of Project Team staff and Sponsors is identified and assigned to each deliverable, and upon completion of the review and edit process, makes a recommendation for approval to the Project Director, who approves each deliverable.

Y

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Ensure there is a clear definition of project deliverables and that deliverables are managed in a consistent manner.

The project currently has a defined process for contractor deliverable review, but this process is not consistently applied to internal (State responsible) project deliverables.

More clearly define internal project deliverables and follow a more formal process, including the development and approval of DEDs for all identified project deliverables.

1) Support discussions with IV&V Contractor to finalize QMS Implementation Strategy (Deliverable IV&V1) including a more robust procedure for DEDs. 2) Once approved, implement QMS Strategy.

The Project currently has a defined process for Contractor deliverable review, but this process is not consistently applied to internal (State responsible) project deliverables. The approved QMS Strategy includes establishing clear definitions for project work products and deliverables.

Include deliverable definitions as part of the detailed QMS policies and procedures.

Current

Previous

Trend

Verify that the defined deliverable review process is consistently administered across the Project.

Deliverable scheduling and progress monitoring is consistent across the project. Use of the DED has not been fully applied to the deliverable review process, with significant focus during the review being on original contract items instead of the DED.

Make the DED process more robust to include and embody the final agreed upon acceptance criteria for each deliverable. In future contracts, particularly the SSI contract, ensure language is included to clearly establish the DED review process so there is no confusion between what is outlined in the DED and the contract. Establish a management and review process for large content deliverables to facilitate and support interim reviews and partial work product approval .

1) Support discussions with IV&V Contractor to finalize QMS Implementation Strategy (Deliverable IV&V1) including a more robust procedure for DEDs. 2) Once approved, implement QMS Strategy.

The QMS Implementation Strategy has been finalized and the Project Team is working with the IV&V Team to develop and roll out the QMS. The implementation of the QMS will include detailed DED and deliverable review processes.

Include a formal deliverable review process as part of the detailed QMS policies and procedures. Include specific guidance for large, complex and multipart deliverables.

Current

Previous

Trend

Y

Y

Y

Y


November 1-30, 2015

of 66 12/10/2015

5.3 Lessons Learned Management Key Metrics


Previous Status

Notes

Lessons Learned Application.

Process in place

The current lessons learned process outlined in the PMP and in use by the Project collects lessons learned, but does not index them to readily allow a proactive search of items relevant to a specific project activity or work track.

Process Observation Summary ‘Lessons Learned’ are defined in the PMP as any useful information or experience gained through the course of the Project that can be applied to a later phase or project activity. Currently, only lessons learned, which have a profound impact on the track are being captured. The Project has a good discipline around identifying, and documenting findings. Significant Lessons are entered into the Project SharePoint portal. New or active Lesson Learned are reviewed at each track’s regular RAIDL (Risk, Action Item, Decision, and Lessons Learned) meeting. At the bi-weekly PMO RAIDL meeting, any new lessons learned items are reviewed.






Category Status

Verify that lessons learned, which could improve processes have been identified.

Lessons learned are documented in SharePoint by the tracks as they are identified and discussed at the regular RAIDL meetings. There is not a clear process for ensuring the lessons are shared between tracks.

Update the lessons learned portion of the RAIDL meeting and SharePoint record to include the track(s) and project activities to which the lesson learned would apply. This will facilitate the ability for the tracks to review applicable lessons learned at the right time during the project.

N/A Lessons learned are documented in SharePoint by the tracks as they are identified and discussed at the regular RAIDL meetings. There is not a clear process for ensuring the lessons are shared between tracks. (No change from initial assessment)

Update the lessons learned portion of the RAIDL meeting and SharePoint record to include the track(s) and project activities where the lesson learned would apply. This will facilitate the ability for the tracks to review applicable lessons learned at the right time during the Project.

Current

Previous

Trend

G G

G

G


November 1-30, 2015

of 66 12/10/2015






Category Status

Verify that lessons learned are managed in a consistent manner across the Project and are at an appropriate level of detail.

While a standard SharePoint tool is used to track the lessons learned, differing levels of detail are captured depending on the track and person capturing the record.

Consider making the documentation of the lesson learned more consistent as part of the review and sharing process.

N/A While a standard SharePoint tool is used to track the lessons learned, differing levels of detail are captured depending on the track and person capturing the record. (No change from initial assessment)

Consider making the documentation of the lesson learned more consistent as part of the review and sharing process.

Current

Previous

Trend

Verify that project or phase closeout reports are created to provide a concise evaluation of the Project. The closeout report should summarize at a high level, the Project objectives, variances or deviations from the plan/baseline encountered, lessons learned and next steps.

Lessons Learned is a component of the Project Management Plan and is being maintained as a part of the Risks, Action Items, Issues, Decisions and Lessons Learned (RAIDL). Close out is a part of the process, and any lessons needing action, can be formally requested to be turned into Action Items.

None. N/A No project phases were closed during the reporting period.


Current

Previous

Trend

G

G

G

G


November 1-30, 2015

of 66 12/10/2015

6. Communications and Outreach

6.1 Communications Management Key Metrics


Previous Status

Notes

Awareness Rating N/A

A formal awareness metric is under development as part of the QMS and Communications Plan. The formal process for collecting and measuring the public and agency awareness of the Project is measured at an event level, not at the overall project level, and while awareness can be extrapolated from event data, it could miss larger –scale agency issues if there is only a limited audience participating in project events.

Awareness Rating Trend

N/A N/A N/A The Project is informally tracking agency awareness, but has not yet developed a specific measure for agency awareness that can be tracked over time.

Process Observation Summary The Project began to update the Communications Management Plan in the month of November, pulling together individual elements which had been separately developed by the OCM Team into a single cohesive plan. The plan includes the expected elements; however, specific measures for agency participation are still under development.

Y Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine whether the Project Communications Plan identifies the correct audiences, contains approaches and strategies to maximize receipt of the targeted messages, and contains effective feedback mechanisms.

The communications plan included in the PMP outlines the major planned communications, their audiences and frequency. Feedback collection processes (surveys, meeting feedback forms, etc.) are in process to solicit input on project activities. In addition, many of the elements of a communication plan are in place, but have not been aggregated into a compressive document.

Revise and update the communications plan to include planned processes and timing for soliciting feedback on communications. Update the plan to include additional communications channels and vehicles identified as necessary from the revised stakeholder assessment.

Develop Pre-DDI Communications Plan (which includes current activities) and route for approval.

The Project is updating the Communications Plan to include the individual communications elements which were previously managed in individual records.

Revise and update the Communications Plan to include planned processes and timing for soliciting feedback on communications. Update the plan to include additional communications channels and vehicles identified as necessary from the revised Stakeholder assessment.

Current

Previous

Trend

Determine if the plan is being successfully executed, observing the plans 'in action'.

The current communications plan is being executed in a thorough manner; the planned communications are being created with appropriate content and delivered as scheduled.

None. N/A The current Communications Plan is being executed in a thorough manner; the planned communications are being created with appropriate content and delivered as scheduled.


Current

Previous

Trend

Y

Y

G

G


November 1-30, 2015

of 66 12/10/2015

6.2 Stakeholder Management Key Metrics


Previous Status

Notes

Engagement Rating. N/A N/A N/A The Project is not currently tracking a metric for Stakeholder engagement.

Engagement Rating Trend.

N/A N/A N/A The Project is not currently tracking a metric for Stakeholder engagement.

Process Observation Summary The Project proactively manages outreach to Agency Stakeholders through multiple media including website updates, directed communications about the Project and planning and execution of a number of workshop and workgroup meetings. The OCM team tracks participation and monitors engagement in these sessions to identify and proactively engage agencies that may require additional support. Formal outreach to external parities is being managed as needed. An overall Stakeholder assessment could significantly benefit the Project Team and communications.






Category Status

Determine if a Stakeholder assessment has been performed for the Project, and assess the completeness and appropriateness of the targeted audiences and messages.

It appears that an informal stakeholder assessment was performed to identify the communications and target audiences for Communications Plan in the PMP. There is significant confusion and misinformation about various elements of the project across some of the key stakeholder groups such as the Legislature and AST.

Perform a new stakeholder assessment as part of revising the communications plan to ensure that all stakeholders are receiving consistent and accurate information regarding the project.

Update the Communications Plan to include the Stakeholder assessment approach and results.

The Project is in the process of updating the Communications Plan to include a full Stakeholder assessment.

Perform a new Stakeholder assessment as part of revising the Communications Plan to ensure that all Stakeholders are receiving consistent and accurate information regarding the Project.

Current

Previous

Trend

R

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Determine whether the change management plan contains approaches and strategies to maximize adoption, assess and manage resistance, assess and increase awareness and effect a successful project implementation.

Many of the elements of a change management plan and strategy exist throughout the project, but they have not been consolidated into a unified strategy and executable plan.

Develop an overall change management strategy and associated plan to guide OCM activities, particularly with respect to stakeholder outreach and engagement.


The Project is in the process of consolidating the individual communications planning elements into a single Communications Plan.

Develop an overall change management strategy and associated plan to guide OCM activities, particularly with respect to Stakeholder outreach and engagement.

Current

Previous

Trend

Verify that outreach efforts exist to ensure that all Stakeholders are aware of the impending change and know their role in the change process. Verify that communication provides appropriate channels for feedback.

The project is performing proactive communication to impacted stakeholders and using feedback mechanisms such as surveys and email to gauge the effectiveness of the project messages.

Update the communications channels including updating the Florida PALM Project website and leveraging various methods of user experience interactions whether email blasts, newsletters, actions items, meeting notices, surveys, etc. Identify and implement feedback mechanisms for each identified stakeholder group.


The Project is performing proactive communication to impacted Stakeholders and using feedback mechanisms such as surveys and email to gauge the effectiveness of the Project messages. The Project is in the process of including feedback mechanisms and updating the communications channels as part of the updated Pre-DDI Communications plan.

Update the communications channels including updating the Florida PALM Project website and leveraging various methods of user experience interactions whether email blasts, newsletters, actions items, meeting notices, surveys, etc. Identify and implement feedback mechanisms for each identified Stakeholder group.

Current

Previous

Trend

Y

Y

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Review plan to determine how the team plans to measure the key components of change (e.g., awareness, buy-in, etc.).

OCM Team is conducting surveys after workshops for feedback, which is one area of measurement, but the team has not identified other key components. Plans to measure key components of change such as awareness and buy-in are not clearly articulated.

Create and implement additional tools for collecting feedback and measuring the effectiveness of the full breadth of stakeholder engagement activities.


The OCM Team is conducting surveys after workshops for feedback, which is one area of measurement, but the team has not identified other key components. Plans to measure key components of change such as awareness and buy-in are being included in the updated Communications Plan.

Create and implement additional tools for collecting feedback and measuring the effectiveness of the full breadth of Stakeholder engagement activities.

Current

Previous

Trend

Determine whether the Project proactively identifies areas at risk for resistance and effects strategies to manage the potential resistance.

The Risk and Issues related to resistance/change management process and workflows are well documented. Supporting documentation is leveraged via SharePoint with summary details provided in the Risk and Issues sections of the site including strategies and potential resistance.

Please refer to the Risk and Issues section of this report for further recommendations.

N/A The Project continues to document risk and issue information on SharePoint and communicate with the impacted parties.


Current

Previous

Trend

Y

Y

G

G


November 1-30, 2015

of 66 12/10/2015

6.3 Content Management Key Metrics

Item Observation Current Status Status Notes

Compliance with Document Archive Policies by team.

N/A N/A N/A Formal guidelines for document management and archive structure, which would guide this metric, are under development.

Compliance with controlled document processes.

N/A N/A N/A Formal guidelines for document management and archive structure, which would guide this metric, are under development.

Process Observation Summary The Project extensively uses a Microsoft SharePoint collaboration site for the tracking and managing of working documents and artifacts. Version control is enabled in SharePoint, providing the ability to track revisions to documents if necessary. While there is good adoption of the use of SharePoint, the directory structure is different for each Project Track. In addition, the timing and guidelines for formal version control and updates of deliverables and other project documents has not been formally adopted and implemented by the Project.






Category Status

Verify that the Project has a clear strategy for maintaining and managing project artifacts.

The project has a strategy which guides the management of project artifacts including work products, deliverables, and KPIs.

None N/A The Project has a strategy which guides the management of project artifacts including work products, deliverables, and KPIs. (No change from initial assessment)


Current

Previous

Trend

Assess the level of detail and adherence to document and deliverable management policies and procedures.

The OCM team has created and published clear document management standards and naming conventions. There is ambiguity around how to version deliverables and other maintained project artifacts.

Include additional detail in the project documentation and deliverable management guidelines to provide clarity and consistency for maintaining version control of deliverables and any other formally tracked document.

Update project documentation and usage standards for SharePoint.

The OCM team has created and published clear document management standards and naming conventions. There is ambiguity around how to version deliverables and other maintained project artifacts. (No change from initial assessment)

Include additional detail in the Project documentation and deliverable management guidelines to provide clarity and consistency for maintaining version control of deliverables and any other formally tracked document.

Current

Previous

Trend

G

G

Y

Y


November 1-30, 2015

of 66 12/10/2015






Category Status

Assess the consistency and ease of finding and accessing project documentation.

Each team has their own SharePoint file structure, leading to confusion and difficulty in finding documents across tracks.

Complete and operationalize the SharePoint structure which is under development, creating a standardized structure and naming for common elements within each track (e.g. status, work plans, deliverables) and enabling flexibility for track-specific content.

Update project documentation and usage standards for SharePoint.

Each team has their own SharePoint file structure, leading to confusion and difficulty in finding documents across tracks. The OCM team has begun to develop a consistent file structure and guideline for the Project’s content.

Complete and operationalize the SharePoint structure, which is under development, creating a standardized structure and naming for common elements within each track (e.g. status, work plans, deliverables) and enabling flexibility for track-specific content.

Current

Previous

Trend

Y

Y


November 1-30, 2015

of 66 12/10/2015

6.4 Collaboration Management Key Metrics


Previous Status

Notes

Partner Engagement N/A N/A N/A The Collaboration Strategy is still in development. This metric will be defined when it is complete.

Process Observation Summary The Florida PALM Project has proactive identified external partners with whom they will require collaboration to be fully successful. Outreach to a number of these entities has already occurred either through individual project activities or through execution of the Communications Plan. A formal Collaboration Strategy for FY15/16 is under development and will include planned engagement with key external Stakeholders including State Agencies, the FFMIS entities, other Public Sector ERP Implementations, and external interest groups.






Category Status

Determine whether the Collaboration Management Strategy and Plan contains approaches and strategies to engage with the right external entities to support the goals of the Project in a productive manner.

The Collaboration Strategy for FY15-16 is in review.

None; evaluate the strategy when the draft is completed.

Update and finalize the FY15-16 Collaboration Strategy.

The Collaboration Strategy for FY15-16 was adopted by the Project in November. It contains content and strategies for internal and external parties.


Current

Previous

Trend

Verify that the metrics for assessing collaboration partner engagement are appropriate and tracked.

The Collaboration Strategy for FY15-16 is in review.

Establish partner collaboration metrics such as participation and level of engagement as part of the rollout of the strategy.

Update and finalize the FY15-16 Collaboration Strategy.

The Collaboration Strategy for FY15-16 was adopted by the Project in November. Specific metrics still need to be developed.

Establish partner collaboration metrics such as participation and level of engagement as part of the rollout of the strategy.

Current

Previous

Trend

G

G

G

Y


November 1-30, 2015

of 66 12/10/2015

Appendix 1. Project Element Value Assessment and Mapping

Value Map Approach Grant Thornton evaluated the elements of Project Management outlined within the Florida PALM PMP along with additional critical elements included from our methodology. Because risks and issues in some project categories will have a larger impact on the overall project, we prioritized the project elements based on their relative impact to the Project. We evaluated each Project Management process against the following three categories to determine an overall priority for the Pre-SSI portion of the Florida PALM Project. These priorities are likely to change to some degree once the integrator is chosen. Each Project Management process was weighted either ‘high’, ‘medium’, or ‘low’ for each of the following categories, with the average providing the overall prioritization.

Correlation to Outcomes – how directly related is the project area to desired outcomes of the Project. (e.g. scope management has a high correlation; if desired functionality is removed from scope, it will have a significant impact) Note that this category is weighted twice as strongly in the overall prioritization.

Visibility – how much external attention is this Project Management area likely to attract

External Impact – is this an area that has direct impact on external Stakeholders, or where external Stakeholders will have a direct impact on the execution of the Project

Process Values The following table displays the Project Management processes, their relative score in each of the value prioritization categories, and the overall priority value. Going forward, these priority values will be used to weight and prioritize the recommendations identified in the body of this report. Value Prioritization Category

Project Management Process

Co

rre

lation

to

Ou

tcom

es

Vis

ibili

ty

Exte

rna

l Im

pact

Ove

rall

Prio

ritizatio

n

1. Mandated Criteria

1.1 Scope Management and Change Control

1.2 Schedule Management 1.3 Cost Management 1.4 Risk Management 1.5 Issue Management

2. Governance

H H H H

H H M H

H H M H

H M H H

H M L M


November 1-30, 2015

of 66 12/10/2015

Value Prioritization Category

Project Management Process

Co

rre

lation

to

Ou

tcom

es

Vis

ibili

ty

Exte

rna

l Im

pact

Ove

rall

Prio

ritizatio

n

2.1 Project Governance 2.2 Decision Management

3. Project Execution

3.1 Performance Management 3.2 Staffing Management 3.3 Action Item Management

4. Procurement and Contracts

4.1 Procurement and Vendor Management

5. Quality Management

5.1 Quality Management 5.2 Deliverable Management 5.3 Lessons Learned Management

6. Communications and Outreach

6.1 Communications Management 6.2 Stakeholder Management 6.3 Content Management 6.4 Collaboration Management

Prioritization Scale: - Low - Medium - High

H M M H

H H H H

H M L M

M M L M

M M L M

M M H M

M H L M

H M L M

L L L L

L H H H

H H H H

L L L L

M L L M

L M H


November 1-30, 2015

of 66 12/10/2015

Appendix 2. Interviews and Meeting Summary List The following meetings and interviews were conducted in development of this report.

Meeting Type Meeting Type Agencies Represented

Collaboration QMS Deliverable Review with PMO PMO IV&V

Collaboration IV&V Strategy Deliverable Review with PMO PMO IV&V

Collaboration Weekly Meeting with AST and PMO to Review Project Status PMO AST IV&V

Collaboration IV&V Team Meeting with Legislature (Senate) to Discuss Project Status and Release Funding Legislature IV&V

Collaboration IV&V Team Meeting with AST to discuss Governance Structure and Status AST IV&V

Collaboration IV&V Team Meeting with Legislature to Discuss Project Structure and Release Funding Legislature IV&V

Collaboration Review and Discuss Comments to the Updated IV&V Implementation Strategy Draft PMO IV&V

Collaboration Meeting with PMO to Review and Discuss DED Revisions for Final Submission PMO IV&V

Collaboration Review PMO Corrective Action Plan Format and Next Steps for the Initial IV&V Monthly Assessment PMO IV&V

Collaboration BPS Team collaboration with FDLE personnel to review the Grant Level 2 requirements and provide FDLE with additional feedback / guidance

BPS Team FDLE IV&V

Collaboration Review of IV&V Implementation Strategy and QMS PMO IV&V

Interview Florida PALM Project JAC Agency Interview (Discussed agency involvement in and perceptions related to the Florida PALM)

JAC IV&V

Interview Florida PALM Project People First Interview (Discussed agency involvement in and perceptions related to the Florida PALM)

People First IV&V


November 1-30, 2015

of 66 12/10/2015

Meeting Type Meeting Type Agencies Represented

Interview Florida PALM Project DMS Agency Interview (Discussed agency involvement in and perceptions related to the Florida PALM)

DMS IV&V

Interview Florida PALM Project FDLE Agency Interview (Discussed agency involvement in and perceptions related to the Florida PALM)

FDLE IV&V

Interview Florida PALM Project DEO Agency Interview (Discussed agency involvement in and perceptions related to the Florida PALM)

DEO IV&V

Interview Florida PALM Project DOT Agency Interview (Discussed agency involvement in and perceptions related to the Florida PALM)

DOT IV&V

Interview Florida PALM Project Legislative Services Office Interview (Discussed agency involvement in and perceptions related to the Florida PALM)

Leg. Services Office IV&V

Planning / Working

PMO Procurement review of Risks, Actions, Issues, Decisions and Lessons Learned (RAIDL)

PMO - Procurement IV&V


November 1-30, 2015

of 66 12/10/2015

Appendix 3. Project Documentation Reviewed The table below lists some of the project artifacts reviewed by the IV&V team in preparation of this assessment.

Category of Documents Reviewed

Documents Reviewed

Project Reporting Project Weekly Reports

Project Quarterly Reports

Steering Committee Meetings

Project Risks, Action Items, Issues, Decisions and Lessons Learned (RAIDL) Dashboard

SharePoint RAIDL:

Risk Log – 100 risks identified and reviewed

Action Item Log – 247 action items captured and reviewed

Issues Log – 7 issues identified and reviewed

Decisions Log – 85 decisions identified and reviewed

Lessons Learned Log – 76 Lessons captured and reviewed

Project Change Requests Log – 7 PCRs logged and reviewed

FL PALM Document Libraries

Project Standards:

Requirements Review: Library for the requirements review process for DFS Accounting and Auditing, Treasury, and Information Systems.

FL PALM All Site Content All Site Content (Summary view):

Action Item Log - Combined action item list for all work tracks including Procurement.

Announcements - List to track upcoming events, status updates or other team news.

Calendar - List to keep informed of meetings, deadlines, and other important events.

Considerations Log

Decision Log - Combined list of BPS, PMO, & Procurement Decision Logs.

Issues Log - Combined list of BPS, PMO, & Procurement Issues Logs

Lessons Learned - Combined list of BPS, PMO & Procurement Lessons Learned Logs

Meeting Log - Meetings with one or more Sponsors and/or any Non-Project Team members.

Project Change Request Log

Risk Log - Combined list of BPS, PMO and Procurement Risk Logs

Business Process Standardization (BPS) Library Documentation

BPS Documentation:

Schedule – Project Scheduled Baseline and Updates

Status Reports – Weekly Work Track Status & 4 Week Look Ahead Reports to the PMO

Policy Issues – Policy Issue tracking log, DEDs, FLAIR and CMS Reports, etc.

Strategic Plan – BPS 2, 9 and 16 Strategic Plan Deliverables and DEDs.

Level 1 – BPS 4, 9 and 8 Deliverables and DEDs.

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Action%20Item%20Log/AllItems.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Announcements/Current%20View.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Calendar/calendar.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Considerations%20Log/AllItems.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Decision%20Log/AllItems.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Issues%20Log/AllItems.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Lessons%20Learned/AllItems.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Communications%20Log/CurrentMeetings.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Project%20Change%20Request%20Log/AllItems.aspx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/Lists/Risk%20Log/AllItems.aspx


November 1-30, 2015

of 66 12/10/2015


Documents Reviewed

Level 2 – BPS 10 and 13 Deliverables and DEDs.

Requirements – BPS 7 Template and BPS 11 and 20

BPS Lists BPS Lists:

Level 1 Requirements List – Business Process Areas Requirements.

Level 2 Topics – Business Process Area Topics.

Level 1 Policy Issues List – Business Process Area Issues.

Project Management Office (PMO) Documents Library

PMO Documents Library:

Project Change Requests

Project Support Tools

Decision Requests

Project Schedule – Baseline Schedule & updated revisions.

Four Week Look-Ahead – Project Schedule 4-week view.

FTE Reports – Staffing

Meeting Documents – Minutes and Agendas

Milestone Report Archive

PMO Contract Management

PMO Forms and Templates

PMO1 Project Status Reports – Deliverable

PMO2 Project Charter – Deliverable

PMO3 Project Management Plan – Deliverable

PMO4 Project Management Review – Deliverable

PMO5 Weekly Review of Track Project Status Reports

PMO6 Monthly Project Status Reports – Deliverable

PMO7 SSI Project Management Requirements SOW

Weekly Status Reports

FLAIR Study - Final

FLAIR Study Assumptions and Timeline

Procurement Lists Procurement Lists:

Contract Change Order Log

Procurement Log

Organizational Change Management (OCM) SharePoint Main Page

OCM Main Page:

Track Schedule – OCM MS Project Team Schedule

Four Week Look Ahead – 4-week calendar view captured in MS Excel tracking milestones, work products and deliverables.

OCM Libraries OCM Libraries:

Administrative – Folders and documents relating specifically to OCM and administrative business

Ideas – Ideas for Track and project improvement or activities that have not been vetted

OCM Weekly Meetings – Agendas and notes from OCM weekly meetings

Schedule – OCM Track Schedule

Status Reports – Weekly Status Reports

Metrics and Data – Files organized by the event from which the information was gathered including collection method, raw data, data evaluations, and final summary reports

OCM Resources – Papers, presentations, graphics, and other tools and information used by OCM team members to provide service and support to the Project and Tracks (e.g., meeting

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2F1%20%2D%20Project%20Change%20Requests&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2F2%20%2D%20Project%20Support%20Tools&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2F3%20%2D%20Decision%20Requests&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2F4%20%2D%20Project%20Schedule&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FFour%20Week%20Look%2DAhead&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FFTE%20Reports&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FMeeting%20Documents&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FMilestone%20Report%20Archive&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO%20Contract%20Management&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO%20Forms%20and%20Templates&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO1%20Project%20Status%20Reports&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO2%20Project%20Charter&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO3%20Project%20Management%20Plan&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO4%20Project%20Management%20Review&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO5%20Weekly%20Review%20of%20Track%20Project%20Status%20Reports&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO6%20Monthly%20Project%20Status%20Reports&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FPMO7%20SSI%20Project%20Management%20Requirements%5FSOW&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FPMO%2FShared%20Documents%2FWeekly%20Status%20Reports&FolderCTID=0x012000D8FD8E521DA22A41BAF0EC6930561A22&View=%7bE96A01D5-DCC9-4F54-9DF3-491236638CCE%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/FLAIR%20Study%20-%20Final.docx

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/PMO/Shared%20Documents/FLAIR%20Study%20Assumptions%20and%20Timeline.docx


November 1-30, 2015

of 66 12/10/2015


Documents Reviewed

facilitation tools, mapping exercises, and change management literature)

Project and Track Support – Support documents and work products produced in , organized by Project and Track

Strategies – Strategic OCM documents (e.g., Collaboration Strategy, FFMIS Engagement Strategy, and Website Content Strategy

OCM Communication Tools & Log

Communication Tools

A&A Newsletter – Articles and information submitted to A&A for publication in their newsletter

Project Standard Documents – Proposed and accepted documents and work products related to project standards (e.g., Project Style Guide and Word Template)

Website – Communication planning documents for the Florida PALM webpages

OCM Communications Log – History of project communications captured in MS Excel format.

SDS Documents SDS Documents:

Status Reports – Weekly Project Status Reports.

Strategy – Data Management Strategy, SDS Pre-DDI Track Strategy.

Admin – Archive, RFQ, SOWs, etc.

Requirements – Draft Requirements, Technical Category Ratings, Category Crosswalk, etc.

FLAIR Study Report State of Florida feasibility study regarding the replacement of FLAIR and CMS with a “Commercial off the Shelf” (COTS) ERP solution.

FLAIR Study Cost Model State of Florida cost study regarding the replacement of FLAIR and CMS with a “Commercial off the Shelf” (COTS) ERP solution.

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/SDS/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FSDS%2FShared%20Documents%2F1%2E0%20Status%20Reports&FolderCTID=0x0120009DF715F0990F6744BF2266D65B8CF76C&View=%7bBC34BC5B-99C9-454B-9B54-A1E7980859AD%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/SDS/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FSDS%2FShared%20Documents%2F2%2E0%20Strategy&FolderCTID=0x0120009DF715F0990F6744BF2266D65B8CF76C&View=%7bBC34BC5B-99C9-454B-9B54-A1E7980859AD%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/SDS/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FSDS%2FShared%20Documents%2F3%2E0%20Admin&FolderCTID=0x0120009DF715F0990F6744BF2266D65B8CF76C&View=%7bBC34BC5B-99C9-454B-9B54-A1E7980859AD%7d

http://dfsintranet.fldoi.gov/capitol/osp/pmoffice/FCR-PreDDI/SDS/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fcapitol%2Fosp%2Fpmoffice%2FFCR%2DPreDDI%2FSDS%2FShared%20Documents%2FRequirements&FolderCTID=0x0120009DF715F0990F6744BF2266D65B8CF76C&View=%7bBC34BC5B-99C9-454B-9B54-A1E7980859AD%7d

http://www.myfloridacfo.com/Division/AA/140429-DOF05-Final-FLAIR-Study-Combined-v100.pdf

http://www.myfloridacfo.com/Division/AA/140409-DOF05-Final-FLAIR-Study-D5-CH2-Model-v103.pdf