© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 1
K I T S O L U T I O N S , L L C . L E S S O N S L E A R N E D
A G E N C Y : H H S
P R O J E C T: D ATA I N F O R M AT I O N T E C H N O L O G Y I N F R A S T R U C T U R E
C O N T R A C T ( D I T I C )
Communication and Project Management Lesson Learned from the DITIC Project
Situation
The DITIC (Data Information Technology Infrastructure Contract) and DACCC (Data Analysis
Coordination and Consolidation Center) are two separate but related contracts enabling CSAP
(Center for Substance Abuse Prevention) to meet GPRA (Government Performance Results Act) and
NOMs (National Outcome Measures) reporting requirements, and at the same time, empower State,
community, and grantee prevention organizations to implement SAMHSA’s Strategic Prevention
Framework (SPF) while improving accountability, performance and effectiveness.
This situation significantly increased the complexity of communication between key stakeholders
(DITIC and its GPO, DACCC and its GPO, as well as POs for other programs) which directly affected
the project management.
Challenge
Due to differences in roles, interests, and backgrounds, these stakeholders have different frames of
reference. They speak to each other; however, they do not always communicate. Often, the same
words have different meanings to different stakeholders. They are speaking different “languages.”
The miscommunication, if not well managed, can perpetuate throughout the IT solution design,
development, and implementation processes. We experienced the serious miscommunication or lack
of communication in the first two years of the contract. This resulted in technical redesigns, repeated
engineering work, and delays in system deployments.
Solution
In the third contract year (2009), a retreat was organized by CSAP in Boston. Most team members
from both DACCC and DITIC spent 2 days together to share each other’s work and discuss how
to improve communication and collaboration. Communication protocols and standard operation
procedures (SOP) were established between the two contractors and GPOs. After the retreat, a joint
monthly DACCC/DITIC project meeting with GPOs from both contracts was established. Project
managers from DACCC and DITIC jointly develop meeting agendas, conduct the meeting, and
prepare meeting minutes. In addition, a project management SharePoint website accessible by all
stakeholders was established to post and track project events, decisions, and progress. DACCC and
DITIC also conducted many system training together to address the needs of grantees.
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 2
The improvement in communication made the project management in the last two and a half
years of the contract much more effective resulting in on-time, high-quality system development,
modification, and deployment. At the end of contract period, both contracts and GPOs felt that the
two contractors worked together like one team.
Lesson Learned
Through this experience, we have learned three valuable lessons:
1) Do not make false assumptions on quality of communication. Below are examples:
•Thecustomersknowwhattheywantsotheyshouldprovidedetailedguidanceonwhatshouldbe
done and in which way. (They usually know the general direction and express their desires but have
limited knowledge on how to make it to work. The contractor needs to listen carefully and translate
the direction and desire into concrete conceptual models and technical design.)
•Customersandotherstakeholdersunderstandthetechnicaldesignissuesandoptionsunder
discussion. (They usually have a general sense of the issues being discussed but don’t have in-depth
knowledge of the subject matter. They also interpret the issues and solutions in their own frame
of reference. The better way is to present multiple solutions in concrete forms, i.e. wireframes,
storyboards, and prototypes so they can react to and make an informed selection.)
•Thee-mails(andattachments)sent(orcc)arereadbyeveryoneonthelist(morelikely,onlythose
directly involved read them, others either did not read thoroughly.)
•Everyonereadthee-mailcommunicationandunderstandsthecontent,(realityismorelike,only
those closely connected to the matter under discussion know what you talk about but with their own
interpretation.)
2) As an IT contractor, we must be fully aware of the different “frames of reference” used by
multiple stakeholders and be able to “translate” the words and statements of other stakeholders into
technical designs and requirements to assure that communication to the engineers is accurate and
precise.
3) We need well established and enforceable SOPs and communication protocols among
stakeholders to assure continuous, in-time, and effective communication.
DUNS 834952038
NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420
CCR CAGE CODE 8A3D3
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 1
K I T S O L U T I O N S , L L C . L E S S O N S L E A R N E D
A G E N C Y : H H S
P R O J E C T: D ATA I N F O R M AT I O N T E C H N O L O G Y I N F R A S T R U C T U R E
C O N T R A C T ( D I T I C )
Do the Right Thing Right Lesson Learned from the DITIC Project
The Situation
The Data Information Technology Infrastructure Contract (DITIC) was a milestone effort in supporting
CSAP’s three major goals of building capacity, enhancing effectiveness, and improving accountability.
Achieving these objectives through this contract was significant to the entire field of substance abuse
prevention in several ways:
(1) DITIC served as CSAP’s coordinated prevention knowledge and information dissemination center,
(2) DITIC became the centralized data collection and management mechanism for monitoring and
managing both block grant and other discretionary grants, and
(3) DITIC was the data repository and report generation center for the field of prevention.
In conjunction with DACCC (Data Analysis Coordination and Consolidation Center), DITIC enabled
CSAP to meet GPRA (Government Performance Results Act) requirements, and at the same time,
empower State, community, and grantee prevention organizations to implement SAMHSA’s Strategic
Prevention Framework (SPF) while improving overall accountability, performance and effectiveness.
As DITIC contractor, KIT Solutions inherited 11 legacy data systems from previous vendors. These
systems were developed by multiple IT teams using different generations of technology over the past
decade. Continuing to perpetuate these fragmented systems would have hindered CSAP’s ability to
achieve the contract objectives stated above. Clearly, the right thing to do was to apply state-of-
the-art information technology to integrate these systems and create a new information technology
infrastructure.
The Challenge
After a successful transition of all the systems from the previous contractor, we received lists of
modifications and enhancement requests from the customer (CSAP) for the various inherited systems.
Due to the length of time for the contract transition, every list KIT received was a high priority, the
pressure to immediately start working on these legacy systems was great. We faced a major challenge:
modify and maintain the 11 systems to satisfy the short term program management needs of
individual systems or conduct an assessment of all legacy systems before doing any further work.
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 2
We proposed to take a pause and conduct a comprehensive assessment of all inherited IT assets
and set a vision for the project. SAMHSA’s Government Project Officer (GPO) supported this
recommendation and allowed KIT to take 2 months to conduct the assessment. The assessment
resulted in a 25-page report that established a new vision and a master schedule for the entire 5-year
contract period. The vision was to build a web portal consisting of three integrated services: (1)
Knowledge and Information Service, (2) Data Collection and Management Service, and (3) Data &
Report Service.
The Solution
Based on the recommendations from the Assessment, KIT was allowed to modify the original
DITIC scope of work using the new vision. Guided by the new vision and applied Service Oriented
Architecture (SOA), we completed the system integration by building an integrated web portal
(Prevention Management Reporting and Training System, or PMRTS) with a Single Sign On (SSO)
portal access provider. This new integrated design provides a centralized one-stop internet service
to multi-level access by CSAP administrators, project officers, program evaluators, and grantee
organizations across all program areas with a fresh and exciting interface and navigation design.
Based on the system evaluation results, the11 fragmented IT systems were either replaced or
upgraded to meet the requirements of the new technology. The contents and functionalities of the
legacy systems were re-organized under the new portal. Data collection, prevention information, and
report generation services are intelligently provided to users based on their access privileges and
grants funding streams. This integrated data portal provides data and reports that satisfy the needs of
GPRA compliance, grantee performance management, cross-site program evaluation, and ad hoc data
analysis and reporting.
Lesson Learned
While the Assessment temporarily delayed the system enhancement tasks for two months, the longer
term impact was very positive. CSAP accomplished the information systems integration within 2
years, a desired goal SAMHSA was unable to achieve in the past decade before DITIC. Through this
experience, we have learned that when you do the right thing right, temporary slow is in the long
term faster and better. As a contractor, we must think outside the box and use our best judgment to
support the mission of the customer. When a good idea is adopted by the customer, the customer
and contractor are working as partners. This type of partnership will continually lead to innovative
solutions while generating valuable Return on IT Investment.
DUNS 834952038
NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420
CCR CAGE CODE 8A3D3
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 1
K I T S O L U T I O N S , L L C . L E S S O N S L E A R N E D
A G E N C Y : H H S
P R O J E C T: D ATA I N F O R M AT I O N T E C H N O L O G Y I N F R A S T R U C T U R E
C O N T R A C T ( D I T I C )
Integrate Two Different Quality Improvement Frameworks Lesson Learned from the DITIC ProjectSituation
The objective of Data Information Technology Infrastructure Contract (DITIC) is to support CSAP in
meeting GPRA (Government Performance Results Act) and NOMs reporting requirements and at the
same time empower State, community, and grantee prevention organizations to implement SAMHSA’s
Strategic Prevention Framework (SPF) while improving accountability, performance and effectiveness.
This is consistent with the Strategic Initiative #7: Data, Outcomes, and Quality in SAMHSA’s 2010 –
2014 Plan (Leading Change: A Plan for SAMHSA’s Roles and Actions). The purpose of this initiative
is “Realizing an integrated data strategy and a national framework for quality improvement in
behavioral health care that will inform policy, measure program impact, and lead to improved quality
of services and outcomes for individuals, families, and communities.”
A key task of DITIC is to collect and manage data that can be used to support the implementation of
SPF and meet the national performance measures required by GPRA and NOMs. The data strategy
must be designed to improve quality of services and yield better outcomes.
Challenge
There are two different approaches to the development of a framework for quality improvement.
One is to set up standards, develop measurements, and then collect data based on measurement to
validate performance. GPRA and NOMs are examples of this approach. The idea behind this approach
is that people improve upon what is being measured. The second approach refers to technology
transfer and the implementation of high quality behavioral health services. In this approach, scientific
logic, implementation steps, and expert guidance are embedded in the data management system.
The five-step Strategic Prevention Framework (SPF) is an example of this approach. A growing body
of scientific literature suggests that intended outcomes of evidence-based practices and programs are
reliably achieved only when there is fidelity to the practice or program as it was designed and tested.
The two approaches have different implications to the type and quality of data collected, and in
turn impact future data analysis. Under the first approach (which is most common today), data
collection is simply a documentation of what happened. In the second approach, however, planning,
implementation, and evaluation data are logically connected. The fidelity of program implementation
can be assessed. To a researcher, this type of planned change data is invaluable but very hard to get.
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 2
However, to practitioners, the planned change process may be too cumbersome and impractical. The
challenge is: how to balance the two approaches in an integrated quality improvement framework?
Solution
The MRT (Management Reporting Tool) further developed under DITIC is an effort to integrate the two
different quality improvement frameworks.
With the guidance from our GPO, the KIT team expanded the use of the COMET system used for the
management, monitoring, and evaluation of the Drug-free Community Grantees to develop the MRT
as an over-arching data collection, management, and reporting platform for managing all discretionary
grant programs in support of an integrated quality improvement framework.
First, grantees from all grant programs must periodically submit NOMs and GPRA data using MRT.
An on-line, real-time report submission, review, revision, and approval service is provided by MRT for
CSAP program officers to monitor, review, and accept performance data submissions. This supports
the quality improvement framework of people improve on what is being measured.
Second, the data system for all grant programs under MRT follow the same SAMHSA SPF program
implementation model consisting of 5 steps: Assessment, Capacity, Planning, Implementation,
and Evaluation. An underlying logic model is enforced (i.e. planning must be linked to assessment
results, implementation activities must be connected to goals and objectives) electronically by the
software system. This supports the technology transfer – implementing evidence-based best practices
framework.
Finally, since both program performance data (GPRA and NOMs) and program implementation data
are stored in an integrated database, we are able to provide data extracts of performance data along
with programmatic data to the DACCC team for ad hoc data analysis as well as for accountability and
GPRA reports preparation. At the same time, we are able to provide program implementation data
to CSAP program officers and senior management for on-going grantee performance management.
Grantee organizations themselves can access both program and performance data in real-time for
continuous quality improvement of their services.
Lesson Learned
The recognition of the two different but related quality improvement frameworks was an important
revelation during our DITIC experience. Clearly, there are pros and cons for each approach. The MRT
experience taught us that it was possible to combine the two approaches to have an integrated quality
improvement framework.
We believe that SAMHSA should consider both approaches and apply them where they fit. There are
opportunities for innovation as SAMHSA establishes its national framework for quality improvement
and supports data collection platforms that allow for their integration. For example, to go beyond just
documenting what happened, we can develop intelligent data systems that guide service providers
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 3
DUNS 834952038
NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420
CCR CAGE CODE 8A3D3
to implement evidence-based best practices where they are available. This will allow us to analyze
the difference between planned change using evidence-based approaches for regular prevention
and intervention programs. At the same time, periodical data collection following established
measurements should also be consistently enforced across programs. The framework and supporting
data collection infrastructure should be flexible enough to accommodate both. The integration of both
approaches can maximize the potential for in-depth and meaningful data analysis and data driven
quality improvement in behavioral health service delivery.
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 1
K I T S O L U T I O N S , L L C . L E S S O N S L E A R N E D
A G E N C Y : H H S
P R O J E C T: D ATA I N F O R M AT I O N T E C H N O L O G Y I N F R A S T R U C T U R E
C O N T R A C T ( D I T I C )
Connect Grant Making and Grantee Performance ManagementLesson Learned from the DITIC ProjectSituation
DITIC (Data Information Technology Infrastructure Contract) and its related contract DACCC
(Data Analysis Consolidation and Coordination Center) were managed by The Division of System
Development (DSD) of CSAP (Center for Substance Abuse Prevention). DSD supports prevention
efforts and programs initiated and managed by both the Division of State Programs (DSP) and
Division of Community Programs (DCP) within CSAP with on-line data collection, program
management tools, data analysis, GPRA reporting, and cross-site program evaluations.
The Grant Making Process: begins with either
1. DSP or DCP as they prepare the grant solicitation RFA (Request for Application) for a specific
program;
2. DSP or DCP evaluates and awards grants to successful applicants;
3. DSD (through DITIC) designs, develops, and deploys data collection and management systems
based on the RFP and GPRA compliance requirements. and if needed, cross-site evaluation design by
the DACCC team;
4. DSD (through DITIC and DACCC) trains both grantees (data submission and reporting) and CSAP
POs (grantee performance management) to use the on-line data collection and reporting tool; and
5. DSD verifies and consolidates data, then conducts an analysis for cross-site evaluation, generates
ad hoc reports requested by CSAP management, and prepares accountability and GPRA reports for
OMB and Congress
Challenge
As the DITIC Contractor, we experienced a disconnection between program divisions (DSP and DCP)
and the system development division (DSD). DSD is responsible for standardizing the data collection
for the National Outcomes Measures (NOMs) and GPRA. DSP and DCP, on the other hand, are more
interested in the data collection needs of each specific grant program. As a result of these different
orientations, we spent a prolonged period of time revising the technical design whilst attempting to
reconcile the differences between the two departments. DSD (through DACCC) needs time (usually
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 2
6 months to a year) to design the cross-site evaluation after the RFA is issued. This led to major
system adjustments based on these new data collection requirements resulting in data collection
interruptions and retraining for grantees.
Solution
Working with the GPOs for the DITIC and DACCC contracts, the two contractors jointly developed
a new inquiry questionnaire which contained pertinent questions about the new grant RFA that
potentially affected the data collection system and evaluation design for the program (see attached
form). The form is approved by the GPOs and presented to both DSP and DCP for consideration.
While it has yet to be utilized, DSD, DSP, and DCP agree the form can fill the communication gap and
enable DSD to be engaged early on the grant making process, which in turn will speed up and smooth
out the evaluation design, implementation, and data collection system development and deployment.
Lesson Learned
Grants management has two dimensions: (1) grant making, and (2) grantee performance
management. The primary objective of grant making is to select the most promising recipients
through a fair, competitive process and assure appropriate fiscal spending. This involves RFA
preparation, solicitation and application, review and decision process, award management and
fiscal oversight. The primary objective of the grantee performance management, on the other hand,
is to monitor and assist grantees to effectively implement the grant and achieve the goals of the
grant program. This involves compliance reporting, implementation of best practices, monitoring
performance and measuring outcomes, and finally, data-driven management decisions making to
effectively allocate limited resources.
The program divisions (DSP and DCP) are involved in both grant making and grantee performance
management. However, DSD is primarily involved in supporting program divisions on grantee
performance management. This is the major cause for the disconnection between program divisions
and DSD. A conscious effort to involve DSD in the early stage of program development and RFA
preparation can help resolve this issue.
DUNS 834952038
NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420
CCR CAGE CODE 8A3D3
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 1
K I T S O L U T I O N S , L L C . L E S S O N S L E A R N E D
A G E N C Y : H H S
P R O J E C T: D ATA I N F O R M AT I O N T E C H N O L O G Y I N F R A S T R U C T U R E
C O N T R A C T ( D I T I C )
Balance Timeliness and Quality of Data in Supporting Prevention Program Management Lesson Learned from the DITIC ProjectSituation
As the contractor of the Data Information Technology Infrastructure Contract (DITIC), KIT Solutions
(KIT) is responsible for collecting, managing, and providing quality prevention program and outcome
data and information that enables CSAP to meet GPRA (Government Performance Results Act)
reporting requirements, while empowering State, community, and prevention grantee organizations to
implement SAMHSA’s Strategic Prevention Framework (SPF) and improve accountability, performance
and effectiveness. The users of prevention data include CSAP senior managers, Division directors,
Program Officers, Evaluators, and Grantee organizations.
Currently, grantees submit data on a biannual or annual basis. This raw data is then cleaned and
used for NOMs and GPRA performance reporting and to provide responses to ad hoc requests by
CSAP staff. The data cleaning process is quite extensive involving (1) communication with grantees
to resolve data discrepancies, (2) matching individual surveys across multiple time periods and, (3)
the identification and resolution of duplicate data entry and other data quality issues. Cleaned data
is then consolidated into program master files that reflect all data submissions to date for a given
program. While the data cleaning process served an important purpose in identifying data issues
which lead to the improvement of data quality, this process added a considerable delay to data
analysis and report generation.
Challenge
As a result of the data submission frequency (annual or biannual) and time delay (typically 5 months
after data submission) due to the data cleaning and preparation process, the performance reports are
completed a year after data submission. While these reports are meeting GPRA and NOMs reporting
requirements, they are not always focused on the needs of daily grant program administration and
grantee performance management and monitoring. CSAP senior management and program officers
need programmatic and performance data on-demand in real time or near real time for daily data
inquiries, ad-hoc reports, and decision making.
In addition, the GPRA modernization act of 2010 also requires federal agencies to use current
information technology to collect, manage, and report data more frequently (quarterly instead of
annually) and to increase the use of performance information in program decision-making.
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 2
Solution
To meet this challenge, we worked closely with our GPO and proposed a plan for developing “real-
time data reporting” (RTDR) capacity under DITIC in 2010. This plan had three key components: (1)
change grant program reporting requirements to increase grantee data submission frequency; and
(2) implement electronic data validation at data submission (front end) to assure data quality and
consistency, and (3) enable meaningful data access using web-based, on-demand report generation
and real-time data visualization.
Since the first component will take a longer period of time and involve efforts from many
stakeholders, we made significant progress in developing the second and third components.
•Wesystematicallyenhanceddatavalidationrulesforallon-linedataentrysystems.
•Wedevelopedanddeployedadatauploadingsubmissionvalidationsystem(on-line“scrubber”)
that only accepts validated data.
•Whilethedataisnoterrorfreeforanalysis,arecentdataqualitycheckbytheDACCCteam
indicated great data quality improvement and backend data cleaning has been minimized.
•WebuiltaMDM(MasterDataManagement)toolwhichcontainsperiodicallyupdateddatafrom
SAMHSA’s Grants Management Information System (SGMIS) including up to date grantee information
and project officer assignments.
•WedesignedaDataMarttohousebothsecondarydatasets(NSDUH,SYNAR,etc.),aswellas
subsets of data from our primary data collection systems (COMET, MRT, etc.)
•WedevelopedaGIS(GeographicInformationSystem)baseddatavisualizationplatform(RTDR)
providing instant data access and display for (1) grant funding, (2) program performance, (3) Targeted
Population, and (4) Prevalence Data.
•RTDRwentliveinJanuary2012andisnowprovidingreal-timedataaccessandreportingtoall
users identified above.
Lesson Learned
Timeliness and quality of Data are both critical to data driven decision support. For program
evaluation and NOMs and GPRA performance reporting, the quality of the data out-weighs the
timeliness of the data. However, for program management and grantee performance management,
timeliness is critical to the data being useful and meaningful. The table below presents the dilemma:
While the most desirable result is validated and real-time data, this is often impractical. Validated
but delayed data satisfies academic standards but has little use for on-going management decision
support. No one wants preliminary and delayed data. Preliminary (with enhanced front end data
validation) real time data can be a practical solution to address a critical need for on-going
L E S S O N S L E A R N E D
© 2012 KIT Solutions, LLC | KIT Solutions, LLC | 5700 Corporate Drive, Suite 530 | Pittsburgh, PA 15237 | 412.366.7188 | kitsolutions.net 3
DUNS 834952038
NAICS 511210, 518210, 541511, 541512, 541519, 541990, 611420
CCR CAGE CODE 8A3D3
management decision support. RTDR development under DITIC is an example of such a solution.
L E S S O N S L E A R N E D
Real (or near real) Time Periodical (Delayed)
Validated Most desirable but Impractical Satisfies academic standards but has
little use for on-going management
decision support
Preliminary Practical and addresses a critical Least desirable and nobody wants it
need for management decision
support
Quality of Data Timeliness of Data