An Assessment of the Impact of the NCI Cancer Biomedical
Informatics Grid (caBIG®)
Board of Scientific AdvisorsAd Hoc Working Group on caBIG®
March 1, 2011
Working Group Charge
caBIG®’s mission: creating an information network enabling all constituencies in the cancer community (researchers/physicians/patients) to share data/knowledge.
Charge to the Working Group: Advise the NCI Board of Scientific Advisors on the goals, accomplishments, challenges, and community outreach of the caBIG® program.
Help caBIG® achieve even greater traction in the cancer research community and identify stumbling blocks and areas that will require greater attention in the future development of the program.
caBIG® areas covered by the assessment: (a) clinical infrastructure, (b) analytical tools for the support of discovery based science, (c) research infrastructure, and (d) program administration, contracts management, budget.isors.
2
Working Group Strategy
Meeting with caBIG® leadership and primary contractors on November 2, 2010
16 teleconferences between November 22, 2010 and February 25, 2011
caBIG® constituencies contacted by WG NCI-designated Cancer Centers listed as caBIG® users by
program Leaders of TCGA labs Cooperative clinical trials groups Industry Strong supporters and “constructive critics”
59 Investigators at 46 institutions interviewed3
Institutions InterviewedAmgen Memorial Sloan Kettering Cancer Center University of California at San Francisco
Comprehensive Cancer CenterBaylor College of Medicine/Dan L. Duncan Cancer Center
Novartis University of California at Santa Cruz
Case Western Reserve University Ohio State University Cancer Center University of Chicago Cancer Research Center
City of Hope National Medical Center/Beckman Research Institute
Oregon Health & Science University Cancer Center
University of Iowa/Holden Comprehensive Cancer Center
Dana Farber/Harvard Cancer Center Roswell Park Cancer Institute University of Medicine and Dentistry of New Jersey
Dartmouth Medical School Salk Institute Cancer Center University of North Carolina at Chapel Hill/Lineberger Cancer Center
Duke Comprehensive Cancer Center Sanford/Burnham Institute for Medical Research
University of Pittsburgh Cancer Center
Eastern Cooperative Oncology Group Stanford University University of Southern CaliforniaEuropean Bioinformatics Institute The Broad Institute of MIT University of Texas MD Anderson Cancer
CenterForteResearch The Jackson Laboratory Cancer Center University of Virginia Cancer CenterFox Chase Cancer Center The Wistar Institute University of Wisconsin Madison/Paul P.
Carbone Comprehensive Cancer CenterFred Hutchinson Cancer Research Center Thomas Jefferson University/Kimmel Cancer
CenterVelos
Georgetown University/Lombardi Comprehensive Cancer Center
University of Alabama at Birmingham Comprehensive Cancer Center
Wake Forest University Comprehensive Cancer Center
H. Lee Moffitt Cancer Center and Research Institute
University of Arizona Cancer Center Wayne State University School of Medicine/The Barbara Ann Karmanos Cancer Center
Inova Health System University of California at San Diego/Rebecca and John Moores Cancer Center
Weill-Cornell Medical School
Johns Hopkins University/Sidney Kimmel Comprehensive Cancer Center
4
Findings in Three Main Areas
Creation and management of standards for data exchange and support of community-based software
Impact and track record of caBIG® initiatives and tools Life science/integrative cancer research tools Clinical trial management system Infrastructure tools Community engagement
Program administration, contracts management, and budget
5
Positive Findings
Clinical and bioinformatics support is critical to achieve NCI goals Prevention, diagnosis, and treatment of cancer
Overwhelming agreement on value of caBIG® initial mission Standards for data exchange and interoperability Support of community-based tools supporting them
Support of initial vision by caBIG® leadership
Addresses a gap in current funding mechanisms6
However……
Long-term Vision driven by technology advances rather than progress in cancer prevention, diagnostics, and treatment Build it and they will come
Lack of applications that demonstrate the value of caBIG® to clinical and basic science investigators
Lack of independent scientific oversight (e.g. Peer review, SAB)
Disconnect between caBIG® leadership and cancer research community resulting from heavy use of contractors to manage the program
7
Standards for Data Exchange and Support of Community-based Tools
Considered caBIG®’s main contribution Catalyzed progress in 3 critical areas:Development of community-driven standards for
data exchange and interoperabilityDevelopment, maintenance, enhancement,
dissemination of tools developed by academic researchers
Community dialog on interoperability of clinical and research software tools
8
caBIG® Life Science/Integrative Cancer Research Tools
Many Bench-to-Bench tools with varying development costs: Reactome ($100K) caARRAY ($9.3M); Surprisingly little correlation between investment and adoption
Most tools were not widely adopted by the community
Main reasons cited for lack of adoption: Tools have been re-engineered too often Tools over-designed and ambitious Tools require significant technical knowledge and dedicated
local informatics support/resources Commercial tools available and easier to use Some tools developed for narrow niches
9
Biospecimen Management Tools
caTissue suite widely used in cancer centers
Adoption facilitated by: NCI mandate Limited availability of commercial software Two caBIG® -sponsored user meetings on caTissue deployment Addresses a non-mission critical need for which legacy systems
were not in place
But needs considerable local customization, lacks needed functionality, only rigorously de-identified data, does not support CHTN standards
10
Overall Impact of caBIG® Life Science Tools
Uneven adoption of tools
Limited impact of most tools
More impact from tools initiated or implemented by academic research institutions and developed at lower overall cost
Less impact from caBIG® initiated tools with highest overall costs
Data sharing and interoperability goals far from being achieved. “Bronze compatibility” not sufficient for interoperability.
11
caBIG® Clinical Trial Management System (CTMS)
Tools for managing clinical data were highest priority for cancer centers
Prior to caBIG®, limited support within some cancer centers for advanced informatics tools
Overall development costs of CTMS at least $50M FY 2004-2010 + at least $50M of ARRA money
Few NCI cancer centers use CTMS tools - Commercial tools remain dominant
CTMS tools support I-SPY trial and Duke/China Trial
12
Main Reasons Cited for Not Adopting caBIG® CTMS Tools
Existence of critical legacy systems
Tools incomplete, too generic, overly complex, “buggy” and require workflow modifications
User interfaces require extensive customization
High maintenance costs for unnecessary features
Tools do not link to AdEERS or MedWatch
Interfaces vary across tools Awkward linkages
Tools do not interoperate
13
caBIG® Infrastructure Tools
Standard vocabularies and data models constitute a valuable caBIG ® effort. However: Process takes too long
Process does not take adequate advantage of widely adopted standards
Resulting standards are very complex and change too rapidly
Imaging tools are only viable tools for exchange of image metadata Adoption in commercial workstations and ACRIN trials
14
caBIG® Infrastructure Tools: caGRID
Did not address need perceived by cancer centers
High development cost (~$9M)
Main reasons cited for not adopting Very complex, difficult to maintain, needs resident experts in
Java
No GUI for basic admin/configuring tasks
“Software churn” in grid architecture/tools
Not adequately security tested
Inadequate harmonization of tools into single architecture, tools don’t interoperate well
15
caBIG® Community Engagement
Broad range of activities
Mixed reactions by community
Deployment Lead program and Knowledge Centers generally viewed positively
Annual meetings viewed as “marketing”
Perception that feedback from community not incorporated
16
caBIG® Program Administration, Contracts Management and Budget
caBIG® managed by NCI Center for Bioinformatics and Information Technology (CBIIT) Two main contractors
Booz Allen Hamilton: program management officeSAIC-Frederick: technical operations, software development
Internal Program Oversight Board – CBIIT senior staff advised by contractors
Subcontracting process for caBIG® projects and initiatives does not require concept clearance BAH or SAIC-F issues RFP, evaluates proposals, selects and
monitors subcontract awards
17
caBIG® Program Administration, Contracts Management and Budget
Strong consensus among interviewees Overly complex management structure
Considerable overhead costs: 25 – 30% of total costs, >$60M in FY 2004-10 + more for ARRA projects
Contractors experts in technology, not science
Internal decision-making/funding decisions are not transparent, not peer reviewed
Potential conflict of interest if same contractors set standards and write/market software
Perception that caBIG® favors “in group”
18
Conclusions
Support for clinical informatics tools and algorithmic advances is mission-critical for NCI
Strong community support for original caBIG®
vision and goals caBIG® accomplishments in data standards offset
by limited success in other areas Overall impact not commensurate with level of
investment
19
Conclusions
caBIG® impact compromised by implementation approaches “Cart-before-the-horse” vision
Technology-centric approach to data sharing
Unfocused expansion
One-size-fits-all approach
Unsustainable business model for both NCI and users
Lack of independent scientific oversight
20
Immediate Tactical Recommendations
1. Institute an immediate moratorium on all ongoing internal and commercial contractor-based software development projects while initiating a mitigation plan to lessen the impact of this moratorium on those using existing caBIG® tools.
2. Institute a one-year moratorium on new projects, contracts and subcontracts by caBIG®.
3. Provide a one-year extension on current caBIG®-supported academic efforts for development, dissemination, and maintenance of new and existing community-developed software tools
21
Immediate Tactical Recommendations
4. Establish an independent oversight committee,representing academic, industrial, and government (NCI, NIH) perspectives to review ongoing and planned initiatives for scientific merit and to recommend effective transition optionsfor current users of caBIG® tools.
5. Conduct a thorough audit of all aspects of the caBIG®
budget and expenditures to recover unspent funds for reprogramming to implement the other recommendations and other NCI priorities.
22
Longer Term Strategic Recommendations
6. Create an independent Scientific Advisory Group (SAG) for NCI biomedical informatics efforts and initiatives that includes scientific, technology and informatics expertise to advise NCI on appropriate informatics priorities, initiatives, business model(s), and resource allocations. It may be appropriate that this group be a subcommittee of the BSA.
7. Refocus caBIG® on its original mission and discontinue all strategic efforts to develop and maintain its own brand of software tools, either directly or indirectly through commercial contractor efforts.
23
Longer Term Strategic Recommendations
8. Separate the clinical informatics and bioinformatics components of the caBIG® program.
9. Use usual and established channels and mechanisms for concept clearance and peer review of NCI biomedical informatics initiatives in the future.
10. Promote interoperability and data sharing by making them key review criteria for grant and cooperative agreement applications and R&D contracts and as requirements for awards
24
Working Group Members
Andrea Califano, Dr.*, Columbia University Medical Center (Chair) Arul Chinnaiyan, M.D.*, Ph.D. *, University of MichiganGeoffrey M. Duyk, M.D., Ph.D., TPG BiotechSanjiv S. Gambhir, M.D., Ph.D.*, Stanford UniversityTim Hubbard, Ph.D., Wellcome Trust Sanger InstituteDavid J. Lipman, M.D., National Library of Medicine, NIHLincoln D. Stein, M.D., Ph.D., Ontario Institute for Cancer ResearchJean Y. Wang, Ph.D.*, University of California, San Diego
Executive Secretary: Olivia T. Bartlett, Ph.D.Committee Management Officer: Ms. Claire L. Harris
* BSA Member
25