managing the business of inefficient oversight: how heavy
TRANSCRIPT
Proprietary & Confidential
1
Managing the Business of Inefficient Oversight:
How Heavy or Light is Your Touch When it Comes to Overseeing Vendors?
Clinical Trial Risk and Performance Management Summit
September 4th, 2019
Proprietary & Confidential
22
SURVEY BACKGROUND
• 32 Questions over 4 sections:1. Company Overview Information
2. Vendor Oversight Practices
3. Internal Oversight Resources
4. Free text (challenges, tools, technologies, future implications)
• Sent to Halloran targeted contact list (not posted for general input)
• Targeted interviews
• Collected responses over 4 weeks
• 40 responses (14.1%); median completion time = 14 minutes
• Survey still open and analysis ongoing
Proprietary & Confidential
33VENDOR OVERSIGHT: YOUR ROLE
(POLLING QUESTION 1)
Which of the following best describes your role?
A. Study team member
B. Other Clinical Operations role
C. Other non-clinical study team member
D. Quality Assurance
E. Vendor Management/Oversight
F. Other
Proprietary & Confidential
88VENDOR OVERSIGHT: CURRENT STATE
(POLLING QUESTION 1)
What is the current state of your organization’s vendor oversight practices?
A. We have an Oversight Plan Template AND implement it well across all of our clinical trials
B. We develop Oversight Plans but don’t follow them.
C. We do oversight and document it, but we do not have a standard Oversight Plan Template.
D. We perform vendor oversight but it is not always documented nor does it follow a standard process
E. We know that we need to do something, but do not know where to start.
F. We do not know what we need to do.
Proprietary & Confidential
99
VENDOR OVERSIGHT: CURRENT STATE
Q: What is the current state of your organization’s vendor oversight practices?
Proprietary & Confidential
1010
VENDOR OVERSIGHT: TOOLS
<100 Employees 100+ Employees
14 16
7 15
11 11
8 12
6 8
4 5
Q: Which of these tools does your organization have to help with Vendor Oversight (check all that apply)?
Proprietary & Confidential
1111VENDOR OVERSIGHT: BARRIERS
<100 Employees 100+ Employees
10 10
8 8
6 8
4 5
3 2
Q: What barriers/challenges do you see with implementing the ICH E6(R2) guidelines around Vendor Oversight (check all that apply)?
Proprietary & Confidential
1212VENDOR OVERSIGHT: RESPONSIBLE FUNCTION
(POLLING QUESTION 2)
What function within your organization is responsible for Vendor Oversight at the trial and/or vendor level (check all that apply)?
A. Centralized Vendor Oversight Group
B. Responsibility resides within each function
C. Clinical Operations
D. Procurement
E. Outsourcing / Vendor Management
F. Other
Proprietary & Confidential
1313
VENDOR OVERSIGHT: RESPONSIBLE FUNCTION
<100 Employees 100+ Employees
17 15
4 8
1 4
0 3
2 0
Q: What function within your organization is responsible for Vendor Oversight at the trial and/or vendor level (check all that apply)?
1 2
Proprietary & Confidential
1515VENDOR OVERSIGHT: TIME COMMITMENT
(POLLING QUESTION 3)
Vendor Oversight (per ICH E6(2)) will require more of my time to manage in the future.
A. Strongly agree
B. Agree
C. Neither agree nor disagree
D. Disagree
E. Strongly disagree
Proprietary & Confidential
1616
VENDOR OVERSIGHT: TIME ALLOTMENT
70.2% Strongly Agree/Agree
65% Strongly Agree/Agree
76.5% Strongly Agree/Agree
Q: Vendor Oversight (per ICH E6(R2)) will require more of my time to manage in the future.
Proprietary & Confidential
1717
VENDOR OVERSIGHT: METHODS USED
<100 Employees 100+ Employees
19 17
17 16
18 14
16 11
17 7
11 13
12 7
12 8
2 0
Q: What types of vendor oversight methods do you employ (check all that apply)?
Proprietary & Confidential
1818
VENDOR OVERSIGHT: METRICS USED
• The following metrics were each listed once: Audit Performance, Change Orders, Co-monitoring Visits, Data Quality, Inactive Sites, LPLV-DBL, Monitor Turnover, Monitoring Plan Compliance, Relationship, Risk Management, Site Activation, Site Compliance, Vendor Meetings
• Non-specific entries not included (i.e. “KPIs”, “SLAs”, etc.)
Q: What do you consider the top 3 most important vendor oversight metrics?
Proprietary & Confidential
1919
VENDOR OVERSIGHT: DATA ACCESS
Q: How easy is it to readily access your patient/operational data?
<100 Employees: 5% (1)100+ Employees: 44% (7)
73% of companies responded that they receive their metric information from the vendor
Proprietary & Confidential
2020VENDOR OVERSIGHT: STUDY MANAGER TIME
(POLLING QUESTION 4)
How much time do you estimate a Study/Trial Manager spends weekly on Vendor Oversight activities for an individual study?
A. < 0.5 day/week
B. 0.5 – 1 day/week
C. 1 - 1.5 days/week
D. 1.5 – 2 days/week
E. 2 – 2.5 days/week
F. 2.5 – 3 days/week
G. 3 – 4 days/week
H. 4 – 5 days/week
I. > 5 days/week
Proprietary & Confidential
2121VENDOR OVERSIGHT: STUDY MANAGER TIME
Q: How much time do you estimate a Study/Trial Manager spends weekly on Vendor Oversight activities for an individual study?
This translates to 10%-60% of a Study
Manager’s time is spent on oversight activities.
Proprietary & Confidential
2323
VENDOR OVERSIGHT: OTHER CHALLENGES
We have sponsor oversight - central team, but then there is study team level - Trial Manager; the study team level is the most challenging due to scope service differences, time to commit, and appropriate experience
Turnover within Vendor organization, retraining of staff.
CROs reluctance to want to have oversight
Implementation of a standard process; Implementing a process that will work across various studies, i.e. not re-inventing the wheel every time we do a study.
Subcontracting by vendors - how to maintain oversight for third party vendors
-Consistency between trials/vendors; - Vendors do sloppy work
Education outside of Clinical Operations; understanding the level of documentation that is "adequate" to pass regulatory inspections.
Organization shifting with the new guidelines fast enough (very slow), findings are arising in QA audits and regulatory inspections, trends and needs to be addressed. Challenges between early and late phase drug trials, template but is geared towards late phase, early has complexities ie oncology. Budget for programmed reports and outputs, vendors and CROs not prepared for this level of work and oversight, too much or too little. Micromanaging vs High level oversight tough balance..
Building consistent, evenly applied tools and templates
Q: What other challenges/concerns has your organization experienced with Vendor Oversight process implementation?
Organizational
Process
Requirements
Tools
Training
Proprietary & Confidential
2424VENDOR OVERSIGHT: EFFECTIVE TOOLS/TECHNOLOGIES
CTMS where all vendor info and metrics get feed into. , graphs, bar charts, are helpful. this is a challenge in a large organization and with multiple vendors, CROs. Preferred vendors list helps but then they are maxed out and overstretched and hard to prioritize with a sponsor gives them all their studies. You loose negotiation on timelines and items slip; CTMS, Spend data, plan templates that are good quality, standardized and risk-based governance model; Having your own CTMS Would love to have CTMS, but have not implemented; CTMS system where we could develop our own metrics rather than going to someone else. Dedicated person to follow up on issues.
Vendor oversight plan, standard reporting metrics that are practical and provide useful data, training of team on a process for effective vendor oversight; Scheduler/tracker for what to review when to generate documentation of completion of tasks (vs. documenting in TCRs and Emails); Oversight trackers used for documentation
Regular detailed meetings with vendors review of every vendor deliverable review of data; Oversight SOP for Sponsor; Joint Quarterly Clinical Quality meetings; TUVA by DDI for RBM; Creating partnerships to improve trust and visibility, plans (joint or sponsor-only), keeping eTMF in-house; monthly check in meetings for governance work well
Data lake, issue management process/tool; CTMS, eTMF, EDC, IVRS, Vendor Portal, and a Sample Management system; eTMF Dashboard with CRO APIs Will implement MediData central monitoring in future
Qlik or Spotfire Dashboards for defined KPIs/SLAs SharePoint Online (or central online file folder) to store metrics, governance materials; Regularly scheduled governance meetings for CROs. Using reverse metrics where applicable to ensure vendor knows we, the sponsor, have skin in the game too. QBRs for other important vendor relationships. Quality Agreements where applicable can be helpful. Have recently been working with AVOCA Quality Consortium and have access to their best practice tools.; Data collection and visualization; Self service analytics tools that do not rely upon IT or specialized vendors for implementation; Data visualization tools; initiate early a risk matrix with triggers and monitoring plan with key data fields to review internal study specific word document with weekly/biweekly updates of 3 parts: overview of study with key time/budget KPI; issues to be resolved (escalation); CRO and site specific KPI
Q: Since ICH E6 (R2) was published, where do you see vendor oversight heading in the future? Do you anticipate any new initiatives or structural changes to accommodate increasing vendor oversight activities?
Systems
CTMS
Documentation
Meetings
Metrics
Proprietary & Confidential
2525
Q: Please add any additional thoughts or comments you may have on vendor oversight that you feel was not addressed but may be relevant
VENDOR OVERSIGHT: ADDITIONAL THOUGHTS
Our organization measures Third Party Supplier Risk Assessment at a corporate, regional, and country level. This is a evolving list of suppliers and helps track and improve oversight practices continuously
More risk-based approaches. More technology applications to have readily available data to mine and trend.
We are working to uplift vendor oversight across the organization and see this approach as a continued evolution to stay in front of requirements as best possible across all necessary vendor categories.
Yes I am currently working on initiatives and any new studies I have started in the last 2 years have an oversight plan, create on my own and shift with updates and priorities.
More structured documentation tools
Implementing risk based oversight, RBM and risk based IR Significant focus on documentation of activities that were previously happening but not necessarily documented, application of a risk based approach to oversight, we are adding significant robustness to our vendor oversight SOPs, plans, tools, and work instructions along with our standard SOPs and plan templates
Risk based oversight. Yes, we are currently working on new initiatives for CRO Oversight and our Risk Based Quality Management.
We are currently working on new initiatives to make this process as efficient as possible.
Time spent over-engineering process and documentation instead of managing the vendor!
More and improved access/ offerings from CROs to obtain real time access to data to view in CRO/vendor platform or internal sponsor owned data lakes/dashboards
More automation
We have always been doing vendor oversight; this is not new. What is new is perhaps a more formal process to document vendor oversight. What would be ideal would be a standardized way to ensure that vendor oversight is being conducted and documented.
Trying to build infrastructure to accommodate. Given our platform we will be looking across sites and across platform not only within study -assume this is the direction regulations will move
Centralized monitoring with monitoring reports and trend analysis. Additional CAPA activities for sites and CROs Fully electronic TMF
Still trying to implement basics
Depends on pipeline
We may choose to do more in house and outsource less.
Yes - activities planned but do not use ICH E6. For device trials, we use ISO 14155, which does not have specific vendor oversight requirements yet.
It must be risk-based. Sponsors can no longer hand-over 100% of any study conduct to a CRO.
I would see it more data driven and risk based
More involved on the Sponsor side. It requires us to know more about what our vendors are doing to meet deliverables and more "behind the scenes" stuff (ie. how they validate their systems, what is their vendor oversight practices, etc) ...things that are not just study-specific per se but business specific.
More budget needed for our central team to manage processes
Vendor oversight will become more important in the future. We are working on new initiatives and structural changes to accommodate consistent vendor oversight
Proprietary & Confidential
2626
Q: What is the current state of your organization’s vendor oversight practices?
VENDOR OVERSIGHT: CURRENT STATE
Having a central sponsor oversight team for supplier relationships is key to help study level teams and controls in place for supplier relationship
Equally so an honest and sincere desire for collaboration between sponsor and CRO.
It's imperative to have a functional senior management oversight contact (or committee if needed) between CRO and sponsor (emphasis on "functional").
Vendor oversight for device trials, per ISO 14155.
We have tried various methods to integrate vendors into sponsor team, making they feel accountable for project but the vendor model doesn't really allow for true integration
Some dashboards and clinical programs have too much data or not emphasize data that will identify issues on compliance for pt safety and data integrity
it would be ideal to be able to quantify the impact of not doing this work earlyInteresting topic. I look forward to seeing the results of the survey.
Industry standard on what is acceptable documentation, with regulatory agency buy-in would be a big help!
It would be great for all of us to work together on how best to address this issue and share best practices, tools, and templates.
Lack of transparency related to CRO internal oversight activities, and documentation of that oversight filed to the eTMF.
Company setting is key. smaller organizations easier to streamline and implement easier than a very large global organization, different layers, roles, SOPs, processes, etc. takes a long time to implement.
Creating and maintaining a solid vendor oversight program is very challenging. It really comes down to the discipline of applying the tools and processes in a consistent manner and keeping the study teams and outsourcing team aligned on the right amount of vendor oversight and when to escalate issues.