kpis and management report - snomed...# new collab agreements 0 2 0 0 1 appropriate representation...
TRANSCRIPT
KPIs and Management Report
Report to the Management Board on Q2 2015
▪ KPIs ▪ Content ▪ Education ▪ HR ▪ CRM & Collaborations ▪ Delivery ▪ Technical Services ▪ MLDS Affiliate Report ▪ Work Item Status Update
Contents
▪ Updates by Line of Business (each with Q2 Achievements, Q3 Key Objectives, Q4 Key Objectives, Key Risks) ▪ Content ▪ Education ▪ CRM ▪ Delivery ▪ Technical Services
KEY PERFORMANCE INDICATORS
Content KPIs (slide 1 of 2)
Request submissions
(target: 70% completed) Q4 2014 Q1 2015 Q2 2015
Actuals completed 66.30% 62.04% 85.79%
Content KPIs (slide 2 of 2)
Red projects Q4 2014 Q1 2015 Q2 2015
Authoring 30% 0% 0%
Mapping 0% 0% 0%
Education KPIs (slide 1 of 3) Presentations available at end of reporting period
Public accessible presentations Course-specific presentations
5 38
Assessments available at end of reporting period Public accessible assessments Course-specific assessments
1 4
E-Learning development and delivery during reporting period Presentations developed Assessments developed Interactive webinars delivered as part of E-Learning courses
20 4 8
Usage of public E-learning materials during reporting period Public accessible presentation views Public accessible assessments taken Average mark for assessment taken
143 243 48%
Course enrollment and completion during reporting period Students starting Foundation Course Students currently taking Foundation Course Students successfully completing Foundation Course Drop out/did not successfully complete Foundation Course Students starting Implementation Course Students currently taking Implementation Course Students successfully completing Implementation Course Drop out or did not successfully complete Implementation Course
521 522 275 99 0 0 0 0
Course demand in this reporting period New Foundation Course applicants New Implementation Course applicants
890 112
Unmet course demand at end of reporting period – Students applying for… Foundation Course with confirmed place allocated in next 6 months Foundation Course without a confirmed place in next six months Implementation Course with confirmed place allocated in next 6 months Implementation Course without a confirmed place in next 6 months
700 0 112 0
Education KPIs (slide 2 of 3)
Education (slide 3 of 3) Helpdesk tickets and response times and resolution rates
Human Resources KPIs
Q2 2015: 2 people leaving (Kathleen Brown and Nilesh Jain), one retirement (Jane Millar), 1 new starter (Louise Jones). Jim Case is not included.
0
5
10
15
20
25
30
35
40
45
50
Q1 2014 Q2 Q3 Q4 Q1 2015 Q2
Number of Departing Employees*
Number of New Employees during Q
Total Number of Employees end of Q
2014 Q2 Q3 Q4 2015 Q1 Q2
# Members 27 27 27 27 27
# Large Vendor Agreements 0 0 0 0 0
# New Collab Agreements 0 2 0 0 1
Appropriate Representation at External Events
100% 100% 100% 100% 100%
% Survey Results Showing Strong Product/Service Levels
n/a n/a 98% (Showcase survey)**
90.5% (Member survey)
n/a
Member satisfaction is at least 80% for CRM (annual measurement)
n/a n/a n/a 95.2% n/a
Member satisfaction of IHTSDO is at least 75% (annual measurement)
n/a n/a n/a 90.5% n/a
Customer Relations & Collaborations KPIs (slide 1 of 2)
2014 Q2 Q3 Q4 2015 Q1 Q2
Customers receive a response to enquiry with confirmation of assignment in 48 hrs
5h12m avg first
response
3h56m avg first respons
e
5h4m avg first
response
5h26m avg first
response
3h21m avg first response
25% of enquiries are resolved in first contact by assigned agent
85% 67% 65% 68% 64%
Avg time to resolution of enquiries is 5 working days/one week
1.4 days 1.27 days
1.5 days 2.75 days 0,94 days
Completing customer engagement plan 75% targets met
n/a n/a n/a On track On track
Collaboration operational plan 2015/16 – 75% targets met
n/a n/a n/a Plan being
updated
Customer Relations & Collaborations KPIs (slide 2 of 2)
SNOMED CT delivered to Members on time (all editions) R=SLA missed by more than 2 working days A=SLA missed by 1 day G=SLA met
Green
SNOMED CT delivered without known defect (all editions) R=Critical defects exist A=Non-critical defects exist G=Zero known defects
Amber
Product release schedule documented and is aligned with content roadmap R=Less than 60% of content projects are aligned with roadmap A=60% to 80% of content projects are aligned with roadmap G=80% of content projects are aligned with roadmap
Green (100% of known projects have been scheduled)
Product lifecycle is documented and products are actively managed R=Less than 60% of content projects are documented and actively managed A=60% to 80% of content projects are documented and actively managed G=80% of content projects are documented and actively managed (Actively managed means product development, maintenance and deprecation processes are in use where applicable)
Green (100% of known products are managed)
Delivery KPIs
Technical Services - KPIs
▪ SLAs are detailed here –
▪ https://confluence.ihtsdotools.org/display/IT/Tooling+Key+Performance+Indicators
Technical Services KPIs Customer Service (internal & external)
Tickets On time resolution (%)
First contact resolution (%)
Average first response (mins)
Average response (mins)
Average resolution (mins)
Q4 2014 75 70 71 342 265 3,098 Q1 2015 160 78 75 236 205 5,556 Q2 2015 103 90 73 146 96.82 2,401
Target 85 75 300 240 2880
Technical Services KPIs Infrastructure Uptime (%)
Target Uptime (%) 99.9
Service Q4 2014 Q1 2015 Q2 2015 IHTSDO Collabnet 99.99% 99.99% 99.85% IHTSDO Confluence 99.98 IHTSDO Daily Build Browser
100% 100% 100%
IHTSDO Freshdesk 100% 100% IHTSDO JIRA 99.98% 99.9% 99.95% IHTSDO Mapping Service 96.14% 94.6% 98.02% IHTSDO MLDS 99.99% 99.93% IHTSDO SNOMED CT Browser
100% 99.97% 100%
IHTSDO Web Site 99.98% 100% 99.97% SIRS 98.77% 99.98% 99.31%
Average 99.35% 99.38% 99.70%
The regular downtime on the Mapping tool was mostly planned downtime for upgrades, and necessary patch fixes. This took place generally outside of normal working hours and had no impact on availability to users
Technical Services KPIs Online Service User Sessions
Q4 2014 Q1 2015 Q2 2015 IHTSDO Web Site 13,876 27,332 43,743 IHTSDO SNOMED CT Browser 14,273 18,441 20,564 MLDS 1,884 1,857 IHTSDO Confluence 174 1,863 1,436 IHTSDO Mapping Tool 1,479 1,601 1,495 IHTSDO Freshdesk Support Portal 602 1,591 1,482 SNOMED In Action 1,296 1,523 2,754
MLDS Affiliate Report 2015 Q2
Work Plan – Work Item Status Update
For Q2 2015: 1) I&E - Document maintenance and updates is reporting as Amber due to some updates not being completed in time for the January release. Resources have been focused on E-learning as a priority. All document updates will be up to date by Q3/Q4. 2) Content - Functioning Project Reporting as Amber as the project is being reviewed to define a new scope to be delivered in 2015. 3) Tooling - Refset tool is reporting as Amber as it is on hold, scope is currently being redefined and external resources being recruited to complete the new scope in 2015. Translation tool is reporting Amber because the development start date has been delayed. RFP for external resource recruitment has been developed to complete scope by end of 2015. Template Based Content authoring is reporting as Red as the focus is now on single concept authoring. Template based authoring will be completed in 2016. 4) Release Management - release service enhancements are reporting as Red and have been rescheduled for 2016. Red = behind schedule and will not complete in 2015 Amber = behind schedule but will complete in 2015 Green: = on track to complete by 31st Dec 2015
UPDATES BY LINE OF BUSINESS
▪ Functioning and abilities – creation of Expert Reference Group ▪ CMT – authoring completed for some domains
Content – 2015 Q2 Achievements
▪ Planning work item delivery for January 2016 International Release ▪ Dentistry – content addition following clinical review ▪ Nursing – addition of nursing procedures content and publication of
SNOMED CT/ICNP linkage table ▪ Publication of updated GP/FP subsets and ICPC-2 map (candidate
baseline and baseline releases) ▪ Functioning and abilities – delivery of concept model for functioning and
editorial guidance based on the Observables project ▪ Microbiology reporting – review of proposed Microbiology reporting
subset prior to publication ▪ Anatomy – updating of content following University of Washington
review ▪ Substances – updating of attribute inheritance and delivery of
substances preview ▪ Drugs – delivery of drug content aligned with model specified and
agreed by Members
Content – 2015 Q3 Key Objectives
▪ Completion of authoring for January 2016 International Release ▪ ICD-11 alignment – continuation of internal review, external clinical
review and content addition ▪ Allergy list – content enhancement ▪ LOINC – delivery of technology preview ▪ Functioning and abilities – delivery of content (including technology
preview of proposed model) ▪ Intervention, plans, orders and goals – content development ▪ Pharmaceutical content development – delivery of agreed SNOMED
CT drug model and initial drug content ▪ Substances – updating of content using substance as a target for
attibute values, delivery of technology preview
Content – 2015 Q4 Key Objectives (slide 1 of 2)
▪ Medical devices – delivery of updated content aligned with biannual GMDN release
▪ Orphanet – gap analysis, content addition and delivery of SNOMED CT/Orphanet linkage table
▪ Consultant Terminologist Program – completing certification of 2012/2014 intakes
▪ Anatomy – updating of content and mapping following the University of Washington review
▪ CMT – content addition in line with donations received ▪ Biannual release of SNOMED CT to ICD-10 and SNOMED CT to
ICD-9CM maps
Content – 2015 Q4 Key Objectives (slide 2 of 2)
▪ Potential disruption of authoring productivity due to development and delivery of new authoring tool, also reduced scope of content additions due to restrictions of new tooling
▪ Lack of feedback on the alpha releases and technology previews will lead to uncertainty of the ability to implement the solution
Content – 2015 Key Risks
▪ Migration to new E-Learning server ▪ More flexible, scalable and lower costs for managed courses ▪ Public access E-Learning services also migrated to new server
▪ Foundation course ▪ 275 people successfully completed course in Q2
▪ 522 people entered course in Q2 ▪ Cumulative intake for year so far: 896
▪ Implementation course ▪ Development delayed slightly but minimal impact on course ▪ 122 applications received from prequalified applicants
▪ 50 will start in July others in October ▪ First ‘pilot’ intake planned will start July 6th
Education – 2015 Q2 Achievements
▪ Foundation course ▪ Current learners to complete course ▪ New intake of 700 people to start 1st September ▪ Optional staff entry to Foundation course opens from 1st August ▪ Applications reopen 24th August … and stays open
▪ Aim to offer everyone a start date within 8 weeks of applying ▪ Implementation course
▪ Development to continuing keep pace with course delivery ▪ Webinar, assignment and assessment delivery at each stage
▪ Content Development Theory course ▪ Development starting with plan to start pilot
intake before end of year ▪ SNOMED CT Expo - Montevideo
▪ Tutorials developed and shared for translation
Education – 2015 Q3 Key Objectives
▪ Foundation course ▪ Revision of course based on feedback ▪ Regular monthly intakes – to match demand ▪ Intakes from July and September complete course
▪ Implementation Course ▪ Development of course materials complete ▪ Pilot intake completion (some learners may be delayed in January) ▪ Second intake starts (70+) ▪ Applications open for more people to join course
▪ Content Development Theory course ▪ Initial ‘pilot’ intake starts at end of year
▪ SNOMED CT Expo - Montevideo ▪ Tutorials delivered
Education – 2015 Q4 Key Objectives
▪ No contingency in team capacity to … ▪ Meet demand to develop and maintain course materials ▪ Provide interactive support and assessments for advanced courses ▪ Provide resilience in the event of unplanned absence of key staff
▪ Difficulties meeting demand for interactive support and assessment of learners on Implementation course ▪ Unpredictability of future demand for E-Learning services ▪ Delays to guidance documentation updates in 2015 due to
efforts to complete E-Learning development and delivery
Education – 2015 Key Risks
▪ Member budgeting process refined and initiated ▪ Prospective Member pipeline updated and Impact
Statements/ targeting financial model developed ▪ Taken part of major regional conferences such as
HIMSS US and eHealth week ▪ Product and services placemat under development ▪ “SCT in Action” redesign project initated ▪ EXPO planning on track ▪ Complete Regional Staffing Models, Scope and Costing ▪ Collaboration hand-over initiated
Customer Relations – 2015 Q2 Achievements
▪ Finalize regional CRM Action Plans based on the regional roadmaps ▪ Finalize Member Budget priorities ▪ Establish Regional Calendar of Meetings and Events ▪ Finalize P&S Placemat ▪ Update NRC Guide ▪ Work with LOB’s to simplify documentation for Members ▪ Develop Regional Marketing Plans
Customer Relations – 2015 Q3 Key Objectives
▪ Delivery against agreed regional CRM Action Plans ▪ Additional intelligence gathered on non-Member
countries and maturity level with regards to standards ▪ Restructure and deliver October Uruguay meetings and
conference ▪ Work on ensuring Member knowledge about translation
policy ▪ Continue to refine the process for MF meetings
Customer Relations – 2015 Q4 Key Objectives
▪ The existing core delivery resources that are mainly centralized in Europe would not be scalable to support Members in Asia Pacific and Americas regions ▪ Insufficient resource capacity to scale up the
engagement as outlined in the vendor engagement strategy ▪ Non-payment of member fees
Customer Relations – 2015 Key Risks
▪ Delivered January 2015 SNOMED CT International Edition ▪ Delivered April 2015 Spanish Edition. ▪ Delivered July 2015 International Edition to IHTSDO Members on
June 30. ▪ External contractor tested Spanish Edition ▪ First external use of Quality Validation Framework API
▪ Contractor validated the Spanish Edition before delivery to IHTSDO
Delivery – 2015 Q2 Achievements
▪ Delivery of in-development SNOMED CT products
Delivery – 2015 Q3 Key Objectives
▪ Delivery of January 2016 SNOMED CT International Edition ▪ Delivery of Spanish Edition, October 30. ▪ Delivery of in-development SNOMED CT products ▪ Continued maturing SNOMED Release Management
function ▪ adoption of new SNOMED product development process, focus
on IHTSDO Member priorities & ROI
Delivery – 2015 Q4 Key Objectives
▪ Risk to the quality of SNOMED CT releases: ▪ New staff are inexperienced in our domain ▪ Newly developed software tends to be unstable
▪ High number of in-flight content development projects requiring tooling that is currently unavailable. ▪ Reduced capacity in Delivery team results in a high risk
of non-delivery of non-prioritized items. This is because two thirds of the team's technical resources are seconded to the Workbench Replacement Project. Mitigation is to manage scope of work until the team returns to full capacity.
Delivery – 2015 Key Risks
▪ Members Licensing & Distribution Service (MLDS) live for more Members use with multi-lingual interface, including Portugal, Denmark, Belgium, New Zealand, Sweden, Estonia, India, Malta, Chile, and Uruguay.
▪ External IT Audit completed and recommendations made (see deck in MB pack for detail) – no major gaps
▪ Re-organization of Technical Services for better governance to include release management, service management and operational IT.
Technical Services 2015 Q2 Achievements
▪ Continued roll out of MLDS for Members throughout Q3, including possibly Hong Kong, China and Singapore. ▪ Continued documentation improvements across all tooling
services ▪ Trial migrations from Collabnet TeamForge to Confluence
completed before end of Q3 ▪ Redesign and enhancement to SNOMED In Action site ▪ Phase 1 of Single Concept Authoring tool complete and tested
by IHTSDO authors ▪ Restart of Reference Set Management Tool
Technical Services 2015 Q3 Key Objectives
▪ IHTSDO content team will no longer need to rely on using the IHTSDO Workbench by end of Q4. This incorporates version 1.0 of the following: ▪ Single concept authoring ▪ Beta of new integrated Content Request Services
▪ Migration away from Collabnet for both infrastructure and TeamForge well before the end of Q4. This includes: ▪ house cleaning of existing Collabnet content and users ▪ moving to JIRA to replace the current use of trackers ▪ moving to Confluence (current product of choice) for
technical documentation and project collaboration
Technical Services – 2015 Q4 Key Objectives
▪ Very tight development and delivery timelines without much room to maneuver (staff illness has already had an impact during Q1 2015) ▪ Costs and timescales affected due to unforeseen
implementation details, especially in areas of new functionality including template based authoring. ▪ Reduced service management capability will impact
services as they are introduced to Members. ▪ As the team is following an open and transparent
development, any mistakes made and lessons learnt are public.
Technical – App & Arch – 2015 Key Risks