Management Report and KPIs

Report to the General Assembly and Management Board on Q1 2015 Produced April 2015

▪  KPIs ▪ Content ▪  Education ▪ HR ▪ CRM & Collaborations ▪ Delivery ▪  Applications & Architecture ▪ MLDS Affiliate Report ▪ Work Item Status Update

Contents

▪ Updates by Line of Business (each with Q1 Achievements, Q2 Key Objectives, Q3-Q4 Key Objectives, Key Risks) ▪ Content ▪  Education ▪ CRM ▪ Collaborations ▪ Delivery ▪  Applications & Architecture

KEY PERFORMANCE INDICATORS

Content KPIs (slide 1 of 2)

(target: 70% completed) Q4 2014 Q1 2015

Actuals completed 66.30% 62.04%

Content KPIs (slide 2 of 2)

Red projects Q4 2014 Q1 2015

Authoring 30% 0%

Mapping 0% 0%

Education KPIs (slide 1 of 2) Presentations available at end of reporting period

Public accessible presentations Course-specific presentations

4 18

Assessments available at end of reporting period Public accessible assessments Course-specific assessments

2 4

E-Learning development and delivery during reporting period Presentations developed Assessments developed Interactive webinars delivered as part of E-Learning courses

18 4 2

Usage of public E-learning materials during reporting period Public accessible presentation views Public accessible assessments taken Average mark for assessment taken

1507 592 37%

Course enrollment and completion during reporting period Students starting Foundation Course Students currently taking Foundation Course Students successfully completing Foundation Course Drop out/did not successfully complete Foundation Course Students starting Implementation Course Students currently taking Implementation Course Students successfully completing Implementation Course Drop out or did not successfully complete Implementation Course

375 368 0 7 0 0 0 0

Course demand in this reporting period New Foundation Course applicants New Implementation Course applicants

880 0

Unmet course demand at end of reporting period – Students applying for… Foundation Course with confirmed place allocated in next 6 months Foundation Course without a confirmed place in next six months Implementation Course with confirmed place allocated in next 6 months Implementation Course without a confirmed place in next 6 months

412 0 0 0

Education KPIs (slide 2 of 2)

Human Resources KPIs

*End Q figures still include employee if employee did not leave until start of following period.

2014 Q1 Q2 Q3 Q4 2015 Q1

# Members 25 27 27 27 27 # Large Vendor Agreements 0 0 0 0 0 # New Collab Agreements 0 0 2 0 0 Appropriate Representation at External Events

100% 100% 100% 100% 100%

% Survey Results Showing Strong Product/Service Levels

71% (Member survey)*

n/a n/a 98% (Showca

se survey)**

90.5% (Member survey)

Member satisfaction is at least 80% for CRM (annual measurement)

81% n/a n/a n/a 95.2%

Member satisfaction of IHTSDO is at least 75% (annual measurement)

73% n/a n/a n/a 90.5%

Customer Relations & Collaborations KPIs (slide 1 of 2)

2014 Q1 Q2 Q3 Q4 2015 Q1

Customers receive a response to enquiry with confirmation of assignment in 48 hrs

n/a 5h12m avg first respons

e

3h56m avg first

response

5h4m avg first

response

5h26m avg first

response

25% of enquiries are resolved in first contact by assigned agent

n/a 85% 67% 65% 68%

Avg time to resolution of enquiries is 5 working days/one week

n/a 1.4 days

1.27 days

1.5 days 2.75 days

Completing customer engagement plan 75% targets met

n/a n/a n/a n/a On track

Collaboration operational plan 2015/16 – 75% targets met

n/a n/a n/a n/a Plan being updated

Customer Relations & Collaborations KPIs (slide 2 of 2)

SNOMED CT delivered to Members on time (all editions) R=SLA missed by more than 2 working days A=SLA missed by 1 day G=SLA met

Green

SNOMED CT delivered without known defect (all editions) R=Critical defects exist A=Non-critical defects exist G=Zero known defects

Amber

Product release schedule documented and is aligned with content roadmap R=Less than 60% of content projects are aligned with roadmap A=60% to 80% of content projects are aligned with roadmap G=80% of content projects are aligned with roadmap

Green (100% of known projects have been scheduled)

Product lifecycle is documented and products are actively managed R=Less than 60% of content projects are documented and actively managed A=60% to 80% of content projects are documented and actively managed G=80% of content projects are documented and actively managed (Actively managed means product development, maintenance and deprecation processes are in use where applicable)

Green (100% of known products are managed)

Delivery KPIs

Technical - Applications & Architecture KPIs Customer Service (internal & external)

Tickets On time resolution (%)

First contact resolution (%)

Average first response (mins)

Average response (mins)

Average resolution (mins)

Q4 2014 75 70 71 342 265 3,098 Q1 2015 160 78 75 236 205 5,556

Target 85 75 300 240 2880

Technical - Applications & Architecture KPIs Infrastructure Uptime (%)

Target Uptime (%) 99.9

Service Q4 2014 Q1 2015 IHTSDO Collabnet 99.99% 99.99% IHTSDO Daily Build Browser 100% 100% IHTSDO Freshdesk 100% IHTSDO JIRA 99.98% 99.9% IHTSDO Mapping Service 96.14% 94.6% IHTSDO MLDS 99.99% IHTSDO SALSA 99.93% IHTSDO SNOMED CT Browser 100% 99.97% IHTSDO Web Site 99.98% 100% SIRS 98.77% 99.98%

Average 99.35% 99.38%

The downtime on the Mapping tool was mostly planned downtime for upgrades, and necessary patch fixes. This took place generally outside of normal working hours.

Technical - Applications & Architecture KPIs Online Service User Sessions

Q4 2014 Q1 2015 Difference IHTSDO Web Site 13,876 27,332 +97% IHTSDO SNOMED CT Browser 14,273 18,441 +29% MLDS 1,884 IHTSDO Confluence 174 1,863 +971% IHTSDO Mapping Tool 1,479 1,601 +8% IHTSDO Freshdesk Support Portal 602 1,591 +164% SNOMED In Action 1,296 1,523 +18%

Technical - Applications & Architecture KPIs Project Success

Target Live Date

Actual Live Date

Workday variance

Notes

IHTSDO Mapping Tool 25 Jul 2014 05 Aug 2014

8

IHTSDO SNOMED CT Browser 01 Oct 2014 18 Sep 2014

-10

IHTSDO Web Site 27 Oct 2014 27 Oct 2014

1

Member Licensing & Distribution Service (MLDS)

01 Dec 2014 12 Jan 2015

31 Delayed to align with new affiliate license

IHTSDO Refset Mgmt Tool - IHTSDO Beta

19 Dec 2014 31 Dec 2014

9

IHTSDO Refset Mgmt Tool - Members Beta

19 Dec 2014 24 Apr 2015

91 Multiple delivery points, rolled over into 2015 for directory functionality and versioning

Member Licensing & Distribution Service (MLDS) ready for Members

20 Feb 2015 20 Feb 2015

1

Average Variance 19 Target 15

MLDS Affiliate Report 2015 Q1

Work Plan – Work Item Status Update

For Q1 2015: 1) I&E LOB - Document maintenance and updates is reporting as Amber due to some updates not being completed in time for the January release. Resources have been focused on E-learning as a priority. All document updates will be up to date by Q3/Q4. 2) Content LOB - Medical Devices work item which relates to the incorporation of medical devices/GMDN content into the International Release is behind schedule. The timeline was to incorporate the content so that some of it would be available in time for the July release. However, the schedule has been changed so that all the content will be incorporated in the January 2016 release. 3) Tooling LOB - Refset tool is reporting as Amber as it may not meet all the specified requirements of Members. Translation tool is reporting Amber because the development start date has been delayed. Both these tools are included in the same work item and therefore counted as 1 Amber as opposed to two. Red = behind schedule and will not complete in 2015 Amber = behind schedule but will complete in 2015 Green: = on track to complete by 31st Dec 2015

57 58 41

13

3 6 14

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

2014 Q3 2014 Q4 2015 Q1

Green Amber Red

UPDATES BY LINE OF BUSINESS

▪  Functioning and abilities – creation of Expert Reference Group ▪ CMT – authoring completed for some domains

Content – 2015 Q1 Achievements

▪  Completion of authoring for July 2015 International Release ▪  Dentistry – content addition following clinical review ▪  Nursing – addition of nursing procedures content and publication of

SNOMED CT/ICNP linkage table ▪  Publication of updated GP/FP subsets and ICPC-2 map (candidate

baseline and baseline releases) ▪  Consultant Terminologist Program – new intake for 2015 ▪  Delivery of agreed refset/subset strategy ▪  Functioning and abilities – delivery of SNOMED CT concept model for

functioning and editorial guidance ▪  Microbiology reporting – publication of Microbiology reporting subset ▪  Anatomy – response to interim review comments from University of

Washington ▪  Substances – delivery of technology preview for Molecular entity content ▪  Publication of candidate baseline of SCT to ICD-10 map

Content – 2015 Q2 Key Objectives

▪  Completion of authoring for January 2016 International Release ▪  ICD-11 alignment – continuation of internal review, external clinical

review and content addition ▪  Allergy list – content enhancement ▪  LOINC – delivery of technology preview ▪  Functioning and abilities – delivery of content (including technology

preview of proposed model) ▪  Intervention, plans, orders and goals – content development ▪  Pharmaceutical content development – delivery of agreed SNOMED

CT drug model and initial drug content ▪  Substances – delivery of technology previews and initial content for

Molecular entity and substances content

Content – 2015 Q3-Q4 Key Objectives (slide 1 of 2)

▪  Medical devices – delivery of updated content aligned with biannual GMDN release

▪  Orphanet – gap analysis, content addition and delivery of SNOMED CT/Orphanet linkage table

▪  Consultant Terminologist Program – completing certification of 2012/2014 intakes

▪  Anatomy – completion of University of Washington review and updating of content and mapping

▪  CMT – content addition in line with donations received ▪  Publication of baseline SCT to ICD-10 map ▪  Biannual release of SNOMED CT to ICD-10 and SNOMED CT to

ICD-9CM maps

Content – 2015 Q3-Q4 Key Objectives (slide 2 of 2)

▪  No contingency in team capacity to deliver the large number of projects underway

▪  Potential disruption of authoring productivity due to development and delivery of new authoring tool

▪  Lack of clarity for the delivery of a clear process for escalation of editorial issues available internally and externally will lead to uncertainty

▪  Potential lack of required skills within the authoring resource to ensure the team can meet the challenging times ahead

▪  Lack of feedback on the alpha releases and technology previews will lead to uncertainty of the ability to implement the solution

Content – 2015 Key Risks

▪  Foundation Course content complete

▪  Total of 375 people entered Foundation Course ▪  45: since 2015-02-01, others since 2015-03-01

▪  Vast majority making excellent progress

▪  Induction of E-Learning Administrator enabling more effective management of courses and feedback

Education – 2015 Q1 Achievements

▪ Current learners complete Foundation Course successfully

▪ New intake to Foundation Course started

▪  Implementation Course development sufficient to launch course

▪  Start of Implementation Course towards end of Q2

Education – 2015 Q2 Key Objectives

▪  Implementation Course development complete ▪ Revision of Foundation Course based on feedback ▪  Third and fourth intake complete Foundation Course ▪  Implementation Course initial intake completes course ▪ Delivery of initial Content Editing Course (content team

dependent) ▪  Face-to-face education at meetings including SNOMED

CT Expo and other events at request of CRM team

Education – 2015 Q3-Q4 Key Objectives

▪ Delay in completion of Implementation Course ▪ Difficulties meeting demand for appropriate interactive

support and assessment of learners on Implementation Course ▪ Unpredictability of future demand for E-Learning services

Education – 2015 Key Risks

▪ Member budgeting process refined ▪  Prospective Member pipeline updated and Impact

Statements/ targeting financial model developed ▪ Member Survey conducted and recommendations

drafted ▪  Topic-based MF calls initiated ▪  KPIs refined ▪  “SCT in Action” redesign plan developed

Customer Relations – 2015 Q1 Achievements

▪  Finalize regional CRM Action Plans based on the regional roadmaps ▪ Discuss Member survey results with LOBs and integrate

them into work plan ▪  Establish Regional Calendar of Meetings and Events ▪  Initiate project to implement redesign of ”SCT in Action” ▪ Work with LOB’s to simplify documentation for Members ▪ Develop Regional Marketing Plans ▪ Complete Regional Staffing Models, Scope and Costing

Customer Relations – 2015 Q2 Key Objectives

▪ Delivery against agreed regional CRM Action Plans ▪  Additional intelligence gathered on non-Member

countries and maturity level with regards to standards ▪ Restructure and deliver October Uruguay meetings and

conference ▪ Work on ensuring Member knowledge about translation

policy ▪ Continue to refine the process for MF meetings

Customer Relations – 2015 Q3-Q4 Key Objectives

▪  The existing core delivery resources that are mainly centralized in Europe would not be scalable to support Members in Asia Pacific and Americas regions ▪  Insufficient resource capacity to scale up the

engagement as outlined in the vendor engagement strategy

Customer Relations – 2015 Key Risks

▪ New Collaboration agreements drafted with agreed joint work areas (for review by MB and signing in Q2) ▪ DICOM –

▪  Development of a DICOM subset ▪  AMA –

▪  QA of SNOMED CT to CPT maps ▪  Management of licensing ▪  Help with clinical engagement (e.g. SNOMED CT/ICD-11 alignment)

▪ Other key achievements: ▪ HL7

▪  Agreement on licensing statements in HL7 materials ▪  Plan for IHTSDO input on binding and content in FHIR feeding in to

HL7 value set work ▪  Start IHTSDO independent review of TermInfo prior to publication

Collaboration – 2015 Q1 Achievements

▪ Other key achievements ▪  ICN

▪ Equivalence table ICNP-SNOMED CT Nursing Diagnoses ▪ Nursing Interventions (INCP) delivered for editing

▪  ADA ▪ Agreement to develop a new Collaboration Agreement in line

with others and proposal to develop a general dentistry subset ▪ Completion of SNODENT/SNOMED CT alignment

▪ WHO ▪ Sign off 2015 workplan, 2014 report and Comms plan ▪ Start of review of Common Ontology for ICD-11 and delivery

plan ▪  JCG discussions about IHTSDO involvement in AMA/WHO

activities

Collaboration – 2015 Q1 Achievements (cont.)

▪  Agreement sign off ▪  MB review and signing AMA and DICOM collaboration agreements ▪  Signing agreement with INSERM re Orphanet

▪  Agreement drafts and sign off ▪  IEEE and IHE – focused on licensing and agreed usage of SNOMED

CT. ▪  ADA

▪  Joint work activities including: ▪  WHO - Plan for ICD-11 Common Ontology; Management ICD-0

alignment; Methodology for usage validation SNOMED CT/ICD-10 map ▪  Use cases analysis for linking SNOMED CT and GS1 products ▪  ICN – Nursing interventions edited for July 2015 release ▪  Start of work with HL7/FHIR on binding of SNOMED CT

▪  Updated Operational plan for MF and MB review and sign off ▪  Put in place RAG reporting for Collaboration-related Statements of

Work

Collaboration – 2015 Q2 Key Objectives

▪ Release of joint products ▪ GP/FP refset and map to ICPC-2 ▪ Nursing Diagnoses ICNP equivalence table and SNOMED CT

subset ▪  Progress joint works

▪ Orphanet incorporation into SNOMED CT and production of a map (ready for January 2016)

▪ Coordination and operational management of all works agreed as part of Collaboration Agreements ▪ Collaboration activities through JIC and input to ISO.

Collaboration – 2015 Q3-Q4 Key Objectives

▪  Lack of access to clinical specialists for SNOMED CT/ICD-11 may delay progress of the work ▪ Dependence on potential and actual collaborating

partners on both establishing updated/new agreements and undertaking agreed work in a timely manner

Collaboration – 2015 Key Risks

▪  Delivered January 2015 SNOMED CT International Edition ▪  Further developed SNOMED Release Management function

▪  Established product lifecycle management for SNOMED CT & derivative products in all lifecycle phases

▪  Established release schedule for all SNOMED CT 2015 deliverables, aligned with Content Roadmap

▪  Introduced a mechanism to track artifacts required for new product development process

▪  Workbench Replacement Project (Authoring tools to replace Workbench) ▪  Formed team to develop business processes and functional

requirements ▪  Formed SME team to provide input on template/batch authoring ▪  Formed project testing team

Delivery – 2015 Q1 Achievements

▪ Delivery of July 2015 SNOMED CT International Edition ▪ Delivery of in-development SNOMED CT products ▪  Functional support for Workbench replacement project

▪  Phase 1, producing July 2015 SNOMED CT release from Terminology Server

Delivery – 2015 Q2 Key Objectives

▪ Delivery of January 2016 SNOMED CT International Edition ▪ Delivery of in-development SNOMED CT products ▪ Continued maturing SNOMED Release Management

function ▪  adoption of new SNOMED product development process, focus

on IHTSDO Member priorities & ROI ▪ Continued functional support to eliminate Workbench

from toolset

Delivery – 2015 Q3-Q4 Key Objectives

▪ Risk to the quality of SNOMED CT releases: ▪ New staff are inexperienced in our domain ▪ Newly developed software tends to be unstable

▪ Reduced capacity in Delivery team results in a high risk of non-delivery of non-prioritized items. This is because two thirds of the team's technical resources are seconded to the Workbench Replacement Project. Mitigation is to manage scope of work until the team returns to full capacity.

Delivery – 2015 Key Risks

▪ Completed purchase of B2i Snow Owl Terminology Server (TS) ▪  Import of ‘in progress’ authoring data from IHTSDO Workbench

into TS in preparation for July 2013 release of SNOMED CT from the TS ▪  Integration of existing Browser to the IHTSDO Terminology

Server ▪ MVP development on Members Licensing & Distribution Service

(MLDS) live for non-member country affiliate Jan 2015 ▪ Members Licensing & Distribution Service (MLDS) live for

Members use with multi-lingual interface, with Denmark & Portugal first to use in March 2015, with Belgium, New Zealand, Sweden, Estonia and India next in line. ▪  IT Policies published including:

▪ Data Protection, and ▪ Software Development Licensing.

Technical – App & Arch – 2015 Q1 Achievements

▪ Continued roll out of MLDS for Members throughout Q2. ▪  IHTSDO Refset Management Service Members public beta

available for members, and integrated with TS ▪ Release of July 2015 run from the IHTSDO Terminology Server ▪  Beta of Template Authoring Service for IHTSDO authoring team ▪ Continued documentation improvements across all tooling

services ▪  Start of work to plan migration off of Collabnet before end of

2015

Technical – App & Arch – 2015 Q2 Key Objectives

▪  IHTSDO content team will no longer need to rely on using the IHTSDO Workbench by end of Q4. This incorporates version 1.0 of the following: ▪  template authoring ▪ batch authoring

▪ Migration away from Collabnet for both infrastructure and TeamForge well before the end of Q4. This includes: ▪ house cleaning of existing Collabnet content and users ▪ moving to JIRA to replace the current use of trackers ▪ moving to Confluence (current product of choice) for

technical documentation and project collaboration

Technical – App & Arch – 2015 Q3-Q4 Key Objectives

▪  Very tight development and delivery timelines without much room to maneuver (staff illness has already had an impact during Q1 2015) ▪ Costs and timescales affected due to unforeseen

implementation details, especially in areas of new functionality including template based authoring. ▪ Reduced service management capability will impact

services as they are introduced to Members. ▪  As the team is following an open and transparent

development, any mistakes made and lessons learnt are public.

Technical – App & Arch – 2015 Key Risks