IT Metrics/Dashboards at Duke:

Curation, Automation, Aggregation?

CSGJanuary 11, 2012

2

Background Initiated metrics effort in 2008 (1 FTE dedicated)

Aligned with finance initially, now integrated with service management team

Focused on availability, capacity, service usage/demand, internal resources

Consult with units on what metrics they should capture Help with collection and analysis of metrics Ensure consistent, universal reporting of data

3

Initial scope and progress Monthly reporting process for managers and

community Internal: Data collection, report development,

management review, publication of detailed report External: availability, usage, performance summaries

Limited range of services historically Email, network/voice, paging, HR, IT security,

telepresence

4

5

6

The challenges of our “curation” process Labor-intensive Monthly periodicity limits immediacy

Fine for long-term trending, but less immediate Reliance on local units to self-report Leveraging Duke’s post-incident review (PIR)

process, but remaining data still sui generis

Curation: work by content specialists immersed in a specialized discipline and imbued with analysis

7

Metrics and newsgathering: Classic curation processes?

8

Duke’s metrics curation challenges No one wants to read “yesterday’s news”

Hard to avoid with curation The stories are too long

New monthly exec-summary focused on trends has helped (like the WSJ’s “What’s News” box) to a point

Focus on putting out a daily paper takes away from the “longreads”

Time required to produce manual reports takes away time expected to be spent on consulting with units to help them be primary data-gatherers

9

Curation vs. automation

10

Automation: leveraging monitoring data

System/network monitoring produces thousands of data points a day – how can we use them?

Left: Daily low-level alert analysis (SPC methodology)

Not shown: Loss of redundancy reports

11

Automation: Leveraging monitoring data

12

Duke’s metrics automation challenges

Automation a great start, but plenty of “curation” still occurs!

Monitoring’s “service dashboard” useful, but not ready to be directly published

Monitoring events don’t/can’t catch everything Human adjustment still needed to raw data

Low-level alert reports are automated… and reviewed by daily operational staff

Ultimately, there’s infinite ways to leverage data, but what do you care about?

13

Curation, automation, aggregation

14

Duke’s aggregation questions What data mix should appear?

Curated data from monthly metrics, live report on high-priority user tickets, performance graphs, availability measures against targets on one screen

Broad enough to be watched; specific enough to be useful

What platform should we use? SharePoint? Improve collection, visualization ServiceNow? Leverage dashboards, APIs, data

structure; easy access to support tickets Javascript/JSON components for easily customized

dashboards? – node.js and d3.js