Can Autonomic Management Systems be Trusted?

A Perspective from Business-Driven IT Management

Jacques Sauvé - UFCGLANOMS 2007 Panel

Trust in IT Management

• Why can I talk about this? What’s my special perspective?• I do research in Business-

Driven IT Management (BDIM)

• Currently looking at ITIL processes and including a "business perspective"

2

An experiment closer to home ...

• How many of you mistrust some result presented at LANOMS 2007 or other conferences?– Why?

• How many of you mistrust results presented in your own papers?– Why?

3

Where does mistrust come from?

• Methodology!– We are deficient in validation

• Questions:– Did you do any kind of validation to see if the problem is

solved?– Do you know (versus hope) that your solution is useful?– Did you cover the problem space well?– Do you know where your solution is good and where it is

not?

• We plant the seeds of mistrust …

4

Why are we deficient?

• Validation easy to do (in theory)– Formulate a hypothesis: "My management software

improves metric X by Y%”– Get a lot of data (from a large user base)– See if it supports the hypothesis– If it does, accept the hypothesis

• Your claims will now be valid when selling the solution

• Sweep the problem space to see what you can claim

5

But the real world creeps in ...

• Marketing people sell their own “hypothesis”• How do you get a large user base to get data?• Pressure to push products to market before

they are validated• How do you test your own baby?• “Of course, it’s good ... I built it!”

6

“Good enough” solutions

• It takes time to validate and by then you have new modules, new problems, ...

• Given the lifecycle of management solutions, how much time can we afford to spend validating?– Our solutions last only a (very few) years

• Validating systems is not like validating general theories– Stop when the validation is “good enough”

7

Conclusion 1: Full scientific validation is usually not an option

• But the following remains true ... – Trust is gained through user experience– Level of trust changes over time– Building trust is a process– Overselling may be good for the bottom line but

may hurt trust

8

How to Lose Trust?

• Is it helpful to get it right 99% of the time?• How about 95%?• Getting it wrong once on a while makes

managers fidgety– “When can I trust it?"

• Losing trust is (almost) immediate and tools get boycotted– Some processes are more critical– Ex. Getting diagnostics wrong

9

Conclusion 2: Sell the right message

• Sell the message that the systems are “useful”, not “always right” or “best”

• Use metrics and baselines to enable comparison– Use aggregate metrics

• But ...

10

Conclusion 3: Automatic Solutions Should not Cause Grand Disasters!

• Example: automatic scheduler loops and can’t choose any job to run

• Example: Automatic capacity allocation should not flip-flop and thrash

• Grand disasters reduce trust to zero

11

What Kind of Metrics Should we Use to Build Trust?

• Are business metrics are better than technical metrics?– Technical: BD throughput, resource utilization,

response time, etc.– Business: business transaction throughput,

financial gain or loss, risk, business impact, etc.

12

Non-Conclusion 1: Are Business Metrics Better for Trust?

• I think business metrics are more useful but they are not directly measurable

• They may or may not help build trust more than easy-to-measure technical metrics– It’s hard to trust models, easier to trust

measurements• Use of business metrics helps the dialog with

business people and should help validation and thus gain trust– This is an untested hypothesis

13

The Effects of Scale

• Humans can’t handle:– Number of elements and– Number of dimensions present in today’s

scenarios

14

Conclusion 4: Autonomic will Rule Because of Scale

• Managers have no choice but to rely to automatic solutions because of scale– Even if trust is not perfect

15

Other Possible Solutions

• Put user in the feedback loop!– To calibrate tools– To make tool learn– To let user decide how much the tool should do• Automate versus suggest actions

16

Thank you.

17