useful decision support: what is it – and why is it so hard to create? arizona association for...

Useful Decision Support:

What is it –

and Why is it so hard to create?

Arizona Association for Institutional Research

Annual Meeting

March 2007

Richard D. HowardUniversity of Minnesota

[email protected]

What is Useful Decision Support (Actionable Knowledge)-

How does it relate to decision making and decision support?

Information Support Circle – Converting data to useful information to inform campus planning and decision making.

Barriers to Effective Decision Support –Why are the data “wrong” and What needs to be done to “fix” them?

Overview

“… is any knowledge that can be put into a design that the human mind can use in a causal manner. " (

http://www.hi.is/~joner/eaps/y3_33875.htm)

Useful Decision SupportActionable Knowledge

ACTION

http://www.hi.is/~joner/eaps/y3_33875.htm

Creating Actionable Knowledge is Decision Support

Primary role is to reduce the risk to the decision maker

Decision Support FocusHigh

PersuadeSeek agreement of values

PrayProcrastinate if possible

PrescribeDescribe action needed

Identify alternative scenarios

Prepare

Low

Desirability of outcome agreement

High

Cau

se-e

ffec

t kn

ow

n

What did you find? -- Technical Knowledge

What does it mean? -- Issues Knowledge

So what? -- Contextual Knowledge

Three Questions –Three Levels of Knowledge

All three types of knowledge must be present to create the most useful decision support .

Information

Knowledge

Intelligence(Actionable Knowledge)

DecisionRational Decision Making Process

Assessment

Data

How is decision support created and what limits its effectiveness?

Information Support Circle

USERCustodian

Steward

Quality Decision Making

Identify and MeasureConcepts

Collect and

Store Data

Restructure and

Analyze Facts

Deliver and Report

Information

Use and Influence

Knowledge

PRODUCER

Decision Maker

Broker

Identification and Measurement

• Defines the area of concern and need.

• ExcitementWhat could be done with knowledge?What events would be evident?What process leads up to these events?

• ExplorationWhat are the key components in the process?How do they tie together?What is known about causality?What is assumed about the situation?

• ClarificationWhat are the key questions?What essential elements of information exist?Alternative ways to measure elements.Costs and benefits of data alternatives.

sacs2295

GeneralizingDelivering and

Reporting

UsingInfluencing and

Decision Making

ModelingIdentifying ConceptsSelecting Measures

CollectingCoding and

Storing

RestructuringAnalyzing and

Integrating

Identification and Measurement Disease

Belief Bulimia: Semi-random gorging and purging of data from data bases and random changes in beliefs about what is important with no direction. • Symptoms:

> Constricted belief structure without linkage to reality.

> Random interactions of users and technicians with frowns.

> Knee-jerk inclusion of data for specific problems.

> No goals set for major activities.

> No sequence of when various things are needed.

Capture and Storage

• Storage of data requires focus and friends.

• Standardization:

Identify critical and key elements and codesDefine and document elements and codesMeasure and verify quality and integrityEstablish on-going process

• Key elements require:

Standard coded representation over data sourcesA standard long nameA standard short nameA standard abbreviation

• Administrative University Data Base elements (AUDB):

Relevant to planning, management, operating or auditingRequired for use by more than one unitIncluded in official administrative report or surveyUsed to derive an element for one or more criteria above


Reporting


Decision Making



Storing


Integrating

Capture and Storage Disease

Data Dyslexia: Inability to recall or recognize the meaning of the data, not knowing where they came from, often confusing one element for another.

• Symptoms:

> Random capture of data as they becomes available.

> Creative coding based on unwritten rules and what works.

> Using one variable for a specific purpose until later.

> Definition depends on who coded the variable.

> Process writes over data when new measure is available.

Data

“Facts” that are meaningless until put into a context, either with other data or in the context of a decision.

The sources of the these “facts” are typically the operating systems that drive the

academic and administrative/support processes of the campus.

Their restructuring and analysis result in the creation of information which should be used to inform planning and decision making.

Census Data

Constitutes a source of consistent data to support reporting, institutional effectiveness,

program reviews, and ad hoc studies

Student Related Data – Same point in time during the academic term

Faculty and Staff Data – Same calendar date for each academic term

Financial Data – Beginning year budget and end of year expenditures

Facilities – Typically once a year

Institutional Administrative Data ManagementInfrastructure

Student

Personnel Financial

Facilities

Standardizing Recodes/Edits

DataDescription/

User Support IADB

Security Service

Management Information Users

Dictionary

Operational Systems

Restructure and Analysis

• Translate from the input resources to outcome concerns.

• Reduce the complexity of the data and focus on specific concern.

• Use various types of analyses:

Description analysis Translate issues into targets and ranges Consider dispersion and associationComparison analysis Alternative when lack absolute standard Can be based on either internal or externalTrend analysis Depends on expectancy of causality Includes events in other situationsModeling analysis Combination of advanced techniques to predict Use leading indicators, multiple measures, and likelihoods.


Reporting


Decision Making



Storing


Integrating

Restructure and AnalyzeDisease

Dimensional Dementia: Results are uninterpretable due to irrational combinations of data using methodology based on the available software.

Symptoms:

> Forgetting the context in which the data were collected.

> Summarizing over data collected on various samples.

> Using most impressive statistics available.

> Cases left over when data bases are integrated.

> Major analyses done on PC with no documentation.

Delivery and Reporting

• Focus on needs of the customer.

• New technologies should:

Maintain batch access to data bases.

Provide processing environment and analyses.

Support retrieval, analyses, and interpretation of internal and external data, based on relevant frames.

Maintain historical types of data.

Comply with external requirements of cross- analyses and integration of data from various sources.

Develop storage and retrieval ability for documents.

• Delivery includes written and verbal reports.

• Reporting includes explaining and generalizing.


Reporting


Decision Making



Storing


Integrating

Delivery and Reporting Disease

Myopic Megalomania: Self-centered, short-sighted delivery of information based on the whims of the deliverer and independent of the user needs.

• Symptoms:

> A firmly held belief of technical superiority.

> Emphasis on the media and method rather than message.

> Disregard of user desires or suggestions of data clerks.

> Massive use of extreme-to ids in reports.

> Constant purchases of individual software by users.

• The key is institutional effectiveness.

• Effective DS requires evidence of use.

• Users must be supported as active learners.

• DS products must be considered in decisions.

• The timing of the decision cycle should be shared.

• DS must be used in shaping future decisions.

• DS should be part of the planning and assessment.

• Information will be only as useful as the weakest point.

• Cooperation is required for continuous improvement.

• Influence comes from reducing the core uncertainty of users.

• Cooperation and sharing is critical for quality.

Use and Influence


Reporting


Decision Making


Storing


Integrating

ModelingConcepts

Selecting Measures

Use and Influence Disease

• Creative Carcinoma: Creating and using facts as needed with First-Liar's Rule, where the fact continues to be quoted until it is a festering sore.

• Symptoms:

> Junior staff frequently provide complicated definitions.

> Executives believe they are invincible.

> The lack of good data is blamed for poor decisions.

> All decisions are last second to avoid disasters.

> Organizational structures are not changed to reflect reality.

Information Support Circle

USERCustodian

Steward

Quality Decision Making

Identify and MeasureConcepts

Collect and

Store Data

Restructure and

Analyze Facts

Deliver and Report

Information

Use and Influence

Knowledge

PRODUCER

Decision Maker

Broker

Two Major Properties

1) Dependency - information created by the process will be only as good as the weakest step in the process.

2) Cooperation - all three roles must function for the good of the institution, none can function in self interest.

Some Thoughts aboutBarriers to EffectiveDecision Support?

Some Institutional Limitations(potential)

Political – lack of trust that the data and analyses are reliable and appropriate

Resources – lack of skills, access to institutional data, time, access to peers

Leadership – inappropriate location in the institutional administrative structure and limited access to decision makers

Institutional Culture – inability/unwillingness to act within the context of strategic goals and assessment information

The Balancing

ctATime Quality

“There are two equally effective ways of keeping aboard in the dark. One is toprovide them with too little information. The other, ironically, is to provide them with too much.”

“Building Better Boards,” by David A. Nadler, Harvard Business Review, May 2004, p.109

Reality

Institutional Belief: Data-informed processes are better

• Different views on most issues can be informed with data.

• Most needed data are available.

• One can create a user capabilityand a comparison group.

• One can obtain appropriate measures and metrics.

• Tools are already available.

• If you monitor the process, you can improve the process.

What Barriers Limit Effective Decision Support at Your Institution?

From the Information Support Audit, check those characteristics that are present at your institution.

People, Processes and Managing Data. (2004) McLaughlin, Howard, et. al. Association for Institutional Research

Data in No Qualitative Context Data in Matching Qualitative Context

Least Effective Decision Support Data Context: Information from Analysis

Requires Technical IntelligenceAnswers “What Did You Find?”

More Effective Decision Support Organizational Context: Structure &

ProcessesRequires Issues Intelligence

Answers “What Does It Mean?”

Most Effective Decision SupportDecision Maker’s Context: Structure & Processes & Values

Requires Contextual IntelligenceAnswers “So What?”

Lo

w D

ecis

ion

-Mak

er P

roxi

mit

y

H

igh

Dec

isio

n-M

aker

Pro

xim

ity

Barriers to Effective DA

• Developers set in their ways.

• Systems/data tied to turf battles.

• Willing to make do with old technology.

• Unwilling to see needs outside operations.

• Already busy taking care of "here and now".

• Waiting for technology to solve the problem.

McLaughlin & McLaughlin, 1989

An Irish Prayer

May those who love us, love us;And those that don't love us,May God turn their hearts.

And if He doesn't turn their heartsMay he turn their ankles,So we'll know them by their limp.

AIR Newsletter, Sept 14, 1992

useful decision support: what is it – and why is it so hard to create? arizona association for...

Documents