using data to bring people together
DESCRIPTION
Peter Radcliffe 2010 Quality FairTRANSCRIPT
Using Data to Bring PeopleTogether
Quality Fair 2010Project and Change Management
Collaborators (PCMC) SessionPeter M. RadcliffeExecutive Director
Office of Planning and AnalysisUniversity of Minnesota
Overview
• Why use data?• Why collaborate?• How can data enhance collaboration?• What are the challenges?• Case study: How has this worked in
practice?• How could you apply this in your own
work?
How do we make good decisions?
Good decisions bring together…• Evidence–What does reality look like?– How do the parts fit together?
• Context– How will stakeholders respond?–What resources are available?–What are our goals?
How do we make bad decisions?
• Evidence is wrong– Data is inaccurate–Model (understanding of what factors lead
to what outcomes) is faulty• Context is wrong– Local culture is different or changing– Resources are unavailable– Contrary to goals of organization
How can data help us?
• More comprehensive than our personalexperience
• More recent than our personalexperience
• Clearer representation of underlyingtrends
• Challenges us to examine our biasesand assumptions
Benefits of collaboration
• Sharing of knowledge and expertise– Not all of the information you need for your
work can be found in your own area– Your team may lack critical skills or
technologies• Improved credibility– Broader collaborations less likely to be
perceived as parochial and self-interested
Challenges to collaboration
• Problem definition–What situation are we trying to change?– Does evidence suggest the problem is real?
• Establishing agreement on outcomes– How do we define success?
• Understanding stakeholders and interests–What does each partner need?– Are interests complimentary, opposed, or
orthogonal
Types and Uses of Data
• Descriptive data–What is the state of the world in this area?
• Metrics–What are our goals and standards of
progress?• Analysis– How do elements connect in this area?– Can show causation or simply correlation
Data, truth, and stories
• Facts may speak for themselves, but theydonʼt say anything interesting on their own
• Data are summaries of a story – they canhelp identify some of the critical points, butthey are not the story themselves
• The full “truth” requires puttingobservations in context
Anecdotes and data
• “The plural of anecdote is data”– Raymond Wolfinger1
• Anecdotes, like data, are summaries ofexperiences which highlight the elements of thestory that are most relevant for the discussion ordeemed most likely to persuade the audience– A data set with a very small “n” (most anecdotes) has
very large error bounds on predictions– A large data set is hard to understand without
statistics to summarize the information, sacrificingsome detail
– Combined, they make a fuller, more engagingargument
1 Nelson W. Polsby. PS, Vol. 17, No. 4. (Autumn, 1984), pp. 778-781. Pg. 779: “Raymond Wolfinger's brilliant aphorism ʻthe plural of anecdote is dataʼ neverinspired a better or more skilled researcher. I e-mailed Wolfinger last year and got the following response from him: "I said 'The plural of anecdote isdata' some time in the 1969-70 academic year while teaching a graduate seminar at Stanford. The occasion was a student's dismissal of a simplefactual statement--by another student or me--as a mere anecdote. The quotation was my rejoinder.“
Trust, Bias, and Integrity
• “Figures won't lie, but liars will figure”– Charles Grosvenor
• Dishonesty with numbers doesnʼt requireoutright fabrication
• Selective use of data sources and definitionscan distort descriptions and analyses
• Best defense is transparency in data sourcesand definitions, and use of institutionalstandards for both whenever possible
• "Grosvenor, Charles H." The Oxford Dictionary of American Quotation. Hugh Rawson and Margaret Miner. OxfordUniversity Press 2008. Oxford Reference Online. Oxford University Press. University of Minnesota - TwinCities. 24 January 2010 <http://www.oxfordreference.com/views/ENTRY.html?subview=Main&entry=t251.e757>
Building the relationship
• Collaboration involves intruding on anotherperson or officeʼs business
• Need to understand their goals and needs,as well as your own
• Must demonstrate you are committed totheir success, not just yours
Case Study: Recreational Sports andInstitutional Research
• Recreational sports had information onfacility usage collected from U-Cardsystem
• Had conducted surveys exploring howstudents felt about the facilities and thebenefits students perceived from usingthem
• Lacked ability to show impact oneducational outcomes without assistance
• Approached Institutional Research forhelp
Project Background•Social Interaction•Tintoʼs Theory of Student Departure•Braxton & Hirschy: Communal Potential
Project BackgroundCRF – High Communal Potential
Social InteractionSocial Integration
PersistenceAcademic Success
Analysis and Preliminary Findings•In 2004, Rec Sportsprovided OIR with threeyears of facility usagedata
•OIR connected thatdata with retentioninformation, and found apositive relationship
•Needed a multivariatemodel to rule outspurious relationship
Research Questions
Is CRF usage associated with better academic performanceand increased likelihood of being retained and graduating,above and beyond other important predictors of academicsuccess?
First-term GPA
First-year retention
Graduating within 5 years
Influence of Visits to a CRF onProbability of Returning for a Second
Year
# of First-term Visits to Campus Recreation Facilities
Significance of collaboration
• The story is that campus recreationalfacilities usage is associated with higherretention, even after controlling for manyother factors
• Benefits of partnership– Recreational Sports strengthened their case
for investment in facilities– Institutional Research gained more
understanding of student retention anddemonstrated the value of their modeling inmaking decisions
Roles data played in collaboration
• Initial descriptive data and bivariateanalysis demonstrated showed promise ofeffort and garnered support fromleadership
• Each office needed data and knowledgeonly the other office could provide
• Analysis validated the understanding ofhow facility usage connected to studentsuccess
• Institutional metrics identified outcomes ofinterest
Try this at home!
• What stories do you and your colleaguestell and hear about why things happen inyour area?
• What do those stories tell you about theimportant concepts to measure?
• How could you explore if thoseexperiences can be generalized?
Keep trying…• Who has the information and skills you
need to access and assemble thatinformation?–May be in local systems in your or other
offices, or in institutional records– You may need to invent a way to collect it
• What does your group need to accomplish,and how does that differ from your whatyour collaborators need?
• How would you measure whether each ofyou is succeeding?
Continuing the conversation
• Join the discussion on Moodle– https://moodle.umn.edu/course/view.php?id=11
30–Will post responses to all questions submitted
on the index cards– Ask additional questions on Moodle or
contribute to the discussion
• Contact Peter: [email protected]
Appendix
Metrics and Agreement on Goals
• At November 2009 Board of RegentsMeeting, the President, Provost, andExecutive Director of Planning andAnalysis presented a system-wideframework for metrics and key indicators
• Framework provides a structure to whichactivity at all levels of the organization canbe aligned
Goals: Mission and CapacityExtraordinary Education – Recruit, educate, challenge, andgraduate outstanding students who become highly motivatedlifelong learners, leaders, and global citizens.
Breakthrough Research – Explore new ideas andbreakthrough discoveries that address the critical problemsand needs of the state, nation, and world.
Dynamic Outreach and Service – Connect the Universityʼsacademic research and teaching as an engine of positivechange for addressing societyʼs most complex challenges.
Mis
sion
Capa
city
World-Class Faculty and Staff – Engage exceptionalfaculty and staff who are innovative, energetic, and dedicatedto the highest standards of excellence.
ExceptionalStudents
ExceptionalInnovation
ExceptionalO
rganizationExceptional
Faculty and Staff
Outstanding Organization – Be responsible stewards ofresources, focused on service, driven by performance, andknown as the best among peers.
Transforming the U Pillars
26
Strategies and Key Indicators: Education
Recruit highly preparedstudents from diversepopulations
Incoming student preparation
Student diversity
Graduation and retention rates
Placement of graduates
Student engagement
Participation in study abroad andinternational experiences
Student development outcomes (in process)
Student learning outcomes (in process)
Extraordinary Education – Recruit, educate, challenge, andgraduate outstanding students who become highly motivated lifelonglearners, leaders, and global citizens.
Develop lifelong learners,leaders and globalcitizens
Challenge, educate andgraduate students
Goa
lSt
rate
gies
Key
Indi
cato
rs
27
Ensure affordableaccess for students ofall backgrounds
Internal support for scholarships
Average net cost for students
Strategies and Key Indicators: Research
Foster an environment ofcreativity that encouragesevolution of dynamicfields of inquiry
Highly cited research publications
National academy members and otherfaculty awards
Major research awards, research centerawards and centers of excellence
Technology disclosures, licenses andstartups
Breakthrough Research – Explore new ideas and breakthroughdiscoveries that address the critical problems and needs of the state,nation, and world.
Develop innovativestrategies to acceleratethe efficient and effectivetransfer of knowledge forthe public good
Goa
lSt
rate
gies
Key
Indi
cato
rs
Research expenditures and competitiveranking
28
Strategies and Key Indicators: Outreach
Promote and secure theadvancement of the mostchallenged communities
Longitudinal changes in communitieswhere the University is actively engaged(in development)
Active partnerships and assessments ofimpact (in development)
Faculty, staff, and student engagement andcommunity service (in development)
Dynamic Outreach and Service – Connect the Universityʼsacademic research and teaching as an engine of positive change foraddressing societyʼs most complex challenges.
Be a knowledge,information, and humancapital resource for thebetterment of the state,nation, and world
Build communitypartnerships thatenhance the value andimpact of the Universityʼsresearch and teaching
Goa
lSt
rate
gies
Key
Indi
cato
rs
29
Strategies and Key Indicators:Faculty and Staff
Recruit and placetalented and diversefaculty and staff to bestmeet organizationalneeds
Quality of incoming faculty and staff
Faculty and staff diversity
Faculty and staff awards and distinctions
Supervisor and departmental supportsatisfaction
Faculty and staff salary and totalcompensation
Employee engagement index
Employee training and development index (indevelopment)
World-Class Faculty and Staff – Engage exceptional faculty andstaff who are innovative, energetic, and dedicated to the higheststandards of excellence.
Recognize and rewardoutstanding faculty andstaff
Mentor, develop, andtrain faculty and staff tooptimize performance
Goa
lSt
rate
gies
Key
Indi
cato
rs
Engage and retainoutstanding faculty andstaff Employee retention index30
Strategies and Key Indicators: Organization
Ensure the Universityʼsfinancial strength
Bond rating: resources and leverage; liquidityand operating cushion
Private giving and endowment
Carbon footprint
Facilities Condition Needs Index (FCNI)
External awards to units for performance,quality, and innovation (in development)
Research space productivity
Crime and perceptions of safety
Faculty and staff satisfaction with supportservices
Outstanding Organization – Be responsible stewards of resources,focused on service, driven by performance, and known as the bestamong peers.
Be responsible stewards ofresources
Goa
lSt
rate
gies
Key
Indi
cato
rs
Focus on quality service
Foster peer-leadingresearch competitiveness,productivity, and impact
Ensure a safe and secureenvironment for theUniversity community
Promote performance,process improvement, andeffective practice
Research proposals and awards
Workplace injuries
31
Technology commercialization agreements
Integrated Metrics Framework
U-WideStrategies
U-WideKey Indicators
U-WideGoals
32
Unit-LevelGoals
Unit-LevelStrategies
Unit-Level Measures
University-Wide
Unit-Level
Criteria for decision-making
1. Centrality to mission2. Quality, productivity, and impact3. Uniqueness and comparative advantage4. Enhancement of academic synergies5. Demand and resources6. Efficiency and effectiveness7. Development and leveraging of resources
http://www1.umn.edu/systemwide/strategic_positioning/decision.html
Useful URLs
• Institutional Research– http://www.irr.umn.edu/
• Management Reporting– http://www.umreports.umn.edu/
• Enterprise Financial System– https://www1.umn.edu/cco/PeopleSoft/v8/fs.shtml
• Data Warehouse– https://dw.umn.edu/index.asp
More useful URLs
• Accountable to U– http://www.academic.umn.edu/accountability
• OVPR Levels and Trends– http://www.oar.umn.edu/trends/index.cfm
• Office of Measurement Services– http://oms.umn.edu/oms/index.php
• Office of Classroom Management– http://www.classroom.umn.edu