class 8: information management, performance measurement & program evaluation uta school of...
TRANSCRIPT
Class 8: Information management, Class 8: Information management, Performance Measurement & Performance Measurement &
Program EvaluationProgram Evaluation
UTA School of Social WorkUTA School of Social Work6371: Community & Administrative Practice6371: Community & Administrative Practice
Dr. Dick SchoechDr. Dick Schoech
Copyright 2009 (permission required before use)Copyright 2009 (permission required before use)
Suggest printing slides for class using: Print | Handouts | 3 slides per page | grayscale optionsSuggest printing slides for class using: Print | Handouts | 3 slides per page | grayscale options
Review of Previous ClassesReview of Previous Classes2.2. Theories, values, perspectivesTheories, values, perspectives
3.3. Problem definition, theories of needProblem definition, theories of need
4.4. Capacities and needs assessmentCapacities and needs assessment
5.5. Evidence based interventions and logic modelsEvidence based interventions and logic models
6.6. Program descriptions, community linkagesProgram descriptions, community linkages
7.7. Mission, Goals and ObjectivesMission, Goals and Objectives
8.8. Information Management, performance Information Management, performance measurement, program evaluationmeasurement, program evaluation
9.9. Budgeting Budgeting §§
Goals of ClassGoals of Class
To examine the basics of program evaluationTo examine the basics of program evaluation To understand the different foci of: To understand the different foci of:
• Process evaluationProcess evaluation• Outcome evaluationOutcome evaluation• Impact evaluationImpact evaluation
To understand the role of measurement & To understand the role of measurement & feedback in managementfeedback in management
To understand organized information concept To understand organized information concept & processes in management& processes in management
Program EvaluationProgram Evaluation
Evaluation OverviewEvaluation Overview
Purposes of evaluationPurposes of evaluation TypesTypes
• Process evaluation (focus on fidelity)Process evaluation (focus on fidelity)• Outcome evaluation (focus on results)Outcome evaluation (focus on results)• Impact evaluation (focus on problem Impact evaluation (focus on problem
reduction)reduction) Evaluation: old vs. new paradigmEvaluation: old vs. new paradigm Issues Issues §§See course pack for PDF on Evaluation
Purposes of EvaluationPurposes of Evaluation
Determination of fidelity Determination of fidelity Provides continuous quality improvement Provides continuous quality improvement
mechanisms (TQM)mechanisms (TQM) Demonstrating accountabilityDemonstrating accountability Gives information for agency decision makingGives information for agency decision making Building stakeholder support with credibilityBuilding stakeholder support with credibility Enhancing future funding potentialEnhancing future funding potential Discovering new knowledge, e.g., cause-effect Discovering new knowledge, e.g., cause-effect
relationshipsrelationships Fits our evidence-informed social work approachFits our evidence-informed social work approach§§
Type: Process/formative/monitoringType: Process/formative/monitoring
Answers: WHO doing HOW MUCH of WHAT to Answers: WHO doing HOW MUCH of WHAT to HOW MANY of WHOM and WHEN are they HOW MANY of WHOM and WHEN are they doing itdoing it
All measures in the process objectives are All measures in the process objectives are covered in the monitoring and trackingcovered in the monitoring and tracking
Measures are:Measures are:• Whether activities are completed as describedWhether activities are completed as described• How much effort things took, e.g., were they How much effort things took, e.g., were they
efficientefficient• Satisfaction with what was doneSatisfaction with what was done• Fidelity, whether intervention implemented Fidelity, whether intervention implemented
correctly, e.g., correct dosage of intervention correctly, e.g., correct dosage of intervention given, services of high quality given, services of high quality §§
Type: Outcome evaluationType: Outcome evaluation
Answers: Who isAnswers: Who is doingdoing how much of whathow much of what to how many of whomto how many of whom, , whenwhen are they are they doing it and withdoing it and with WHAT RESULTSWHAT RESULTS
Measures are:Measures are:• Client change, e.g., Goal Attainment Client change, e.g., Goal Attainment
Scales (GAS)Scales (GAS)• cost/benefitcost/benefit• Effectiveness measuresEffectiveness measures§§
Type: Impact EvaluationType: Impact Evaluation
•Results for Results for person/family/communityperson/family/community
•Unintended side effects on Unintended side effects on person/family/communityperson/family/community
•Measures are:Measures are: If condition changed If condition changed If problems reducedIf problems reduced If need reduced If need reduced §§
Evaluation: Evaluation: Old vs. New Tech ParadigmOld vs. New Tech Paradigm
Imposed from outsideImposed from outside Tedious and onerous Tedious and onerous
activityactivity Results sit unused on Results sit unused on
shelfshelf Fear of what evaluation may Fear of what evaluation may
saysay Complex and expensiveComplex and expensive Takes time from real workTakes time from real work True results are impossible True results are impossible
to quantifyto quantify
Desired from insideDesired from inside Integral to way of providing Integral to way of providing
services so supported by techservices so supported by tech A continuous process of A continuous process of
organizational learningorganizational learning Requires collaboration from all Requires collaboration from all
parties parties Can requires expert Can requires expert
consultation in design and consultation in design and interpretation phasesinterpretation phases
Documents success & how to Documents success & how to improve §improve §
Old New
Evaluation IssuesEvaluation Issues Cost (1-5% of budget)Cost (1-5% of budget) Source (inside or outside)Source (inside or outside) Accuracy or for showAccuracy or for show How results used and disseminated, e.g., public How results used and disseminated, e.g., public
or not publicor not public How to measure change/outcomes (performance)How to measure change/outcomes (performance) Whether client change or lack of change is due to Whether client change or lack of change is due to
program or other causes Public/funders not program or other causes Public/funders not understanding success, 5% change 6 months understanding success, 5% change 6 months after treatment may be very goodafter treatment may be very good
ConclusionConclusion
Evaluation is being taken more Evaluation is being taken more seriouslyseriously
Outside evaluation often requiredOutside evaluation often required
A much less rigorous process if A much less rigorous process if implementing a proven modelimplementing a proven model
Evaluation is changing to support Evaluation is changing to support proactive management & decision proactive management & decision supportsupport
Information Information management and management and Decision SupportDecision Support
Book was as different from the Book was as different from the scroll as computer is from book so scroll as computer is from book so support needed. Click for videosupport needed. Click for video
http://www.milkandcookies.com/link/57946/detail/
Management using Information technologyManagement using Information technology
Information & IT as a basic resourceInformation & IT as a basic resource Information management termsInformation management terms Information system componentsInformation system components Change via 8 step process, 3 approaches, Change via 8 step process, 3 approaches,
several agency structuresseveral agency structures Guidelines for successGuidelines for success Organizational Learning/IntelligenceOrganizational Learning/Intelligence Future of Technology in Social Work Future of Technology in Social Work §§
Information/technology as basic resourceInformation/technology as basic resource
Treat info as other basic resource, e.g., $s, people, Treat info as other basic resource, e.g., $s, people, etc.etc.
Info easy resource to change & manage (compared to Info easy resource to change & manage (compared to people, task, goals, structure)people, task, goals, structure)
Computers add power to information management, Computers add power to information management, but errors can be bigger and more disastrousbut errors can be bigger and more disastrous
Managing info more difficult if shared across Managing info more difficult if shared across departments & outside agenciesdepartments & outside agencies
Information resource is easily sabotaged, e.g., Information resource is easily sabotaged, e.g., garbage in—garbage outgarbage in—garbage out
Technology keeps changing requiring constant Technology keeps changing requiring constant attention attention §§
Information Management TermsInformation Management Terms Database vs. Database Mana SystemDatabase vs. Database Mana System
• Database=a collection of data (records in fiscal, client, Database=a collection of data (records in fiscal, client, services, and staff files) with minimal redundancy and services, and staff files) with minimal redundancy and easy manipulation and access.easy manipulation and access.
• DBMS=software to manage a database (MS-ACCESS)DBMS=software to manage a database (MS-ACCESS)
• Data Warehouse = Department where all data Data Warehouse = Department where all data resides, is managed, analyzed, and disseminatedresides, is managed, analyzed, and disseminated
Computer vs. Information SystemComputer vs. Information System• MIS=system of people, procedures, and equipment for MIS=system of people, procedures, and equipment for
collecting, storing, manipulating, protecting, and collecting, storing, manipulating, protecting, and reporting informationreporting information (can be manual)(can be manual)
ASP, Internet, Intranets, extranets ASP, Internet, Intranets, extranets §§
IT as ChangeIT as Change
Treat as any other changeTreat as any other change
Info mana develops an info model of agency like Info mana develops an info model of agency like financial mana develops a money model of agency. financial mana develops a money model of agency. Agency goals/obj drive info & $, not vice versa.Agency goals/obj drive info & $, not vice versa.
A 2-3 year effort of hard work by allA 2-3 year effort of hard work by all
Social workers are experts in change but have Social workers are experts in change but have problems with technology changeproblems with technology change
All change involves resistance & emotional loss of All change involves resistance & emotional loss of old way of doingold way of doing
Reduce change problems by following a process §Reduce change problems by following a process §
8 Step Development Process8 Step Development Process
1.1. Assess Preparedness & FeasibilityAssess Preparedness & Feasibility
2.2. Analyze Existing SystemAnalyze Existing System
3.3. Conceptual DesignConceptual Design
4.4. Detailed Design & DevelopmentDetailed Design & Development
5.5. System Testing & Agency PreparationSystem Testing & Agency Preparation
6.6. Convert from old to new systemConvert from old to new system
7.7. EvaluationEvaluation
8.8. Operation, maintenance & modificationOperation, maintenance & modification
1: Explore feasibility/preparedness1: Explore feasibility/preparedness
A key step often neglected Communicate about effort to all stakeholders Establish steering committee Estimate resources of money, time, expertise Determine commitment of key individuals Estimate potential impacts (positive & negative) Define IT purpose, objectives, timetables,
responsibilities, evaluation Assess the expectations/reactions of those impacted Circulate preparedness/feasibility report §§
2: 2: Assess Situation (systems Analysis)Assess Situation (systems Analysis)
Identify the major situations/tasks/decisions system will support & what is needed to support them
Define the characteristics of the information needed, its source, and collection methods
Analyze current input forms, data processing, storage files, and output reports
Describe logical routing or flow of information from collection to dissemination
Review similar efforts in other agencies Prepare systems analysis report §§
3: Prepare conceptual Design3: Prepare conceptual Design
Finalize IT scope, goals, and objectives
Develop alternative conceptual designs, i.e., files, data manipulation, forms and reports
Apply restrictions to designs (volume, security, confidentiality, turn around time, statistical capabilities, growth potential, ties to other systems)
Apply resources to designs (money, time, expertise)
Translate designs into software and hardware Detail the advantages, disadvantages, and assumptions of
alternate designs §§
4: Detailed Design & Development4: Detailed Design & Development
Set up controls and technical performance standards for chosen design
Select the software for the chosen design
Select the hardware to match the software
Design and develop input forms, data manipulations & processing operations, file specifications, database structures, indexes, error checks, storage and backup mechanisms and procedures, and output reports
Prepare programming documentation and instruction manuals §§
5: Test System & Prepare Agency5: Test System & Prepare Agency
Prepare system operators, users, and others to receive the system
Develop agency policy and procedural changes necessary for the new system
Develop performance criteria and testing plan
Test programming, forms, operational procedures, instructions, reports, and use of outputs
Educate and train system operators, system users, and others affected §§
6: Implement6: Implement
Develop and approve conversion plan, e.g., stop old system when new system starts or run old and new systems simultaneously for comparison
Incorporate IT into standard operating procedures, e.g., performance appraisals, new employee orientation
Reorganize staff and space if necessary
Convert from old to new equipment, new processing methods, and new proceduresInsure all systems and controls are working §§
7: Monitor and evaluate7: Monitor and evaluate
Compare system performance with initial system objectives, e.g., if system improved client services
Relate benefits and costs to initial estimates
Measure satisfaction with the system
Determine if system outputs are used in decision making §§
8: Operate, maintain & modify8: Operate, maintain & modify
Prepare backup and emergency plans/procedures
Complete documentation e.g., instructions for adding to, deleting from, or modifying system
Assign persons responsible for data integrity, system maintenance, new software appropriateness, virus protection, etc.
Provide continuous training of users Continue to add desired enhancements and to maintain
and debug the system §§
Which Development ApproachWhich Development Approach Top down – specify total system firstTop down – specify total system first Bottom up – Design small systems Bottom up – Design small systems
and then link them into a total and then link them into a total systemsystem
Prototyping – Build as you go and Prototyping – Build as you go and build overall system as needed build overall system as needed §§
Structuring Expertise: Power Balance
Top management
IT Committee
Application Steering Committee
Information Manager/Department
A balance of power exists between the IT committee and the information manager. Disagreements are settled by top management.
Structuring Expertise: Committee Control
Top management
IT Committee
Application Steering Committee
Information Manager/Department
The application steering committee and information manager reports directly to the IT committee (typically the recommended structure).
Structuring Expertise: Management control
Top management
IT Committee
Application Steering Committee
Information Manager/Department
All computing committees are under the information management department. This structure is not usually recommended because it gives too much control to the information manager and makes access to top management by user representatives more difficult.
Guidelines for SuccessGuidelines for Success
IT needs separate high level deptIT needs separate high level dept Planning with stakeholders is essentialPlanning with stakeholders is essential Centralize control, distribute powerCentralize control, distribute power Avoid relying on small # techniciansAvoid relying on small # technicians 10/40/50 rule and 80/20 rule10/40/50 rule and 80/20 rule Expect frustrating and time consuming Expect frustrating and time consuming (2-(2-
3 years for basic system3 years for basic system §§
IT system development issuesIT system development issues
What % of budget to spend on ITWhat % of budget to spend on IT Social workers are experts in change but have Social workers are experts in change but have
problems with technology changeproblems with technology change All change involves loss/resistanceAll change involves loss/resistance IT involves emotional loss of old way of doingIT involves emotional loss of old way of doing How to get the necessary expertise when IT pays How to get the necessary expertise when IT pays
much more than social workmuch more than social work Security, privacy, & confidentiality are big issues Security, privacy, & confidentiality are big issues
when automating and sharing information when automating and sharing information §§
Current vs. FutureCurrent vs. Future
Networked PC on desks Beginning to web enable Basic MIS developed Information as text & #s User passively receives
system outputs High cost info for a few Worker’s info better than
client’s You search for information
Convergence & connectivity (phone, computer, TV)
Information as text, pictures, movies, VR, etc.
Increased interactivity Client’s info better than
worker’s Information finds you Interconnected agencies Infotainment §
Current Future
Organizational Intelligence & DSSOrganizational Intelligence & DSS
Agency has capacity to accumulate expertise Agency has capacity to accumulate expertise
Agency becomes smarter as each case is workedAgency becomes smarter as each case is worked
Intelligence automatically available when and where Intelligence automatically available when and where needed and in the format neededneeded and in the format needed
Intelligent agencies linked to form virtual support Intelligent agencies linked to form virtual support system with global knowledgesystem with global knowledge
Much can be done automatically, e.g., automatic Much can be done automatically, e.g., automatic sharing of information sharing of information
Interactive models (Decision support systems) to Interactive models (Decision support systems) to scenario decision making scenario decision making §§
Examples DP & CAPExamples DP & CAP
DPDP Human services information systemHuman services information system Web Phone System Direct practice—depression Direct practice—panic & anxiety Direct practice—marriage counseling Online therapy Dr. Judith -- consumer beware -- consumer beware
CAPCAP Move on, , Meet up, Kiva, , Kiva,
ResourcesResources
Community Tool boxCommunity Tool box NASW/ASWB Technology StandardsNASW/ASWB Technology Standards Journal of Tech in Human ServicesJournal of Tech in Human Services TechSoup for technology support and TechSoup for technology support and
products for agenciesproducts for agencies HUSITA (HUman Services Information HUSITA (HUman Services Information
Technology ApplicationsTechnology Applications FOSS (free & open source software)FOSS (free & open source software)