modeling education with work domain analysis: a work in progress
DESCRIPTION
Modeling Education with Work Domain Analysis: A Work in Progress. Lee Nickles September 11, 2002. Outline. Defining Educational Evaluation Obstacles to Performing an Evaluation Work Domain Analysis as Solution Process of Building the Model. What is Educational Evaluation?. - PowerPoint PPT PresentationTRANSCRIPT
Outline
• Defining Educational Evaluation
• Obstacles to Performing an Evaluation
• Work Domain Analysis as Solution
• Process of Building the Model
What is Educational Evaluation?
“Evaluation in the context of educational systems is briefly defined as examining the effectiveness of an educational system (or component of that system) in meeting learning and teaching goals.”
Based on:
(Bloom et al, 1971; Stevens, et al [NSF], 1993; Walker, 1997)
Formative vs. Summative Evaluation
• Formative – During development and implementation
– Examines system to reveal problems and improvements
• Summative– At the end of the system’s life
– Examines overall success of the system
• My focus: Formative
Formative Evaluation is a “Good Thing™”
• Similar to Quality Control, leads to continuous improvement
• Changes can be made in the current life cycle of the system
Evaluation Obstacles (1)
• Education systems have multiple, interacting components– Work together to produce learning
• Isolating which component produced what or (how much) learning is difficult– Testing each individually is too great a burden– Components build on each other...
Evaluation Obstacles (2)
• There is a lack of validated measurement instruments for evaluation– There may not be a test that analyzes whether
or not your specific web site produced learning– You may not have the time or resources to
generate a valid measurement
Validation Solution
• Measurement Triangulation– Use multiple measurements of different types
to measure the thing you want to evaluate• Subjective Opinion/Attitude
• Performance
• Process/Behavior
– If they give similar/confirming readings, you can have greater confidence in your result
Triangulation Problem
• How do we apply this practically?
• That is, how do you know your measurements are triangulating properly?– They are measuring slightly different things
• Need a structured way to apply triangulation
Proposed Solution: Work Domain Analysis
• Interacting components– Specifically examines tightly coupled systems,
with many relationships between the components
• Structure to apply triangulation– Provides the abstraction hierarchy framework
to guide measurement triangulation
Work Domain Analysis - Abstraction Hierarchy
• Rasmussen, 1985
• Modeling tool
• Broad and deep view of the system
• Way to represent the levels of abstraction of a system together and how components, functions, and goals connect to each other
Levels of System Abstraction
Can describe a system at levels of abstraction from the physical
Concrete Abstract
Physical Objects
System Functions
Goals and Objectives
Means-Ends Relationships
Items in one level of abstraction are related to others by means-ends relationships
1: Goals
2: Functions
3: Physical Form
Means: Level 3 tells HOW Level 2 is accomplished
Ends: Level 1 tells WHY Level 2 is performed
Example for Community of Scholars
Levels ofAbstraction
Components
GeneralPurpose
Develop an intellectually stimulating environment in thebasement of the ESM
AbstractPurpose
Increase awareness of everyone’s research in basement of ESM,Provide forum for “scholarly banter”
GeneralFunction
Share ongoing research, Learn about new research applications,Critique research, Find areas for collaboration
PhysicalFunction
Presentations, discussion of presentations
Physical Form Students (speakers, participants), Advisors, Meeting space (fortalks/discussion), PowerPoint presentations
Education and the AH
• Problem: the AH hasn’t really been applied thoroughly to intentional systems– Intentional systems: systems where actions are
based on the user’s intentions more than physical constraints
• Some say it can’t be done (Wong et al, 1998)• Same say it can (Hajdukiewicz et al, 1999)• No really good examples in the literature
Building an Abstraction Hierarchy to Describe Education
• I’ve gone through this process for some time (too long?)
• Many iterations, back and forth with Dr. Pritchett
• NOT Done!
• Process may be as informative as the product, so here is (approximately) the process...
Early Versions
• Considered various ways an AH has been represented– Separating very different domains that are part of
the same system
– Think about how cognitive functions, social activities, and information flow should be represented
– Systems based on user intention rather than physical constraints
Early Framework - Domains
DOMAINSLevels ofAbstraction
Social/Interaction
Cognition Education
Course Goals
“FunctionalGoals”EducationalFunctionsAtomicComponents
Difficult to Separate/Categorize things
I noticed something...
• Process Control– Ongoing process
– Goal: keep process stable
– Change is bad
• Education– Defined start/end of
educational system
– Goal: produce learning in the students
– Change is desired
AH has typically been applied to process control
Education as Change Over Time
STAGESLevels ofAbstraction
Acquire factsin coursematerial
Acquire skillsfor course
Apply facts& skills tonewproblems
Awareness ofinterconnec-tions betweentopics
Course Goals
“FunctionalGoals”EducationalFunctionsAtomicComponents
Education as Pragmatic Change Over Time
GeneralPurpose
Learn IE Knowledge/Skills
Freshman Sophomore Junior SeniorAbstractPurpose
Learn basicdesign, math,etc. (courselist)
(course list)
GeneralFunction
(courseobjectives)
PhysicalFunctionPhysical Form
Is it this simple? Not very general!
Education as Pragmatic Domains
Levels ofAbstraction
Domain of Student Domain of Instructor
GeneralPurpose
Acquire marketable skills Grow Intellectually Socialization
Grow as instructor Propagate domain Job requirement
AbstractPurpose
Student learns X Uphold Honor Code
GeneralFunction
Knowledge transmission Skill acquisition Grading
Low-levelFunctions
Assignments Tests Lecture Reading
Atomic Form Assignment problems Lecture notes Case Studies Exams SyllabusE-mail Textbook
Interesting, but is the domain only interesting at the top level? Did we lose the element of change?
Three Dimensional AH?
• Maybe this was going to take multiple dimensions of domains to represent
• About this time, I switched from pen and paper to Excel to create AHs and used the IT Web site for ISyE 4009 as a case
Multi-Dimensional, “Unit” Based
Unit 1 Course Unit (Lecture? Week?) TopicUnderstand how w e, as engineers, can design information systems to create effective w ork processesLearn how to identify and design for the needs of w orkers and organizationsCreate machine interface designs that establish and support good w ork practicesBe able to identify and communicate the properties of a machine's interface
General Purpose
Understand the limitations of human operators under a variety of situations
(Course Objectives)
Understand the principles of human-integrated systems evaluation - and apply them to your project design
Conceptual Purpose ?? Course Introduction, Learn Procedures
Introduction, Procedure Following Concepts, Preflight Inspection Experiment, Procedures and Roles of Humans
Cognitive Functions Assessment of learning
Acquire domain knowledge (AL), Practice for skill acquisition (AL, LFB) Acquire declarative knowledge (IT)
Low-Level Functions
Work exam, grade exam, review exam solution
Work homework, grade homework, review homework View powerpoint slides
Atomic Elements
IT Web website, syllabus.doc, Spring2002Exam1Solutions.doc homework1.doc, homework2-12.doc
Introduction.ppt, ProcedureFollowing.ppt, PreflightInspectionExperiment.ppt, ProceduresAndRolesOfHumans.ppt
What is a unit?
Multi-Dimensional, Week BasedWeek 4 Course Unit Topic
Understand how w e, as engineers, can design information systems to create effective w ork processesLearn how to identify and design for the needs of w orkers and organizationsCreate machine interface designs that establish and support good w ork practicesBe able to identify and communicate the properties of a machine's interface
General Purpose
Understand the limitations of human operators under a variety of situations
(Course Objectives)
Understand the principles of human-integrated systems evaluation - and apply them to your project design
Conceptual Purpose
Modeling Work I, Decision Making - Naturalistic Models
Work Models Overview, Physical Models, Artifact Models, Cultural Models, Sequence Models, Flow Models, Non B&H Models, Tactical and Strategic Decision Making
Cognitive/ Educational Functions
Practice (AL, LFB), Assessment of learning Practice (LFB) Acquire declarative knowledge (IT)
Low-Level Functions
Work Project (Part A), grade project, review project
Work homework 3, grade homework, review homework View powerpoint slides
Atomic Elements
IT Web website, syllabus.doc, DescriptionOfProject.ppt, DescriptionOfProjectPartA.ppt homework3.doc
ModelingWorkOverview.ppt, HelpfulModelsandStructurestoGuideContextualInquiry.ppt, FlowModels.ppt, SequenceModels.ppt, CulturalModels.ppt, ArtifactModels.ppt, PhysicalModels.ppt, TacticalStrategicDecisionMaking.ppt
This is getting somewhere! But weeks are not general.
Multi-Dimensional, Assessment Based
A2 Course Assessment Topic
Course Objectives
Learn how to identify and design for the needs of w orkers and organizations; Be able to identify and communicate the properties of a machine's interface
Bloom's Level
Knowledge, Comprehension, Application, Analysis Knowledge, Comprehension, Application
Cognitive/ Educational Functions
Research, reflective learning, problem-based learning
Acquire declarative knowledge, Reflective learning
Low-Level Functions
Research, Critical analysis, Applying principles View PowerPoint slides, reflective thinking
Atomic/IT Web Elements Homework 2
Procedure Following Concepts; Preflight Inspection Experiment; Procedures and Roles of Humans
An assessment (homework, exam, project, etc.) as an essential part of course structure
Today: Bring Back Domains, Widen Scope
StudentAssignment 1 Course Assessment Content
Course Objectives
Understand the principles of human-integrated systems evaluation - and apply them to your project design
Cog./Edu. Functions Discovery Learning (sort of) Acquire declarative knowledge
Low-Level Functions Research, Identify terms, submit results
View/Read PowerPoint slides, reflective thinking, ask questions of instructor, meet in study groups, attend lecture, participate in lecture
Atomic Elements
IT Web, instructor, students, classroom, presentation equipment, syllabus/calendar, informal student groups
Homework 1, student submissions, job listings
Introduction Lecture, Results of Employment Ad Keywords
InstructorAssignment 1 Course Assessment Content
Course Objectives
Understand the principles of human-integrated systems evaluation - and apply them to your project design
Cog./Edu. Functions Discovery Learning (sort of) Transmit declarative knowledgeLow-Level Functions Create assessment, compile results
Create PowerPoint slides, lecture, answer questions, lead discussion
Atomic Elements
IT Web, instructor, students, classroom, presentation equipment, syllabus/calendar Homework 1, student submissions
Introduction Lecture, Results of Employment Ad Keywords
Where do I go now?
• Maybe stop now and model an entire course with this method– Are there things we’re leaving out?– What can we leave out?
• Consider, how useful is this for evaluation?– Can we easily generate an AH for a course?– Is it general enough to apply broadly?– Is it specific enough to be useful?