education data sciences and the need for interpretive skills

Education Data Sciences and the Need For Interpretive Skills

Philip Piety, John Behrens, Roy PeaAmerican Education Research Association Annual Meeting

Monday, Apr 29 - 10:35am Parc 55 San Francisco / Divisadero Room

Some Driving Questions

• What kind of profession will education data sciences be?

• What are its ancestor, sister, and adjoining disciplines?

• Which kinds of skills and dispositions are important for preparing future practitioners and scholars?

Our Sociotechnical Thesis

• Data exist inside a social context; shaped by and shaping that context.



• Interpretation is not technical. It is itself socially situated with goals, predispositions/ biases, and norms.



• Interpretation is not technical. It is itself socially situated with goals, predispositions/ biases, and norms.

• Professional communities have developed valuable ways to reason from imperfect evidence. We can leverage/translate them to this new sociotechnical terrain.

Overview

1. Quantitative shifts in evidentiary artifacts (a digital ocean) in education

2. Qualitative shifts in educational focus

3. Some contributing/relevant disciplines

4. Interpretive skills, how education data scientists should approach data analysis?

QUANTITATIVE SHIFTS IN EDUCATION EVIDENCE

Computer

Adaptive testing

Assessment Technology

Computing Technology

Central “Mainframe“ Computing

Personal Computin

g

Devices

Tabulating Technology

Cloud Technology

Services

Traditional fixed response, short task assessments

Analog Paper-based (Textbooks, worksheets, and manual classroom tools)

Analog Portfolio

Classroom Technology

Th

e D

igit

al

Ocean

Distributed Integrated Assessment

Systems

Dramatic Growth in Artifacts

Digital Classroom

Technology

1850s 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 20101850s 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010

The Digital Ocean• Test scores• Interim assessments• In class, formative assessments• Growth models• Student collaboration• Conversation records from classroom

talk and online tools • Student work, including rich and

multimodal demonstrations of knowledge and competency (essays, presentations, etc.)• Records of after-school experiences• Records of informal learning • Activity traces from digital media (in

school, out of school, etc.)

• Demographics• Student-teacher relationships (TSDL) • School improvement plans/goals• Classifications (ex: proficiency groups)• Video records of teaching• Annotated/evaluated records of

teaching• Teacher evaluations• Individual Education Plans (IEPs) and

personalized learning maps• Geospatial information

(mapping and trends)• Attendance and rosters (more

important than you think!)• FERPA/privacy blocks

Where to Begin?

Studying Oceans

Studying Oceans

Influenced by concurrent work with behrens, Mislevy, and DiCerbo for the Learning Analytics Workgroup.

Studying Oceans

Structures & Interrelationships


Studying Oceans


Diachronic/Change Processes


Studying Oceans


Diachronic/Change Processes

Variations in Affordance


…AND QUALITATIVE SHIFTS IN EDUCATION ORIENTATION

Qualitative Shifts

Qualitative Shifts

1. Reorientation of center of control

2. Broader focus on competencies

3. Blended/pers-onalized learning

Social Networks &Teams

Mobile Technology

Evidence and Transparency

Institution Focus

Teacher Control

Institutional ReorientationInstitutions and Teachers

Social Networks &Teams

Mobile Technology

Evidence and Transparency

Institution Focus

Teacher Control

Networks and Students

Institutional Reorientation

SocialNetworksLearning

Networks

LearningCommuni

ties.

ExpertSources

Open Ed.Resources

Families

Institutions and Teachers

Related to the Education Data Movement

Emphasis on Broader Competencies

WHAT DO STUDENTS KNOW?

Cognitive• Cognitive processes

and strategies• Knowledge• Creativity

Intrapersonal• Intellectual openness• Work ethic and

conscientiousness• Positive core self-

evaluation

Interpersonal• Teamwork and

collaboration• Leadership

• Critical thinking• Information literacy• Reasoning• Innovation

• Flexibility• Initiative• Appreciation for

diversity• Metacognition

• Communication• Collaboration• Responsibility• Conflict resolution

Emphasis on Broader Competencies

WHAT DO STUDENTS KNOW?

Cognitive• Cognitive processes

and strategies• Knowledge• Creativity

Intrapersonal• Intellectual openness• Work ethic and

conscientiousness• Positive core self-

evaluation

Interpersonal• Teamwork and

collaboration• Leadership

Dig

ital M

edia

tion

• Critical thinking• Information literacy• Reasoning• Innovation

• Flexibility• Initiative• Appreciation for

diversity• Metacognition

• Communication• Collaboration• Responsibility• Conflict resolution

Artifacts

Blended/Personalized Learning• Blend the best of

face-to-face/online.• Incorporate interaction and

dynamic material coupled with metadata and paradata to enable feedback.

• Leverage embedded diagnostic assessments & interactive data visualization tools.

• “Learning algorithms” match content/activities/ teaching approaches with learner’s needs.

• Connect the in/out of school learning for complete picture of student’s development.

The World of Ed Data Scientists

• Oriented towards new kinds of education models while often working with data that comes from earlier models of education.

• Not only producing evidence (data jocks), but also change agents.

• Will be need to be innovators and draw off of different kinds of disciplines.

HOW DO WE ASSEMBLE AN EDUCATION DATA SCIENCES?

Considering Six Adjoining Disciplines

Education Data Sciences

1. Growing interest from leading universities, foundations, USED

2. Journals, conferences, & programs now emerging

3. What is the disciplinary focus? What counts as rigor and success? From where are faculty?

EducationData

Sciences

Statistical Data Analysis

Statistical Data

Analysis

EducationData

Sciences

• Much of the digital ocean is compatible with statistical analysis.

• Exploratory data analysis (ex: Tukey with satellite data in 70s asked many questions that are being asked today about “big data”

• Already established (entrenched) in Education power structures

• Can produce strong claims

Learning Technology

Statistical Data

Analysis

EducationData

Sciences

Classroom/ Learning

Technology

• This area is seeing an explosion in media for:• Inquiry• Communication• Construction• Expression

• This is where the data we want most often come from…

Learning Sciences

Statistical Data

Analysis

EducationData

Sciences

Classroom/ Learning

Technology

Learning Sciences

• What does big data mean for socio-technical multimodal learning?

• Socio-cultural and cognitive theories influence/informed by data technologies

• A design science for education practice

Information Sciences

Statistical Data

Analysis

EducationData

Sciences

Classroom/ Learning

Technology

Learning Sciences


• Data visualizations and HCI

• Info. architectures that undergird data systems• Codes, classifications• Boundary objects

• In schools, media centers evolving with data specialists

Organization/Management Sciences

Statistical Data

Analysis

Organization & Mgmt Sciences

EducationData

Sciences

Classroom/ Learning

Technology

Learning Sciences


• Education full of designed processes

• Blended learning models essentially re-structuring of org. practices

• Inter-organizational functions changing:• States-districts• Special education


Statistical Data

Analysis


EducationData

Sciences

Classroom/ Learning

Technology

Learning Sciences


Decision Sciences

• Established field uses large bodies of data to support org. decisions

• As volume/quality of education data increase, more situations where decision sciences can be applied emerging

HOW DO WE ASSEMBLE AN EDUCATION DATA SCIENCES?

The Seventh, Generative Discipline




Classroom/ Learning Technology

Learning Sciences


Decision Sciences

Computer Science and EDS

Computer Science





Learning Sciences

Decision Sciences



Computer Science





Learning Sciences

Decision Sciences

Machine Learning

Data Mining

Hum-Comp. Interaction &Visualization

Natural Language Processing

Computational Statistics



Data Scientist Definition

INTERPRETIVE SKILLS Reasoning from Digital Age Evidence

Flashlights, Imperfect Lenses

Approaching Digital Age Data Analysis

• Broad fluency with a range of qualitative/quantitative methods

• Ethics, privacy, and confidentiality (FERPA+)• Technology acumen and ability to reason from

imperfect evidence

Five Core Principles

1. All analytic processes are socially situated and iterative

2. Data is a mediational tool in an iterative process of discovery

3. Data is an imperfect lens for context and for interactions within that context

4. Organizational/systems thinking helps expand the reach of Education data science

5. Ethical as well as legal considerations are important.

Education Data Sciences and the Need For Interpretive Skills

Philip Piety, John Behrens, Roy PeaAmerican Education Research Association Annual Meeting

Monday, Apr 29 - 10:35am Parc 55 San Francisco / Divisadero Room

Contact: [email protected]