institutional reflection on readiness: the learning analytics readiness instrument (237810993)

8/11/2019 Institutional Reflection on Readiness: The Learning Analytics Readiness Instrument (237810993)

http://slidepdf.com/reader/full/institutional-reflection-on-readiness-the-learning-analytics-readiness-instrument 1/80

Institutional Reflection on Readiness:The Learning Analytics Readiness Instrument

May 19, 2014

Kimberly Arnold, University of Wisconsin-Madison

Steven Lonn, University of Michigan

Matthew D. Pistilli, Purdue University

Who is attending this seminar with you?

A) I’m attending by myself

B) There are a small handful of us here

(< 5)C) There are many people here (6+)

Who are we?

Where are we from?Why are we here?

Introductions and Purpose

Introduction

What is Learning Analytics?

The Learning Analytics Readiness Instrument

Breaking Down the Factors

Concluding Thoughts

What is Learning Analytics:

Definitions

Definitions of Learning Analytics Using analytic techniques to help target instructional,

curricular, and support resources to support theachievement of specific learning goals (van Bareneveld,Arnold, & Campbell, 2012)

the process of developing actionable insights throughproblem definition and the application of statisticalmodels and analysis against existing and/or simulatedfuture data (Cooper, 2012)

The use of data, statistical analysis, and explanatoryand predictive models to gain insights and act oncomplex issues (Bichsel, 2012)

Challenge: How do you find the student at risk?

https://reader010.{domain}/reader010/html5/0613/5b2057b131b74/5b2057b683499.jpg

https://reader010.{domain}/reader010/html5/0613/5b2057b131b74/5b2057b79bc43.jpg

Challenge: How do you find the student at risk?

Analytics is about…

Actionable intelligence

Moving research to practice

Basis for design, pedagogy, self-awareness

Changing institutional culture

Understanding the limitations andrisks

http://i.imgur.com/nZArTnc.jpg

New Possibilities

Using data that exists on campus

Taking advantages of existing programs

Bringing a “complete picture” beyond academics

Focusing on the “Action” in “Actionable Intelligence”

The Concept of LA Readiness

What is “Readiness”

• What things do you think of when

you hear Institutional Readinessfor Learning Analytics?

• Discuss / Brainstorm

– What does the institution need?

• For whom? When?

Institutional Readiness

- Share Out -

• What did you / your team list?

• What is the highest / first priority?

– Why?

• What is already in place vs. needs to be

purchased / developed / trained?

• Are their larger / cultural issues to discuss

before logistical issues can be addressed?

How ready is your institution to

implement learning analytics?

A.We’re ready!

B.We need a little planning, but we are

basically ready.C.We’ve talked about learning analytics,

but made no plans.

D.We are not ready.

Readiness (ˈred-ē-nis)

noun1 willingness to do something.

2 the state of being fully prepared

for something.

3 immediacy, quickness, or promptness.

• Adopting learning analytics requiressignificant investment of:

– Time

– Resources – Money

http://www.flickr.com/photos/gadgetgirl70/2918038830/

Readiness for

Learning Analytics Will not happen

accidentally

Logistics can

be daunting

Institutional

reflection is

criticalhttp://ieatgrass.com/2011/05/ready‐set‐go/

The Need for Reflection

• Comprehensive understanding

• Multiple perspectives needed

– Cross-disciplinary

– Diverse expertise & skills• More realistic understanding

of resources

• Limit bias that inadvertentlyinterjected

“Thinking” designed by

Ashley Fiveash

The Noun Project

Assessing Readiness

Analytics “Maturity”Educause’s (2012) Center for Analysis and Research’s Analytics Maturity Index

Framework for Creation

Provide an institutional profile with

readiness indicators

Serve as a prescriptive diagnostic

Situated within L.A. literature

Formative instrument focusing on“3 Ps”

Parsimony

Practicality

Proactive

http://www.flickr.com/photos/tschiae/8417927326/

The Learning Analytics

Readiness Instrument(LARI)

Item Development Questions based on:

Literature

ECAR Maturity Index

Practitioners’ experience

Original Factors:

ABILITY DATAGOVERNANCE/

INFRASTRUCTURE

CULTURE PROCESS

What is the most important factor to

consider when preparing for a learning

analytics implementation?

A. Ability

B. DataC. Culture and process

D. Governance and infrastructure

E. Readiness perception

Participants• 33 participants across 9 institutions

(8 US, 1 Canadian)• R1s

• Already established foothold in LA

Exploratory Factor Analysis Began with 139 items

Varimax rotation with Kaiser normalization Exclusion criteria of |.30|

42 survey items eliminated

Second factor analysis 7 additional items eliminated

New Factors

ABILITY

GOVERNANCE /INFRASTRUCTURE

CULTURE

PROCESS

New Factors

ABILITY ABILITY

CULTURE

PROCESS

New Factors

ABILITY

CULTURE

PROCESS

New Factors

ABILITY

CULTURE

PROCESS

New Factors

ABILITY

CULTURE /

PROCESS

ABILITY

CULTURE

PROCESS

New Factors

ABILITY

CULTURE /

PROCESS

OVERALL

READINESS

PERCEPTIONS

ABILITY

CULTURE

PROCESS

Exploratory Factor Analysis

Final 90 survey items

Explained 55.7% of variation

Chronbach’s alpha of .946

Factor # of Items Median Skewness Kurtosis α

Ability 19 -.038 -.401 -.640 .940

Data 22 .228 -1.157 .686 .945Culture &

Process23 .146 -.420 .073 .898

Governance &

Infrastructure15 .164 -1.299 2.131 .896

OverallReadiness

Perceptions11 .006 -.734 1.421 .841

Results

Ability Data

Culture / Process Governance / Infrastructure

Readiness Perceptions

Institutional Differences(at least 2 respondents)

A B C E F G I

Ability Data

A B C E F G I

Ability Data

A B C E F G I

Limitations• Sample size

• Institution type

• Policy implication in international

settings

Breaking Down the Factors

Breaking Down the Factors: Data

Data is…

Foundational

Abundant

Diverse

Data is often…

Abundant (confusing)

Not archived

Dispersed

Siloed

Inaccessible

Data considerations…

“Big” or “small”

Validity and reliability of data sources

Storage, archival, and access

Ownership/stewardship/utilization

LARI Items

The extent to which my institution currently

collects admissions data (e.g., prospect

demographics, interests, applicationevaluation)

At my institution, the primary “owner” (the

person / group responsible for granting accessand stewarding the data) of the following data

sources is....

At my institution there are routine scripts /processes for refreshing / updating institutional

reports.

Reflection Questions…

On what topic(s)/issue(s) do you plan to

focus and why?

What data is important to gather? How will

you know if you have the “right” data? The

“best” data?

What are the challenges associated with

gathering, assembling, and maintaining

Breaking Down the Factors: Ability

Which abilities and expertise are most

important for a successful learning analytics

initiative?

A. Technical architecture

B. Analytics proficiencyC. Strategic leadership

D. Instructional design

Abilit

Ability

Wide-ranging abilities and expertise will

need to accessible

Different abilities will need to be accessed

at different points

Abilities needed will be different dependingon goals

Abilit

Ability

Technical expertise (architecture, coding,

development)

Analytic proficiency

Strategic leadership

Student support skills Instructional and curriculum expertise

Researchers and evaluators

Abilit C id ti

Ability Considerations

Technical expertise (architecture, coding,

development)

Analytic proficiency

Strategic leadership

Student support skills Instructional and curriculum expertise

LARI Items

My institution has professionals with

mathematical/statistical experience in deriving

and validating statistically sound algorithms(e.g., simulations, cross validation).

My institution has professionals who have

specialized analytics training. My institution has professionals with technical

knowledge / expertise in documenting data

store / warehouse structure and datadictionaries.

Reflection Questions

How do you identify where expertise exists?

How will you manage “territorial disputes”

between departments/units that have staff

with needed expertise?

To what extent is your executive leadershipinvolved in moving this project to fruition?

The Learning Analytics Readiness Instrument (LARI)

How would you describe your institution’s approach to using data on a day‐

to‐day basis?

A) We ALWAYS use data to bolster actions we’re going to take

B) Data is available, and sometimes we access it to inform our decision

C) Data is available, but we usually don’t refer to it

D) Data is difficult to get, so we only ask for it when absolutely necessary

E) Data? What’s data? We don’t need data to make good decisions

• Shared vision of what analytics is

• Shared vision of student success

• Awareness and acceptance of data‐driven decision

making

• Goal

orientation

implementing/using

analytics• Stakeholder support of the effort

• Policies and practices amenable to examining/using

student data

• Communication between units about data/data use is

commonplace

Culture

• Stakeholder involvement in the effort

• Project

management

resources• Sustaining new initiatives

• Data and report use in practice

• Established procedures for collecting, organizing, storing, accessing, sharing, using, and securing data

Process

• My institution has the ability to store, manage, and analyze increasingly large volumes of

data.• Institutional reports routinely inform

institutional decisions.

• There is

a clear

vision

institution

can make changes to help students be more

successful academically.

• My institution

measurable

objectives

will indicate if success has been achieved.

• What are the different policies that exist on

campus regarding access to and use of data?

• To

extent

currently

reported

campus?

determination

what is reported, and the audience(s) to whom

those reports go? How are those reports used

recipients?

extent

information presented trustworthy?

• In

institution

defined

student

success?

multiple

definitions

exist, are they competing?

Campus

Discussion

Institutional Report Out

Breaking down the factors:

Governance & InfrastructureThe Learning Readiness Instrument (LARI)

How would you characterize your current

governance and infrastructure around learning

analytics?

A) A benefit. Resources and professionals are

ready to assist in LA efforts.

B) A hindrance. No cohesive rules or resources.

Individuals can block progress.

C) A mixed bag. Some elements work great, someneed a lot of work.

Governance

• Broad, long-term involvement of LA efforts

• Sustainability

• Policy, goals ownership

(related to Culture & Process) – Policies that allow appropriate use

Infrastructure

• Professional knowledge / expertise

– Building & maintaining large data warehouses

– Documenting data sources

– Query building, extraction – Manipulating & combining data sources

• Customer Service

– Diverse constituencies

– Documentation, tutorials

LARI Items - Governance

LARI Items Governance

• My institution has had conversations regarding

the sustainability of an analytics effort.

• People from multiple offices across my institution

are involved in the effort.

• My institution views learning analytics as a long

term investment, rather than a short term

expense.

• My institution has IT professionals who know

how to support analytics.

LARI Items - Infrastructure

• My institution has professionals with expertise in:

– Building / Maintaining complex data stores /warehouses

– Query building and data extraction

– Manipulating data from multiple sources and

platforms to conform to institutional

specifications

– documenting data store / warehouse structure

and data dictionaries

Worksheet

Discuss the following questions:

1. What institutional data is collected, curated, andmaintained for use in research projects?

2. How accessible are the subject matter experts on

institutional data?3. What institutional policies exist that govern the use of

varying kinds of data?

1. To what extent do these policies compete with one

another?

2. Who has the ability to determine which policies apply or

are superfluous to the project?72

Institutional Report Out

Future of the LARI

Next Steps

• Continue iterating the instrument

• Create tailored feedback and automate

– Overall profile score; domain scores

– Low, medium, high readiness

– Broad,

domain‐

specific

feedback – Prompts for exploration

• Build out scaffolds and activities

• Identifying international

partners

Beta Partners

• Lead individual

American

institution

• 10‐15 individuals in various roles representing

the diversity of job roles associated with

analytics

• Provide honest and constructive feedback

• Participate

follow‐

usefulness of report

• To become involved, email

lari.pilot@gmail.com

Expectations Reality

Plug and Play

Immediate results

Solve every problem –

Universal adoption

Everyone would love it!

Fits, starts, reboots

Mostly long termoutcomes

Solve some problems,

create some newproblems

Lackluster use

Not everyone loved it

Institutional Challenges

Data in many places, “owned” by manypeople/organizations

Different processes, procedures, and regulationsdepending on data owner

Sustainability

Everyone can see potential, but all want something slightlydifferent

“can’t you just…”

Faculty participation is essential

Staffing can be a challenge

In the end…

It is challenging

It can be difficultBUT

It is doable

This is the purpose of the LARI

Please consider joining the beta

lari.pilot@gmail.com

Institutional Reflection on Readiness:

The Learning Analytics Readiness Instrument

Kimberly Arnold, University of Wisconsin-Madison

Steven Lonn, University of MichiganMatthew D. Pistilli, Purdue University

institutional reflection on readiness: the learning analytics readiness instrument (237810993)

Documents

teacher evaluation instrument · 2019. 6. 20. ·...

instrument rating - airplane airman certification standards...

using the early development instrument to support school...

gismo simulation study objective key instrument and geometry...

user manual tristimulus reflection analyser tra-2000 · the...

provider readiness to offer programmes using … · web...

reflection of light -...

using the early development instrument to support school...

readiness scorecard – market readiness series

organizational change questionnaire climate of change,...

reflection stone / reflection stone grand

the green it readiness (g-readiness) of organizations- an...

measuring school readiness : the early development...

reflection & refraction -...

reflection cmsc 432. 2 issues what is reflection history of...

waves – topic 4 chapters 26 reflection & refraction...

the calculus concept readiness (ccr) instrument ...6...

reflection and refraction. reflection the law of reflection...

soft matter interfaces (smi) beamline (12id) - instrument...

reflection for learning, learning for reflection...