Large Effects With Small Efforts: What Does it Take to Leverage User Data?
Joe Zucca, Director for Planning and Organizational Intelligence, Penn Libraries
SLA, Philadelphia, April 2, 2014
PrologueProducing Large Effects With Small Effort?
Count of User Type Column Labels
Row Labels FAC Other Staff StudentGrand Total
Canvas Features 217 2 53 16 288
Access 200 2 58 69 329
Site Request 103 41 144
Administration 8 1 7 16
Account Request 1 1 2
Other 1 4 5
Grand Total 530 6 163 85 784
Courseware Problem Incidents: Frequency by Incident & User Group
How Well Are They Managing Complexity (in the LMS)?
Migrate content 41
Settings 29
Course content 27
Assignments 21
Copy content 17
Other (Features) 15
Announcements 14Conversations/messages 13
Multimedia 11
LTI Tools (other) 8
Quizzes/surveys 6
Gradebook 6
Modules 4
Discussions 3
Turnitin 1
Learning Objects 1
Average of lagColumn Labels
Row Labels FAC Staff Student Other Grand Total
Administration 3.0 5.4 5.0 3.9
Canvas Features 1.9 4.0 0.6 27.5 2.4
Site Request 1.4 1.8 1.5
Access 1.3 0.4 1.0 1.5 1.1
Account Request 0.0 7.0 3.5
Other 0.0 3.3 3.2
Grand Total 1.6 2.2 0.9 11.7 1.8
Courseware Problem Incidents: AVG resolution time by Incident & User Group
How Well Are We Managing Complexity (in the LMS)?
Quizzes/surveys 9.2
Multimedia 4.0
Gradebook 3.8
Assignments 2.4
Migrate content 2.0
Copy content 1.8
Course content 1.6
Settings 1.6
Turnitin 1.0
Modules 0.5
LTI Tools (other) 0.4
Other (Features) 0.4
Announcements 0.4Conversations/messages 0.3
Discussions 0.0
E-reserves 0.0
Learning Objects 0.0
The Challenge: The Scope & Complexity of Service
What products and services does your library offer or directly support for the university?
Acquisition and licensing services Access to massive e-content Discovery service Supply chain services [local circulation & third party - ILL] Inventory control Research | Instructional services Courseware service (e.g., Blackboard) Learning environments (e.g. MOOCs) Author service Copyright compliance service Technology support services (e.g., desktops, digital media,
scanning) Digitization services (image production, description, curation) Metadata service Data services Repository services (internal and public-facing digital
archives) Researcher intelligence services (e.g., VIVO)
COUNTER
SFX
SOLRlog
ILLIAD
VIVO
Canvas LMS
BePress
FootPrints
Fedora Repo
Banner IDM
RelaisGoogle Analytics
ARES
HR db
STAFF
SpringShare
AEON
ILSAPACHE
log
EZPROXYlog
AnalyticsRepository
A B.I. Framework
AnalyticsRepository
Extract Trnsfrm Load
Data Mart AnalysisEnterprise Products &Services
Resolvers
a framework for:
Extracting activity data from systems
Transforming those data into readable, normalized formats
Loading transformed/normalized payload into a repository
Supporting analysis through local and collaborative dissemination channels.
https://github.com/upenn-libraries/metridoc-app
Abstracts 4 key functions, exposes interfaces for interoperability
Target Source, e.g. Relais, Illiad, ILS
Ingest Log
Parse
Format
Refined output
1. Extract
Resolution Sources e.g. IdM, WorldCat
Refined output
Resolve Codes & IDs
Normalize
2. Transform
Query Srvc
Data Repo
3. Load
MetriDoc Tiered Architecture
User Interface
LocalData Stores
Results Document
Query Document
4. Query
Economic failure of German colonies in AfricaUsing Google Fusion TablesPapers on reaction kinetics on ethylene oxide hydrationHistory of marketing listerinemonetization of contemporary art especially digital artThe folklore of the supernatural in the mediaArab nationalism at the beginning of the 20th century, and relations with the Ottoman Empire.Medicaid eligibility rules by state dataUsing GIS for studying health in neighborhoodsResearching Democratic party's views on charter schools and high-stakes testing.Health of pregnant women in prisonLocate the full list of proscribed authors under the Yuanyou ban 20thh-century China.Pediatric Tuberculosis Caregivers Attitudeshistory of Chinatown developments in the U.S./CanadaHow the Illiad reflects political realities in Bronze Age Greece.Economic effects Sykes-Picot agreement in Syria.The effect of the Nike settlement on labor relations in the footwear industry around the world.educational linguistics, semiotics, reflectivity and meta-languagetype 1 diabetes and telecommunicationsNative American slaveholders (specifically the Cherokee)Import substitution in Brazil under Vargashousehold interventions for asthma prevention in urban settingdata sets on toilets in MozambiqueThe changing Confucian-influenced perceptions among Song-dynasty writers of the Kitans.Familial lineages in Commedia del ArteWalter Mischel and original marshmallow, gratification studiesResearch on democratically elected Presidents of Brazil since the 1980s.Why Employment Non-Discrimination Act is taking so long to pass.Economic consequences of French colonialism in North Africa.Raster creation in arc gisPossible use of Shoah VHA in class about the Holocaust in Italy.Relation between U.S. intervention in Philippines and Cuban independence.glycogen storage disease and mortalityPutting together an annotated bibliography on the Cristero rebellion.Researching medieval pilgrimages to HebronLliterature by Christian missionaries in the Middle East during the late 19th and early 20th centuries.
EpilogueRaising the Organizational I.Q.
The O.I.Q. NOT about
Software, IT Systems, Network Plumbing, Assessment Plans, or Culture…
…it IS about creating an infrastructure for learning.
AnalyticsRepository
Services Analytics Organizational Knowledge
Understand how users work, what they want, need, and expect, and what’s expected of them
And What Is It We’re Learning to Do?
Guide the library through the morass of strategic choices…
Improve our response to change and the value of our services
Heighten the observational power of our organizations.
“…the key requirement for institutional success is to move from scalable efficiency to scalable learning.
Said differently, the rate of learning, innovation, and performance improvement within the institution must match (or exceed) that of the surrounding environment
if the institution is to survive (or thrive).”
-- John Hagel III and John Seely Brown The New Organization Model: Learning at Scale HBR Blog Network
Joe Zucca | [email protected] Libraries
With the support of
Producing Large Effects With Small Efforts
MetriDoc: https://github.com/upenn-libraries/metridoc-app