experimenting with a machine generated annotations pipelineucla library experimenting with machine...
TRANSCRIPT
![Page 1: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/1.jpg)
Joshua Gomez, Head of Software Development & Library Systems, UCLA LibraryCNI Fall Membership Meeting, December 10, 2019
Experimenting with a Machine Generated Annotations Pipeline
![Page 2: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/2.jpg)
Experimenting with Machine Generated Annotations
UCLA Library
WHO AM I AND WHY AM I HERE?
December 10, 2019 2
INTRODUCTION
![Page 3: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/3.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
Joshua Gomez, Head of Software Development & Library Systems
December 10, 2019
3
Introduction
• Joined UCLA Library in Fall 2018• Lecturer in Information Studies since Spring 2016• Formerly at the Getty Research Institiute
Tasked with: 1. Modernizing the software development process2. Reorganizing the dev team3. Improving the user experience of our digital systems4. Bringing the systems portfolio under control
![Page 4: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/4.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
To improve our quality and velocity, adopt a DevOps culture, including changes to:
December 10, 2019
4
Modernizing Library Software Development
• Structure: small focused dev teams; embedded Ops and UX (a matrixed org)
• Process: Agile planning; testing (both code and usability)
• Tools: containerization & orchestration; CI/CD pipelines
• Architecture: evolutionary (event-driven & microservices)
• Strategy: experimentation & evidence-based decisions
![Page 5: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/5.jpg)
Experimenting with Machine Generated Annotations
UCLA Library
WHAT’S IT ALL ABOUT?
December 10, 2019 5
MGAP EXPERIMENT OVERVIEW
![Page 6: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/6.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
MACHINE GENERATED ANNOTATIONS PIPELINE
December 10, 2019
6
MGAP Experiment
Research QuestionCan commercial image tagging services improve the digital library’s metadata?
AssertionWe will measure improvement via user testing.
HypothesisThe tags will be of no use to scholars (expert users) conducting research, but they
will be of use to casual (nonexpert) users just looking for interesting images.
![Page 7: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/7.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
UCLA DIGITAL LIBRARY
December 10, 2019
7
MGAP Context
• New platform under development (https://digital.library.ucla.edu/)
• Uses IIIF protocol for images• See Project Mirador for demo of a IIIF-powered viewer
• Web Annotations is a sister protocol to IIIF
• Enables anyone to annotate any web resource
• Could be used for transcription, translation, crowd-sourced metadata,
scholarly dialogue, etc.
![Page 8: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/8.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
THE LABS – A SOFTWARE DEV SUBTEAM
December 10, 2019
8
MGAP Team
• Mission: conduct experiments; build prototypes• Projects should be strategic!
• Team Members:
Dev Ops UX
Kristian Allen Anthony Vuong Tinuola Awopetu
Mark Matney Sharon Shafer
![Page 9: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/9.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
FIRST EXPERIMENT - MGAP
December 10, 2019
9
MGAP Motivation
• Strategic Motivations:
• Answer the question: Can tags improve the digital library?
• Gain experience with Web Annotations (for future user functionality)
• Gain experience with event-driven systems (for future software architecture)
• Gain experience conducting usability tests (part of UX Strategic Plan)
• Try out usertesting.com
![Page 10: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/10.jpg)
Experimenting with Machine Generated Annotations
UCLA Library
HOW DID WE DO IT?
December 10, 2019 10
MGAP SETUP
![Page 11: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/11.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
TOOLS
December 10, 2019
11
MGAP Setup
• Queue processing: Python, Celery, RabbitMQ
• Image tagging: Google Vision, AWS Rekognition, Clarafai, Azure CV*
• Annotation storage: Elucidate
• Search Index: Apache Solr
• Search interface: Blacklight
• Deployment: Docker, AWS
![Page 12: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/12.jpg)
UCLA Library
Experimenting with Machine Generated Annotations July 22, 2019
12
MGAP Architecture
![Page 13: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/13.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
DIGITAL IMAGE COLLECTIONS
December 10, 2019
13
MGAP Content
• LA Daily News Negatives• 5,172 images
• Will Connell Papers• 502 images
• Walter E. Bennet Photographic Collection
• 79 images
![Page 14: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/14.jpg)
Experimenting with Machine Generated Annotations
UCLA Library
PART I: IN-PERSON
December 10, 2019 14
MGAP TESTING
![Page 15: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/15.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
IN-PERSON
December 10, 2019
15
MGAP User Testing
• Participants: 9 UCLA staff (mostly from the library)• Note: It's quite difficult to recruit students during finals week!
• Setup: • 5 instances of the digital library interface
• 1 unaltered (base case)• 3 with tags from individual services• 1 with combined tags from all three services
• Each user sees 2 browser windows:• Base case• 1 of the 4 alternatives
![Page 16: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/16.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
IN-PERSON
December 10, 2019
16
MGAP User Testing
• Test 1: Specific search• Participants given exact terms (sunglasses, chinatown, cat)• Perform search in both interfaces• Asked to rate relevancy of results
• Test 2: Scenario• Participants given topic (wardrobe selection)• Perform searches with their own terms• Asked to rate relevancy of results
• Test 3: Free-form• Participant performs a search of their own• Asked to rate relevancy of results
![Page 17: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/17.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
IN-PERSON
December 10, 2019
17
MGAP User Testing
• Findings• Users want to scan lots of images at once
• Will page through results, unlike text-based searching
• Users expect more semantic understanding by the system• Synonyms should produce the same results (cat, feline)• Tags should be members of ontological hierarchies (cat, mammal)
• Users did not understand results ranking• Some tags were irrelevant
![Page 18: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/18.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
IN-PERSON
December 10, 2019
18
MGAP User Testing
• Limitations• Participant pool may be "biased"
• Library staff are not representative of library users
• Image set was not very large • Did not specifically ask to compare across the two interfaces
• Challenges• Recruiting• Scheduling
![Page 19: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/19.jpg)
Experimenting with Machine Generated Annotations
UCLA Library
PART II: ONLINE
December 10, 2019 19
MGAP TESTING
![Page 20: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/20.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
ONLINE
December 10, 2019
20
MGAP User Testing
• Platform: https://usertesting.com • Test creation similar to building a form
• Benefits: • Panel of ~1.5 million users/testers
• No more recruiting!• Demographic filtering capability
• Tests are unmoderated• No more scheduling woes!
• Display and audio recording
• Drawbacks:• Tests must be very well defined• Cost $$$$
![Page 21: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/21.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
ONLINE
December 10, 2019
21
MGAP User Testing
• Participants: 21 total• 1 dry run• 5 for each of the 4 test setups
• Setup: • Same as in-person setup• 5 instances of the digital library interface
• 1 unaltered (base case)• 3 with tags from individual services• 1 with combined tags from all three services
• Each user sees 2 browser windows:• Base case• 1 of the 4 alternatives
![Page 22: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/22.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
ONLINE
December 10, 2019
22
MGAP User Testing
• Test 1: Specific search• Same as in-person Test 1• Added: Compare relevancy between the two interfaces
• Test 2: Compare tags• Ask user to look at tags for a given image in all four interfaces• Compare relevancy across the sets of tags
• Test 3: Scenario• Participants given topic (wardrobe selection) • Perform searches with their own terms • Asked to rate relevancy of results
• Test 4: Free-form • Participant performs a search of their own• Asked to rate relevancy of results
![Page 23: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/23.jpg)
UCLA Library
Experimenting with Machine Generated Annotations December 10, 2019
23
MGAP User Testing
![Page 24: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/24.jpg)
UCLA Library
Experimenting with Machine Generated Annotations December 10, 2019
24
MGAP User Testing
![Page 25: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/25.jpg)
UCLA Library
Experimenting with Machine Generated Annotations December 10, 2019
25
MGAP User Testing
![Page 26: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/26.jpg)
UCLA Library
Experimenting with Machine Generated Annotations December 10, 2019
26
MGAP User Testing
![Page 27: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/27.jpg)
UCLA Library
Experimenting with Machine Generated Annotations December 10, 2019
27
MGAP User Testing
![Page 28: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/28.jpg)
UCLA Library
Experimenting with Machine Generated Annotations
ONLINE
December 10, 2019
28
MGAP User Testing
• Challenges• Volume of respondent data is large
• 15-45 minutes of video for each user (20x)• Limitations
• Image set was not very large and narrow in scope • Participant size was too small for statistics• Poor choice of instrument
• We tried to get quantitative data from user testing, which is really a qualitative research method
![Page 29: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/29.jpg)
Experimenting with Machine Generated Annotations
UCLA Library
WHAT'S NEXT?
December 10, 2019 29
CONCLUSION
![Page 30: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/30.jpg)
UCLA Library
Experimenting with Machine Generated Annotations December 10, 2019
30
MGAP Outcomes• Findings
• Inconclusive, but results suggest our hypothesis was correct• Tags do not provide a strong benefit; may actually confuse users• Of the 3 services, Google seems to perform best and AWS worst
• Decisions• Do not add CV tags to the digital library (yet)• Alternative tag-based image portal may be useful to casual users
• Achievements• The experiment was inconclusive, but the project was a success!
• Achieved strategic goals of training team on desired skills• Discovered UX problems during testing; led to enhancements on
production site
![Page 31: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/31.jpg)
UCLA Library
Experimenting with Machine Generated Annotations December 10, 2019
31
Future Work• AI & Libraries
• Try more metadata enhancement experiments• Event-driven systems
• PubSub for all repositories (catalog, Dataverse, Dig. Lib.)• Aggregated index and single search for all library data
• IIIF & Web Annotations• Custom collections• Scholarly annotations• Time-based media support
• Library UI improvements• Website redesign• More user testing!!!
![Page 32: Experimenting with a Machine Generated Annotations PipelineUCLA Library Experimenting with Machine Generated Annotations To improve our quality and velocity, adopt a DevOps culture,](https://reader035.vdocument.in/reader035/viewer/2022063004/5f7ada1a0dd1587c254f5b39/html5/thumbnails/32.jpg)
Experimenting with Machine Generated Annotations
UCLA Library
Thank You
December 10, 2019 32