Automating Authority Work

Mike Monaco

Coordinator, Cataloging Services

May 14, 2018

Automating authority work, or,

Be your own authority control vendor

Mike MonacoCoordinator, Cataloging ServicesThe University of Akronmmonaco@uakron.edu

Ohio Valley Group of Technical Services Librarians2018 Conference, May 13-15, 2018Hesburgh LibrariesThe University of Notre DameSouth Bend, Indiana

Who are you?John Carroll University (2001-2004)

Part-time AV catalogerAkron-Summit County Public Library (2001-2004)

Substitute librarianCleveland Public Library (2004-2016)

Catalog librarianThe University of Akron (2016- )

Coordinator, Cataloging Services

The University of Akron Libraries

University Libraries

Bierce Library

Science & Technology Library

Archival Services

(Separate Units)

Wayne College Library

Akron Law Library

Center for the History of Psychology

Authority control at UA● 1995 migration and vendor (BNA) supplied one-time authority processing

● Local authority work put on hold in expectation contracting with a vendor…which never happened

● Authority work resumed early 2000s

○ Full authority control for tangible items only

○ Shift to batches of e-resources over time made authority work for batches overwhelming

○ 2013: Budget 80:20 electronic:tangible

○ 2018: ratio is about 95:5

What this is NOT about

Automated authority control within the ILS

Working with an authority control vendor

What this IS

Grabbing the “low-hanging fruit” for batches of records

When traditional authority work is not practical

(the item is not in-hand

or headings reports are too vast to address individually)

Wouldn’t it be nice if...

The “Headings used for the first time” report could export a list of the headings, and we could batch search OCLC for records?

The tool box● MarcEdit● OCLC Connexion Client● Excel (or other program for sorting textual lists)● pgAdmin (or similar for a SQL query, III/Sierra only)● A rudimentary grasp of Regular Expressions● EditPad (or similar RegEx-compatible text editor: Google Sheets, EmEditor)

The process1. Before loading, correct variant headings (with MarcEdit)2. After loading, extract headings from report (with SQL query or ILS’s output)3. Separate names and subjects (in a spreadsheet or text editor) 4. Remove extraneous data (with RegEx-capable editor)5. Batch search for authority records (in Connexion Client)6. Load authority records

Validate HeadingsMarcEdit can check name and subject fields against LC authorities in the Linked Data Service, and automatically correct headings that match a variant (“Use for”) heading*.

*NB: The process is imperfect!

Because this is an extra step, we’ve been comparing record sets from various vendors to determine which ones really benefit.

Selected Vendor Loads (March 2017-March 2018)

Record Source

Records per load

Invalid headings per load

Variants changed per load

Invalid heading:Record ratio

Variant:Record ratio

Alexander Street Press

293 117 18 0.399279 0.060566

EBSCO 76992 75668 181 0.982807 0.002348

Films on Demand

2509 1019 175 0.406314 0.069911

Kanopy 9960 4946 397 0.496628 0.039815

Proquest EBC

13086 2309 101 0.17647 0.0077

World Share 31 7 0.7 0.232114 0.023772

III Sierra

SQL query of Headings used for the first time report

https://mmonaco-uakron.tinytake.com/sf/MjUwMDQxMF83NTIyNTY0

Headings used for the first time

Hundreds or even thousands of entries after batch loads...

SQL query*

Results...

In Excel...Be sure to import as Unicode

(UTF-8) if your ILS is encoding characters as Unicode rather than MARC8!

Sort the termsSorting A-Z arranges the headings by field group tag and MARC tag

(a=names, b=other names, d=subject)

So

a100-b730 : names used as names

d600-d630 : names used as subjects

d650- : subjects

NoticeYou can’t feed this raw data into a batch search in Connexion Client

In EditPad(or other RegEx-enabled editor)

Strip out MARC tags, delimiters, punctuation, etc.

Find/replace using RegEx(.*\|a)

(\|db\. ca\. |\|db\. |\|d\. ca\.|\|dd\. |\|dca\. |-ca\. |\|dfl\. ca\. |\|dfl\.)

(\|e.*|\|4.*|\|0.*)

(\|x.*|\|v.*|\|z.*)

(\|.|\|$)

(;|:|\(|\)|\?| and | or |&c\.|&| in | an |,| the | for | on | so | with | to | by |”|’ | ‘| be | that |\.{3}| near )

Names(.*\|a)

Everything before |a

(\|db\. ca\. |\|db\. |\|d\. ca\.|\|dd\. |\|dca\. |-ca\. |\|dfl\. ca\. |\|dfl\.)

AACR2 abbreviations

b. d. fl. ca.

(\|e.*|\|4.*|\|0.*)

Relator terms, URIs

(\|.|\|$)

Any remaining delimiters and subfield codes

(;|:|\(|\)|\?| and | or |&c\.|&| in | an |,| the | for | on | so | with | to | by |”|’ | ‘| be | that |\.{3}| near )

Punctuation, operators, and stopwords that foil OCLC searches

Names as subjects(\|x.*|\|v.*|\|z.*)

Subdivisions

Converting SQL output to batch searchable text file with RegEx

https://mmonaco-uakron.tinytake.com/sf/MjU4ODk4OF83Nzg3NTMy

Name headings

(.*\|a)

(\|db\. ca\. |\|db\. |\|d\. ca\.|\|dd\. |\|dca\. |-ca\. |\|dfl\. ca\. |\|dfl\.)

(\|e.*|\|4.*|\|0.*)

(\|.|\|$)

(;|:|\(|\)|\?| and | or |&c\.|&| in | an |,| the | for | on | so | with | to | by |”|’| be | that |\.{3}| near )

Names (can be skipped)(.*\|a)

(\|db\. ca\. |\|db\. |\|d\. ca\.|\|dd\. |\|dca\. |-ca\. |\|dfl\. ca\. |\|dfl\.)

(\|e.*|\|4.*|\|0.*)

(\|x.*|\|v.*|\|z.*)

(\|.|\|$)

(;|:|\(|\)|\?| and | or |&c\.|&| in | an |,| the | for | on | so | with | to | by |”|’| be | that |\.{3}| near )

Names as subjects(.*\|a)

(\|db\. ca\. |\|db\. |\|d\. ca\.|\|dd\. |\|dca\. |-ca\. |\|dfl\. ca\. |\|dfl\.)

(\|e.*|\|4.*|\|0.*)

(\|x.*|\|v.*|\|z.*)

(\|.|\|$)

(;|:|\(|\)|\?| and | or |&c\.|&| in | an |,| the | for | on | so | with | to | by |”|’| be | that |\.{3}| near )

Topical subjects (can be skipped)(.*\|a)

(\|db\. ca\. |\|db\. |\|d\. ca\.|\|dd\. |\|dca\. |-ca\. |\|dfl\. ca\. |\|dfl\.)

(\|e.*|\|4.*|\|0.*)

(\|x.*|\|v.*|\|z.*)

(\|.|\|$)

(;|:|\(|\)|\?| and | or |&c\.|&| in | an |,| the | for | on | so | with | to | by |”|’| be | that |\.{3}| near )

Sirsi/Dynix Symphony

List unauthorized tags report

Slightly different procedure to clean these up1. Open .txt file in editor2. Delete header of report 3. Find/Replace to delete page headers (“Tags With UNAUTHORIZED

Headings / Produced on Sat Jul 1 17:00:11 2017”)4. Separate name and topical headings5. RegEx to remove other data

So far so good...(.*\|a)

Uh oh...(\|e.*|\|4.*|\|0.*)

Misses “|?UNAUTHORIZED” by itself.

Only captures it if preceded by |e |4 |0

(\|e.*|\|4.*|\|0.*|\|\?.*)

\|\?.*

captures

“|?” followed by anything

Problems

Spaces

Scraps

Portions of “|?UNAUTHORIZED” that wrapped to new line

AsterisksOutput changes any diacritics to them

Line breaksName/Title headings are especially likely to get broken up. Here, the delimiter was even separated from the subfield code “t”

Workaround1. FIND: \* REPLACE:[nothing] to

delete asterisks2. Use EditPad’s “Extras” to delete blank

lines, duplicate lines, etc.3. Depending on the number of items,

you might close up split lines by hand.

Searching in batches

Searching a batch of terms in Connexion Client

https://mmonaco-uakron.tinytake.com/sf/MjU4OTAyMF83Nzg3NjE2

Batch searching“Use default index” settings

nw: for names/titles

su: for topics/geographic terms

Maximum number of matches to download: 1 (Tools>Options>Batch)

Batch searchingNOTE:

Your local save file has a maximum capacity of 10,000 records, so don’t search more than that many strings!

Successful name searches (of 1941 entries)

Names, names as subjects, and subjectsIII requires name headings that are to be used as subjects to be loaded separately from name headings to be used as names!

SirsiDynix does not have this issue.

A four month test

I’m very pleased with hit rate on names!

Total headings extracted

Hits in batch search Success rate (ARs found for heading)

Names 36,244 21,760 60 %

Names as Subjects 3,795 896 23.6 %

Subjects 29,147 1,516 5.2 %

A four month test

Main issue:

Name/Title headings often not established

Total headings extracted

Hits in batch search Success rate (ARs found for heading)

Names 36,244 21,760 60 %

Names as Subjects 3,795 896 23.6 %

Subjects 29,147 1,516 5.2 %

A four month test

Main issues:

Sierra treats subdivided subject headings as single headings, inflating report (3.4 should correct this issue!)

Music headings (instruments, Arranged) often valid but not established in an AR

Total headings extracted

Unique hits in batch search

Success rate (ARs found for heading)

Names 36,244 21,760 60 %

Names as Subjects 3,795 896 23.6 %

Subjects 29,147 1,516 5.2 %

Is it worth it?Typically this process (excluding Validate Headings in MarcEdit) takes less than an hour and resolved 36% of unauthorized headings -- averaging over a thousand ARs a week. Several “known issues” in Sierra made me place this project on hold until they are fixed however.

So consider

1. the number of new headings you normally see in a report, 2. the quality of your incoming bib records (can the headings be authorized?)

and 3. the capability of your ILS to use authority records effectively

Questions?If you decide to test this process at your institution or discover any refinements, please let me know!

mmonaco@uakron.edu

Useful toolsHelp with regular expressions: Regular Expressions 101 http://regex101.com

RegEx enabled text editors: Editpad Lite https://www.editpadlite.com

EmEditor https://www.emeditor.com/

pgAdmin free software: https://www.pgadmin.org/download/

This presentation: https://events.library.nd.edu/ovgtsl2018/talk/monaco.shtml