Converting Local Portal Data to MySpace
Anne L. Highsmith, Consortia Systems Coordinator &
Prabha Acharya, Programmer Analyst IITexas A&M University
Converting Local Portal Data to MySpace ELUNA 2007
Points to Cover
Background Data Conversion Issues Testing on a Cloned Database Conversion Process Results and Future Plans
Converting Local Portal Data to MySpace ELUNA 2007
Background (1)
TAMU had a local Electronic Resources Locater (ERL) function to list ejournals and databases.
Users searched ERL and clicked on links to access eresources
Users could store the links in their MyPortal record on the library’s website
Converting Local Portal Data to MySpace ELUNA 2007
Background (2)
Metalib went into production at TAMU in January 2006 as Search Now.
TAMU had discussions with Ex Libris staff as to how MyPortal links could be converted to Metalib First proposal, for Ex Libris staff to do a major
conversion, not feasible Decided to wait for Metalib 4.0 and X-Portal
APIs to store data to MySpace
Converting Local Portal Data to MySpace ELUNA 2007
Rude awakening!
Fall of 2006, Ex Libris staff informed TAMU there would be no API to store to MySpace
Alternatives Margaritas? Convert them ourselves?
Converting Local Portal Data to MySpace ELUNA 2007
Conversion options
Assumed conversion would have to be: User-initiated Title by title
Discovered a way do the conversion directly, without involving the end user
Converting Local Portal Data to MySpace ELUNA 2007
Data Conversion Issues
Patrons Journals Databases
Converting Local Portal Data to MySpace ELUNA 2007
Data Conversion Issues - Patrons
How to match the MyPortal patron with the Metalib user? MyPortal patrons identified by uin (University
ID Number) Our Metalib implementation uses uin as user
login Needed to match the uin with the
z312_source_id
Converting Local Portal Data to MySpace ELUNA 2007
Data Conversion Issues - Journals
How to match MyPortal journals with Metalib? Basic journal data stored in z11 table
Wrote Perl script (journals.pl) to extract journal identifier and issn from z11
Used MS Access to match MyPortal data, keyed by uin & issn, with Metalib data
User journal data stored in z312_jnl_001 Wrote Perl script (journals_by_patron.pl) to
extract z312_source_id (uin) and journal identifier from Metalib user record in z312 table
Converting Local Portal Data to MySpace ELUNA 2007
Data Conversion Issues - Databases
How to match MyPortal databases with Metalib? During creation of Metalib IRDS, stored the
ERL key in the IRD cataloger note Loaded IRD extract into MS Access
Matched Metalib user extract with IRD extract Basic database data stored in z122 table
Wrote Perl script (databases_by_patron.pl) to extract z312_source_id (uin) and Metalib identifier (TEX #)
Converting Local Portal Data to MySpace ELUNA 2007
Data Conversion Issues - Final
As a result of matching ERL data and Metalib data, created a file that contained 1 or 2 entries for each user whose data had to be converted: Tab-separated record beginning with uin and
list of database id numbers Tab-separated record beginning with uin and
list of journal id numbers
Converting Local Portal Data to MySpace ELUNA 2007
Testing on a Cloned Database
Converting Local Portal Data to MySpace ELUNA 2007
Testing – Cloning the database (1)
TAMU has a test Metalib DB on separate server
Cloned production DB to test DB so could test in a realistic environment mpsync - documented in Appendix D in the
System Administrators Guide & Knowledge Base entry #1240.
exp_current_library & imp_current_library, documented in 7.8 in the SysAdmin Guide
Further doc – “Working with Test/Failover Installations of MetaLib”
Converting Local Portal Data to MySpace ELUNA 2007
Cloning – Export from Production
Created files on prod for export to test Run mpsync to copy IRDs
TAMU has approx. 4500 IRDS Elapsed time – 58 sec.
Run export current library for vir00 TAMU has 32,000+ user records Elapsed time – 1 hr.
Run export current library for jnl01 TAMU has approx. 44,000 journal records Elapsed time – 2:25 min.
Converting Local Portal Data to MySpace ELUNA 2007
Cloning – Import into Test
Import IRDS created by mpsync Elapsed time – 3 hrs. 688 new records added Matched existing keys. Appeared to create
some duplicates Import User Data
Elapsed time – 40 min. Drops & re-adds tables
Import Journal Data Elapsed time – 3 min. Drops & re-adds tables
Converting Local Portal Data to MySpace ELUNA 2007
Conversion Process
Quality control File preparation Loading new patrons Convert databases and ejournals
Converting Local Portal Data to MySpace ELUNA 2007
Conversion – Quality Control
Wrote a Perl script (pre_migration.pl) to collect stats and set up QC measures. For each user it reports: Users who don’t have Metalib record # of databases and journals already in Metalib # of databases and journals to convert # of users that have more than maximum
number of databases or ejournals
Converting Local Portal Data to MySpace ELUNA 2007
Conversion – File Preparation
Testing demonstrated that It takes about 3 seconds per resource to be migrated.
With about 13000 resources that would be about 10 hours
So, wrote Perl script (breakup_myportal_file.pl) to break the input file into several files of about 250 records each.
Converting Local Portal Data to MySpace ELUNA 2007
Conversion – Loading new patrons
Since TAMU uses external authentication rather than patron load, some older MyPortal patrons not in Metalib database
Using file created during QC stage, create_xml_patron_load.pl formats user records for Metalib patron load utility Set expiry date to 1 year in future Based on a review of records created on that
date, approx. 65 have since logged in.
Converting Local Portal Data to MySpace ELUNA 2007
Conversion - Convert databases and ejournals (1) Run databases_by_patron.pl to get
current lists from user records for comparison
Converting Local Portal Data to MySpace ELUNA 2007
Conversion - Convert databases and ejournals (2) Using save_to_bookbag.cgi, for each
user Log in to Metalib Send request to Metalib pds to get pds handle Send request to Metalib, using just obtained
pds handle, to get session id Construct url of the form
http://metalibhost.edu/V/sessionid?pds_handle=pdshandle&func=find-db-add-res&resource=resourceid
Converting Local Portal Data to MySpace ELUNA 2007
Conversion - Convert databases and ejournals (3) Set session id in cookie For each database
Check to see if it duplicates db already stored If not, store db to user’s clipboard
For each journal Store to user’s record
Converting Local Portal Data to MySpace ELUNA 2007
Conversion - Convert databases and ejournals (4) Do QC to make sure that all of the
appropriate databases and ejournals are stored
Exit the session Error logging revealed that for some users, 1
database or ejournal was not stored – these were re-run as an added input file
Converting Local Portal Data to MySpace ELUNA 2007
Final results
Processing all files took most of the week, off and on
Each file, limited to 250 records, took about 1 hour to process
Only post-production problem had nothing to do with conversion – ezproxy error.
Once users have logged into library portal, can now go directly to My Databases or My Ejournals list within Metalib
Converting Local Portal Data to MySpace ELUNA 2007
Future plans
Conversion of Voyager My Bookbag to e-shelf
Converting Local Portal Data to MySpace ELUNA 2007
Contact & download info
Anne L. Highsmith
Consortia Systems Coordinator
Texas A&M University
URL to download perl code, cgi, etc.http://libraryasp.tamu.edu/users/ahighsmi/
presentations/save_to_bookbag.zip