documenting internet2 an it perspective eric celeste university of minnesota (twin cities) libraries...
TRANSCRIPT
![Page 1: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/1.jpg)
Documenting Internet2an IT perspective
Eric CelesteUniversity of Minnesota (Twin Cities)
Librariesfor the Coalition for Networked Information
6 December 2005
...or... A joyful romp with Heritrix, JavaScript, & Spotlight!
![Page 2: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/2.jpg)
background...
• DI2 brought together– University of Minnesota (CBI)– University of Michigan (SI)– Internet2
• web crawling only a small part
• the “save everything” approach
![Page 3: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/3.jpg)
briefly…
• on crawling with spiders• on Heritrix and JavaScript• on Spotlight and local files• on sinkholes and strategies
![Page 4: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/4.jpg)
spiders on the web
![Page 5: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/5.jpg)
pages
![Page 6: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/6.jpg)
links
![Page 7: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/7.jpg)
hosts & domains
![Page 8: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/8.jpg)
robots.txt
![Page 9: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/9.jpg)
scope
![Page 10: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/10.jpg)
seeds
![Page 11: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/11.jpg)
excluded pages
![Page 12: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/12.jpg)
![Page 13: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/13.jpg)
![Page 14: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/14.jpg)
![Page 15: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/15.jpg)
![Page 16: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/16.jpg)
![Page 17: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/17.jpg)
![Page 18: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/18.jpg)
done!
![Page 19: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/19.jpg)
our crawler
• Heritrix, from the IA• aiming for broad deployment, Archive-It
• cross-platform, many users• simple setup, sophisticated options
• generates ARC files
![Page 20: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/20.jpg)
from ARC to archive
• keep originals intact• a few large files to manage• can serve a mirror from the master
• can extract files for research• solution requires Perl, PHP, JavaScript, MySQL
![Page 21: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/21.jpg)
processing...
• for mirroring online– optimizing and indexing with Perl
– loading into MySQL database– presenting via PHP
• for using on local disk– extracting files from ARC
![Page 22: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/22.jpg)
joys of javascript...
• modifies the page after loading
• HTML almost unmolested• changes explicit in code
![Page 23: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/23.jpg)
are we there yet?
• make the archive obvious• yet intrude as little as possible
![Page 24: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/24.jpg)
global research locally• a web site in your pocket• applying local tools• maintaining browse-ability• Apple’s Spotlight one of many
![Page 25: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/25.jpg)
sinkholes / strategies• partnership with institution
– config, IP, retention
• crawling far from perfect– no creation dates, exclusions– sticky traps, scripted pages (AJAX)
• scripts still immature– better demarcation– more self-contained (not at /)
![Page 26: Documenting Internet2 an IT perspective Eric Celeste University of Minnesota (Twin Cities) Libraries for the Coalition for Networked Information 6 December](https://reader036.vdocument.in/reader036/viewer/2022062518/56649f2c5503460f94c475c7/html5/thumbnails/26.jpg)
still...
• capture & save what we can• keep it as “original” as possible
• stay flexible for the future• have fun in the present!