web archiving poses challengesbesser.tsoa.nyu.edu/howard/talks/18composers-buckland.pdf ·...

1/18/18

1

MakingWebArchivingWorkforStreamingMedia:

ArchivingtheWebsitesofContemporaryYoungComposers

HowardBesser,NYUh3p://besser.tsoa.nyu.edu/howard/Talks/

Besser-BerkeleySeminar1/19/2018 1

MakingWebArchivingWorkforStreamingMedia

•  Backgroundissuesandproblems•  TheProject

– OurTechnicalCollaboraLon– OurCollaboraLonwithContentCreators&restricLons

– Architectures&Workflows– Howthingsmaylook– EvaluaLon

•  ImpactbeyondthisProject


BACKGROUNDISSUESANDPROBLEMS


WebArchivingposeschallenges

•  Anygivenwebpagemaybeupdatedfrequently

•  Weblinksconstantlybreak(404errors)•  Fewtools/servicesexistfor“Curated”webarchiving(Archive-It,CDL’sWAS),andtheyrequiresignificanttraining/experiencetolearn,butwedohaveint’l-acceptedformat(WARC)


ManyparametersneedtobesetforWebArchiving

•  Frequencyofcrawls•  Depthofcrawls(#ofhops)•  StarLngpointsofcrawls(seeds)


Otherissuesfordevelopinggoodcrawls

•  Qualitycontrol/assurance•  Workflows•  Fidelitytooriginalwebpages•  Howenduserwillnavigateandviewit


1/18/18

2

Archive-It

•  TheleadingapplicaLon/serviceforcuratedwebarchivinginNorthAmerica

•  RunbytheInternetArchive,andismuchmoretargetedandcuratedthantheirWayBackMachine

•  IsbasedonCrawlersohwaredevelopedbyIA(Heritrix)in2003-2004

•  IsverypooratcapturingstreamingaudioorvideoaswellasinserLngitproperlyintoacomposedwebpage-


Archive-ItIssuesw/StreamingMedia






Archive-Itscreenshotsgeneratedaspartofourproject-

•  ByLorenaRamirez-Løpez


Archive-ItIssuesw/StreamingMediaFireFoxversion39.0.ScreenshotofTarikO’Regan’ssitetaken2015/10/05


1/18/18

3





Archive-ItIssuesw/StreamingMediaFireFoxversion39.0.ScreenshotofTedHearne’swebsitetaken2015/10/05


Somesourcesofstreamingissues

•  Problemswithcapturingresourcesresidingon3rdpartyservices(YouTube,Vimeo,Soundcloud)

•  ProblemswithhowfaithfullytheA/VmaterialsarecapturedandplacedbyArchive-It

•  ProblemswithwebsitesgeneratedthroughsitebuildingplamormssuchasSquarespace


OtherIssueswe’retryingtosolve

•  DiscoveringURLsgeneratedbyJavascript


THEPROJECT


1/18/18

4

ArchivingComposerWebsitesh3p://www.nyu.edu/about/news-publicaLons/news/2015/03/27/nyu-libraries-to-team-with-internet-archive-to-preserve-high-

quality-musical-content-on-the-web.html

•  Collect,preserve,&makeavailableWebsitesofComposers

•  $480,000grantfromMellonin2015toNYULibrary/MIAP/InternetArchive

•  Dealingwiththeissuethatcontemporarycomposerwebsitesgoupanddown(andalsoincorporaterelaLonship-buildingbtwncomposerandfans)

•  AddressingtheproblemsofcollecLngstreamingmedia•  AlsoselecLvelycollecLnghigh-qualityversionsthatareusedtogeneratethestreams,andallowingfutureresearcherstosee/hearthehigherqualityversions


ArchivingComposerWebsites


•  DevelopgoodandongoingrelaLonshipsbtwnLibrariesandComposers

•  DevelopTrust–  fordevelopingcollecLons,andconLnuingtoaddtothem–  forPolicyreasons

•  Examinewhattypeoferrorstakeplace–  howfaithfullyaudiovisualmaterialsarebeingcaptured–  howresourcesthatresideonthird-partyweb-services(YouTube,Vimeo,Soundcloud)are(not)displayedwithinArchive-It’sinterface

–  IssueswwebsitesgeneratedthroughsitebuildingplamormssuchasSquarespace

•  Findwaystofixthoseerrors

MetricsAccomplished(asofJan2016)

•  172Composersitescrawled,scoped,assessedforquality,&analyzedforproblems(feedingintoIAdevelopmentwork)

•  800QA/QCreportsgenerated•  IniLalwebarchivingagreementfrom165Composers(25fromNPR’s100)

•  IdenLfiedwebsiteinfrastructuresencounteredandcreatedaclassificaLonmatrix-


WebsiteInfrastructureencountered


ProjectTeam•  JeffersonBailey(InternetArchive)•  HowardBesser(MIAP)•  LoriDonovan(InternetArchive)•  AprilHathcock(Lib/ScholComm)•  NicoleGreenhouse(Lib/ACM)•  CarolKassel(Lib/DLTS)•  Sco3Statland(MIAP)•  DonaldMennerich(Lib/ACM/DLTS)•  DavidMillman(Lib/DLTS)•  CourtneyMumma(InternetArchive)•  RobinPreiss(Lib/AFC)•  LorenaRamirez(MIAP)---specialthanks!•  MichaelStoller(Lib/C&RS)•  KentUnderwood(Lib/AFC)•  ChelaSco3Weber(Lib/AFC)--departed


OURTECHNICALCOLLABORATION:CRAWLING


1/18/18

5

NYU/IACollaboraLon


NYU/IACollaboraLon


TradiLonalCrawlers


•  Archive-ItandotherwebarchivesuseHeritrix•  Followlinks,capturemostwebcontent•  Lesssuccessfulwithstreamingvideoanddynamiccontentexecutedinthebrowser

•  Umbrahelps

BROZZLER!

“browser” | “crawler” = BROZZLER

Logo: Noah Levitt Besser-BerkeleySeminar1/19/2018 28


BrozzlerSystemArchitecturev1


1/18/18

6

BrozzlerModel

•  job:collecLonofseeds•  seed:principalunitofcrawlconfiguraLon

–  onebrowserworksononeseedataLme(politeness)–  seedhasitsownconfiguraLon,alsoinheritsfromparentjob

•  page:atomicunitofcrawlingfrombrozzlerperspecLve

•  url:onlybrowsers,warcproxhavetodealwitheveryurl


Warcprox:WARC-wriRnghSpproxy

•  man-in-the-middleforh3ps•  asynchronous:WarcWriterThread

– writeswarcrecords– savesdeduplicaLoninfo– updatesstaLsLcs


Otherpieces

•  pythonwayback•  Rethinkdb(distributeddocumentstore)


StreamcapturereliesonYoutube-dlh3ps://rg3.github.io/youtube-dl/supportedsites.html


OURCOLLABORATIONWITHCONTENTCREATORS,IPISSUES


YoungComposersCorpus

•  BeganwithNPR’s2011listof“100ComposersUnder40”

•  91of100haveownself-containedsites•  Asof5/2016hadwri3enagreementswith165Composers(25ofthemfromNPR’slist)

•  Willrecruit10ofthemforenhancedarchiving(uncompressed;be3erthanwhatisonwebsite)–  Thiswillrequireanaddedappendixtocontract/agreement(whichmayinvolvedarkarchivingand/orrestrictedaccess)


1/18/18

7

BuildingrelaLonshipswithComposers

•  EngagethemwiththeideaofpreservingtheirWebsite

•  Aretheywillingtogiveusricherversionsofcontentontheirsite?

•  Aretheywillingtomakeall(orjustpart)ofthecontentfreelyaccessible?Dotheywanttoembargosomecontentinadarkarchive?

•  DonorAgreement/Contract-


DonorAgreement/Contract

•  Havebeenworkingonthiswithlawyersforoverayear

•  Havehadfairlystablelanguageinitandsomecontractsalreadysignedandreturned

•  Doesdefaulttoallowinguscompleterightsforreformaungandforallowingresearcherstosee/hearallhighqualityversionsatminimumon-site– AndthusfarallComposerscontactedhaveagreedtothoseprinciples(butnotnecessarilytothecontractuallanguage)


ContractIntrotentaLvelanguage

•  NYUandComposerwishtoestablishlong-termpreservaLonofthematerialslistedatthehighestpossiblequality.TheParLeswishtoenterintothisAgreementtoestablishguidelinesandstandardswithregardtoongoingandfuturelibraryprocessesrelatedtosuchpreservaLon.


ElementsintheContract

•  Whatisbeingacquired•  TermsofTransfer•  TermsofuserAccess•  Rights&ResponsibiliLes(bothNYU&Composer)

•  Appendixdescribingeachitem(format,content,amount,otherper4nentdescriptors)

•  AppendixwithAccessRestricLons-Besser-BerkeleySeminar1/19/2018 40

4possibleLevelsofStreamingAccess

•  Availableforcopy-protectedstreamingfromtheNYULibraries’websitewithunrestrictedaccessbythegeneralpublic.

•  Availableforcopy-protectedstreamingfromtheNYULibraries’website–  withaccesslimitedtoregisteredNYUfacultyandstudentsand–  toexternalresearcherswitheligibilitytouseNYULibraries’archivalresourcesaccordingtoNYULibraries’generalaccesspolicies,withpasswordauthenLcaLon,onoroffcampus.

•  Availableforcopy-protectedstreamingonNYULibrariespremises,atdesignatedworkstaLons,withaccessmediatedbyNYULibrariespersonnel.

•  NotavailableforstreamingunLladesignatedfuturedate.


TentaLvepiecesoftheContract•  TheuncompressedmasterfilesofMaterialslicensedforinclusionwillbemadeavailabletotheLibrariestoenabletheresearchanddevelopmentofhigherqualitytoolsandprocessesforarchivingontheWebandsuccessortechnology.Theresultanthigh-qualitycopiesofComposer’swebsite—incorporaLngthebestqualitymediafiles—willbepreservedashistoricaldocumentsinthearchive,whichwillbeaccessibleworldwideontheWeborsuccessortechnologyasastorehouseofculturalmemoryandavehicleforresearchandscholarship.ComposerretainsexisLngrightstohisorherMaterials,subjecttothelicensegrantedinthisAgreement.


1/18/18

8

TentaLvepiecesoftheContract•  non-exclusiveworldwide,perpetual,irrevocable,royalty-freerighttoproduce,use,copy,anddistributeDerivaLveWorks

•  strictlylimitedtoreforma3eddigitalfilesortoexcerptsandabridgements(suchasthumbnails)createdforthetechnicalpurposesofbuilding,preserving,andprovidingaccesstotheWebarchiveovertheWorldWideWeboritssuccessor

•  maybeusedonlyforthenon-profiteducaLonalandresearchpurposesprovidedunderthisAgreement

•  Agreementdoesnotaffectortransferanycopyrightsorotherintellectualpropertyrights


ARCHITECTURE&WORKFLOWS


Architecture&Workflows

•  TheFindingAidsaregeneratedfromArchiveSpace(whichcontainsrichmetadata)

•  ThereisanoverallComposersFindingAid,aswellasaseparateFindingAidforeachcomposer(lisLnginventoryandwebarchives,andlinktoassets)

•  WebarchiveisstoredinArchive-It;richercontentinNYURepository

•  ConnecLonsbuiltoffofArchiveSpaceback-endAPIDemoSite


Sohware&ServiceComponents

•  IA’sArchive-It•  NYUdigitallibraryinternalcomponents

– Aeonforworkflowmanagement– ArchiveSpace– EAD


RecentDevelopmentwork

•  Supplyingaseparateaudioplayer?•  HiringaDigitalArchivist•  SLllworkingonpreciseformsofnavigaLonbtwnArchiveSpace,Archive-It,andrichercontentwithinNYU’sdigitalrepository

•  ExampleofworkdoneonIA’sAPI-


InterimworkonAPItoIA•  WhatIAneedsfromNYUAPI

–  APIURL–  CredenLals(username,password)->AuthenLcaLonToken()–  RepositoryID–  ResourceID

•  WhatIAwillreturnasJSONarray–  UnitTitle–  Creator–  DataExpression–  ExtentStatement–  TechCharacterisLcs–  [SomethingBasedonAccessRestricLon,i.e.canitbestreamed]???

•  WeSpeakEtruscan,1993May21,23.5MB,1AIFFfileStereouncompressed16bit/44.1K

•  TheDreamofInnocenceIII,1998March26,150MB,1AIFFfileStereouncompressed16bit/44.1K


1/18/18

9

HOWTHINGSMAYLOOK


QuerypathssLllunderdevelopment


OneopLonforUserQueries

•  UserbrowsesthroughArchive-It•  UserseesthatA/Vcontentexists(andinsomecases,itwillincluderichercontent,butsomeofthatmightbeaccess-restricted)

•  Archive-IthandsoffusertoNYU(eitherdirectlytoA/Vcontent,ortoFindingAid)


OneopLonforQueries


OneopLonforhighqualitycontent

•  OnarchivedwebsitepagelisLngcomposer’scontent,userseesamessagethathigherqualitycontentisavailable,with:– AccessrestricLons,ifapplicable– Linktorelevantfindingaid–  (lookinglikefollowingimage)-

Besser-BerkeleySeminar1/19/2018 53 Besser-BerkeleySeminar1/19/2018 54

1/18/18

10

DemofromAPIsideh3p://composers.dlib.nyu.edu/


FromtheLibraryFindingAidsideh3p://dlib.nyu.edu/findingaids/html/fales/mss_479/


FromtheLibraryFindingAidside(cont)


FromtheLibraryFindingAidside(ContainerList)


FromtheLibraryFindingAidsideh3p://dlib.nyu.edu/findingaids/html/fales/mss_460/dscaspace_7951feea619b6c41436c556e0674d1c8.html


FromtheArchive-Itsideh3ps://archive-it.org/collecLons/7872


1/18/18

11

FromtheArchive-Itsideh3ps://archive-it.org/collecLons/7872?

q=h3p%3A%2F%2Fwww.bitrosie.com&show=SeedVideos&fc=seedId%3A1157594


FromanydirecLon,usermightneedtoauthenLcate


SOMEOTHERINTERNALTRACKING

Besser-BerkeleySeminar1/19/2018 63 Besser-BerkeleySeminar1/19/2018 64

CrawlRecords


EVALUATION


1/18/18

12

EvaluaLonforImprovement

•  ComposersandtheirsaLsfacLonwiththewaysinwhichaudienceswillbeabletoviewarchivesoftheirwebsites

•  Researchers,andwhetherthecontentandfuncLonalityofthesewebarchivesworksforthem

•  Tweakingwhatwedoinordertobe3erserveCreatorsandResearchers


ScheduleandMethodologyforEvaluaLon

•  Jan2018—Scheduleone-on-oneinterviewswithsetsofcomposersandResearchers

•  Feb-Mar2017—Onehourindividualsessionswith10Composersandalsowith10Researchers,havingthemlookattheuserinterfaceandconductqueries–  Composers:AretheysaLsfiedwithhowaudienceswillbeabletoviewthe

archivalcopiesoftheirwebsites?Isitbe3erorworsethantheirownlivesites?AretheysaLsfiedwiththeaudioandvideoplacementandquality(aswellasopLons)?AretheycontentwiththeDonorAgreement?Whatchanges/improvementsmightbemadetoanyofthese?

–  Researchers:Cantheyfindwhattheyneedinthewebarchive?Isitdifficult(clunky)touse?Whatpartsdon’tworkwelloraren’tintuiLve?WewanttoidenLfywhatchangesinthecontent,funcLonality,ornavigaLonfeatureswouldimprovetheiruserexperience

•  Apr-May2017—ConstrucLonofEvaluaLonSummarycontainingthelistofimprovements/changesthatshouldbemadetotheArchivingproject

•  June-Aug2017—Implementthechanges


IMPACTBEYONDTHISPROJECT


ImpactBeyondthisProject•  Archive-Itwillbeabletobe3erhandlestreamingmedia,anddisplayitinpropercontext

•  WewillhavearchitecturesandworkflowsforArchive-Ittointeractwithricherlocalresources(aswellasexamplesofhowinteracLonandnavigaLoncanproceedbtwnArchive-It,ArchiveSpace,FindingAids,andaninternaldigitalrepository)

•  ModelsforinteracLonbtwncreatorsandcollecLngorganizaLonswillhavebeendeveloped(incldonoragreements)

•  Wewillhavepreserved100+++websitesofyoungcomposers


MakingWebArchivingWorkforStreamingMedia

•  h3p://besser.tsoa.nyu.edu/howard/Talks/•  h3p://www.nyu.edu/about/news-publicaLons/news/

2015/03/27/nyu-libraries-to-team-with-internet-archive-to-preserve-high-quality-musical-content-on-the-web.html

•  h3p://archive.org/~nlevi3/reveal.js/•  h3p://composers.dlib.nyu.edu/•  h3ps://rg3.github.io/youtube-dl/supportedsites.html


web archiving poses challengesbesser.tsoa.nyu.edu/howard/talks/18composers-buckland.pdf ·...

Documents