archive-itbesser.tsoa.nyu.edu/howard/talks/18composers-iipc.pdf · archive-it besser-iipc...

11/13/18

1

Archivingwebsitescontainingstreamingmedia:theMusic

ComposerProject

HowardBesser,NYUh4p://besser.tsoa.nyu.edu/howard/Talks/

Besser-IIPC 13/11/2018 1

Archivingwebsitescontainingstreamingmedia:

theMusicComposerProject

•  TheProblemwithHeritrixandArchive-It•  TheProject

–  OurTechnicalCollaboraLon–  OurCollaboraLonwithContentCreators&restricLons–  Architectures&Workflows–  Howthingslook–  EvaluaLon

•  ImpactbeyondthisProject

•  CaveatI:Thisisanin-progressreport;theprojectisunfinished•  CaveatII:Iamnotinvolvedinsystemarchitecture&hand-offs,somaynot

beabletoanswerdetailedquesLonsintheseareas


PROBLEMSWITHHERITRIXANDARCHIVE-IT


Archive-It

•  TheleadingapplicaLon/serviceforcuratedwebarchivinginNorthAmerica

•  RunbytheInternetArchive,andismuchmoretargetedandcuratedthantheirWayBackMachine

•  IsbasedonCrawlersoZwaredevelopedbyIA(Heritrix)in2003-2004

•  IsverypooratcapturingstreamingaudioorvideoaswellasinserLngitproperlyintoacomposedwebpage-


Archive-ItIssuesw/StreamingMedia




11/13/18

2



Archive-Itscreenshotsgeneratedaspartofourproject-

•  ByLorenaRamirez-Løpez


Archive-ItIssuesw/StreamingMediaFireFoxversion39.0.ScreenshotofTarikO’Regan’ssitetaken2015/10/05






Archive-ItIssuesw/StreamingMediaFireFoxversion39.0.ScreenshotofTedHearne’swebsitetaken2015/10/05


11/13/18

3

Somesourcesofstreamingissues

•  Problemswithcapturingresourcesresidingon3rdpartyservices(YouTube,Vimeo,Soundcloud)

•  ProblemswithhowfaithfullytheA/VmaterialsarecapturedandplacedbyArchive-It

•  ProblemswithwebsitesgeneratedthroughsitebuildingplajormssuchasSquarespace


OtherIssueswe’retryingtosolve

•  DiscoveringURLsgeneratedbyJavascript


THEPROJECT


ArchivingComposerWebsitesh4p://www.nyu.edu/about/news-publicaLons/news/2015/03/27/nyu-libraries-to-team-with-internet-archive-to-preserve-high-

quality-musical-content-on-the-web.html

•  Collect,preserve,&makeavailableWebsitesofComposers

•  $480,000grantfromMellonin2015toNYULibrary/MIAP/InternetArchive

•  Dealingwiththeissuethatcontemporarycomposerwebsitesgoupanddown(andalsoincorporaterelaLonship-buildingbtwncomposerandfans)

•  AddressingtheproblemsofcollecLngstreamingmedia•  AlsoselecLvelycollecLnghigh-qualityversionsthatareusedtogeneratethestreams,andallowingfutureresearcherstosee/hearthehigherqualityversions


ArchivingComposerWebsites


•  DevelopgoodandongoingrelaLonshipsbtwnLibrariesandComposers

•  DevelopTrust–  fordevelopingcollecLons,andconLnuingtoaddtothem–  forPolicyreasons

•  Examinewhattypeoferrorstakeplace–  howfaithfullyaudiovisualmaterialsarebeingcaptured–  howresourcesthatresideonthird-partyweb-services(YouTube,Vimeo,Soundcloud)are(not)displayedwithinArchive-It’sinterface

–  IssueswwebsitesgeneratedthroughsitebuildingplajormssuchasSquarespace

•  Findwaystofixthoseerrors

Somemethodsused

•  BeganwithNPR’slistof100importantcomposersunder40,andaugmetedthelistwithfacultyandlibrariansuggesLons

•  IdenLfiedwebsiteinfrastructuresencounteredandcreatedaclassificaLonmatrix-


11/13/18

4

WebsiteInfrastructureencountered


ProjectTeam•  JeffersonBailey(InternetArchive)•  HowardBesser(MIAP)•  LoriDonovan(InternetArchive)•  AprilHathcock(Lib/ScholComm)•  NicoleGreenhouse(Lib/ACM)•  CarolKassel(Lib/DLTS)•  Sco4Statland(MIAP)•  DonaldMennerich(Lib/ACM/DLTS)•  DavidMillman(Lib/DLTS)•  CourtneyMumma(InternetArchive)•  RobinPreiss(Lib/AFC)•  LorenaRamirez(MIAP)---specialthanks!•  MichaelStoller(Lib/C&RS)•  KentUnderwood(Lib/AFC)•  ChelaSco4Weber(Lib/AFC)--departed


OURTECHNICALCOLLABORATION:CRAWLING


NYU/IACollaboraLon


TradiLonalCrawlers


•  Archive-ItandotherwebarchivesuseHeritrix•  Followlinks,capturemostwebcontent•  Lesssuccessfulwithstreamingvideoanddynamiccontentexecutedinthebrowser

•  Umbrahelps

BROZZLER!

“browser” | “crawler” = BROZZLER

Logo: Noah Levitt Besser-IIPC 13/11/2018 24

11/13/18

5


BrozzlerSystemArchitecturev1


BrozzlerModel

•  job:collecLonofseeds•  seed:principalunitofcrawlconfiguraLon

–  onebrowserworksononeseedataLme(politeness)–  seedhasitsownconfiguraLon,alsoinheritsfromparentjob

•  page:atomicunitofcrawlingfrombrozzlerperspecLve

•  url:onlybrowsers,warcproxhavetodealwitheveryurl


Warcprox:WARC-wriOnghPpproxy

•  man-in-the-middleforh4ps•  asynchronous:WarcWriterThread

– writeswarcrecords– savesdeduplicaLoninfo– updatesstaLsLcs


Otherpieces

•  pythonwayback•  Rethinkdb(distributeddocumentstore)


StreamcapturereliesonYoutube-dlh4ps://rg3.github.io/youtube-dl/supportedsites.html


11/13/18

6

OURCOLLABORATIONWITHCONTENTCREATORS,IPISSUES


YoungComposersCorpus

•  BeganwithNPR’s2011listof“100ComposersUnder40”

•  91of100haveownself-containedsites•  WithinayearofstarLngwehadwri4enagreementswith165Composers(25ofthemfromNPR’slist)

•  Plannedtorecruit10ofthemforenhancedarchiving(uncompressed;be4erthanwhatisonwebsite)–  Thiswillrequireanaddedappendixtocontract/agreement(whichmayinvolvedarkarchivingand/orrestrictedaccess)


BuildingrelaLonshipswithComposers

•  EngagethemwiththeideaofpreservingtheirWebsite

•  Aretheywillingtogiveusricherversionsofcontentontheirsite?

•  Aretheywillingtomakeall(orjustpart)ofthecontentfreelyaccessible?Dotheywanttoembargosomecontentinadarkarchive?

•  DonorAgreement/Contract-


DonorAgreement/Contract

•  Workedonthiswithlawyersforwelloverayear•  Havehadfairlystablelanguageinitandmanycontractsalreadysignedandreturned

•  Doesdefaulttoallowinguscompleterightsforreformasngandforallowingresearcherstosee/hearallhighqualityversionsatminimumon-site– AndthusfarallComposerscontactedhaveagreedtothoseprinciples(butnotnecessarilytothecontractuallanguage)


ContractIntrotentaLvelanguage

•  NYUandComposerwishtoestablishlong-termpreservaLonofthematerialslistedatthehighestpossiblequality.TheParLeswishtoenterintothisAgreementtoestablishguidelinesandstandardswithregardtoongoingandfuturelibraryprocessesrelatedtosuchpreservaLon.


ElementsintheContract

•  Whatisbeingacquired•  TermsofTransfer•  TermsofuserAccess•  Rights&ResponsibiliLes(bothNYU&Composer)

•  Appendixdescribingeachitem(format,content,amount,otherper4nentdescriptors)

•  AppendixwithAccessRestricLons-Besser-IIPC 13/11/2018 36

11/13/18

7

4possibleLevelsofStreamingAccess

•  Availableforcopy-protectedstreamingfromtheNYULibraries’websitewithunrestrictedaccessbythegeneralpublic.

•  Availableforcopy-protectedstreamingfromtheNYULibraries’website–  withaccesslimitedtoregisteredNYUfacultyandstudentsand–  toexternalresearcherswitheligibilitytouseNYULibraries’archivalresourcesaccordingtoNYULibraries’generalaccesspolicies,withpasswordauthenLcaLon,onoroffcampus.

•  Availableforcopy-protectedstreamingonNYULibrariespremises,atdesignatedworkstaLons,withaccessmediatedbyNYULibrariespersonnel.

•  NotavailableforstreamingunLladesignatedfuturedate.


TentaLvepiecesoftheContract•  TheuncompressedmasterfilesofMaterialslicensedforinclusionwillbemadeavailabletotheLibrariestoenabletheresearchanddevelopmentofhigherqualitytoolsandprocessesforarchivingontheWebandsuccessortechnology.Theresultanthigh-qualitycopiesofComposer’swebsite—incorporaLngthebestqualitymediafiles—willbepreservedashistoricaldocumentsinthearchive,whichwillbeaccessibleworldwideontheWeborsuccessortechnologyasastorehouseofculturalmemoryandavehicleforresearchandscholarship.ComposerretainsexisLngrightstohisorherMaterials,subjecttothelicensegrantedinthisAgreement.


TentaLvepiecesoftheContract•  non-exclusiveworldwide,perpetual,irrevocable,royalty-freerighttoproduce,use,copy,anddistributeDerivaLveWorks

•  strictlylimitedtoreforma4eddigitalfilesortoexcerptsandabridgements(suchasthumbnails)createdforthetechnicalpurposesofbuilding,preserving,andprovidingaccesstotheWebarchiveovertheWorldWideWeboritssuccessor

•  maybeusedonlyforthenon-profiteducaLonalandresearchpurposesprovidedunderthisAgreement

•  Agreementdoesnotaffectortransferanycopyrightsorotherintellectualpropertyrights


ARCHITECTURE&WORKFLOWS


Architecture&Workflows

•  TheFindingAidsaregeneratedfromArchiveSpace(whichcontainsrichmetadata)

•  ThereisanoverallComposersFindingAid,aswellasaseparateFindingAidforeachcomposer(lisLnginventoryandwebarchives,andlinktoassets)

•  WebarchiveisstoredinArchive-It;richercontentinNYURepository

•  ConnecLonsbuiltoffofArchiveSpaceback-endAPIDemoSite


SoZware&ServiceComponents

•  IA’sArchive-It•  NYUdigitallibraryinternalcomponents

– Aeonforworkflowmanagement– ArchiveSpace– EAD


11/13/18

8

UnfinishedDevelopmentwork

•  Supplyingaseparateaudioplayer?•  SLllworkingonpreciseformsofnavigaLonbtwnArchiveSpace,Archive-It,andrichercontentwithinNYU’sdigitalrepository

•  WhatwillbeontheworkstaLonforitemsthatneedtobelookedaton-site?

•  Issueswithstreamsthatwerenotcaptured•  ExampleofworkdoneonIA’sAPI-


InterimworkonAPItoIA•  WhatIAneedsfromNYUAPI

–  APIURL–  CredenLals(username,password)->AuthenLcaLonToken()–  RepositoryID–  ResourceID

•  WhatIAwillreturnasJSONarray–  UnitTitle–  Creator–  DataExpression–  ExtentStatement–  TechCharacterisLcs–  [SomethingBasedonAccessRestricLon,i.e.canitbestreamed]???

•  WeSpeakEtruscan,1993May21,23.5MB,1AIFFfileStereouncompressed16bit/44.1K

•  TheDreamofInnocenceIII,1998March26,150MB,1AIFFfileStereouncompressed16bit/44.1K


HOWTHINGSMAYLOOK


QuerypathssLllunderdevelopment


OneopLonforUserQueries

•  UserbrowsesthroughArchive-It•  UserseesthatA/Vcontentexists(andinsomecases,itwillincluderichercontent,butsomeofthatmightbeaccess-restricted)

•  Archive-IthandsoffusertoNYU(eitherdirectlytoA/Vcontent,ortoFindingAid)


OneopLonforQueries


11/13/18

9

OneopLonforhighqualitycontent

•  OnarchivedwebsitepagelisLngcomposer’scontent,userseesamessagethathigherqualitycontentisavailable,with:– AccessrestricLons,ifapplicable– Linktorelevantfindingaid–  (lookinglikefollowingimage)-

Besser-IIPC 13/11/2018 49 Besser-IIPC 13/11/2018 50

DemofromAPIsideh4p://composers.dlib.nyu.edu/


FromtheLibraryFindingAidsideh4p://dlib.nyu.edu/findingaids/html/fales/mss_479/


FromtheLibraryFindingAidside(cont)


FromtheLibraryFindingAidside(ContainerList)


11/13/18

10

FromtheLibraryFindingAidsideh4p://dlib.nyu.edu/findingaids/html/fales/mss_460/dscaspace_7951feea619b6c41436c556e0674d1c8.html


FromtheArchive-Itsideh4ps://archive-it.org/collecLons/7872


FromtheArchive-Itsideh4ps://archive-it.org/collecLons/7872?

q=h4p%3A%2F%2Fwww.bitrosie.com&show=SeedVideos&fc=seedId%3A1157594


FromanydirecLon,usermightneedtoauthenLcate


SOMEOTHERINTERNALTRACKING

Besser-IIPC 13/11/2018 59 Besser-IIPC 13/11/2018 60

11/13/18

11

CrawlRecords


EVALUATION


EvaluaLonforImprovement

•  ComposersandtheirsaLsfacLonwiththewaysinwhichaudienceswillbeabletoviewarchivesoftheirwebsites(improvingusability)

•  Researchers,andwhetherthecontentandfuncLonalityofthesewebarchivesworksforthem(contentpresentaLon

•  Tweakingwhatwedoinordertobe4erserveCreatorsandResearchers

•  Findingoutwhethercapturesreallyworked


FindingssLllbeinganalyzed

•  Streamingcapturesappearmoresuccessful,butwesLllexperiencesomestreamingcaptureproblems

•  NeedfurtherexploraLontoseetheprecisecauseofthecrawler/captureissues(&recLfythemifpossible)


CrawlerIssues(brokenheaderlinks)


CrawlerIssues(failedvideocapture)


11/13/18

12

CrawlerIssues(videocapturefailure)


CrawlerIssues(Flashvideoissue)


CrawlerIssues(videocapturedwithoutaudio)


CrawlerIssues(brokenvideolinks)


CrawlerIssues(1audionotcaptured)


CrawlerIssues(audionotcaptured)


11/13/18

13

CrawlerIssues(audiofailure&anchorproblem)


CrawlerIssues(parLalcapturefailure)


CrawlerIssues(incompleteloading)


CrawlerIssues(Captureissues)


CrawlerIssues(unknownproblems)


CrawlerIssues

•  Campjulie.com:–  Anycapturedate:IfveryslowloadLme,hardtotellifwasworkingornot,sosomesubjects

gaveup.[Siteownersaysthisisinherenttosite,somightnotbeacaptureproblem.]–  Discrepanciesbetweenwhenonehopoutiscapturedornot.

•  Kmariekim.com:–  Sep26,2017capture(latestcapture):A4emptstoplaymusicfromarchivedtumblrpagefrom

variousplajorms(youtube,soundcloud,etc.).•  Bitrosie.com:

–  Allcapturedates:linkstakeroughly5minutes(assumedbrokenatfirst)•  Adelefournet.com/video/:

–  Sep12,2017capture:VideoerroraZerroughly10seconds.Stopsplaying"BeretsofMaryJeanPlace",andstartsplayinganothervideowithopeningLtle"BarrancoDistrict,Lima,Peru".Therestofthevideosonthepagedonotplay.Linkto"BeretsofMaryJeanPlace"ontheInternetArchivealsoplaysincorrectvideo("BarrancoDistrict,Lima,Peru").

•  MichaelRobinsonarchivedwebsite:Errormessage


11/13/18

14

EvaluaLonResults•  ThesubjectswerebasicallysaLsfiedwiththecaptures,but

hadverymanysuggesLonsforimprovementswithlabeling,searching,display,andperformance.MostalsowantedaddiLonalfuncLonality.

•  ManyofthesubjectswereconfusedbetweencapturedsitesandtheFindingAidsforthem.InaddiLon,thewords“Papersof”incollecLonLtlesbaffledpeoplewhentheywerelookingforrecordings,notpapers.

•  Bothusersandsiteownerswereunclearaboutthescopeofcontentthathadactuallybeencollected.Onesiteownerexpresseddisappointmentthatreviewsthattheylinkedtowerenotcaptured.Andonlyonesubjectfiguredouthowtonavigatetoasuggested“liveweb”pagethathadnotbeennotarchived.


FuncLonalityrequestedbyusers•  Mostsubjectswantedmoremetadatadisplayed.Examplesincluded:

displayingadescripLonoftheComposersProjectandlikelycontentsontheiniLalstartpage;displayofaudio/videorun-Lmeinsteadoffilesize;descripLon,thumbnails,excerptsformaterialrestrictedtoonsiteuse(sothattheycoulddecidewhetherornottheyreallyneededtomakeasitevisit);morefieldsshowninvariousdisplays(bothinlistsandinlinkstoessence).

•  BothsiteownersrespondedposiLvelytotheideaofprovidingasitemapwithacollapsingmenuoflinks.

•  Mostsubjectswantedasearchbox.AndmostwantedtobeabletoimmediatelysortamulL-columndisplaylistbyanycolumnoftheirchoosing.

•  Onesubjectfounditmisleadingwhenarestrictedobjectlinkedtoanewpage.

•  Onesiteownerpreferredthattheirdigitalobjectsbeorganizedbyproject,ratherthaninanundifferenLatedlistofeverydigitalobjectontheirsite.


FuncLonalityrequestedforlocalworkstaLons

•  Abilitytotakescreengrabs•  AccesstoaddiLonalbrowserwindow

•  Previewframewhenscrubbing(fastforwarding)throughvideomaterial

•  Useoftheirownlaptoporanotherwindow•  DisplayofLmecode•  And2subjectsspecificallyrequestedthe

–  abilitytoslowvideo/audiofiletotranscribe–  abilitytodroppin/a4achnotestospecificpointinvideo/audiofile


IMPACTBEYONDTHISPROJECT


ImpactBeyondthisProject•  TherewillbeanalternaLvetoHeritrixforcapturing

streamingmedia,andArchive-Itwillideallybeabletobe4erhandlestreamingmedia,anddisplayitinpropercontext

•  WewillhavearchitecturesandworkflowsforArchive-Ittointeractwithricherlocalresources(aswellasexamplesofhowinteracLonandnavigaLoncanproceedbtwnArchive-It,ArchiveSpace,FindingAids,andaninternaldigitalrepository)

•  ModelsforinteracLonbtwncreatorsandcollecLngorganizaLonswillhavebeendeveloped(incldonoragreements)

•  Wehavepreserved100+++websitesofyoungcomposers


Archivingwebsitescontainingstreamingmedia:theMusicComposer

Project

•  h4p://besser.tsoa.nyu.edu/howard/Talks/•  h4p://www.nyu.edu/about/news-publicaLons/news/

2015/03/27/nyu-libraries-to-team-with-internet-archive-to-preserve-high-quality-musical-content-on-the-web.html

•  h4p://archive.org/~nlevi4/reveal.js/•  h4p://composers.dlib.nyu.edu/•  h4ps://rg3.github.io/youtube-dl/supportedsites.html


archive-itbesser.tsoa.nyu.edu/howard/talks/18composers-iipc.pdf · archive-it besser-iipc...

Documents