archive-itbesser.tsoa.nyu.edu/howard/talks/18composers-iipc.pdf · archive-it besser-iipc...

11/13/18

Archivingwebsitescontainingstreamingmedia:theMusic

ComposerProject

HowardBesser,NYUh4p://besser.tsoa.nyu.edu/howard/Talks/

Besser-IIPC 13/11/2018 1

Archivingwebsitescontainingstreamingmedia:

theMusicComposerProject

•  TheProblemwithHeritrixandArchive-It•  TheProject

–  OurTechnicalCollaboraLon–  OurCollaboraLonwithContentCreators&restricLons–  Architectures&Workflows–  Howthingslook–  EvaluaLon

•  ImpactbeyondthisProject

•  CaveatI:Thisisanin-progressreport;theprojectisunfinished•  CaveatII:Iamnotinvolvedinsystemarchitecture&hand-offs,somaynot

beabletoanswerdetailedquesLonsintheseareas

PROBLEMSWITHHERITRIXANDARCHIVE-IT

Archive-It

•  TheleadingapplicaLon/serviceforcuratedwebarchivinginNorthAmerica

•  RunbytheInternetArchive,andismuchmoretargetedandcuratedthantheirWayBackMachine

•  IsbasedonCrawlersoZwaredevelopedbyIA(Heritrix)in2003-2004

•  IsverypooratcapturingstreamingaudioorvideoaswellasinserLngitproperlyintoacomposedwebpage-

Archive-ItIssuesw/StreamingMedia

11/13/18

Archive-Itscreenshotsgeneratedaspartofourproject-

•  ByLorenaRamirez-Løpez

Archive-ItIssuesw/StreamingMediaFireFoxversion39.0.ScreenshotofTarikO’Regan’ssitetaken2015/10/05

Archive-ItIssuesw/StreamingMediaFireFoxversion39.0.ScreenshotofTedHearne’swebsitetaken2015/10/05

11/13/18

Somesourcesofstreamingissues

•  Problemswithcapturingresourcesresidingon3rdpartyservices(YouTube,Vimeo,Soundcloud)

•  ProblemswithhowfaithfullytheA/VmaterialsarecapturedandplacedbyArchive-It

•  ProblemswithwebsitesgeneratedthroughsitebuildingplajormssuchasSquarespace

OtherIssueswe’retryingtosolve

•  DiscoveringURLsgeneratedbyJavascript

THEPROJECT

ArchivingComposerWebsitesh4p://www.nyu.edu/about/news-publicaLons/news/2015/03/27/nyu-libraries-to-team-with-internet-archive-to-preserve-high-

quality-musical-content-on-the-web.html

•  Collect,preserve,&makeavailableWebsitesofComposers

•  $480,000grantfromMellonin2015toNYULibrary/MIAP/InternetArchive

•  Dealingwiththeissuethatcontemporarycomposerwebsitesgoupanddown(andalsoincorporaterelaLonship-buildingbtwncomposerandfans)

•  AddressingtheproblemsofcollecLngstreamingmedia•  AlsoselecLvelycollecLnghigh-qualityversionsthatareusedtogeneratethestreams,andallowingfutureresearcherstosee/hearthehigherqualityversions

ArchivingComposerWebsites

•  DevelopgoodandongoingrelaLonshipsbtwnLibrariesandComposers

•  DevelopTrust–  fordevelopingcollecLons,andconLnuingtoaddtothem–  forPolicyreasons

•  Examinewhattypeoferrorstakeplace–  howfaithfullyaudiovisualmaterialsarebeingcaptured–  howresourcesthatresideonthird-partyweb-services(YouTube,Vimeo,Soundcloud)are(not)displayedwithinArchive-It’sinterface

–  IssueswwebsitesgeneratedthroughsitebuildingplajormssuchasSquarespace

•  Findwaystofixthoseerrors

Somemethodsused

•  BeganwithNPR’slistof100importantcomposersunder40,andaugmetedthelistwithfacultyandlibrariansuggesLons

•  IdenLfiedwebsiteinfrastructuresencounteredandcreatedaclassificaLonmatrix-

11/13/18

WebsiteInfrastructureencountered

ProjectTeam•  JeffersonBailey(InternetArchive)•  HowardBesser(MIAP)•  LoriDonovan(InternetArchive)•  AprilHathcock(Lib/ScholComm)•  NicoleGreenhouse(Lib/ACM)•  CarolKassel(Lib/DLTS)•  Sco4Statland(MIAP)•  DonaldMennerich(Lib/ACM/DLTS)•  DavidMillman(Lib/DLTS)•  CourtneyMumma(InternetArchive)•  RobinPreiss(Lib/AFC)•  LorenaRamirez(MIAP)---specialthanks!•  MichaelStoller(Lib/C&RS)•  KentUnderwood(Lib/AFC)•  ChelaSco4Weber(Lib/AFC)--departed

OURTECHNICALCOLLABORATION:CRAWLING

NYU/IACollaboraLon

TradiLonalCrawlers

•  Archive-ItandotherwebarchivesuseHeritrix•  Followlinks,capturemostwebcontent•  Lesssuccessfulwithstreamingvideoanddynamiccontentexecutedinthebrowser

•  Umbrahelps

BROZZLER!

“browser” | “crawler” = BROZZLER

Logo: Noah Levitt Besser-IIPC 13/11/2018 24

11/13/18

BrozzlerSystemArchitecturev1

BrozzlerModel

•  job:collecLonofseeds•  seed:principalunitofcrawlconfiguraLon

–  onebrowserworksononeseedataLme(politeness)–  seedhasitsownconfiguraLon,alsoinheritsfromparentjob

•  page:atomicunitofcrawlingfrombrozzlerperspecLve

•  url:onlybrowsers,warcproxhavetodealwitheveryurl

Warcprox:WARC-wriOnghPpproxy

•  man-in-the-middleforh4ps•  asynchronous:WarcWriterThread

– writeswarcrecords– savesdeduplicaLoninfo– updatesstaLsLcs

Otherpieces

•  pythonwayback•  Rethinkdb(distributeddocumentstore)

StreamcapturereliesonYoutube-dlh4ps://rg3.github.io/youtube-dl/supportedsites.html

11/13/18

OURCOLLABORATIONWITHCONTENTCREATORS,IPISSUES

YoungComposersCorpus

•  BeganwithNPR’s2011listof“100ComposersUnder40”

•  91of100haveownself-containedsites•  WithinayearofstarLngwehadwri4enagreementswith165Composers(25ofthemfromNPR’slist)

•  Plannedtorecruit10ofthemforenhancedarchiving(uncompressed;be4erthanwhatisonwebsite)–  Thiswillrequireanaddedappendixtocontract/agreement(whichmayinvolvedarkarchivingand/orrestrictedaccess)

BuildingrelaLonshipswithComposers

•  EngagethemwiththeideaofpreservingtheirWebsite

•  Aretheywillingtogiveusricherversionsofcontentontheirsite?

•  Aretheywillingtomakeall(orjustpart)ofthecontentfreelyaccessible?Dotheywanttoembargosomecontentinadarkarchive?

•  DonorAgreement/Contract-

DonorAgreement/Contract

•  Workedonthiswithlawyersforwelloverayear•  Havehadfairlystablelanguageinitandmanycontractsalreadysignedandreturned

•  Doesdefaulttoallowinguscompleterightsforreformasngandforallowingresearcherstosee/hearallhighqualityversionsatminimumon-site– AndthusfarallComposerscontactedhaveagreedtothoseprinciples(butnotnecessarilytothecontractuallanguage)

ContractIntrotentaLvelanguage

•  NYUandComposerwishtoestablishlong-termpreservaLonofthematerialslistedatthehighestpossiblequality.TheParLeswishtoenterintothisAgreementtoestablishguidelinesandstandardswithregardtoongoingandfuturelibraryprocessesrelatedtosuchpreservaLon.

ElementsintheContract

•  Whatisbeingacquired•  TermsofTransfer•  TermsofuserAccess•  Rights&ResponsibiliLes(bothNYU&Composer)

•  Appendixdescribingeachitem(format,content,amount,otherper4nentdescriptors)

•  AppendixwithAccessRestricLons-Besser-IIPC 13/11/2018 36

11/13/18

4possibleLevelsofStreamingAccess

•  Availableforcopy-protectedstreamingfromtheNYULibraries’websitewithunrestrictedaccessbythegeneralpublic.

•  Availableforcopy-protectedstreamingfromtheNYULibraries’website–  withaccesslimitedtoregisteredNYUfacultyandstudentsand–  toexternalresearcherswitheligibilitytouseNYULibraries’archivalresourcesaccordingtoNYULibraries’generalaccesspolicies,withpasswordauthenLcaLon,onoroffcampus.

•  Availableforcopy-protectedstreamingonNYULibrariespremises,atdesignatedworkstaLons,withaccessmediatedbyNYULibrariespersonnel.

•  NotavailableforstreamingunLladesignatedfuturedate.

TentaLvepiecesoftheContract•  TheuncompressedmasterfilesofMaterialslicensedforinclusionwillbemadeavailabletotheLibrariestoenabletheresearchanddevelopmentofhigherqualitytoolsandprocessesforarchivingontheWebandsuccessortechnology.Theresultanthigh-qualitycopiesofComposer’swebsite—incorporaLngthebestqualitymediafiles—willbepreservedashistoricaldocumentsinthearchive,whichwillbeaccessibleworldwideontheWeborsuccessortechnologyasastorehouseofculturalmemoryandavehicleforresearchandscholarship.ComposerretainsexisLngrightstohisorherMaterials,subjecttothelicensegrantedinthisAgreement.

TentaLvepiecesoftheContract•  non-exclusiveworldwide,perpetual,irrevocable,royalty-freerighttoproduce,use,copy,anddistributeDerivaLveWorks

•  strictlylimitedtoreforma4eddigitalfilesortoexcerptsandabridgements(suchasthumbnails)createdforthetechnicalpurposesofbuilding,preserving,andprovidingaccesstotheWebarchiveovertheWorldWideWeboritssuccessor

•  maybeusedonlyforthenon-profiteducaLonalandresearchpurposesprovidedunderthisAgreement

•  Agreementdoesnotaffectortransferanycopyrightsorotherintellectualpropertyrights

ARCHITECTURE&WORKFLOWS

Architecture&Workflows

•  TheFindingAidsaregeneratedfromArchiveSpace(whichcontainsrichmetadata)

•  ThereisanoverallComposersFindingAid,aswellasaseparateFindingAidforeachcomposer(lisLnginventoryandwebarchives,andlinktoassets)

•  WebarchiveisstoredinArchive-It;richercontentinNYURepository

•  ConnecLonsbuiltoffofArchiveSpaceback-endAPIDemoSite

SoZware&ServiceComponents

•  IA’sArchive-It•  NYUdigitallibraryinternalcomponents

– Aeonforworkflowmanagement– ArchiveSpace– EAD

11/13/18

UnfinishedDevelopmentwork

•  Supplyingaseparateaudioplayer?•  SLllworkingonpreciseformsofnavigaLonbtwnArchiveSpace,Archive-It,andrichercontentwithinNYU’sdigitalrepository

•  WhatwillbeontheworkstaLonforitemsthatneedtobelookedaton-site?

•  Issueswithstreamsthatwerenotcaptured•  ExampleofworkdoneonIA’sAPI-

InterimworkonAPItoIA•  WhatIAneedsfromNYUAPI

–  APIURL–  CredenLals(username,password)->AuthenLcaLonToken()–  RepositoryID–  ResourceID

•  WhatIAwillreturnasJSONarray–  UnitTitle–  Creator–  DataExpression–  ExtentStatement–  TechCharacterisLcs–  [SomethingBasedonAccessRestricLon,i.e.canitbestreamed]???

•  WeSpeakEtruscan,1993May21,23.5MB,1AIFFfileStereouncompressed16bit/44.1K

•  TheDreamofInnocenceIII,1998March26,150MB,1AIFFfileStereouncompressed16bit/44.1K

HOWTHINGSMAYLOOK

QuerypathssLllunderdevelopment

OneopLonforUserQueries

•  UserbrowsesthroughArchive-It•  UserseesthatA/Vcontentexists(andinsomecases,itwillincluderichercontent,butsomeofthatmightbeaccess-restricted)

•  Archive-IthandsoffusertoNYU(eitherdirectlytoA/Vcontent,ortoFindingAid)

OneopLonforQueries

11/13/18

OneopLonforhighqualitycontent

•  OnarchivedwebsitepagelisLngcomposer’scontent,userseesamessagethathigherqualitycontentisavailable,with:– AccessrestricLons,ifapplicable– Linktorelevantfindingaid–  (lookinglikefollowingimage)-

Besser-IIPC 13/11/2018 49 Besser-IIPC 13/11/2018 50

DemofromAPIsideh4p://composers.dlib.nyu.edu/

FromtheLibraryFindingAidsideh4p://dlib.nyu.edu/findingaids/html/fales/mss_479/

FromtheLibraryFindingAidside(cont)

FromtheLibraryFindingAidside(ContainerList)

11/13/18

FromtheLibraryFindingAidsideh4p://dlib.nyu.edu/findingaids/html/fales/mss_460/dscaspace_7951feea619b6c41436c556e0674d1c8.html

FromtheArchive-Itsideh4ps://archive-it.org/collecLons/7872

FromtheArchive-Itsideh4ps://archive-it.org/collecLons/7872?

q=h4p%3A%2F%2Fwww.bitrosie.com&show=SeedVideos&fc=seedId%3A1157594

FromanydirecLon,usermightneedtoauthenLcate

SOMEOTHERINTERNALTRACKING

Besser-IIPC 13/11/2018 59 Besser-IIPC 13/11/2018 60

11/13/18

CrawlRecords

EVALUATION

EvaluaLonforImprovement

•  ComposersandtheirsaLsfacLonwiththewaysinwhichaudienceswillbeabletoviewarchivesoftheirwebsites(improvingusability)

•  Researchers,andwhetherthecontentandfuncLonalityofthesewebarchivesworksforthem(contentpresentaLon

•  Tweakingwhatwedoinordertobe4erserveCreatorsandResearchers

•  Findingoutwhethercapturesreallyworked

FindingssLllbeinganalyzed

•  Streamingcapturesappearmoresuccessful,butwesLllexperiencesomestreamingcaptureproblems

•  NeedfurtherexploraLontoseetheprecisecauseofthecrawler/captureissues(&recLfythemifpossible)

CrawlerIssues(brokenheaderlinks)

CrawlerIssues(failedvideocapture)

11/13/18

CrawlerIssues(videocapturefailure)

CrawlerIssues(Flashvideoissue)

CrawlerIssues(videocapturedwithoutaudio)

CrawlerIssues(brokenvideolinks)

CrawlerIssues(1audionotcaptured)

CrawlerIssues(audionotcaptured)

11/13/18

CrawlerIssues(audiofailure&anchorproblem)

CrawlerIssues(parLalcapturefailure)

CrawlerIssues(incompleteloading)

CrawlerIssues(Captureissues)

CrawlerIssues(unknownproblems)

CrawlerIssues

•  Campjulie.com:–  Anycapturedate:IfveryslowloadLme,hardtotellifwasworkingornot,sosomesubjects

gaveup.[Siteownersaysthisisinherenttosite,somightnotbeacaptureproblem.]–  Discrepanciesbetweenwhenonehopoutiscapturedornot.

•  Kmariekim.com:–  Sep26,2017capture(latestcapture):A4emptstoplaymusicfromarchivedtumblrpagefrom

variousplajorms(youtube,soundcloud,etc.).•  Bitrosie.com:

–  Allcapturedates:linkstakeroughly5minutes(assumedbrokenatfirst)•  Adelefournet.com/video/:

–  Sep12,2017capture:VideoerroraZerroughly10seconds.Stopsplaying"BeretsofMaryJeanPlace",andstartsplayinganothervideowithopeningLtle"BarrancoDistrict,Lima,Peru".Therestofthevideosonthepagedonotplay.Linkto"BeretsofMaryJeanPlace"ontheInternetArchivealsoplaysincorrectvideo("BarrancoDistrict,Lima,Peru").

•  MichaelRobinsonarchivedwebsite:Errormessage

11/13/18

EvaluaLonResults•  ThesubjectswerebasicallysaLsfiedwiththecaptures,but

hadverymanysuggesLonsforimprovementswithlabeling,searching,display,andperformance.MostalsowantedaddiLonalfuncLonality.

•  ManyofthesubjectswereconfusedbetweencapturedsitesandtheFindingAidsforthem.InaddiLon,thewords“Papersof”incollecLonLtlesbaffledpeoplewhentheywerelookingforrecordings,notpapers.

•  Bothusersandsiteownerswereunclearaboutthescopeofcontentthathadactuallybeencollected.Onesiteownerexpresseddisappointmentthatreviewsthattheylinkedtowerenotcaptured.Andonlyonesubjectfiguredouthowtonavigatetoasuggested“liveweb”pagethathadnotbeennotarchived.

FuncLonalityrequestedbyusers•  Mostsubjectswantedmoremetadatadisplayed.Examplesincluded:

displayingadescripLonoftheComposersProjectandlikelycontentsontheiniLalstartpage;displayofaudio/videorun-Lmeinsteadoffilesize;descripLon,thumbnails,excerptsformaterialrestrictedtoonsiteuse(sothattheycoulddecidewhetherornottheyreallyneededtomakeasitevisit);morefieldsshowninvariousdisplays(bothinlistsandinlinkstoessence).

•  BothsiteownersrespondedposiLvelytotheideaofprovidingasitemapwithacollapsingmenuoflinks.

•  Mostsubjectswantedasearchbox.AndmostwantedtobeabletoimmediatelysortamulL-columndisplaylistbyanycolumnoftheirchoosing.

•  Onesubjectfounditmisleadingwhenarestrictedobjectlinkedtoanewpage.

•  Onesiteownerpreferredthattheirdigitalobjectsbeorganizedbyproject,ratherthaninanundifferenLatedlistofeverydigitalobjectontheirsite.

FuncLonalityrequestedforlocalworkstaLons

•  Abilitytotakescreengrabs•  AccesstoaddiLonalbrowserwindow

•  Previewframewhenscrubbing(fastforwarding)throughvideomaterial

•  Useoftheirownlaptoporanotherwindow•  DisplayofLmecode•  And2subjectsspecificallyrequestedthe

–  abilitytoslowvideo/audiofiletotranscribe–  abilitytodroppin/a4achnotestospecificpointinvideo/audiofile

IMPACTBEYONDTHISPROJECT

ImpactBeyondthisProject•  TherewillbeanalternaLvetoHeritrixforcapturing

streamingmedia,andArchive-Itwillideallybeabletobe4erhandlestreamingmedia,anddisplayitinpropercontext

•  WewillhavearchitecturesandworkflowsforArchive-Ittointeractwithricherlocalresources(aswellasexamplesofhowinteracLonandnavigaLoncanproceedbtwnArchive-It,ArchiveSpace,FindingAids,andaninternaldigitalrepository)

•  ModelsforinteracLonbtwncreatorsandcollecLngorganizaLonswillhavebeendeveloped(incldonoragreements)

•  Wehavepreserved100+++websitesofyoungcomposers

Archivingwebsitescontainingstreamingmedia:theMusicComposer

Project

•  h4p://besser.tsoa.nyu.edu/howard/Talks/•  h4p://www.nyu.edu/about/news-publicaLons/news/

2015/03/27/nyu-libraries-to-team-with-internet-archive-to-preserve-high-quality-musical-content-on-the-web.html

•  h4p://archive.org/~nlevi4/reveal.js/•  h4p://composers.dlib.nyu.edu/•  h4ps://rg3.github.io/youtube-dl/supportedsites.html

archive-itbesser.tsoa.nyu.edu/howard/talks/18composers-iipc.pdf · archive-it besser-iipc...

Documents

iipc illmer consulting ag - guidance to... · iipc...

iipc illmer consulting ag - new and... · produced by: dr....

iipc illmer consulting ag - update on gips 2020.pdf ·...

aleph archives iipc presentation

iipc: project re dds: digital archeology (may 9, 2011)

topic&5&–transport transportservices&and&protocols& ·...

wrf/dart(–applicaon( examples(

lv motors: hazardous area applicaon

iipc illmer consulting ag · produced by: dr. stefan j....

regionally significant industrial area applicaon

broadcasng video in dense 802.11g networks using applicaon

profiling web archives iipc ga 2015

hard content, fab front-end @ iipc 2014

iipc illmer consulting ag - irr attribution.pdf · produced...

united states ofamerica criminal no.: iipc^' i

the iipc web curator tool -...

towards multidimensional web archive access (iipc 2016)

iipc - decision oriented... · 2017-12-27 · net of...

http:// webarchiv czech web archive iipc 2007, paris

iipc illmer consulting ag - holdings... · produced by: dr....