foldershare: building a data sharing cloud on drupal 8 for ... · node : 25,639 (drupal core 8.5.0)...
Post on 08-Aug-2020
4 Views
Preview:
TRANSCRIPT
AmitChourasia,DavidNadeau&MichaelNormanSanDiegoSupercomputerCenter,UCSanDiego
Projectcode: dibbs.seedme.org/downloads
or
drupal.org/projects/foldershare
drupal.org/projects/smalldata
Trialwebsite: sandbox.seedme.org
Projectwebsite:dibbs.seedme.org
FolderShare:BuildingadatasharingcloudonDrupal8forresearchers
PresentaHonOverview1. Background&moHvaHon2. Architecture3. SeedMe2plaPorm
FolderShare:VirtualfilesystemforDrupal
» EnHtydatamodel&accesscontrol» Filemanagement&security» ViewsintegraHon» UI&Commandplugins» FileformaUers» Webservices
SmalldatamoduleandAPI
– VisualizaHon4. Targetusers/Usecases5. Screenshots6. Demo
SeedMe1
a.k.a.StreamencodeexploreanddisseminateMyexperiments• VideoencodingfromvisualizaDonimageswasthemainfocus
• BasedonDrupal7• InproducHonasPlaPormasaService(PaaS)
SeedMe2:Datasharingbuildingblocks
EvoluHonoftheoriginalSeedMeproject(Completerewrite)• DataorganizaDon,sharinganddatamanagementisthemainfocus
• IncorporatesuserfeedbackfromoriginalSeedMeproject• BasedonDrupal8• BuiltfordistribuHon(opensource)andextension
SeedMeProjects
SeedMe2’sfocus• Enablerapidaccesstodatatoconsumabledataovertheweb
Mustbehandledbystockwebbrowser(Upload<2GBperfile)Mustbedisplayableonmanydevices(PhonetoPC)
• RichpresentaDonSupportcustomizaHon/extension
• DiscoverabilityIndexandsearchfiles,descripHons,etc..
• ThreewayinteracDonWebbrowserCommandlineAPI
Datamanagementstumblingblocks
Transfer Storage CollaboraJon AutomaJonAccesscontrol
ButwhataboutPresentaJonandDiscovery?
Datamanagementstumblingblocks
Transfer Storage CollaboraJon AutomaJonAccesscontrol
Issuesduetocontentdispersion
DescripJoninsomeone’smindDataintheCloud Discussiononemails
ThreeD’s:Data,DescripGon,Discussion
FilesystembasedsoluDons
RelatedsoluHons
ContentmanagementsystembasedsoluDonsHubZeroFigShare
MiddlewareGlobusIRodsNEWT
SoNwarerepositoriesGitHubSVNCVS
FilehosDngClouddrives(Dropbox,etc)WebDAV
ToolsSCPFTP
LimitaDonsofexisDngsoluDons Lackextensibility Lacksupportforrichcontent(descripHon,discussion,etc…) Lackindependentdevelopersupport Lack3wayinteracHonviawebbrowser&commandline&API Resourcerestricted
Workflow
Update as desired
Description
CreateProject
Sharing
Add foldersView
SearchDownload
Web browser, Command line, REST or App
Sign In
1 2
3
4
Upload files
Users
Drupal8
Architecture
Modules
• Virtualfilesystem• Accesscontrol• Hierarchicalstorage• Commandplugins• UIanddisplay• Search/index• Webservices
• FileformaUers• QuickVisualizaHon
Webserver(Apache+PHP)
Database(MySQL)
Webbrowser
RESTclients
Commandline
Smalldata(PHPlibrary)
Drupal8
Contributed
Modules
e.g.
Federated
AuthenDcaDon
viaOIDC
module
ProjectcontribuHons
Drupal(ContentManagementSystem)
• Modularandextensiblearchitecture(1,000+communitycontributedmodulesforversion8)• Widelyusedinindustry,academiaandgovernment(thirdmostpopularCMSonweb)• Largedeveloper&supportcommunity(4,000contributorsforDrupal8core+thousandsmore)• Securityadvisoryandupdatesforcoreandstablecontributed
moduleseverymonth• VersaHledeploymentopHons(personalhosHng,insHtuHonalhosHng,cloudhosHng)
FolderSharemodule
• Virtualfilesystem(fieldable):– EnHtytype&API– Accesscontrols– Usagetracking– Views,displays,breadcrumbs,forms– PluginsforfieldformaUers,search,views,acHons,andqueueworkers
• Configurablebysites– e.g.Keywords,comments,flags,DOIs
• Extensiblebydevelopers
Files&folders
• Childrenpointtoparents– ParentIDsenablefastqueriesforallchildrenofafolder
– RootIDsenablefastqueriesforaccesscontrolsandbreadcrumbs
parenHdpointstoimmediateparent
rooHdpointstotopfolder
Accesscontrols
• Drupalaccount-based• Permissions+accesscontrollistontopfolders– Listofusersthatcanviewandauthor
• TopfoldercontrolsenHrehierarchy– SimplerthandesktopOSes– Similartofilesharingservices– Fasttocheckaccess
Filestorage
• Foldersonlyexistindatabase• Filesdescribedindatabase&storedondisk
• Diskdirectory!==folderhierarchy– BeUerforsecurityandloadbalancing– Fileshavegeneratednames
• Avoidsnamecollisions• Avoidscharactersetandnamelengthlimits
Views
• Listpersonal,public,andsharedfiles&folders– Pages&embeddedviewsinfolderpages
• Integrateddesktop-likeUI– Selectfilesandfolders– Thenchoosemenucommand
• ThreeUIvariants:– NoscripHng– ScripHngbutnoAJAX– ScripHngwithAJAX
Plugins
• FieldformaUers– Foldernames,enHtyreferences,MIME-typeicons
• Search– Indexandpresentresults
• Queueworker– Updatefolderhierarchysizesinbackground
• AcHons&customcommands– MenuUIitemstoadd,delete,etc
Totallinesofphpcode
Node:25,639(Drupalcore8.5.0)Foldershare:50,156(Alphaversion)
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
v1 v2 v3 v4 v5 alpha
Linesofcode
Release
Foldersharecodemain
PHP
Docs
0
2000
4000
6000
8000
10000
12000
v1 v2 v3 v4 v5 alpha
Linesofcode
Release
Foldersharecodemisc
JS
YML
CSS
TWIG
TXT
Codetrivia
FoldershareAPIDocumentaHon
SmallDataAPI&Module
• Structureddataparsers&writers– Tables,trees,andgraphs– JSON,CSV,TSV,TXT,etc.
• FieldformaUers– Light-weightvisualizaHon– Lineplots,barcharts,piecharts,etc.
SmallData
FORADMINISTRATORS
FolderShareconfiguraDonAdminmenuStructure>FolderShare
FolderShareconfiguraDonFieldsManagefields,forms&display
ConfiguraHonlocatedinadminmenuStructure>FolderShare
FolderShareconfiguraDonFiles
StoragelocaHon&uploadrestricHons
FolderShareconfiguraDonInterface
Commandplugins
FolderShareconfiguraDonLists
ManagelisHngoffileandfolders
FolderShareconfiguraDonSearch(opDonal)
FolderShareconfiguraDonSecurity
ManagesharingcapabiliHes
FolderShareconfiguraDonWebservices(opDonal)
ManageRESTcapabiliHes
FolderShareRESTseWngsRequiresRESTUIcontributedmodule
ManageRESToperaHons
FolderShareUsageAdminmenuReports>FolderShareusage
FolderSharepermissions
FORUSERS
Topfoldersownedbyyou
Topfolderssharedwithyou
Publictopfolders
Menu
Theselistsdisplaytoplevelfolders
FoldersmayhaveadescripHon
MenuopHons–withnoselecHon
MenuopHons–withselecHon
Sharingformtorestrictaccess
SortablelisHngoffilesandfolders.
Differentusers
Breadcrumbsshowspath
EveryfolderandfilemayaddadescripHon.(FormaUedtextfieldakaBodyfieldinDrupal’sNode)
AddcustomsfieldssuchasComments,Keywords/tags,taxonomytoFolderShare.
TheFolderShare
enDtyisfieldable.
MenuopHonschangeonselecHon
Viewsubfolder
Breadcrumbsshowspath
VisualizaHonscanbeswitchedinteracHvelytodifferentcharttypes
QuickvisualizaHonofCSV&JSONfiles
SamplecommandlineinteracHonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'
help
SamplecommandlineinteracHonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'
help
ls --help ls / ls -l "/Classification Collection" ls -l "/Classification Collection/Preliminary Results"
mkdir --help mkdir "/test" mkdir "/test/data"
upload --help upload "plots/villi.png" "/test" upload "plots/composite.png" "/test” Upload "sample-small-data/OpenGL mesh memory use.csv" "/test/data" Upload "sample-small-data/Image classification breakdown schedule.json" "/test/data"
download --help download "/test" "/Users/amit/downloads"
rm --help rm -rf "/test”
SeedMe2–Targetusers/Usecases• Researchers
– CollaboraHonhubforaprojectorpersonalwebsite• Customprojectrepositories
– IncludeprojectspecificcustomizaHon(e.g.taxonomy,keywords)• Developers
– IntegrateyourscienHficapplicaHon• Sciencegateways
– Datasharing– Datapublishing– Dataescrowservice
• CyberInfrastructureproviders– OfferSeedMe2plaPormtoyouruserbase/communiHes
Realusagebyphysicists
Deployment/serviceopHonsDIY-Runyourowninstance(Yourbranding+domain)• Onyourownhardware• Cloudhosted
Provider/vendorrunsaninstance• YourinsHtuHon• CIcentersorITservices• Commercialvendors
Cloudservice(Ourdomain&branding)
Explorerswelcome(webbrowserneeded)Trialwebsite:sandbox.seedme.org
Tryityourself!
DevelopersinvitedDownloadcode:h_ps://dibbs.seedme.org/downloads
Team
AmitChourasia,DavidNadeau,DmitryMishin&MichaelNormanSanDiegoSupercomputerCenter|UniversityofCaliforniaSanDiego
SeedMe
Acknowledgements
AllusersandapplicaHonintegratorsfortheirvaluablefeedback
NaDonalScienceFoundaDon
ThismaterialisbaseduponworksupportedbytheNaHonalScienceFoundaHonunderGrantNo.ACI-1235505andACI-1443083
"Anyopinions,findings,andconclusionsorrecommendaHonsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNaHonalScienceFoundaHon."
Talk to usamit sdsc.edu
Keen to learn about potential uses cases in your work
Tutorial: Thursday afternoonBuilding data centric website for research projects and research groups
Requirements
1. Installdocker2. Downloadstart/finalimages(~1.2GB)
top related