globus: making it easy to move, share and publish...
TRANSCRIPT
![Page 2: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/2.jpg)
Thank you to our sponsors!U . S . D E PARTMENT O F
ENERGY
2
![Page 3: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/3.jpg)
...and Thank YOU!
5major services
13national labs use Globus
190 PBtransferred
10,000active endpoints
30 Bnfiles processed
45,000registered users
99.9%uptime
60+institutional subscribers
1 PBlargest single
transfer to date
3 months longest
continuously managed transfer
130federated
campus identities
10,000active users/year
![Page 4: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/4.jpg)
Agenda
• Overview for those new to Globus• Survey of “recent” enhancements• Brief demonstration• Questions
4
![Page 5: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/5.jpg)
Globus delivers…
Big data transfer, sharing,publication, and discovery…
…directly from your own storage systems…...via software-as-a-service
5
![Page 6: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/6.jpg)
Data management today
• Move?• Share?• Discover?• Reproduce?
![Page 7: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/7.jpg)
Globus and the research data lifecycle
Researcher initiates transfer request; or requested automatically by script, science gateway
1
InstrumentCompute Facility
Globus transfers files reliably, securely
2
Globus controls access to shared
files on existing storage; no need
to move files to cloud storage!
4
Curator reviews and approves; data set
published on campus or other system
7
Researcher selects files to share, selects user or group,
and sets access permissions
3
Collaborator logs in to Globus and accesses shared files; no local
account required; download via Globus
5
Researcher assembles data set;
describes it using metadata (Datacite& domain-specific)
6
6
Peers, collaborators search and discover datasets; transfer and share using Globus
8
Publication Repository
Personal Computer
Transfer
Share
Publish
Discover
• Only a Web browser required
• Use storage system of your choice
• Access using your campus credentials
7
![Page 8: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/8.jpg)
Another perspective...
Campus Bridging…within and beyond campus
boundaries
![Page 9: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/9.jpg)
Bridge to national cyberinfrastructure
Move datasets to supercomputer, national facility
Move results to campus (…)
![Page 10: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/10.jpg)
Bridge to instruments
• Clear staging store• Data cleansing• Pre-processing• ...
Amazon Glacier
High durability,low cost store
RawSourceData
Analysis store
![Page 11: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/11.jpg)
Bridge to your collaborators
EC2
Cloud stores
Local/Campus stores
• Group management• Share creation• Permissions configuration
![Page 12: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/12.jpg)
Bridge to community/public
• Review à Update àResubmit cycle
Multi-step curation workflow
• DOI
Persistentidentifier
• Datacite metadata• Domain metadata• Provenance info• ...
Describe…• PDF/A• HDF• …
(Re)format…
Replicated, publicrepositories
![Page 13: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/13.jpg)
Why use Globus?
• Ease of use: consistent UI across systems• “Fire-and-forget” file transfer• Maximized WAN throughput• Secure access w/multi-tier security model• No/low-overhead external collaboration • Rapid deployment w/standard packages• Highly automatable: CLI, RESTful API
13
![Page 14: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/14.jpg)
What’s new?
• New authentication service• Endpoint search and bookmarks• Management console• Data publication and discovery• Storage connectors• Globus Platform-as-a-Service
14
![Page 15: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/15.jpg)
Demonstration
15
![Page 16: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/16.jpg)
Globus Connect…Makes your storage system a Globus endpoint
16
![Page 17: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/17.jpg)
Globus Connect Personal
• Installers do not require admin access• Zero configuration; auto updating• Handles NATs
17
![Page 18: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/18.jpg)
Globus Connect Server
• Enable access for all users with local accounts• Create endpoint on practically any filesystem• Native packages: RPMs and DEBs
Local users
18
Local Storage System(HPC cluster, campus server, …)
Globus Connect Server
MyProxyCA
GridFTP Server
OAuthServer
![Page 19: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/19.jpg)
Storage connectors
• Standard storage connectors (Posix)– Linux, Windows, MacOS– Lustre, GPFS, OrangeFS, etc.
• Premium storage connectors– HPSS– HDFS– S3– Ceph RadosGW (S3 API)– Spectra Logic BlackPearl– Google Drive (coming soon)
19
![Page 20: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/20.jpg)
Desktop
GlobusCloud
Firewall
ScienceDMZ
Globus Platform-as-a-Service
• Move portal storage into Science DMZ, with Globus endpoint
• Leave Portal Web server behind firewall
• Globus handles the security and data heavy lifting 20
GlobusTransferService
PortalWebServer (Client)
GlobusAuth
Browser
User’sEndpoint(optional)
PortalEndpoint
OtherEndpoints
HTTPS
GridFTP
REST OtherServices
GlobusWebHelperPages
![Page 21: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/21.jpg)
Sample python data portal
21https://github.com/globus/globus-sample-data-portal
![Page 22: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/22.jpg)
Coming Soon
• HTTPS support– Synchronous alternative GridFTP– Implements same fine-grained access control model– Greatly simplifies sharing/transfer for “small” datasets– Supports standard browser behaviors such as
viewing known file types in the browser– Allows integration with clients and web apps to further
leverage existing research storage systems
• New command line interface– Locally installed Python client– Similar to AWS command line tools
22
![Page 23: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/23.jpg)
Globus Subscriptions• Standard Subscription
– Shared endpoints– Data publication– HTTPS support*– Management console– Usage reporting– Priority support– Application integration
• Branded Web Site• Premium Storage Connectors
– Amazon S3, Ceph, HPSS, Spectra, Google Drive*, …
• Alternate Identity Provider (InCommon is standard)23*Available in early 2017
![Page 24: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/24.jpg)
New pricing model
24https://www.globus.org/subscriptions
![Page 25: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/25.jpg)
Representative subscribers
25
![Page 26: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/26.jpg)
Learn more on the GlobusWorld Tour
• www.globusworld.org/tour/• Webinar: October 6th, 1pm CDT• SC16: Full-day developer tutorial (Booth #436)
26
![Page 27: Globus: Making It Easy to Move, Share and Publish …symposium2016.oscer.ou.edu/oksupercompsymp2016_talk_vas...2016/09/21 · ...and Thank YOU! 5 major services 13 national labs use](https://reader036.vdocument.in/reader036/viewer/2022070711/5ec7e9b5eab61f50f12970b2/html5/thumbnails/27.jpg)
Join the Globus community
• Access the service: globus.org/login• Create a personal endpoint:
globus.org/app/endpoints/create-gcp• Create a server endpoint:
docs.globus.org/resource-provider-guide• Need help? [email protected]• Subscribe: globus.org/subscriptions• Follow us: @globusonline
27