etadatasynchronization ack cyfronet agh kraków, …cgw'18, october 22-24, krakÓw. c....
TRANSCRIPT
![Page 1: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/1.jpg)
CGW'18, October 22-24, KRAKÓW
METADATA SYNCHRONIZATIONPROTOCOL FOR ADECENTRALIZED NETWORK OFDATA PROVIDERS
Łukasz Opioła | Łukasz Dutka | Renata G. Słota | Jacek Kitowski
ACK CYFRONET AGH, Kraków, Poland
AGH University of Science and Technology, Kraków, PolandFaculty of Computer Science, Electronics and TelecommunicationsDepartment of Computer Science
![Page 2: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/2.jpg)
CGW'18, October 22-24, KRAKÓW
Global data access for modern science
Challenges of global data access
Metadata synchronization protocol - requirements
Proposed concept of metadata synchronization protocol
Conclusions
1
2
3
4
5
AGENDA
![Page 3: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/3.jpg)
CGW'18, October 22-24, KRAKÓW
GLOBAL DATA ACCESS FOR MODERN SCIENCE
GROUP 1
GROUP 2
GROUP 3
![Page 4: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/4.jpg)
CGW'18, October 22-24, KRAKÓW
CHALLENGES OF GLOBAL DATA ACCESS
GROUP 1
GROUP 2
GROUP 3
![Page 5: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/5.jpg)
CGW'18, October 22-24, KRAKÓW
CHALLENGES OF GLOBAL DATA ACCESS
GROUP 1
GROUP 2
GROUP 3
Transparent data access
Bob
Bob
![Page 6: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/6.jpg)
CGW'18, October 22-24, KRAKÓW
CHALLENGES OF GLOBAL DATA ACCESS
GROUP 1
GROUP 2
GROUP 3
Transparent data access
Cross-border collaboration
BobAlice
Bob
![Page 7: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/7.jpg)
CGW'18, October 22-24, KRAKÓW
CHALLENGES OF GLOBAL DATA ACCESS
GROUP 1
GROUP 2
GROUP 3
Transparent data access
Cross-border collaboration
Security & efficiency BobAlice
![Page 8: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/8.jpg)
CGW'18, October 22-24, KRAKÓW
CHALLENGES OF GLOBAL DATA ACCESS
GROUP 1
GROUP 2
GROUP 3
Transparent data access
Cross-border collaboration
Security & efficiency
Openness of the network
BobAlice
GROUP 4
James
![Page 9: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/9.jpg)
CGW'18, October 22-24, KRAKÓW
CHALLENGES OF GLOBAL DATA ACCESS
GROUP 1
GROUP 2
GROUP 3
Transparent data access
Cross-border collaboration
Security & efficiency
Openness of the network
Autonomy (lack of trust)
GROUP 4
! !!
!
!
!
![Page 10: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/10.jpg)
CGW'18, October 22-24, KRAKÓW
CHALLENGES OF GLOBAL DATA ACCESS
GROUP 1
GROUP 2
GROUP 3
Transparent data access
Cross-border collaboration
Security & efficiency
Openness of the network
Autonomy (lack of trust)
Reflecting existing hierarchiesGROUP 4
![Page 11: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/11.jpg)
CGW'18, October 22-24, KRAKÓW
METADATA SYNCHRONIZATION PROTOCOL
![Page 12: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/12.jpg)
CGW'18, October 22-24, KRAKÓW
GROUP 1
GROUP 2
GROUP 3
GROUP 4
METADATA SYNCHRONIZATION
![Page 13: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/13.jpg)
CGW'18, October 22-24, KRAKÓW
Decentralized (retaining autonomy)
Globally consistent
Scalable – hundreds of providers
Secure – decentralized AAI*
Location discovery mechanisms
METADATA SYNCHRONIZATION - REQUIREMENTS
* AAI – authentication and authorization infrastructure
![Page 14: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/14.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION
Metadata synchronization protocolbased on
hybrid, multi-tier architecture
![Page 15: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/15.jpg)
CGW'18, October 22-24, KRAKÓW
Gathers knowledge using metadata sync
Manages data on underlying storages
PROPOSED SOLUTION – PROVIDER SCOPEProvider service – basic building block
AAI – Authentication and Authorization Infrastructure
![Page 16: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/16.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION – ZONE SCOPEZone service – oversees a group of providers
Gathers and serves metadata
Trusted authority & mediator
AuthN & AuthZ center
Macaroon based tokens
Reflects existing hierarchies
![Page 17: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/17.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION – GLOBAL SCOPECooperation between Zones – hybrid architecture
Local authority center
Decentralized, P2P backbone
ZONE A ZONE B
PROVIDER 1 PROVIDER 2 PROVIDER 3
Local authority center
![Page 18: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/18.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION – METADATA SYNCMetadata handled by the Zone of origin – local AuthN & AuthZ center
CREATE / UPDATE / DELETE
READ
AAI AAI
CREATE / UPDATE / DELETE
CREATE / UPDATE / DELETE
READ
READ
ZONE A ZONE B
No concurrent modification conflicts Metadata sync overheads evenly distributed
PROVIDER 1 PROVIDER 2 PROVIDER 3
![Page 19: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/19.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION – METADATA SYNCMulti-tier client-server architecture
PROVIDER 1
PROVIDER 2
PROVIDER 3
ZONE A ZONE B
web interface web interfacenative client
![Page 20: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/20.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION – METADATA SYNCPublish / subscribe and multi-tier caching
FETCH & SUBSCRIBE
USER META
CACHE
CACHE CACHE
CACHE
CACHEZONE A ZONE B
PROVIDER 1
PROVIDER 2
PROVIDER 3
web interface web interfacenative client
FETCH & SUBSCRIBE
FETCH & SUBSCRIBE
FETCH & SUBSCRIBE
FETCH & SUBSCRIBEUSER META
USER META
USER META
USER META
Vast performance improvement
![Page 21: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/21.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION – METADATA SYNCPublish / subscribe and multi-tier caching
PUBLISHPUBLISH
PUBLISH
PUBLISH
PUBLISH
CACHE
CACHE CACHE
CACHE
CACHEZONE A ZONE B
PROVIDER 1
PROVIDER 2
PROVIDER 3
web interface web interfacenative client
MODIFIED
CACHE
CACHE CACHE
CACHE
CACHE
Eventual consistency
![Page 22: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/22.jpg)
CGW'18, October 22-24, KRAKÓW
PROPOSED SOLUTION – AAIMacaroon-based decentralized authorization
TOKEN
location https://zone-a.com
public bd3f798c749b
<caveats> …
sig a7e108fa16baeffab0
ZONE A ZONE B
DISCOVERYMECHANISM
PROVIDER 1
PROVIDER 2
PROVIDER 3
web interface web interfacenative client
FETCH & SUBSCRIBE
FETCH & SUBSCRIBE
FETCH & SUBSCRIBE
![Page 23: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/23.jpg)
CGW'18, October 22-24, KRAKÓW
A) Open
B) Restricted
C) Isolated
PROPOSED SOLUTION – ZONE MODE
![Page 24: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/24.jpg)
CGW'18, October 22-24, KRAKÓW
METADATA SYNC PROTOCOL IN GLOBAL DATA ACCESS
ZONE 1
ZONE 2
ZONE 3
GROUP 1
GROUP 2
GROUP 3
GROUP 4ZONE 4
![Page 25: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/25.jpg)
CGW'18, October 22-24, KRAKÓW
Zone service acting as central authority and SP serverSynchronization Protocol (SP) for single zone scopeMacaroon based AAI for single zone scopeData provider service employing the SPCross Zone cooperation supportSync protocol supporting global, cross zone scale
CONCLUSIONS Global data access can be achieved by creating a decentralized network of data providers
We propose a metadata synchronization protocol for such network, based on hybdrid, multi-tier architecture with P2P backbone
Proposed concept is being implemented in Onedata, a distributed virtual file system
![Page 26: ETADATASYNCHRONIZATION ACK CYFRONET AGH Kraków, …CGW'18, October 22-24, KRAKÓW. C. HALLENGES OF GLOBAL DATAACCESS. GROUP 1. GROUP 2. GROUP 3 Transparent data access Cross-border](https://reader036.vdocument.in/reader036/viewer/2022062921/5f0513b67e708231d411254a/html5/thumbnails/26.jpg)
CGW'18, October 22-24, KRAKÓW
THANK YOU
https://onedata.org
The authors gratefully acknowledge the financial support of this work from AGH-UST