use of storage inside switch - terena.org · use of storage inside switch switch, 11.02.2014...
TRANSCRIPT
Use of storage inside SWITCH
SWITCH, 11.02.2014
Jens-Christian Fischer [email protected]
Thorsten Kleindienst [email protected]
© 2013 SWITCH
• Zürich CLA, ETHZ • Zürich RZ, ETHZ, 600 m distance to CLA • Lausanne, Amphimax, 220 km distance to Zürich
Locations
2
© 2013 SWITCH
• 28 HP ProLiant • 2 SPARC Enterprise T5220 • 2 SPARC Enterprise M4000 • 3 SPARC T3,4 • 6 Sun Blade T6320 Server modules • 15 Apple Xserves
Real server
3
© 2013 SWITCH
• 151 Debian • 53 Solaris • 23 Ubuntu • 10 Scientific Linux • 5 Windows • 2 Suse
Virtual server
4
© 2013 SWITCH
• 2 NetApp FAS2040, 45 TB netto space • 2 NetApp FAS2240, 40 TB netto space • 2 NetApp FAS3020, 6 TB netto space
Distributed on disk types • 75 TB SATA • 16 TB SAS • 0.4 TB SSD, FlashPool Cache
Central storage
5
© 2013 SWITCH
• KVM images via NFS, live migration possible • Solaris zones via iSCSI, no live migration • Unix-Homes via NFS • Shared Storage for miscellaneous projects • Backup of databases, e.g. Domain Registry
Usage of storage
6
© 2013 SWITCH
• 5 TB for backup on disk • 3 TB for KVM images • 2 TB for Unix-Homes, Project-shares, ... • 0.5 TB for internal mail • 0.2 TB for solaris zones
Consumption of storage on NetApp Filer
7
© 2013 SWITCH
• 95% asynchronous by means of Snapmirror • 5% synchronous for mail by NetApp MetroCluster
Replication
8
© 2013 SWITCH
• “Pre-Production/ownCloud” – 6 Physical servers – 192 cores – Ceph cluster: 140 TB Raw
– Status: Up and running
• Production – 2 Datacenters (Zurich, Lausanne) – 32 OpenStack Compute Nodes – 512 cores – 32 Ceph Storage Nodes – 768 disks, around 1.5 PB Raw Storage
– Status: Hardware has been ordered, go live: June 2014
OpenStack / Ceph
9
© 2013 SWITCH
• Integration into OpenStack – Glance – images/snapshots
• 1 TB – Cinder – volumes
• 2.5 TB
• RadosGW – Used for S3 compatible applications (SWITCHtube)
• 200GB
Ceph @ SWITCH
10
© 2013 SWITCH
• ownCloud based service for academia • Potentially 100’000 users • Go Live: 1.4.2014
• Storage: – 20 TB (for now) volume on Ceph – Exposed via NFS server
– 20 TB Backup volume
SWITCHdrive
11
© 2013 SWITCH 12 ceph
OpenStack Virtualisiert
rbd volume
web1 web2 web3
ha1
nfs1
Architektur Diagramm SWITCHbox v0.4jcf, 8.1.2014
ldap1
reg
Internet
ha - HA Proxy Load Balancerreg: Registration / LDAP account creation
web: Apache Web Servernfs: NFS Server
ldap: LDAP diectory Serverdb: Postgres DB Server
db1
AAI
SWITCHdrive Architecture
© 2013 SWITCH 13
SWITCHdrive – Preliminary findings
• Good support from ownCloud • Found (and experienced) a number of bugs – working with
support and development to fix them • Performance of PHP application not satisfactory (500 –
1000ms to upload a tiny file) – up to 1000 SELECT statements for one file upload
• Scalability concerns: – Database size (projected: 128 - 256 GB) – Storage (single mountpoint) – Application server performance for single clients (no parallelism due
to session locking)
© 2013 SWITCH
• Stock Ubuntu NFS Server on a VM • 2 vCPUs, 4 GB RAM • 20 TB Volume for data (backed by RBD) • 20 TB Volume for file system backups (again Ceph RBD)
• At one point we asked – is this server a bottleneck?
NFS Server Performance
14