ceph@mimos: growing pains from r&d to deployment

Ceph@MIMOS: Growing Pains from R&D to Deployment

Jing Yuan LUKEAdvanced Computing Lab

Ceph Day Kuala Lumpur, 22 August 2016

• MIMOS – A Brief Overview• Distributed Object Storage in the era of Big Data• Our Ceph Journey• Moving Forward• Concluding Remark

Copyright © 2016, MIMOS Bhd 2

Outline

3

MIMOS: An Overview

Enhancing ICT industry growth through indigenous technologies

R&D in ICThttp://www.mimos.my

Copyright © 2016, MIMOS Bhd

MIMOS R&D Labs

Data Centric Approach

4Copyright © 2016, MIMOS Bhd

MIMOS R&D Labs

Data Centric Approach

5

INFORMATION SECURITY

PHOTONICSNANOELECTRONICS

ADVANCED ANALYSIS & MODELING

ARTIFICIAL INTELLIGENCE

WIRELESS COMMUNICATIONS

ADVANCED INFORMATICS

ADVANCED COMPUTING

MICRO-ELECTRONICS /ENERGY

ACCELERATIVE TECHNOLOGY

INTELLIGENT INFORMATICS

USER EXPERIENCE



Enterprise DataStore

Structured Data

DATA Sources in the Big Data Era


Structured Data


Sensors

Social Networks

Unstructured Data

MobilieDevices

Wearables

Tags

DATA Sources in the Big Data Era

DATAThe new Currency


Structured Data


Sensors

Social Networks

Unstructured Data

MobilieDevices

Wearables

The TSUNAMI

Tags

Copyright © 2016, MIMOS Bhd9


KnowledgeThe new Discovery

Enlightenment

Copyright © 2016, MIMOS Bhd10


KnowledgeThe new Discovery

Enlightenment

More data generatedMore storage required


Distributed Object Storage in Big Data Era

Source: SNIA : Swift Object Storage adding EC (Erasure Code), 2014


Distributed Object Storage in Big Data Era

Source: SNIA : Swift Object Storage adding EC (Erasure Code), 2014

• Other potential benefits• Scales

• Capability• Capacity

• Hardware agnostics• Self-healing

• Replication• Erasure Codes


Our Ceph Journey

2013

2014

20152016

• R&D• PoC/Testbed


Our Ceph Journey

2013

2014

20152016


• Which solution/platform that can provide:– Support to existing cloud initiatives, is it well

accepted by:• OpenStack• OpenNebula• Others

• Provide different ways to access the backend– Web services– Block like access– File System

• Highly Available– Active-active

• Linux upstream support


Our Ceph Journey

2013

2014

20152016


12

3MIMOS

TPMHPCC2 HPCC1

32

1

MIMOS KHTP


Our Ceph Journey

2013

2014

20152016


• Big Data Storage Event

• First Internal Deployment


Our Ceph Journey

2013

2014

20152016




Big Data Storage @ Big Data Week KL 2014


Our Ceph Journey

2013

2014

20152016




• Simple Backup/Archiving– First attempt to use Ceph in a small

production cluster– Backup application: BackupPC– Access: CephFS– Challenge: BackupPC creates a lot of small

files (several kB), saw plenty of space wasted (due to the default 4MB object size), solution create a dedicated pool and use extended attributes to assign mount point to different pool and reduce object size

– Moving forward: considering RBD and Bacula


Our Ceph Journey

2013

2014

20152016




• Mi-ROSS 1.0 development

• More deployments for:– VDI– Government

agencies, law enforcement agency, etc.

– Expanding internal cluster


2013

2014

20152016








Our Ceph Journey


2013

2014

20152016








• VDI– Challenges:

• 60+ Windows based VMs accessed via RDP, for s/w development environment, lots of I/O (code check-in/out, compiling, etc.)

• 30+ development VMs– Solution: Used a experimental feature, i.e. KV

based datastore instead of the typical journal based datastore

Our Ceph Journey


2013

2014

20152016








• Storage for Cloud Deployment and General Uses– Mi-Cloud (MIMOS Cloud Platform)

• Ceph provides datastores for and to VMs– Support for multiple workloads via Mi-ROSS

• File Sharing (NFS and SAMBA)

MyHDW

Our Ceph Journey


Our Ceph Journey

2013

2014

20152016









• Both Mi-Cloud and Mi-ROSS supporting 600+ VMs internally


2013

2014

20152016









• Both Mi-Cloud and Mi-ROSS supporting 600+ VMs internally

24

Our Ceph Journey

• As we grow we have learnt about various operation issues, e.g.– Impact on performance as we add/remove disks into the cluster

• How to work around it– Dealing with different network switches

• Different brands/models often implement the same protocols slight differently– Monitoring and Maintenance

• Developing monitoring agents/plug-ins• Developing our own SOP

– Fine tuning• Ceph• Kernel


Lessons learnt from the Journey

• A NAS appliance like building on top of Ceph providing:– Ceph Management

• Pool• RBD• CRUSH (planned)• Keys (planned)

– File sharing• SAMBA• NFS• iSCSI (in progress)

– Dashboard• Utilization• Health• Other Ceph related statistics

– Other Storage Services• “Dropbox” (planned)• Backup (planned)


Mi-ROSS

• Distributed Storage for Disaster Mitigation and Smart Cities

• How can distributed block storage such Ceph and be used to address Edge Computing– Network Latency– Ad-Hoc-ness of the edge devices– Etc


Moving Forward

• Distributed Object Storage such Ceph can address the data deluge from the era of Big Data and IoT

• MIMOS’ Advance Computing Lab would like welcome all to collaborate to further enhance this exciting open solutions


Concluding Remark


Thank You

ceph@mimos: growing pains from r&d to deployment

Technology