openstack days taiwan 2016 0712
TRANSCRIPT
![Page 1: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/1.jpg)
1
~ Architecture of our public clouds ~
OpenStack Days TaiwanJul 12, 2016Naoto Gohko (@naoto_gohko)GMO Internet, Inc.
How is GMO Ineternet using OpenStack
for Public CloudSlide URLhttp://www.slideshare.net/chroum/openstack-days-taiwan-2016-0712-public-cloud-arch
ConoHa public cloud (lang zh)https://www.conoha.jp/zh/
ConoHa public cloud (lang en)https://www.conoha.jp/en/
![Page 2: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/2.jpg)
2
Public Clouds
We are offering multiple public cloud services.
![Page 3: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/3.jpg)
3
Physical Servers
Running VMPhysical Server
1508
25294
Created VM
Running Infrastructure (2015/10)
137223
![Page 4: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/4.jpg)
4
OpenStack service development team
![Page 5: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/5.jpg)
5
Cloud service development team: (abount 30 people)– OpenStack Neutron team: 4 people
• Neutron driver / modification / engineering– Cloud API development team: 5 people
• Public API validation program• OpenStack modification / scaduler programing / keystone
– Cloud Infra. development team: 11 people• Security engineering / glance driver / cinder driver / nova
additional extensions / construction of OpenStack infra.– Applicatoin cloud service development team: 5 people
• Billing engineering / staff tools / GMO AppsCloud web GUI
Additional engineering team: many people (30 ~) – QA Team / Server Engineering Team / GUI development Team– Network Engineering Team / SaaS development Team– CRM backend and billing Team
Cloud service development team: Now(2016)
![Page 6: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/6.jpg)
6
Cloud service development team: Office(2016) #1
Neutron TeamAnd
Cloud API Team
Cloud Infra. TeamAnd
AppsCloud Team
![Page 7: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/7.jpg)
7
Cloud service development team: Office(2016) #2
Neutron TeamAnd
Cloud API Team
Cloud Infra. TeamAnd
AppsCloud Team
![Page 8: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/8.jpg)
8
Limied number of people.But, we have to run a lot of OpenStack service clusters.
![Page 9: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/9.jpg)
9
Service developmemt historyby OpenStack
![Page 10: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/10.jpg)
10
Swift cluster
GMO Internet, Inc.: VPS and Cloud servicesOnamae.com VPS (2012/03) : http://www.onamae-server.com/Forcus: global IPs; provided by simple "nova-network"
tenten VPS (2012/12)http://www.tenten.vn/Share of OSS by Group companies in Vietnam
ConoHa VPS (2013/07) : http://www.conoha.jp/Forcus: Quantam(Neutron) overlay tenant network
GMO AppsCloud (2014/04) : http://cloud.gmo.jp/OpenStack Havana based 1st regionEnterprise grade IaaS with block storage, object storage, LBaaS and baremetal compute was provided
Onamae.com Cloud (2014/11)http://www.onamae-cloud.com/Forcus: Low price VM instances, baremetal compute and object storage
ConoHa Cloud (2015/05/18) http://www.conoha.jp/Forcus: ML2 vxlan overlay, LBaaS, block storage, DNSaaS(Designate) and original services by keystone auth
OpenStack Diablo
on CentOS 6.x
NovaKeystoneGlance
Nova network
Shared codes
Quantam
OpenStack Glizzly
on Ubuntu 12.04
NovaKeystoneGlance
OpenStack Havana
on CentOS 6.x
KeystoneGlance
Cinder
Swift
Swift
Shared cluster
Shared codes KeystoneGlanceNeutron
Nova SwiftBaremetal compute
NovaCeilometer
Baremetal compute
Neutron LBaaS
ovs + gre tunnel overlay
Ceilometer
Designate
SwiftOpenStack Junoon CentOS 7.x
NovaKeystoneGlanceCinder
Ceilometer Neutron LBaa
SGMO AppsCloud (2015/09/27) : http://cloud.gmo.jp/2nd region by OpenStack Juno based Enterprise grade IaaS with High IOPS Ironic Compute and Neutron LBaaS
Upgrade Juno
GSLB
SwiftKeystone Glance
CinderCeilometer
NovaNeutron
IronicLBaaS
![Page 11: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/11.jpg)
11
Dark age for the Cloud suppliers
![Page 12: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/12.jpg)
12
OpenStack Swift: shared cluster
![Page 13: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/13.jpg)
13
Swift Hardware: Object nodes• Boot: SSD x2• HDD: 4TB x12• E3-1230 v3 @ 3.30GHz• Memory 16GB• 10GbE x2 (SFP+)
(Intel NIC)ASUSTeK COMPUTER INC.RS300-H8-PS12
![Page 14: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/14.jpg)
14
Hardware: LVS-DSR and reverse-proxy(Layer7) nodes• Boot: SSD x2
• E3-1230 v3 @ 3.30GHz• Memory 16GB• 10GbE NIC x1 (Intel NIC)Supermicro microblade8 blade nodes type
![Page 15: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/15.jpg)
15
Hardware: swift-proxy nodes• Boot: HDD x6 (1.7TB)
– Ceilometer Log disk– (Swift all request billing data)
• E5620 @ 2.40GHz x2 CPU• Memory 64GB• NIC: 10GbE SFP+ x2(Intel NIC)System x3550 M3 (old IBM)
Hardware: account/container-server nodes• Boot: HDD x2• Account/Container storage: SSD x2• E5620 @ 2.40GHz x2 CPU• Memory 64GB• NIC: 10GbE SFP+ x2(Intel NIC)System x3550 M3 (old IBM)
![Page 16: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/16.jpg)
16
Swift cluster (Havana to Juno upgrade)
SSD storage:container/account server at every zone
![Page 17: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/17.jpg)
18
swift proxy
keystone
OpenStack Swift cluster (5 zones, 3 copy)
swift proxy
keystoneLVS-DSRLVS-DSR HAProxy(SSL)HAProxy(SSL)
Xeon E3-1230 3.3GHzMemory 16GB
Xeon E3-1230 3.3GHzMemory 16GB
Xeon E5620 2.4GHz x 2CPUMemory 64GB
swift objects
swift objects
Xeon E3-1230 3.3GHz
swift accountswift container
Xeon E5620 2.4GHz x 2CPUMemory 64GB, SSD x 2
swift objects
swift objects
Xeon E3-1230 3.3GHz
swift accountswift container
Xeon E5620 2.4GHz x 2CPUMemory 64GB, SSD x 2
swift objects
swift objects
Xeon E3-1230 3.3GHz
swift accountswift container
Xeon E5620 2.4GHz x 2CPUMemory 64GB, SSD x 2
swift objects
swift objects
Xeon E3-1230 3.3GHz
swift accountswift container
Xeon E5620 2.4GHz x 2CPUMemory 64GB, SSD x 2
swift objects
swift objects
Xeon E3-1230 3.3GHz
swift accountswift container
Xeon E5620 2.4GHz x 2CPUMemory 64GB, SSD x 2
![Page 18: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/18.jpg)
19
swift objectsswift objects
swift objectsswift objects
swift objectsswift objects
swift objectsswift objects
swift objectsswift objects
swift proxy keystone
Havana AppsCloudswift proxy keystone
Grizzly ConoHa
HavanaTo Juno
swift account
swift container
swift account
swift container
swift account
swift container
swift account
swift container
swift account
swift container
swift proxy keystone
Juno ConoHaswift proxy keystone
Juno AppsCloud
Swift cluster: multi-auth and multi-endpoint
swift proxy keystone
Juno Z.com
![Page 19: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/19.jpg)
20
Swift shared cluster: ex)
![Page 20: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/20.jpg)
21
OpenStack history of computing environment
![Page 21: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/21.jpg)
22
Oname.com VPS(Diablo) • Service XaaS model:
– VPS (KVM, libvirt)• Network:
– 1Gbps• Network model:
– Flat-VLAN (Nova Network), without floting IP(no L3)
– IPv4 only• Public API
– None (only web-panel)• Glance
– Public image only.
OpenStack service: Onamae.com VPS(Diablo)
![Page 22: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/22.jpg)
23
ConoHa(Grizzly)• Service XaaS model:
– VPS + Private networks (KVM + ovs)• Network model:
– Flat-VLAN + Quantam ovs-GRE overlay
– IPv6/IPv4 dualstack• Network:
– 10GE wired(10GBase-T)
• Public API: None (only web)• Glance
– Only Public image• Cinder: None• ObjectStorage
– Swift (After Havana)
OpenStack service: ConoHa(Grizzly, 2013/07)
![Page 23: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/23.jpg)
24
Grizzly• Quantam Network:
– It was using the initial version of the Open vSwitch full mesh GRE-vlan overlay network with LinuxBridge Hybrid
ButWhen the scale becomes large, Localization occurs to a specific node of the communication of the GRE-mesh-tunnel(with under cloud network(L2) problems)(Broadcast storm?)
OpenStack service: ConoHa(Grizzly)
![Page 24: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/24.jpg)
25
• Service XaaS model:– KVM compute + Private VLAN networks + Cinder + Swift
• Network:– 10Gbps wired(10GBase SFP+)
• Network model: – IPv4 Flat-VLAN + Neutron LinuxBridge(not ML2) + Cisco Nexsu L2 sw/port
driver– Brocade ADX L4-LBaaS original driver
• Public API– Provided the public API
• Ceilometer (Billing)• Glance : Provided(GlusterFS)• Cinder : HP 3PAR(Active-Active Multipath original) + NetApp• ObjectStorage : Swift cluster • Bare-Metal Compute
– Modifiyed cobbler bare-metal deploy driver – Cisco Nexsus switch bare-metal networking driver (L2 tenant NW)
OpenStack service: GMO AppsCloud(Havana)
![Page 25: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/25.jpg)
26
OpenStack service: GMO AppsCloud model
compute
vm
NIC
Vlan network
bridge
NIC vlan
tap
vNIC
Vlan network
vNIC
bridge
vlan
tap
compute
NIC
bridge
NIC vlan
bridge
vlan
public networkNeutron LinuxBridge model(very Fast, simple is Best) this cloud is optimized services for the GAME server.
![Page 26: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/26.jpg)
27
Cisco Nexsus L2 sw/Port manage driver(self made)• L2 resource is limited / SW CPU
– MAC ADDRESS– VLAN per Network– VLAN per Port
Allowed VLAN to trunked port is allowed only VLAN to be used in LinuxBridge in VM/Baremetal Compute node.
– Baremetal : link aggregation port– Port discovery using by lldp
• Cisco Nexsus NX-OS– Server:
LACP : port-ChannelActive-Active link aggreration
![Page 27: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/27.jpg)
29
Nova-baremetal(havana)/Ironic(juno) ansibleBaremetal networking• Bonding NIC + lldp discovery• Taged VLAN• allowd VLAN + dhcp native VLAN
![Page 28: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/28.jpg)
30
GMO AppsCloud(Havana/Juno)
![Page 29: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/29.jpg)
31
Public API security and load balance:• LVS-DSR• L7 reverse-proxy• API validation wrapper
![Page 30: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/30.jpg)
32
public API
Web panel(httpd, php)
API wrapper proxy(httpd, phpFramework: fuel php)
Nova API
Customer sys API
Neutron API Glance API
OpenStack API for input validation
Customer DB
Keystone API
OpenStack API
Cinder APICeilometer API
Endpoint L7:reverse proxy
Swift Proxy
![Page 31: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/31.jpg)
33
public API: step 1, step 2)
step 1) LVS-DSR (L4) is received https(tcp/443) packet, then forward api-reverse-proxy real IP’s.
step 2) HAProxy has valid API ACL and backend server configurations.IF HAProxy allowed POST “/v2.0/tokens”, then the request call to ext-api-wrapper0[12].
![Page 32: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/32.jpg)
34
public API: step 3), step 4)
step 3) ext-api-wrapper0 [12], it is a php program.request URI and header, and the input value of json of the body was confirmed by php, and then call the real OpenStack API as the next processing.
step 4) OpenStack API that is checked the input value will be run.
![Page 33: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/33.jpg)
35
OpenStack Juno cluster: • ConoHa (Juno) and Z.com
cloud• AppsCloud (Juno)
![Page 34: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/34.jpg)
36
Tokyo
Singapore
Sanjose
# ConoHa has data centers in 3 Locations
![Page 35: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/35.jpg)
37
Tokyo Singapole
User/tenant User/tenant
API ManagementKeystone API
API Management
Keystone API
API ManagementKeystone API
Token Token
Tokyo SanJoseSingapore
API Management
Keystone API
API Management
Keystone API READ/
WRITEREAD READ
TokenToken Token
Do not create/delete
users
Do not create/delete
users
Our Customer baseUser administration
# User-registration is possible in Japan only
DB Replication DB ReplicationUser/tenant User/tenantUser/tenant
R/W R/W
![Page 36: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/36.jpg)
38
OpenStack Juno: 2 service cluster, released
Mikumo ConoHa Mikumo Anzu
Mikumo = 美雲 = Beautiful cloud
New Juno region released: 10/26/2015
![Page 37: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/37.jpg)
39
• Service model: Public cloud by KVM• Network: 10Gbps wired(10GBase SFP+)• Network model:
– Flat-VLAN + Neutron ML2 ovs-VXLAN overlay + ML2 LinuxBridge(SaaS only)
– IPv6/IPv4 dualstack• LBaaS: LVS-DSR(original)• Public API
– Provided the public API (v2 Domain)• Compute node: ALL SSD for booting OS
– Without Cinder boot • Glance: provided• Cinder: SSD NexentaStore zfs (SDS)• Swift (shared Juno cluster)• Cobbler deply on under-cloud
– Ansible configuration• SaaS original service with keystone auth
– Email, web, CPanel and WordPress
OpenStack Juno: 2 service cluster, released
• Service model: Public cloud by KVM• Network: 10Gbps wired(10GBase SFP+)• Network model:
– L4-LB-Nat + Neutron ML2 LinuxBridge VLAN– IPv4 only
• LBaaS: Brocade ADX L4-NAT-LB(original)• Public API
– Provided the public API• Compute node: Flash cached or SSD• Glance: provided (NetApp offload)• Cinder: NetApp storage• Swift (shared Juno cluster)• Ironic on under-cloud
– Compute server deploy with Ansible config• Ironic baremetal compute
– Nexsus Cisco for Tagged VLAN module– ioMemory configuration
![Page 38: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/38.jpg)
40
OpenStack Cinder Block storage:
ConoHa: NexentaStor(SDS)AppsCloud: NetApp
![Page 39: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/39.jpg)
41
NexentaStor zfs cinder: ConoHa cloud(Juno)
Compute
![Page 40: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/40.jpg)
42
NetApp storage: GMO AppsCloud(Havana/Juno)If you are using the same Cluster onTAP
NetApp a Glance and Cinder storage, it is possible to offload a copy of the inter-service of OpenStack as the processing of NetApp side.
• Create volume from glance image
((glance the image is converted (ex: qcow2 to raw) required that does not cause the condition)
• Volume QoS limit: Important function of multi-tenant storage• Uppper IOPS-limit by volume
![Page 41: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/41.jpg)
43
OpenStack Ironic: Only AppsCloud:• Undercloud Ironic deploy• Multi-tenant Ironic deploy
![Page 42: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/42.jpg)
44
Ironic with undercloud: GMO AppsCloud(Juno)For Compute server deployment.Kilo Ironic and All-in-one• Compute server: 10G boot• Clout-init: network• Compute setup: Ansible
Under-cloud Ironic(Kilo):It will use a different network and Ironic Baremetal dhcp for Service baremetal compute Ironic(Kilo).(OOO seed server)
Trunk allowed vlan, LACP
![Page 43: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/43.jpg)
45
Ironic(Kilo) baremetal: GMO AppsCloud(Juno)Boot baremetal instance• baremetal server
(with Fusion ioMemory SanDisk)• 1G x4 bonding + Tagged allowed
VLAN• Clout-init: network + lldp• Network: Nexsus Cisco
Allowd VLAN security
Ironic Kilo + Juno: Fine• Ironic Python driver• Whole Image write• Windows: OK
![Page 44: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/44.jpg)
46
• Service model: Public cloud by KVM• Network: 10Gbps wired(10GBase SFP+)• Network model:
– Flat-VLAN + Neutron ML2 ovs-VXLAN overlay + ML2 LinuxBridge(SaaS only)
– IPv6/IPv4 dualstack• LBaaS: LVS-DSR(original)• Public API
– Provided the public API (v2 Domain)• Compute node: ALL SSD for booting OS
– Without Cinder boot • Glance: provided• Cinder: SSD NexentaStore zfs (SDS)• Swift (shared Juno cluster)• Cobbler deply on under-cloud
– Ansible configuration• SaaS original service with keystone auth
– Email, web, CPanel and WordPress
OpenStack Juno: 2 service cluster, released
• Service model: Public cloud by KVM• Network: 10Gbps wired(10GBase SFP+)• Network model:
– L4-LB-Nat + Neutron ML2 LinuxBridge VLAN– IPv4 only
• LBaaS: Brocade ADX L4-NAT-LB(original)• Public API
– Provided the public API• Compute node: Flash cached or SSD• Glance: provided (NetApp offload)• Cinder: NetApp storage• Swift (shared Juno cluster)• Ironic on under-cloud
– Compute server deploy with Ansible config• Ironic baremetal compute
– Nexsus Cisco for Tagged VLAN module– ioMemory configuration
![Page 45: Openstack days taiwan 2016 0712](https://reader035.vdocument.in/reader035/viewer/2022062901/58f9b353760da3da068bd501/html5/thumbnails/45.jpg)
47
Fin.