2016 10-26 docker meetup - kubernetes on open stack
TRANSCRIPT
Mario Siegenthaler
11th Docker Switzerland User Group, 26.10.2016
a field report
Ferdinand Hübner
Kubernetes on OpenStack
About Us
450
ops & dev
...
Solutions for the
Government
(Cantons)
450
ops & dev
About Us
450
...
Ferdinand Hübner
Mario Siegenthaler
CTO development
Solutions for the
Government
(Cantons)
Project Goals
Testing Automated
Exploration
(Production)
Continuous Delivery
User Acceptance
on
premise
production
quality
Requirements
easy setup
+ persistent data
+ “useful” stability
Technology Evaluation
Quelle: classroomclipart.com
kubernetes
Infrastructure
Plattform
Application
packaging
Classification of Applications
Analysis - Dimensions
storage
communication scalability
Analysis - Storage
storage cheap
fast
durable volatile
large small application-level
replicated
storage-level
replicated
Analysis - Communication
communication
load
balancing
security
external
access
Analysis - Scalability
scalability auto
scale
multi
AZ
vertical
only
heterogenous
(Master/Slave)
stateless
stateful
Excluded Functions
storage
communication scalability huge scale
auto scale multi AZ
redundancy
backup
fast storage dynamic scaling of
persistent replicas
IMPLEMENTATION
Implementation goals
Figure out what makes kubernetes tick
Consider AuthN and AuthZ
Cluster creation: fast, easy, automated
AuthN
● We chose certificates over tokens
● Password files were never an option
● Create simple and easy to use PKI tools
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● If you’re authenticated, you can do everything or nothing
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● Attribute Based Access Control
● Cumbersome - requires API server restart
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● Roles Based Access Control
● In Alpha
● Kubernetes API Objects
AuthZ: Kubernetes capabilities
policy files
(ABAC)
RBAC webhook Always
● Delegate to a webservice
● Implement a bridge to your corporate directory
AuthZ: Our choice
Always
● Freedom and responsibility for developers
● Separate teams with multiple clusters
Cluster overview
Ingress: Layer 4 and Layer 7
● Expose services easily with DNS and HTTPS
● Existing implementations are written in go
● LBaaS missing on our OpenStack
● We can’t use Let’s Encrypt
● Ingress spec doesn’t cover Layer 4
Ingress: Layer 4 and Layer 7
● Implementation: Java, Akka, HAProxy
● Assign services to ingress nodes
● Register and refresh DNS in SkyDNS cluster
● Wildcard SSL certificate
● HAProxy targets service endpoints
Ingress: Layer 4 and Layer 7
Custom with drawbacks, but good enough for now
kind: Service metadata: annotations: bedag.ch/ingress-bedag: mysvc.xy.bedag.ch bedag.ch/ingress-mode: https
Ingress: Next steps
Implement:
● Service Load-Balancer
● Ingress
Or:
● Use existing implementations
● Write addons
Persistent Volumes
● Plugin-Based architecture
● Started with NFS on K8S
● Moved to Ceph
● Reference Volumes by ID
kubernetes
Ceph
cinder
NFS
Ceph Cluster
● Traditional disks on OpenStack nodes
● No special tuning
● Sequential Read/Write is OK
● Write-Latency is awful
etcd WAL fsync duration
local SSDs
etcd WAL fsync duration
local SSDs
our Ceph
CREATING CLUSTERS
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
Infrastructure as code
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
Infrastructure as code
● We don’t want snowflakes in our datacenter
Creating clusters: goals / objectives
Cluster creation: fast, easy, automated
● (Our own) documentation gets outdated and fuzzy
Infrastructure as code
● We don’t want snowflakes in our datacenter
Immutable servers
OS choice
“CoreOS Linux is the leading container operating system,
designed to be managed and run at massive scale, with
minimal operational overhead.“
-- CoreOS, Inc
(Our pick at) Infrastructure as code
define a server entirely
through its cloud-config.yaml
by writing code that
templates it
Immutable servers
Throw away servers that
● need a configuration change
● need updating
● misbehave
Creating clusters and servers
cluster name and server role
associate ips, create certificates, create storage
template cloud-config.yaml
boot server, attach storage
Logging, Monitoring
● fluentd daemon set
● ships to graylog
● not using docker’s native
GELF output
● prometheus
● grafana
Kubernetes configuration files
Configuration files in git repositories
● Changes can be traced
● Easy to deploy to other clusters
● Deployment can be automated
Conclusion
great, powerful and stable!
perfect for prototyping and testing
storage can be a challenge
QUESTIONS
Contact Us
linkyard.ch
Mario Siegenthaler
Consulting Software Architecture & DevOps
Partner
Ferdinand Hübner
CTO Software Development
www.bedag.ch