Taking the Helm - Ancestry’s Journey to Kubernetes

Paul MacKay, Software Engineer

Our JourneyWhere we come from.

What we have learned along the way.

Where we are in the journey.

Where we come from.

We’re a science and technology company with a very human mission.

Data drives our business

• 20 billion historical records

• 90 million family trees

• 10 billion profiles

• 175 million shareable photos, documents and written stories

• 9 petabytes of data

• 4 million members in the AncestryDNA® network

• 37 million 3rd cousin or closer matches

Technologies

Microsoft Windows ®

C# .NET ®

SQL Server®, IIS, MSMQ, TFS, etc.

Java, Node.js, Python running on Linux

Private data center

1,000s of servers/VMs running 100s of services

REST-based macro to micro size services

Deployment Processes

Continuous integration/deploymentDeployment management tools

Virtual MachinesTypically one (macro/micro) service per VM

Long deployment times to roll services to production20-50 minutes

How Our Journey Started to Change

Began experimenting with Docker.Docker Compose

Created a “Docker agent” for remote deployment.

Demonstrated how easy it is to deploy and scale up services.Deployment times drastically reduced from current methods.

Easier to deploy services of any size (macro to micro)

Showed greater density using current computing resources.

Created and deployed our own micro services using Docker.

What we learned along the way.

Adopting New Technologies is Hard

Developers are comfortable with how things are done today.We think it is faster doing it the “old” way.

It is hard to see the advantages of changing to something new.

Change has real cost.Change takes time away from developing new features.

Change is disruptive to schedules.

Early Discoveries

Many opinions about the appropriate size of a service.

Normal Linux distros are just too big.Not specifically built for Docker.

Too large of a footprint.

Too many packages to keep updated.

Docker is best supported on newer Linux kernels.

Need to train Windows developers Linux concepts/tools.

The size of a service cannot be dictated

Container orchestration is hard to do it right.

Adopting New Technologies or Paradigms

Understand current technologies, processes and paradigms.

Need a “patron”.

Own something “to be real”.

Create a partnership with pilot teams and be agile.

Determining the Size of a Service

Be pragmatic; do not break up a service just to break up a service.

Remember the cost of managing many services.

Network latencies

Many things to worry about (e.g. monitoring, coordinated deployments, scaling)

Ask, “will this really be used independently by other services?”

Does it make sense for the service to exist by itself

Be pragmatic, not dogmatic.

Linux Built for Containers

Running containers is a first-class citizen

Updates are holisticCan be automatically pushed to machines.

Can easily revert back to the previous version.

Less is moreFewer packages means fewer vulnerabilities

Infrequent need for direct access to the machines

Kubernetes to the Rescue

Created a small ”sandbox” cluster.

Gathered “committed” pilot teams.

Daily standupsAddress problems/concerns early

Provided Docker and Kubernetes trainingDeveloped templates and scripts

Where we are in the journey.

Conventions/Standards

Developed deployment standardsNamespace for each service

Naming conventions (functionalGroup-serviceName)

One container per pod

Start with wide privileges and narrow as neededAllow deployment all the way to production

Secrets are controlled by operations/security

Separate clusters for each environment (dev, stage, prod)

Use intra-cluster DNS for micro-services to reduce network latencies

Production Environment

Private Registry (Quay Enterprise)

Monitoring (Prometheus)

LoggingCustom cluster-wide log forwarder

Namespace Portal

Authorization (webhook)

Require soft resource quotas

Quick Start Tools

Created tool to help teams quickly deployWorks across all cluster environments

Provide ”best practices” and conventions

Transparent – can generate standard resource files

Created scripts to insert secrets into namespacesLabels are used to version secrets

Cluster backup/restore scripts

Scripts to easily create clusters in various environments

Allow easy deployment of any size of service

Our Journey So Far

Several clustersPrivate data center and in the cloud

Hundreds of namespaces and servicesHundreds of pods

Macro to micro size services

Live production traffice.g. “We’re Related App”

Made up of 14 micro services

Easiest deployment path for developers

The Power of Kubernetes

Programmers have REPL (Read-Eval-Print-Loop)

Kubernetes now gives us CDEL (Compile-Deploy-Execute-Loop)

Unlock your past. Inspire your future

Questions