docker & ecs: secure nearline execution

Scala, ECS, Docker: Delayed Execution @Coursera

ECS & Docker:Secure Async Execution @

Brennan Saeta

1

- General platform, not just for single course types . - Advance pedagogy - Transformative education?2

The Beginnings 2012

10courses1 million learners worldwide

4partners

3

Education at Scale

1,800courses18 million learners worldwide

140partners

4

OutlineEvolution of Courseras nearline execution systemsNext-generation execution framework: IguazIguaz application deep dive: GrID evaluating programming assignments

Key TakeawaysWhat is nearline execution, and why it is usefulBest practices for running containers in production in the cloudHardening techniques for securely operating container infrastructure at scale

A history of nearline execution

Let me paint a picture for you. It's the wild wild west of 2012 silicon valley. Like gold miners from yesteryear, the weight of hopes, dreams and promises of affordable high quality education pushed a small team of mostly Stanford undergrads to build a platform for global learning.8

Coursera Architecture (2012)

PHP Monolith

Everyone was working around the clock, and we needed to get something shipped quickly. We started with a stateless PHP-based monolith backed by a sharded array of MySQL servers. This architecture enabled the small team to quickly build out the fundamental features of the learning platform. We built forums, video lectures, in video-quizzes, assessments, and more in this architecture. Thanks to some good engineering, it scaled beautifully and had great availability.

But then, we started getting these weird feature requests that we couldn't effectively build in this monolithic architecture.

9

Early days - RequirementsVideo re-encoding for distribution

Grade computation for 100,000+ learners

Pedagogical data exports for courses

Since joining Coursera, I've learned a few things. One of which is that instructors are humans. Another, is that procrastination is a global phenomenon. Instructors would upload their video lectures hours before they needed to be released. We needed to quickly optimize them for distribution across the internet and to our low-bandwidth users. However, our webtier was not well suited for this long-running job.

Additionally, as we built our platform, we wrote a function that would compute a user's grade as they progressed through the course. However, as courses ended, we needed to re-compute everyone's grades in order to issue certificates of completion. We had no way of doing this effectively within a web request.

Finally, a key promise of MOOCs is pedagogical innovation derived from large learner behavior datasets. Our early instructional teams were begging us to release data on their own courses10

Coursera Architecture (2012)

PHP Monolith

The PHP monolith had a lot of really useful code. We had a sharded database abstraction, common data models, and libraries such as the grade computation function. We had so many new features to build, so we wanted to avoid re-writing all of that.

So, we did the easy-expedient thing, ...11

Cascade Architecture

PHP Monolith

PHP MonolithCascade

Cascade Architecture

PHP Monolith

PHP MonolithCascade

Queue

Copy of online serving codebase polling a queue.Restarts required due to memory leaks in PHP runtime.Code updates were infrequent and painful.

13

Upgrading to ScalaRe-architecting delayed execution for our 2nd generation learning platform.

Already in 2012, we realized the need to move off of PHP. After many lengthy debates on the comparitive merits of static types, concurrency, and performance, and after experimenting with toy Python, Go, and Java services, we eventually settled on Scala for our primary server-side technology. By 2013, we began completely re-architecting the learning platform from the ground up.

As part of this migration, we re-built our nearline execution framework in Scala.14

Upgrading to the JVMLeverage mature Scala & JVM ecosystems for code sharingJVM much more reliable (no memory leaks)New job model: scheduled recurring jobs.Named: Saturn

Code sharing: - JARs - Packages - DI-abstractions, such as Guice Modules

... Now, as part of the migration, we changed the mental model for running a job. We realized that running some code on a regular cadence is a useful building block for platform features. Developers would write their jobs, and schedule them to run on a regular, recurring basis.15

Saturn Architecture

Service A

Service B

Service C

C*Online ServingScala/micro-service architecture

C*

As we moved to a modern, Scala, microservices-based architecture, we invested heavily in the tool-chain, from common libraries to automated deployment.

We still were aggressively under-resourced, so we wanted to re-use as much of that as possible.16

Saturn Architecture

Service A

Service B

Service C

C*Online ServingScala/micro-service architecture

Saturn

C*

As a result, Saturn is just another HTTP microservice, that serves no HTTP requests. When the server boots up, it forks a background thread to run the jobs. These jobs can easily interact with the other microservices in our architecture, just like any other microservice.

For high availability, we always run at minimum 3 replicas of every service across 3 availability zones. While this works fine for the other microservices where each incoming request is sent to one replica, this is a big problem for Saturn. We do not want 17

Saturn Architecture

Service A

Service B

Service C

C*

Saturn

C*ZK Ensemble

Saturn Architecture

SaturnLeaderZK Ensemble

Service A

Service B

Service C

C*

C*

... Now the conventional wisdom is that if you have a problem, and then you introduce zookeeper, you now have 2 problems. While zookeeper may be seen as an architecture anti-pattern, Saturn had much bigger issues.19

Problems with SaturnSingle master meant nave implementation ran all jobs in same JVMHuge CPU contention @ top of the hourOOM Exceptions & GC issues

20

Enter: Docker

Containers allow for resource isolation!

CC-by-2.0 https://www.flickr.com/photos/photohome_uk/1494590209

21

Supported FeaturesPlatformSaturnDockerAmazonECSIguazRun codeResource IsolationClusters /HAGreatdeveloper workflowScheduledJobs

Saturn - https://upload.wikimedia.org/wikipedia/commons/c/c7/Saturn_during_Equinox.jpg

22



23



24

Additional requirementGreat developer workflow:Easy developmentEasy deployment Reliable runtime

Key point: minimal amount of work required to get their job done. Abstract away not just VMs / instances / clusters / etc., but also difficulties of code sharing & scheduling & deployment.25


Most important feature: great developer workflow. Developers care about the product features they need to ship. They dont care if underneath the hood its running on containers, VMs or bare metal, so long as there is: - Easy development - Automated deployment - Reliable runtime

26

Supported FeaturesPlatformSaturnDockerAmazonECS???Run codeResource IsolationClusters /HAGreatdeveloper workflowScheduledJobs


27

Solution: IguazMarissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0

- Where Iguazu name comes from28

Solution: IguazFramework & service for asynchronous executionOptimized Scala developer experience for Coursera

Unified scheduler supports:Immediate execution (nearline)Scheduled recurring execution (cron-like)Deferred execution (run once @ time X)Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0

Nearline execution, or almost immediate execution of non-interactive jobs that interact with online serving systems.29

Iguaz ArchitectureIguaz FrontendIguaz SchedulerIguaz BackendCassandraServicesServicesIguaz Admin

Iguaz Workers

SQSECS API

DevsUsers

30


Iguaz Workers

SQSQueueECS API

DevsUsers

31


Iguaz Workers

ECS API

DevsUsers

SQSQueue

32


Iguaz Workers

ECS API

DevsUsersZK EnsembleSQSQueue

33


Iguaz Workers

ECS API

DevsUsersZK Ensemble

SQSQueue

Now, I want to talk about an important implementation detail. In particular, why do we put this queue here right in the middle of a nice, clean, normal microservice? We do not need to have a queue for communication between the two halves of Iguazu. It could be a simple function call; when a request comes in, we could have the Iguazu microservice immediately turn around and schedule with the ECS API before responding.

Recall, the big problem with Saturn is that at the top of the hour, dozens of jobs would kick off, and wed exhaust all available resources. But, a nearline system is intentionally not an online system. In an online system, requests must be served immediately. But ia nearline architecture, the framework and scheduler is allowed to delay the execution of the jobs. We leverage a Queue to buffer up the bursty nature of incoming jobs. As a result, a nearline system can be provisioned at less than peak capacity. In fact, a nearline cluster can be provisioned on a gradient between peak capacity and average capacity, allowing a tradeoff between latency and cost.34

Autoscale, autoscale, autoscale!

When moving to a cloud-native architecture, you will be brainwashed into using autoscaling. There is a good reason for that. This is because autoscaling is a really good practice for online, latency-sensitive microservices. Even more important than saving money, Autoscaling enforces immutable infrastructure, and high degrees of automation resulting in a modern, flexible and highly available architecture. Those benefits translate over to nearline environments. We autoscale not just the control plane, but the worker pool as well.

However, autoscaling a cluster with long running jobs is much more challenging than low latency API servers. While scaling up is easy, scaling down safely is harder. You dont want to terminate an EC2 instance thats running a non-idempotent job! To solve this problem, we dont use the default Amazon ECS scheduler. Instead, Iguazu has its own scheduler that is integrated with the Amazon Autoscaling API to avoid scheduling new jobs on instances scheduled for termination.35

Autoscaling Iguaz ECSIguazuECS APIAutoscalingEC2WorkerEC2WorkerShutdownLifecycle NotificationPoll WorkerJob StatusAll finishedProceedTerm-inateEC2Worker

Failure in Nearline SystemsMost jobs are non-idempotent

Iguaz: At most once executionTime-bounded delay

Future: At least once executionWith caveats

Unfortunately, while we can work to avoid premature terminations, the reality is that jobs will fail to complete. The hardware could fail, power could go out, it could try and use too much memory, and there may be bugs. When designing distributed systems, you must architect for failure right from the start.

In our experience, many of these nearline jobs make API calls, and have a large number of side effects (e.g. sending emails). Re-running a failed job could have serious consequences. 37

Iguaz adoption by the numbers~100 jobs in production>1000 runs per day>100 different job schedules

Coursera is a very data-informed company; we always look to numbers to track our progress and validate our successes. Coursera developers have authored over an order of magnitude more jobs than in any of our previous systems. Developers take advantage of scheduled recurring jobs, and many jobs have multiple different schedules associated with them. As a result, were constantly running jobs on our cluster.38

Iguaz ApplicationsNearline Jobs

Pedagogical Instructor Data ExportsSystem IntegrationsCourse MigrationsScheduled Recurring Jobs

Course RemindersSystem IntegrationsPayment reconciliationCourse translationsHousekeepingBuild artifact archivalA/B Experiments

While numbers can tell a very insightful story, I think in this context they are too difficult to interpret appropriately. I find it more illustrative to look at how we use Iguazu to truly understand how ubiquitously applicable nearline architectures can be.

39

While containers may help you on your journey, they are not themselves a destination.CC-by-2.0 https://www.flickr.com/photos/usoceangov/5369581593

When you decide to build a new website, you almost never start with int main(). We always build on top of higher-level frameworks; theres no need to re-write HTTP parsing libraries, cookie libraries, or database connection pools. The same principles apply to containers and nearline jobs. Saying Im using containers to build my app is like saying Im using HTTP to build my app. While its a great foundation, often a higher level of abstractions results in increased developer productivity. So, while containers may be an integral component of your architecture, or even necessary to the solution, they are not sufficient! Good architects should think about even higher levels of abstraction.40

Writing an Iguazu Jobclass AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM

def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } }}

While Iguazu can invoke and run arbitrary containers, in practice almost all jobs use the most important feature of Igauzu: the developer-optimized higher level framework. This is what a toy job looks like. Lets break it down.41









Testing an Iguazu job

46

The Hollywood Principle applies to distributed systems.CC-by-2.0 https://www.flickr.com/photos/raindog808/354080327

The Hollywood principle says, Dont call me, Ill call you. Normally, you hear about it in the context of IoC frameworks, dependency injection, and UI or app toolkits. But it absolutely applies to distributed systems as well. Thinking back to Cascade (the initial PHP framework), if a developer wanted to test their new job, they must create a new queue, reconfigure their local copy of Cascade to talk to their new private queue, insert the job information into the queue, and wait for their job to eventually be run.47

Deploying a new Iguazu JobDevelopermerge into master done

Jenkins Build StepsCompile & package job JARPrepare Docker imagePushes image into registryRegister updated job with Amazon ECS API

At Coursera, we practice a DevOps (or actually NoOps) approach. All developers deploy their own code hundreds of times a week via automated tools and custom webapp tools. 48

Invoking an Iguaz Job// invoking a job with one function call// from another service via REST framework RPC

val invocationId = iguazuJobInvocationClient .create(IguazuJobInvocationRequest( jobName = "exportQuizGrades", parameters = quizParams))

A clean environment increases reliability.CC-by-2.0 https://www.flickr.com/photos/raindog808/354080327

Now, back in 2012, we totally laughed at PHP for it's horribly unreliable runtime full of memory leaks. But in Iguazu, we're actually worse. We don't just throw away the whole process, we throw away the whole file system, and the rest of the container. But, actually, this is a really good idea.

Longer-running, resource intensive jobs tend to leave a disproportionate amount of garbage in their wake. It's common to use temporary files on disk & a variety of other resources, such as temporary files as part of our pedagogical data exports. By allocating a new container instance from the container image, the system ensures a consistent environment and freeing developers from file bookkeeping in the same way a garbage collector frees developers from memory management.

PHP was on to something after all!!!50

Evaluating Programming AssignmentsAn application of Iguaz

Now, I'd like to delve into the flagship application of Iguazu: Evaluating programming assignments.51

Design GoalsElastic InfrastructureNo MaintenanceNear Real-timeSecure Infrastructure

Procrastination is a global phenomenon. We regularly see an order of magnitude increase in submission rates right before assignment deadlines. We needed an elastic service backed by a shared pool of resources to efficiently evaluate programming assignments in a cost effective manner.54


Our online serving environment benefits greatly from immutable infrastructure and high degrees of automation to radically reduce operations and maintenance overhead. We wanted to apply these same lessons to evaluating programming assignments.55


For pedagogical reasons, we would like to provide feedback as quickly as possible. Ideally, we are able to execute fast graders and turn around their scores within 60 seconds at the 90th percentile.56

Solution: GrIDPatrick Hoesly (https://www.flickr.com/photos/zooboing/5665221326/) CC-BY-2.0 Service + framework for gradingprogramming assignments

Builds on Iguaz

Named for Trons digital frontierBackronym: Grading Inside Docker

High-level GrID Architecture

LearnersGrIDIguaz

S3 BucketECS APIs

Grading Machines

VPC Firewalls

Coursera Production AccountCoursera GrID Grading Account


LearnersGrIDIguaz

S3 BucketECS APIs

Grading Machines

VPC Firewalls

Coursera Production AccountCoursera GrID Grading Account


LearnersGrIDIguaz

S3 BucketECS API

Grading Machines

VPC Firewalls

Production AcctGrID Grading Account


LearnersGrIDIguaz

S3 BucketECS API

Grading Machines

VPC Firewalls

Production AcctGrID Grading Account

Thanks to Iguazu, the GrID service itself is only ~1k LoC. 61


Because were operating on a shared pool of resources, we need to bake security into the infrastructure. This also has the added benefit of making the system robust to less byzantine occurrences. But, what does Secure Infrastructure even mean?62

Programming Assignments

The Security Challenge

Compiling and running untrusted, arbitrary code on our cluster in near real time.

Would you like to compile and run C code from randompeople on the Internet on your servers?

By a show of hands, who of you would like to run arbitrary C code from random people on the internet on your servers?

While you may think this insane security challenge only applies to these crazies from Coursera, it turns out that this applies far more broadly.64

FROM redisFROM ubuntu:latestFROM janes-image

Most Dockerfiles start with from ubuntu, or from redis or from jane-doe-on-github. That one little innocent-looking line pulls in effectively arbitrary binaries & code to run on your container infrastructure. What this means is that: in practice, if you have container-based infrastructure at your organization, you should prepare to defend against arbitrary code running within your containers.

65

Security AssumptionsRun arbitrary binaries

Instructor grading scripts may have vulnerabilities Grading code is untrusted

Unknown vulnerabilities in Docker and Linux name-spacing and/or container implementation

66

Security GoalsPrevent submitted code from:impacting the evaluation of other submissions.disrupting the grading environment (e.g., DoS)affecting the rest of the Coursera learning platform

Grading assignment submissionsCC-by-2.0 https://www.flickr.com/photos/dherholz/4367511580/

Now, containers are very new, and security is sometimes very impenetrable. So, lets instead talk about something thats old, and much more straight forward. Babies. The first picture I have of a gaggle of small children is something along the lines of this picture. Each one warmly swaddled in their own tub, happy as can be. When I initially thought of grading programming assignments, I had a similar image. Each submission happly running along within their own container. Reality will quickly disabuse of these foolish notions.

https://www.google.com/search?espv=2&biw=2560&bih=1468&tbm=isch&sa=1&q=babies+hospital+&oq=babies+hospital+&gs_l=img.3..0j0i30j0i5i30l3j0i8i30l5.4194.4194.0.4783.1.1.0.0.0.0.74.74.1.1.0....0...1c.1.64.img..0.1.74.mKcYVszmBgo#imgrc=BRbfAc8Wi9uf2M%3A

68

CPUCPUCPUCPURAMAlices ContainerAlices SubmissionGraderBobs ContainerBobs SubmissionGraderMallorys ContainerMallorys SubmissionGraderKernelDisk

CPUCPUCPUCPURAMAlices ContainerAlices SubmissionGraderBobs ContainerBobs SubmissionGraderMallorys ContainerMallorys SubmissionGraderKernelDisk

CPUcgroupsCPUcgroupsRAM cgroupsAlices ContainerAlices SubmissionGraderBobs ContainerBobs SubmissionGraderMallorys ContainerMallorys SubmissionGraderKernelDisk

CPUcgroupsCPUcgroupsRAM cgroupsAlices ContainerAlices SubmissionGraderBobs ContainerBobs SubmissionGraderMallorys ContainerMallorys SubmissionGraderKernelDisk

CPUcgroupsCPUcgroupsRAM cgroupsAlices ContainerAlices SubmissionGraderBobs ContainerBobs SubmissionGraderMallorys ContainerMallorys SubmissionGraderKernelDisk blkio limits & btrfs quotas

CPUcgroupsCPUcgroupsRAM cgroupsAlices ContainerAlices SubmissionGraderBobs ContainerBobs SubmissionGraderMallorys ContainerMallorys SubmissionGraderKernelDisk blkio limits & btrfs quotas

Attacks: Kernel Resource ExhaustionOpen file limits per container (nofile)

nproc Process limits

Limit kernel memory per cgroup

Limit execution time

CPUcgroupsCPUcgroupsRAM cgroupsAlices ContainerAlices SubmissionGraderBobs ContainerBobs SubmissionGraderMallorys ContainerMallorys SubmissionGraderKernel cgroups, ulimitsDisk blkio limits & btrfs quotasNetwork

Attacks: Network attacksAttacks:Bitcoin miningDoS attacks on other systemsAccess Amazon S3 and other AWS APIs

Defense:Deny network access

Docker Network Modes

NetworkDisabled too restrictiveSome graders require local loopbackFeature also deprecated

--net=none + deny net_admin + audit networkIsolation via Docker creating an independent network stack for each container

github.com/coursera/amazon-ecs-agent

CC-by-2.0 https://www.flickr.com/photos/valentinap/253659858

Once we have all of these systems configured, graders can run happily within the containers.

Now, some of you functional programmers may have picked up on something: grading is an idempotent operation. But as it turns out, with GrID, its even better. Because we have hermetically sealed the grading containers, we have transformed messy business of evaluating programming assignments into effectively a pure function in the functional programming sense. It has almost zero extra input from the outside world! Containers are really cool!81

CC-by-2.0 https://www.flickr.com/photos/jessicafm/2834658255/

CC-by-2.0 https://www.flickr.com/photos/donnieray/11501178306/in/photostream/

https://www.flickr.com/photos/donnieray/11501178306/in/photostream/83

Defense in DepthMandatory Access Control (App Armor)Allows auditing or denying access to a variety of subsystems

Drop capabilities from bounding setNo need for NET_BIND_SERVICE, CAP_FOWNER, MKNOD

Deny root within container

If you ignore all the name-spacing and container mumbo-jumbo, at the core processes running within containers are just linux processes, and so the standard security techniques apply.84

Deny Root Escalations

We modify instructor grader images before allowing them to be runClears setuidInserts C wrapper to drop privileges from root and redirect stdin/stdout/stderr

Run cleaning job on another Iguaz clusterRun Docker in Docker!

Docker 1.10 adds User Namespaces

85

If all else fails

Utilizes VPC security measures to further restrict network accessNo public internet accessSecurity group to restrict inbound/outbound accessNetwork flow logs for auditingSeparate AWS accountRun in an Auto Scaling groupRegularly terminate all grading EC2 instances

Now, there are a number of unknown vulnerabilities not included in this defense.86

Other Security Measures

Utilize AWS CloudTrail for audit logs

Third-party security monitoring (Threat Stack)No one should log in, so any TTY is an alert

Penetration testing by third-party red team (Synack)

Baby monitor graphics?87

Lessons Learned - GrID

Building a platform for code execution is hard!Carefully monitor disk usageRun the latest kernelsLatest security patchesbtrfs wedging on older kernelsDefault Ubuntu 14.04 kernel not new enough!

88

Reliable deploytooling pays for itself.

Public Domain: https://www.flickr.com/photos/mustangjoe/20437315996/in/photolist-x8YA2b-4CHj67-8Cjveb-bC2UPc-ibCEkV-aswFR8-gmv5Vj-4r5sPk-4CHiyy-92qQGf-28i54x-5LfUcS-opNLAM-7QTwNd-d7HmTA-efZc4Y-brT6Uv-d7Hnfd-5sARbG-5vvzmv-aqn5Li-DTWCYi-7XMsUo-8m1fUK-uj58iZ-D2nADa-78SpzZ-6BJGaL-4BrcEY-ne6BDJ-9FhXQ6-9QALSm-4EP8Hb-6h14wn-5nTnpt-7groVi-4EP8VW-8Qv9zx-6bCq1k-a7E8EJ-adFoNW-5Rp7Pb-s8otHi-7xSqsJ-4JZiUA-qW6wFZ-7XJdzg-jiYBq5-9hJ5Vo-ySx3Uo

89

Thank you!

Brennan Saetagithub/saeta@[email protected] Chengithub/frankchn@[email protected]

GrID leadIguaz Lead

Questions?

Brennan Saetagithub/saeta@[email protected] Chengithub/frankchn@[email protected]

GrID leadIguaz Lead

docker & ecs: secure nearline execution

Software