mesos gets pluggable - introducing mesos modules

Kapil Arya & Niklas Nielsen

Mesos Gets Pluggable Introducing Mesos Modules

© 2015 Mesosphere, Inc. 2

Niklas Nielsenniklas@mesosphere.io

Kapil Aryakapil@mesosphere.io

© 2015 Mesosphere, Inc. 3

Mesos Modules & HooksModules & HooksArwwwww

© 2015 Mesosphere, Inc. 4

How and why modules was introduced in Mesos

Our humble thoughts on how modules and extensibility in Mesos can evolve in the future

How Mesos Modules work and give you concrete examples of modules in action

© 2015 Mesosphere, Inc.

Modules

5

© 2015 Mesosphere, Inc.

Different organizationsDifferent needs

6

● Hardly anyone run clusters the same way○ Different scales○ Different hardware○ Different workloads○ Different external tooling○ Different security needs

One cluster with turbo chargers please

© 2015 Mesosphere, Inc.

• Mesos was built with this in mind!• The subsystems are lightweight insight

and control over HTTP

• Excellent for tooling around

• Different subsystems can be enabled and configured in a modular way

• Most notable: Isolation mechanisms

Good news!

7

© 2015 Mesosphere, Inc.

New “extensions” to subsystems like isolators had to be upstreamed

But…

● Mesos can be made even more customizable and extendable

● Not all organizations can share their work

● Support proprietary and experimental integrations

● Not create bespoke forks of Mesos

However...

8

© 2015 Mesosphere, Inc.

• Tie into and control task launch

• Dynamically setup execution environments

• Pass signatures through Mesos

• All of this, transparently to the framework and user

We needed it to support bespoke security subsystems

9

© 2015 Mesosphere, Inc.

• Be able to extend and replace any component in Mesos• Allocator algorithms

• Authentication mechanisms

• Advanced scheduling features like oversubscription

• Anything!

The general thought of Modules was bigger

10

Imagine ifI could write my own?

© 2015 Mesosphere, Inc.

Modules are old news

Many large software systems support libraries to

• Extend behavior• Isolate and abstract complexity• Make this a configuration rather than a

build exercise

For example

• Browsers (Firefox)• Server software (Apache Webserver)• Linux kernel

11

Wish I had modules already

© 2015 Mesosphere, Inc.

What is a module anyway?

Module, plugin, extension, library …

Adds or replace a full component

For example:

• An isolator (works together with existing ones) in the agent

• The allocator and authenticators in the master

12

© 2015 Mesosphere, Inc.

And how about hooks?

More often than not, you don’t want to replace a full component

Just want to tie into events and their context

For example:

• Launch task requests at the master

• Launch task requests at the agent

• Exit and cleanup events

13

Psst - I just launched a task

© 2015 Mesosphere, Inc.

And who is using it?

Powering new exciting features and integrations!

• Oversubscription modules• Static (fixed) estimator

• Dynamic estimator and QoS Controller, project Serenity

• Networking integration with project Calico

14

© 2015 Mesosphere, Inc. 15

Module Mechanics

© 2015 Mesosphere, Inc.

A demo!

16

● A hook module that tags TaskStatus messages

© 2015 Mesosphere, Inc.

Components

17

Isolator InterfaceIsolator Module

Hook Module H1

Hook Module H2

Hook Interface

Mesos Master/Agent

Module library

ModuleManager

Module spec JSON

Initialization

Initialize subsystems

use module objects

Module libraryinitialize modules

get module object

readspec

call hooks

© 2015 Mesosphere, Inc.

Initialization

18

● First phase: ○ load module libraries○ compatibility checks, etc.○ libprocess not available

● Second phase○ initialize a specific module○ module-specific parameters○ libprocess available

© 2015 Mesosphere, Inc.

class TestHook : public Hook{public: Result<Labels> slaveTaskStatusLabelDecorator( const FrameworkID& frameworkId, const TaskStatus& status) { Labels labels; if (status.state() == TASK_RUNNING) { Label* newLabel = labels.add_labels(); newLabel->set_key("whereami"); newLabel->set_value("mesoscon"); } return labels; }};

A Hook Module

19

// Create and return an object or TestHook type. static Hook* createHook(const Parameters& parameters){ // Any initialization checks go here.

return new TestHook();}

// Declares a Hook module named ‘org_apache_mesos_TestHook'mesos::modules::Module<Hook> org_apache_mesos_TestHook( MESOS_MODULE_API_VERSION, MESOS_VERSION, "Apache Mesos", "modules@mesos.apache.org", "Test Hook module.", NULL, createHook);

© 2015 Mesosphere, Inc.

{ "libraries": [ { "file": "/path/to/libmodule.so", "modules": [

{ "name": "org_apache_mesos_TestHookModule", "parameters": [ { "key": "agent_addr", "value": "agent.host.domain" }, { "key": "...", "value": "..." } ] } ] } ]}

Specifying Modules to Master/Agent

20

© 2015 Mesosphere, Inc.

● Build without building Mesos○ Just have Mesos installed

● Modules compile into a shared libraries○ Multiple modules per library

● Specify modules on command line:mesos-agent.sh <master-parameters> --modules=file:///path/to/modules.json --isolation=”my_isolator” --hooks=”my_hook”

Using Modules

21

© 2015 Mesosphere, Inc.

● Add/replace a full component● Implement the interface● Asynchronous (actor model)

● Existing modularized interfaces:○ Allocator○ Authentication○ Authorizer○ Isolator○ QoSController○ ResourceEstimator

Replacement Modules

22

© 2015 Mesosphere, Inc.

● Listen/Intercept interesting calls● Occasionally modify the behavior

○ Trigger initialization/cleanup● Allows us to “tag” certain tasks, statuses, etc.

● Two broad categories○ Task launch sequence○ Status updates

Hook Modules

23

© 2015 Mesosphere, Inc.

● Co-exists with the parent process ○ separate thread of execution

● Create Master/Agent http “listen” endpoints● No callbacks

Anonymous Modules

24

One module to rule them all!

© 2015 Mesosphere, Inc.

● Do not block○ Hooks are synchronous○ Use libprocess/pthreads

● Exit semantics○ Avoid assertions

Writing Modules

25

© 2015 Mesosphere, Inc.

● Logs○ stdout/stderr

● Run debug module with non-debug Master/Agent○ gdb

Debugging

26

What crashed the Master?

© 2015 Mesosphere, Inc.

● Dependency on other modules● Compatibility within set of modules● Upgrade path

○ rebuild modules when updating Mesos

Dependency and Compatibility

27

© 2015 Mesosphere, Inc. 28

Future Work

© 2015 Mesosphere, Inc.

● Safeguard against unsafe modules○ Limit data exposure○ Execute modules in a separate process

● Module certification● ACL’s● Runtime functionality checks

○ whitelist services○ can it add routes or not

Better Safety and Security

29

© 2015 Mesosphere, Inc.

● More module interfaces● Load/Unload a module without rebooting Master/Agent● Upgrade path● Express dependability on other modules● Inter-module communication● Non-C++ modules

Future Work

30

Thanks for listening!kapil@mesosphere.io niklas@mesosphere.io

❏ Documentation: http://mesos.apache.org/documentation/latest/modules/

❏ Modules repo: https://github.com/mesos/modules

❏ Mailing list: modules@mesos.apache.org