RPaaS DevOps:Lessons from using Cloudfoundry in Production

Oct/26/2013

Waldemar QuevedoPaaS DevOps team. Rakuten, Inc.

2

About me

• Waldemar Quevedo

• From Mexico

• Joined Rakuten in 2010 as an App developer

• Joined the RPaaS team in 2012 in its earlier stages

• Github: https://github.com/wallyqs

3

WARNING

• No funny pictures in these slides! Sorry!

4

What is RPaaS?

• Internal PaaS at use in Rakuten

• Built on Cloudfoundry

• Started to service production applications since mid 2012

• Derek Collison, its creator is a technical fellow of our team

5

Example site: tech.rakuten.co.jp

6

7

Why have an application Platform as a Service?

• Natural evolution of how to develop, release and maintain applications.

• Improve productivity by automating the provisioning of resources on demand.

• Scale horizontally (more instances)• Scale vertically (more cpus, memory, etc..)• Guarantees high availability

8

Example use case: Scaling

How to go from this,

aaa.bbb.ccc.ddd:80 tech.rakuten.co.jp

to this,

aaa.bbb.ccc.ddd:9393 tech.rakuten.co.jpaaa.bbb.ccc.ddd:9395 tech.rakuten.co.jpaaa.bbb.ccc.ddd:9291 tech.rakuten.co.jpaaa.bbb.ccc.ddd:8888 tech.rakuten.co.jp

• …in a matter of seconds?

9

Increasing instances

10

Increasing instances

11

Benefits from an application PaaS

• Improves agility of the application team

• Application team can focus on the application not maintaining servers.

• Simplified operations

12

Simplified operations

13

Benefits from an application PaaS

• Enforces automation and better practices for developers.

• Reduces the amount of workflows required to get things done.

• Integration with internal environment.

• And many more…

14

Topic of the day: How to adopt an application PaaS

• An application PaaS is all goodness for developers, since they can stop caring about servers.

• Instead, they can rely on the RPaaS team!

15

Usual DevOps metaphor: Vampires vs. Werewolves

• Vampires (吸血鬼 )app developersprovide value

• Werewolves ( オオカミ人間 )sys admins protect value

16

Vampires vs. Werewolves

• A PaaS admin is somewhere in the middle.

• We take care of the servers on behalf of users but in the end we also care about:• How the user is actually using the runtimes.• Demand from application runtimes available.• Support the users when they run into problems

with one of the runtimes.

17

Different metaphor with PaaS: Villagers and Samurai

• Villagers ( 村人 )Application team can survive without PaaS but has lots of issues (-agility, -flexibility, -availability)

• Samurai ( 侍 )Solves the issues of the application team by providing a better environment.

18

In order to deliver to users…

• RPaaS team approach: • “you should know everything”

1. Know your stack (Cloudfoundry, Nginx, Services)

2. Know your environment (infrastructure, the network…)

3. Know your users (runtimes: Ruby, Java, support)

4. Know your capacity (metrics & trends)

19

Know your stack

• We needed to learn the internals of Cloudfoundry (OpenSource software FTW)

• CloudController• DEA• HealthManager• Router• NATS

20

Cloudfoundry

21

Know your stack

• Specially need to be prepared for failure scenarios:

• How long does it take for the HM to execute the application failover when a DEA goes down?

• What happens when there is a NATS failure?

• How long does it take for the Router to stop balancing an application when the app goes down?

22

Know your environment

• Remember the ‘Fallacies of distributed computing’http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing

• Need to be familiar with your infrastructure, the network etc…• Don’t be afraid of tcpdump

• Share your findings about the environment with the users• Each datacenter is a different beast

23

Know your environment

• Again, need to be prepared for trouble scenarios:• Storage will fail

• Physical hosts will fail

• That VM that you expected to never to restart, will restart (hint: NATS)

24

Know your users

• Insight: When introducing a PaaS into your company, you need an extra effort during the adoption phase.• ‘Do things that Don’t Scale’:

http://paulgraham.com/ds.html

25

Know your users

• At the time we started Cloudfoundry was relatively new technology• Roadmap mismatch: Private PaaS vs Public

PaaS• Must features for our users were different:

• Logging, monitoring, billing, team based deployment

• Need highly available services• These were blockers we need to comply with

so that teams would start adoption of the PaaS

26

Know your capacity

• re: ‘Auto-scale’: • You usually don’t want to do it• You do need to automate provisioning of

resources though

• Monitor the trends and provision according to them:• DEAs (app resources)• Disks• Services (cache, dbs)• Traffic (increase max qps limit)

27

Managing user expectations

• “Under promise and over deliver”• Need to make it very clear for users what is

possible and what is just not feasible according to the current limitations of the system.

• The concept of the cloud is cause of confusion sometimes• “it should be elastic”• “it should auto scale”• “it should self heal”

28

Managing user expectations

• “The RPaaS team will fix it for us”• It helps becoming familiar with popular libraries

for different runtimes to give good advice to the users on how to solve their issues in the environment you provide.

• Learn to say no sometimes

• Ideally: Make PaaS users collaborators• Provide a straightforward way for users to

have a single node setup of the PaaS.

29

Wrapping up

• The PaaS field continues to evolve

• The application PaaS we run is still first,second generation technology

• We already sort of see traces of the next evolution

30

Wrapping up

• Keep up with the trend• When we started it seemed Ruby was the

language from the cloudhttps://twitter.com/mart_brooks/status/273115028008341506

31

Wrapping up

• Nowadays, it seems that Go is looking out to displace it little by little.• Biggest gains seem to be performance, and it

being so practical to deploy.• Also, docker.

https://twitter.com/derekcollison/status/245522124666716160 http://www.slideshare.net/derekcollison/go-conference-japan

32

Thanks!

• Questions? Comments?