provisioning oracle fusion middleware environments with chef and puppet
DESCRIPTION
Provisioning Oracle Fusion Middleware Environments with Chef and Puppet This session presents case studies and experiences involving automated provisioning of Oracle Fusion Middleware environments with the popular DevOps tools Chef and Puppet. In addition, it discusses experiences in orchestrating multinode environments with these tools, together with others such as MCollective and some custom-built tooling. The presentation also covers issues such as installing, creating domains, patching, configuring resources such as JDBC, and deploying applications. It also spends a little time on how this provisioning can contribute to building an environment for cloud-based automated acceptance testing.TRANSCRIPT
Provisioning Oracle FMWEnvironments with Chef & PuppetCON7629
Mark Nelson, Edwin BiemondFusion Middleware Platform TeamOctober 02, 2014
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 3
• Mark Nelson– Architect, Oracle
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 4
• Edwin Biemond– Consulting Member Technical Staff, Oracle
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 5
Agenda
Current State
Experiences
Continuous Delivery
Testing
Wrap Up, Q & A
1
2
3
4
5
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 6
Agenda
Current State
Experiences
Continuous Delivery
Testing
Wrap Up, Q & A
1
2
3
4
5
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
What is possible Now!!!
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 8
Focus on Puppet & Chef, which are the most popular provisioning tools
• > 10.000.000 nodes• Fastest growing companies in the US• Stand-alone (Client) & Master (Server)• Open Source & Enterprise / Premium– Free < 10 Nodes (Puppet)– Free < 5 Nodes (Chef)
• Manage Linux, Solaris & Windows• Cloud plugins, VMware, Google, Openstack,
Azure & Amazon
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 9
Chef components overview
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 10
Next major release• SOA architecture• Services• Easy to scale up• Own release lifecycle
• Runs in a JVM
• 3 times faster
• Build with clojure, same as PuppetDB
• JRuby
Puppet component overview
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 11
• Administrator / Enterprise focus• Forge.puppetlabs.com > 2700
Modules• Language, Puppet DSL & Ruby• Manifest-> Some Puppet DSL Code • Puppet Modules• Facter• Type/Provider -> Ruby
• Developer focus• Supermarket.getchef.com> 1600
Cookbooks• Language, Chef DSL & Ruby• Recipe -> Some Chef DSL Code• Chef Cookbooks• Ohai• Resource/Provider– Light -> Chef DSL– Heavy -> Ruby
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 12
file{'/etc/resolv.conf':
content => template('mod/my_resolv.conf.erb'),
owner => 'root',
group => 'root',
mode => '0644',
}
$files = ['bacon', 'eggs', 'sausage']
file{$files:
content => "#{title} is delicious!",
}
package{'bash':
ensure => latest,
}
template '/etc/resolv.conf' do
source 'my_resolv.conf.erb'
owner 'root'
group 'root'
mode '0644'
end
['bacon', 'eggs', 'sausage'].each do |type|
file "/tmp/#{type}" do
content "#{type} is delicious!"
end
end
package 'bash' do
action :upgrade
end
Some DSL Code
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 13
So how does it work!!• We have to create some recipes/manifests which describes the To Be
situation & not how to achieve it • Execute chef-apply or puppet apply• Chef/Puppet retrieves the current state and determines what the necessary
actions should be• Executes the actions (apply only deltas)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 14
Agenda
Current State
Experiences
Continuous Delivery
Testing
Wrap Up, Q & A
1
2
3
4
5
15Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
ExperiencesChef & Puppet challenges & solutions
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 16
WebLogic 12c Changes• All jars, no O.S. specific installer– Jars can be used on every Operating System
• Shared oracle_common which is part of WebLogic Infrastructure– ServiceBus installer +/- 400Mb
• Repository Creation Utility is now part of WebLogic Infrastructure– FMW Database repository tightly connected to the domain (because of OPSS)
• WebLogic Restful Management Services
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 17
Oracle Tools like OPatch, RCU or BSU are not idempotent• 2nd Times always fails– It breaks Chef or Puppet– Hard to handle this with the standard Puppet/Chef exec resource– Slow, inventory command is always faster then apply and let it crash
• For example OPatch utility– Detect current state with opatch lsinventory• Every utility has a different output ( XML, CSV or text ouput )
– Determine the right action– Skip, Install or Remove the patch with opatch apply– Check the outcome
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 18
One Time• Not really the Chef or Puppet way• One big or many WLST scripts• Use exec of Chef or Puppet, almost
impossible to control (Facter/Ohai)• Execute this once or need to catch
the BeanAlreadyExists exception• Re-use your old scripts• Easy to get out of sync
Model driven• WebLogic Chef/Puppet Resources• The Model is always up-to-date• Need to know some Ruby• Map WLST or the WLS Rest
Interface (12.1.3) to Chef or Puppet Attributes• 4 WLST scripts per WLS resource,
retrieve, create/update and destroy• Big effort to have all WLS resources
One time provisioning or Model driven
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 19
• BSU, OPatch, AdminServer & ManagedServer Control• Machine, Domain, Server, Channels• (Dynamic) Cluster, Coherence,
Server Template• User, Group, Authentication
providers• Deployment
• Datasource• File Persistence• JMS server, JMS module,
Connection Factory, Queue, Topic, SubDeployment, Quota, Foreign Server• SAF agents, Imported Destination,
Remote Context, Error handler• Workmanagers• Virtualhost
Puppet WebLogic Resources ( Total of 37)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 20
Puppet apply Puppet resource wls_server SoaServer1 --edit
Model driven resource demo
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 21
WLST Performance
• Startup of WLST takes times-> even with skipWLSModuleScanning
• Connect() or readDomain(‘xx’) is slow
• With > 10 WLST scripts or with multiple Domains you will have some bad performance
• Don’t schedule a Chef or Puppet run every XX minutes
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 22
WLST Domain Daemon process
• For Every domain
• Is already connected to the Domain
• Auto start & destroy in every Puppet/Chef run
• More complexity -> need to know when it is finished or when it fails
• Can’t handle invalid WLST/Python scripts
• It’s Fast -> 2 Domains with 8 clusters and > 100 Queues in just a few minutes ( normal 30-60 minutes)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 23
Naming of WebLogic artifacts• In Chef or Puppet a resource should be unique. There can be only one
Firewall Service or one Package Bash• In WebLogic you can have multiple Domains and a Queue only has to
unique inside a JMS Module.• Need to have a WebLogic Title convention and implement Title pattern in
all the Chef/Puppet Resources
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 24
Orchestration of FMW HA Environment• A High Available FMW environments needs at least– A Database configuration with Oracle RAC or Oracle Dataguard ( 10 a 30 min ) –WebLogic AdminServer node which depends on the Database ( 5 a 25 min )– 2 or more WebLogic nodes which depends on the AdminServer ( 5 a 10 min )
• Chef or Puppet can’t orchestrate multi-node environments• Can implement some try / catch together with a repeat & wait – AdminServer should wait for a Database with a FMW Repository–WebLogic Nodes should wait for a Domain pack on the AdminServer
• We need something on top of Chef/Puppet which can handle this
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 25
Puppet & Chef Sandbox environments
Oracle VirtualBox
Vagrant
Desktop
Shared Folder Puppet / Chef code
VM
VMTemplate
Puppet/ChefClient
Oracle
JDK
WebLogic
Database
GithubForge
Supermarket
yum / wget
• Out of the box with Chef Test Kitchen• Puppet -> requires some Vagrant
configuration
• Easy to simulate environments• Destroy & Re-Create• Direct feedback, Change Log levels• Remote shell access• Only need some MB of configuration data• Retrieve the latest Modules or Cookbooks
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 26
Oracle Big Files
• Download & Extract zip files requires a lot of memory or a big swap file
• Takes a lot of time to download plus extract & requires disk space
• It is slow in every run because Puppet/Chef file resource checks for differences (locally and remote)
• Use a Shared Drive
• Skip the File resource step with some Puppet/Chef Facts
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 27
Agenda
Current State
Experiences
Continuous Delivery
Testing
Wrap Up, Q & A
1
2
3
4
5
28Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Continuous Delivery
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 29
Separate environment specific variables from your code• Puppet – Facter– Parameter manifest class– Hiera ( Multiple levels Node, Env, Common ) with a YAML, Database, JSON Backend
• CHEF–OHAI– Roles– Environments– Databags– Attribute
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 30
Promotion of infrastructure• Versioning (Git Suite -> Gitbucket, Gitlab or Github) for infrastructure
configuration & Development code– Issues– Pull Requests
• Multi environment on the master (Development, Test, Staging, Prod)• Retrieve Modules/Cookbooks from Git or use Forge or Supermarket– Puppet -> Puppet Librarian, R10k– Chef -> Berkshelf, Chef Librarian
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 31
Model your environments• Define Building blocks or Roles– Better to understand• just assign to nodes
– Easier to maintain• Consist of small modules• Optimal re-use
– Auto-adapt to the environment• Developer desktop• High Available production environment
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 32
Standardize your VM Base ImagePacker.io is a tool for creating identical machine images for multiple platforms from a single source configuration–Works with PXE boot– Download ISO/Net install– Installs Chef or Puppet client– Different post-processors• Vagrant for VMware or Oracle Virtualbox• Amazon• Docker• OpenStack• Google
–Only have to provide a Kickstart file
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 33
Continuous Environment Overview – Modules/Cookbooks
Git• Clone
Code• RVM (Parallel)• Chef version
• Lint, RSpec
Smoke Test• Vagrant• Librarian
Approve• Archive• Version
Distribute• Repository
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 34
Continuous Environment Overview – Building blocks
Git• Clone
Smoke Test• Vagrant• Librarian
Approve• Archive• Version
Distribute• Repository
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 35
Continuous Environment Overview – Test 2 Production
Test
VMs
Cookbooks
RecipesProductionStaging
Test Environment 1
Test Environment 2
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 36
Agenda
Current State
Experiences
Continuous Delivery
Testing
Wrap Up, Q & A
1
2
3
4
5
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 37
Test/Code Quality frameworks• Rubocup for Ruby coding style
• Chef– Foodcritic for Chef coding style– ChefSpec/Serverspec
• Puppet– Puppet-Lint for Puppet coding style– Rspec
• RVM/Bundler to test the different Ruby & Puppet/Chef combinations• Sublime or IntelliJ plugins
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 38
Test against different versionof• Ruby• Chef/Puppet
Test for
• Lint• RSpec Tests• Code Coverage
Opensource
Test in the cloud with Travis CI
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 39
Robot Framework• Test your FMW environment if it contains what you expect.– Can’t fully rely on Puppet or Chef output
• With Robot you can add tests in readable and understandable text• Libraries– Selenium for WebLogic Web applications tests– HTTP/Rest Requests for WebLogic Restful Management Services
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 40
Test WebLogic applications with Robot
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 41
Agenda
Current State
Experiences
Continuous Delivery
Testing
Wrap Up, Q & A
1
2
3
4
5
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 42
• JDK7– URandom fix, JCE Security policy
• ORAWLS– Install, Patch, RCU, Domain, Cluster, WLS Native Types
• ORADB, ORA_RAC– Install, ASM, RAC, Patch, Instance, RCU 11g
• ORACLE– DB Native Types like Tablespace, Role, User & Init params
• > 10k Downloads
• Many Contributions/Pull requests
• Many Big Enterprise companies are using it
• Developers with Vagrant/Virtualbox
Download locations
• Forge.puppetlabs.com
• Github.com/biemond
• Github.com/hajee
Opensource Puppet Modules
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
ConclusionFinally we have total control over FMW Environments without any human interaction.
And experience the provisioning of new FMW environments in minutes and push changes to DTAP
44Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Q & [email protected]@oracle.com
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 45