with ansible and git umut bozkurt, 20.11...standardizing oracle environment with ansible and git...
TRANSCRIPT
−Problem
− Increasing number of databases
− Increasing security requirements
− Consistent number of DBAs
−Solution
− Increasing number of DBAs
− Standardization
− Automation
30
40
50
60
70
80
90Oracle Database Size (TB)
About me…
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank4
−Umut Bozkurt
− Database Engineer
− Swiss National Bank
−More than 18 years of Oracle Database experience
− from database development to Oracle hardware
Oracle Infrastructure
−Current platform: IBM AIX, LPAR
−Next platform: Red Hat, VMware ESX
−Oracle technologies
− Single instance
− Multitenant
− Data Guard
− Automatic Storage Management
− Oracle Enterprise Manager
− Automation with shell scripts
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank6
What is Ansible?
− Ansible is an automation language and engine
− Ansible is agentless and uses OpenSSH or WinRM
− Ansible terminology:
− Playbooks files contains plays, in YAML
− YAML is a data serialization standard that can be used in
conjunction with all programming languages
− Plays consists of tasks and ensures that managed hosts
are in a particular state
− Tasks calls modules
− Tasks runs sequentially
− Playbooks are executed from the control node
− Managed hosts are listed in the inventory
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank7
- name: httpd server installation
hosts: all
become: yes
become_user: root
tasks:
- name: httpd server is installed
yum:
name: httpd
state: present
- name: httpd is started and enabled
service:
name: httpd
state: started
enabled: true
What is Ansible Tower?
− Ansible Tower is a web console and REST API
− Ansible Tower is a commercial offering from Red Hat
− Ansible AWX is the upstream project of the Ansible Tower
− Ansible Tower features:
− Role based access control
− Job scheduling
− Workflows
− Credential management
− Logging and auditing
− Real time and historical job status reporting
− Notifications
− REST API
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank8
Ansible Tower Organization
−Credentials
− Git repository credentials
− Server credentials
− Projects
− Git connection
− Git credential
− Git branch
− Inventories
− Managed hosts
− Groups
− Variables
Templates
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank9
Security Concept - Personal Users
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank10
Authentication
AD account 1 +
Smart Card certificate
Authentication
AD account 2 +
Smart Card certificate
Authentication
Unix account + RSA SecurID
Authorization
Database Server privileges
Change Management
># Database Server># ---------------> logged in as umut> sudo su - oracle
># Database Server># ---------------> logged in as umut> sudo su -
># Database Server># ---------------> logged in as umut> sudo su -
># Jump Host># ---------------
> ssh umut@dbsrv
Authentication
AD account 3
−Running Ad-Hoc commands only with the personal accounts
Security Concept - Jobs
− Scheduling jobs only with a technical user and a privileged
credential
− Scheduler user
− Only execute privileges on the assigned template
−Credential
− Encrypted and stored in the Ansible Tower database
− Can only be used from the Tower GUI or API
− Exclusive for the scheduler user
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank11
# Database Server# ---------------> cat authorized_users...from "10.24.23.11" ssh-rsa AKCB3NzaC1yc...
# Database Server# ---------------> cat sshd_config... AllowUsers [email protected]...
# Database Server# ---------------> sudo su -
Security Concept - Operations
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank12
− Execute privileges on a template allows running the template. Viewing or modifications are
not possible
− Surveys allows operations team or end users to run the templates with input parameters
Standardization
− IT standardization is a strategy for minimizing the IT
costs by keeping the hardware and software as
consistent as possible
−Our goal is to consistently push and (if changed)
overwrite both our scripts and configuration files to the
database servers
− This will lead us to
− easy automation
− increase security
− reduce human errors
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank14
># Development Server># ------------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
Standardization with Ansible and Git
−Git
− Playbooks, inventory scripts
− Files to push to the servers
− LDAP
− Server and lifecycle information
−CMDB
− Oracle instance information
−Oracle Enterprise Manager
− Monitoring
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank15
># Database Server># ----------------
># Database Server># ----------------
># Database Server># ----------------
LDAP CMDB
># Ansible Tower ># Rest API># ---------------
curl -X GET …
Git Projects
Files to push to the servers
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank16
oracle_distribution
oracle_automation
Scripts for automation, like playbooks,
dynamic inventory scripts,..
Protected branches cannot be deleted and cannot be force pushed
Inventory
−Managed hosts are defined in the inventory
−Hosts are organized by groups
− The inventory can be defined either
− in a static text file
− dynamically by scripts, from the external sources
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank17
# My Static Inventory# -------------------
[PROD]
dbserver01
dbserver02
dbserver03
[TEST]
dbserver05
dbserver06
[AVALOQ]
dbserver01
dbserver05
[SAP]
dbserver03
dbserver06
Instance Information in the Inventory
− In most cases, extending the inventory with the instance
information is helpful
− Instance information can be gathered either
− directly from the managed hosts (custom facts)
− from the external inventories
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank18
# My Static Inventory# -------------------
[PROD]
dbserver01 instance=AAAL HOME=/u01/...
dbserver02 instance=DB01 HOME=/u01/...
dbserver03 instance=PRQ HOME=/PRQ/...
[TEST]
dbserver05 instance=AAAQ HOME=/u01/...
dbserver06 instance=SRQ HOME=/SRQ/...
[AVALOQ]
dbserver01 instance=AAAL HOME=/u01/...
dbserver05 instance=AAAQ HOME=/u01/...
[SAP]
dbserver03 instance=PRQ HOME=/PRQ/...
dbserver06 instance=SRQ HOME=/SRQ/...
Ansible Facts
− Facts are
− variables that are automatically discovered on the
managed host
− collected before executing the first task
− Fact information can be accessed later in the tasks
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank19
PLAY [Install Oracle PDB] *************
TASK [Gathering Facts]***************
ok: [birs1z.snb.ch]
TASK [Create PDB] *******************
changed: [birs1z.snb.ch]
PLAY [Install Oracle PDB] *************
TASK [Gathering Facts]***************
ok: [server1.ubozkurt.com]
TASK [Create PDB] *******************
changed: [server1.ubozkurt.com]
Custom Facts
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank20
−Custom facts
− allows us to gather additional information (like instance information)
from the managed hosts
− are either static files or scripts stored in the managed host
−Custom fact scripts must
− be executable
− have the extension ".fact”
− output in JSON
− be placed in /etc/ansible/facts.d directory on the managed host
>./local_script.fact
{"instances": [
{"name": "DBMS01CD","oracle_home": "…","version": "19c"
},{
"name": "DBMS02CD","oracle_home": "…","version": "18c"
}]
}
Dynamic Inventory
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank21
− Inventory scripts
− output in JSON
− mandatory arguments:
− --list
− --host <hostname>
− runs on the Ansible controller
− does not accept input parameters. But
you can work with environment
variables
> ./snb_invent.py --list
{“production": {
"hosts": ["prdserver1.snb.ch","prdserver2.snb.ch"
],"vars": {}
},“development": {
"hosts": ["devserver3.snb.ch"
]"vars": {}
}}
> ./snb_invent.py \--host prdserver2.snb.ch
{"instances": [
{"name": "DBMS01CD","oracle_home": "…“,"version": "19c"
},{
"name": "DBMS02CD","oracle_home": "…“,"version": "18c"
}]
}
Clone Git Repository to the Ansible Tower
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank22
># Ansible Tower>------------------
> ls -la tower_fs/oracle_distribution
.gitu01_app_oraclehome_oracle
- name: Clone Git project oracle_distribution
hosts: localhost
connection: local
gather_facts: false
tasks:
- name: Clone dev branch to Ansible Tower file system
git:
repo: 'http://ansitower:vk6z@gitlab/dbms/oracle_distribution.git'
dest: /tower_fs/
version: dev
force: yes
depth: 1
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank23
># Database Server>------------------
> ls -l /u01/app/oracle/snb
basenvbindbsetchistoryinstalllog -> /var/log/dbmsrcvsqltmp
- name: Oracle File Distribution
hosts: all
become: yes
become_user: oracle
gather_facts: true
gather_subset: min
tasks:
- name: Synchronize from the Tower file system to the managed host
synchronize:
src: "/tower_fs/oracle_distribution/u01_app_oracle/snb/{{ item }}"
dest: "/u01/app/oracle/snb/"
checksum: yes
delete: yes
times: yes
recursive: yes
rsync_opts: "--chown=oracle:dba"
loop:
- basenv
- bin
- install
- rcv
- sql
Distribute files from Ansible Tower to the Servers
Configuration files with Jinja2 Templates
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank24
− A template contains variables which are replaced by their values when the template is
rendered
− Ansible uses Jinja2 templating
<title>{% block title %}{% endblock %}</title><ul>{% for user in users %}
<li><a href="{{ user.url }}">{{ user.username }}</a></li>{% endfor %}</ul>
listener.ora with Jinja2 Templates
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank25
# Jinja2 Template # -----------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST={{ ansible_fqdn }})(PORT=1599)))
SID_LIST_LISTENER=(SID_LIST={% for instance in hostvars[inventory_hostname].instances %}(SID_DESC=(ORACLE_HOME={{ instance.oracle_home }})(SID_NAME={{ instance.name }})){% endfor %})
# Listener.ora# -----------------------------------------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbserver.snb.ch)(PORT=1599)))
SID_LIST_LISTENER=(SID_LIST=(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS01CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS02CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home18c)(SID_NAME=DBMS03CD)))
# Output of the Dynamic Inventory # Script# -------------------------------
> ./snb_invent.py --host dbserver.snb.ch{"instances": [{"name": "DBMS01CD","oracle_home": "…","version": "19c"
},{"name": "DBMS02CD","oracle_home": "…","version": "19c"
},{"name": "DBMS03CD","oracle_home": "…","version": "18c"
}]}
# Task in the playbook# ------------------------------------ name: creating listener.ora
template:src: "/tower_fs/oracle_distribution/db00_app_oracle/snb/templates/listener.j2"dest: "/u01/app/oracle/network/admin/listener.ora"owner: oraclegroup: dbamode: '0640'force: yes
Modules
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank27
- name: httpd server installation
hosts: all
become: yes
become_user: root
tasks:
- name: httpd server is installed
yum:
name: httpd
state: present
- name: httpd is started and enabled
service:
name: httpd
state: started
enabled: true
># Ansible Tower
># -------------
> cat /usr/lib/python2.7/site-packages/ansible/modules/system/service.py
#!/usr/bin/python
...
options:
name:
description:
- Name of the service.
type: str
required: true
state:
description:
- C(started)/C(stopped) are idempotent actions that will not run
commands unless necessary.
- C(restarted) will always bounce the service.
- C(reloaded) will always reload.
- B(At least one of state and enabled are required.)
type: str
choices: [ reloaded, restarted, started, stopped ]
...
Module Support
− Four types of modules:
− Core: Maintained and supported by Ansible
engineering team
− Network: Maintained and supported by Ansible
Network team
− Certified: Maintained by Red Hat partners. First
level contact is Red Hat
− Community: Not supported under an Ansible
engine subscription
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank28
Documentation of the synchronize module
−Organizations requires regulations about using the community modules
Oracle Modules
−One of the most popular community module for Oracle is
Ansible-Oracle-Modules, developed by Mikael Sandström
− Can be downloaded from GitHub
−Oracle also provides Oracle Cloud Infrastructure Ansible
modules
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank29
> ls ansible-oracle-modules
oracle_asmdgoracle_asmvoloracle_awroracle_datapatchoracle_dboracle_factsoracle_gi_factsoracle_grantsoracle_joboracle_jobclassoracle_jobscheduleoracle_jobwindoworacle_ldapuseroracle_opatchoracle_parameteroracle_pdboracle_privsoracle_profileoracle_redooracle_roleoracle_rsrc_consgrouporacle_servicesoracle_sqloracle_stats_prefsoracle_tablespaceoracle_user
− Automating database tasks without logging in to each
server
− Database software installation
− Database creation
− Database configuration
− Database patching and upgrade
− Database cloning
− ...
− Base for a self-service platform
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank30
># Database Server># ---------------
> Creating User
># Database Server># ---------------
> Cloning PDB
># Database Server># ---------------
> Creating Database
Self Service Portal
># Ansible Tower ># Rest API># ---------------
curl -X GET …
Oracle Infrastructure Automation with Ansible
># Database Server># ---------------
> Creating User
># Ansible Tower ># Rest API># ---------------
curl -X GET …
Applying Release Update
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank31
- name: Start the instance and apply datapatch
shell: |
export ORACLE_HOME="{{ item.oracle_home }}"
export ORACLE_SID="{{ item.name }}"
$ORACLE_HOME/bin/sqlplus / as sysdba << EOF
startup
EOF
$ORACLE_HOME/OPatch/datapatch -verbose
register: shell_result
changed_when: >
( "'Installing patches...' in shell_result.stdout" ) or
( "'ORACLE instance started.' in shell_result.stdout" )
failed_when: "'ORA-' in shell_result.stdout"
with_items: "{{ hostvars[inventory_hostname]['instances'] }}"
when: item.oracle_home == "/db00/app/product/db19c/db00"
- name: Copy RU from the control host to the servers and unzip
unarchive:
copy: yes
src: "/ansible_fs/p30125133_190000_Linux-x86-64.zip"
dest: "/db00/app/snb/tmp_for_apply_psu"
owner: oracle
group: dba
- name: Apply RU with Ansible-Oracle-Modules
oracle_opatch:
oracle_home: "/db00/app/oracle/product/db19c/db00"
patch_base: "/db00/app/snb/tmp_for_apply_psu/30125133"
opatch_minversion: '12.2.0.1.17'
conflict_check: yes
stop_processes: no
state: present
Ansible Execution
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank32
># Ansible Tower># ---------------
> ansible-playbook db.yml -e "oracle_sid=orcl"
># Managed Host># ----------------
> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory
SSH
−Most of the modules requires python installed on the managed host. Modules can also
have additional requirements
− All playbooks are executed as awx user on the control node
># Managed Host># ----------------
> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory
Developing Ansible Modules
− You can use any language; however python is the most suitable
− Ansible modules strive for idempotency:
− You can run a playbook on the same host multiple times. When your systems are in the correct
state, the playbook should not make any changes
− Idempotency is not always possible especially when you are calling external programs, like
oracle setup or grid infrastructure root scripts
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank33
> mkdir /u01/app
> If (! -d /u01/app); then mkdir /u01/app ; fi
x
Simple Create/Remove Directory Module with bash and python
−Default custom module directory in Ansible Tower is
/usr/share/ansible/plugins/modules
− Ansible calls custom scripts with a parameter file
−Output should be a JSON object
− “failed” should be true in case of a failure; exit value is
not enough
− “changed” key is for idempotency
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank34
> ./my_bash_script.ksh /tmp/variables
{ "changed" : false,"msg": "ERROR:cannot create directory…","failed": true }
- name:create directory with a bash modulemy_bash_script:
dirname: "/tmp/new_dir"state: present
#!/usr/bin/ksh
source $1
echo ${dirname}echo ${state}
Simple Create/Remove Directory Module with bash and python
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank35
#!/usr/bin/bash
source $1
typeset -l state=${state:-present}
if !([ "${state}" = "absent" ] || [ "${state}" = "present" ]) then
echo "{ \"failed\": true,"
echo "\"msg\": \"ERROR: state must be absent or present\"}"
exit 1
fi
if [ "${dirname}" = "" ] ; then
echo "{ \"failed\": true, "
echo " \"msg\": \"ERROR: please provide dirname \" }"
exit 1
fi
#!/usr/bin/python
import os
def main():
module = AnsibleModule(
argument_spec = dict(
dirname = dict(required=True, type='str'),
state=dict(default='present',choices=['present','absent'])
)
)
dirname = module.params["dirname"]
state = module.params["state"]
Simple Create/Remove Directory Module with bash and python
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank36
if [ "${state}" = "absent" ] && [ -d "${dirname}" ] ; then
vRetMsg=$( rm -rf ${dirname} 2>&1 )
if [ $? -eq 0 ] ; then
echo "{ \"changed\": true }"
else
echo "{ \"failed\": true, "
echo " \"msg\": \"ERROR: ${vRetMsg}\" }"
exit 1
fi
elif [ "${state}" = "present" ] && [ ! -d "${dirname}" ] then
vRetMsg=$( mkdir ${dirname} 2>&1 )
if [ $? -eq 0 ] ; then
echo "{ \"changed\": true }"
else
echo "{ \"failed\": true, "
echo " \"msg\": \"ERROR: ${vRetMsg}\" }"
exit 1
fi
else
echo " { \"changed\": false }"
fi
exit 0
if state == 'absend' and os.path.exists(dirname):
try:
os.rmdir(dirname)
except OSError as err:
module.fail_json(msg="OS error: {0}".format(err))
else:
module.exit_json(changed=True)
elif state == 'present' and not os.path.exists(dirname):
try:
os.mkdir(dirname)
except OSError as err:
module.fail_json(msg="OS error: {0}".format(err))
else:
module.exit_json(changed=True)
else:
module.exit_json(changed=False)
from ansible.module_utils.basic import *
if __name__ == '__main__':
main()
Simple Create/Remove Directory Module with bash and python
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank37
TASK [Create directory] ************************************************
changed: [srv1.ubozkurt.com]
TASK [Create directory second time] ************************************************
ok: [srv1.ubozkurt.com]
TASK [Create directory without having permission on the fs] ************************************************
fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: mkdir: cannot create directory ‘/umut_new_dir’: Permission denied"}
...ignoring
TASK [Invalid input parameter] ************************************************
fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: state must be absent or present"}
...ignoring
tasks:
- name: Create directory
my_bash_script:
dirname: "/tmp/umut_new_dir"
state: present
- name: Create directory second time
my_bash_script :
dirname: "/tmp/umut_new_dir"
state: present
- name: Create directory without having permission on the fs
my_bash_script:
dirname: "/umut_new_dir"
state: present
ignore_errors: true
- name: Invalid input parameter
my_bash_script:
dirname: "/tmp/umut_new_dir"
state: "invalid_value"
ignore_errors: true
Summary
− Ansible is a simple, powerful automation engine
− There are many use cases for automating daily tasks
−Role based access control helps to integrate Ansible Tower in to the organizations
−Organization wide usage requires a good security concept
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank38