ovm on exadata -...
TRANSCRIPT
![Page 1: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/1.jpg)
OVM on Exadata
Living in A virtual world
Experiences from the field : Pieter Van Puymbroek
Philippe Fierens
![Page 2: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/2.jpg)
About PieterSailor Musician Engineered systems Oracle DBA dad Recently Joined the ACE program
working for Exitas Oracle Platinum Partner in BELGIUM
And ….
![Page 3: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/3.jpg)
likes to be in control
![Page 4: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/4.jpg)
Dreams at night ;-)
![Page 5: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/5.jpg)
About Philippe
Philippe Fierens Independent Oracle DBA Architect Oracle SuperCluster fan Oracle ACE and most of all DAD & husband ;-) AND
…
pfierens.blogspot.com@pfierens
![Page 6: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/6.jpg)
BBL
![Page 7: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/7.jpg)
3MembershipTiers• OracleACEDirector• OracleACE• OracleACEAssociate
bit.ly/OracleACEProgram
500+TechnicalExpertsHelpingPeersGlobally
Connect:
Nominateyourselforsomeoneyouknow:acenomination.oracle.com
@oracleace
Facebook.com/oracleaces
![Page 8: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/8.jpg)
About Exitas
Platinum partner Support full red stack - Consultancy - Managed services - Reselling
Team of 25 dba’s www.exitas.be
Quality. Passion. Personality
![Page 9: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/9.jpg)
• What
• Why
• How / Implementation
• Issues
Agenda
![Page 10: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/10.jpg)
Small Recap OVM
https://docs.oracle.com/cd/E50245_01/E50249/html/vmcon-ovm-arch.html
![Page 11: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/11.jpg)
openstack
high availability
vm manager Web gui ?OVF
VM templates
Live MigrationRapid VM provision cloning
Windows Solaris guest
Unboxing the exadata OVM
![Page 12: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/12.jpg)
VMWare / real OVM on Exadata ?
![Page 13: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/13.jpg)
You will not find any of these :
OVM manager
Load balancing
High Availability : in RAC we trust ;-)
Live Migration
What is it not?
![Page 14: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/14.jpg)
What is it then ?
![Page 15: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/15.jpg)
Manage Expectations
WARNING
![Page 16: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/16.jpg)
Better name is Virtualised exadata
See domU’s bonded in 1 cluster as a bare metal exadata
HA is foreseen by Oracle Grid infrastructure
Cell’s are NOT virtualised
Limit Licenses -> Trusted domains
Why?
![Page 17: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/17.jpg)
Flexibility
most operations on VM level not online* :
No Memory ballooning (mem = maxmem) —> due to IB sr-iov
CPU adding removal —> as long as vcpus < maxvcpus (License!) ASM / Celldisks can be modified online as traditional Decent planning required
Just because you can, doesn’t mean you should
*not supported in 12.1.2.2.1
![Page 18: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/18.jpg)
Licensing
Only use options in the environments needed EE and storage cell licenses as you’re used to Limit number of CPU licensed Capacity on demand is possible you pay for hwm of max_vcpu ! Checked with OEM
![Page 19: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/19.jpg)
SecurityDesigned to isolate environments from each other :
vlanning on compute nodes supported X6-2 allows to add NIC partition keys can be used on IB
cellwall is still present ssh can be shutdown on the cells managed through ExaCLI Enterprise linux / compute nodes best practices still apply!
![Page 20: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/20.jpg)
Caging
Resource manager = mandatory
Cgroups not supported for DB processes ->example Oracle DB + Oracle R => put in own DomU
Cage your os processes (the ones which aren’t allowed to cap with groups)
![Page 21: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/21.jpg)
Mos Support Addict ;-)
You love to open MOS tickets ;-)
![Page 22: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/22.jpg)
High level overview
vexa123-clu1exa123-scan1exa123adm01vm01 exa123adm02vm01
vexa123-clu3exa123-scan3
exa123adm01vm03exa123adm02vm03
vexa123-clu4exa123-scan4exa123adm01vm04 exa123adm02vm04
vexa123-clu2exa123-scan2
exa123adm01vm02 exa123adm02vm02
exa123db01 exa123db02
![Page 23: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/23.jpg)
IB
http://downloads.openfabrics.org/downloads/Media/IBUG_2014/Thursday/PDF/12_IB_VIRT.pdf
Dom0 DomUDomU
HCAonPCIe
PFdriver VFdriver VFdriver
PFdriver VF PF
Hypervisor
Server
![Page 24: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/24.jpg)
Disk layout example
12 disks per cell 4 data diskgroups 4 reco disk groups data1 + data3 were extended reco4 was extended Dbfs_dg diskgroup
![Page 25: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/25.jpg)
Project / real world checkpoint
Move 2 OVM Upgrade to 12.1
Application upgrade
April QFSDP 2015 because bare metals were running that version
![Page 26: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/26.jpg)
Issue 1First install done by ACS OEDA
•dos2unix on xml files created by OEDA •usernames / groups cannot have _ eg rac_dba
—> fixed in later versions
GMT issue •symlink in /usr/share/zoneinfo to GMT (GMT) due to GMT ilom settings
•adapt cell.conf —> fixed in later versions
![Page 27: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/27.jpg)
Issue 1ssh key equivalence failing
•primary groupname and oracle user had the same name, we needed to adapt /etc/profile and /etc/bashrc to make sure the umask is always 022 in the vm’s
—> still valid in image 12.1.2.3.1 / OEDA august2016—> Finally fixed in OEDA april 2017
![Page 28: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/28.jpg)
Issue 2Oracle 12c quite some issues
•expdp impdp •truncate generate ORA-0600 •DBMS_STATS •wrong results •Cursor S wait for X
bottom line quite some one-offs to put on top of QFSDP April 2015
![Page 29: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/29.jpg)
Issue 3performance
insert append slower on OVM cluster_database = false
performance ok cluster_database = true
1 instance or 2 instances up performance not ok
weeks and weeks and lots of escalation meeting further
![Page 30: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/30.jpg)
Issue 3 contCpu scheduling issue : RAC uses other code path then single instance CPU flags not passed correctly to DomU’s (X5)
result : Two month delay GO Live
details here : Performance slower in OVM when compared with Baremetal (Doc ID 2080843.1)
—> Also Valid in Exadata X6-2
![Page 31: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/31.jpg)
Issue 4OEDA doesn’t assign groups for : syskm sysdg bckdba
Solution: edit $ORACLE_HOME/lib/config.c + relink
Recently an enhancement request was opened for this
![Page 32: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/32.jpg)
Issue 5Stopping a domU takes up to 1hour !!!!! -> fixed in QFSDP OCT 2015
![Page 33: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/33.jpg)
Issue 6unplugging public network did not trigger VIP failover
-> GI 12.1.0.2 image 12.1.2.1.1 -> 12.1.2.1.1 not supported anymore
-> no fix but must set ping targets
srvctl modify network -k 1 -pingtarget “<UsefulTargetIP(s)>”
Check certification matrix it changes over time mos 888828.1
![Page 34: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/34.jpg)
Things to considerAvoid swapping from DomU’s Check 888828.1 for critical issues -> If necessary PATCH! Kernel panic’s seen whilst coupled to ZFSSA -> new Firmware on the HCA’s loaded Always use latest OEDA-> using OEDA upgrades rpms (eg. exadata-ovmutils,…) having a lot of OVM’s means : each OVM needs GI +DB + OS upgrade
![Page 35: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/35.jpg)
OEDA - Always use the latest version
—> get it from mos-note: 888828.1 —> Nov 2017 v171130 - Patch 27179375
- GIMR not default installed —> ENABLEMGMTDBCONFIG: false -> true
- upgrades rpms - ovmwatch - ovmwatch - exadata-ovmutils - …
![Page 36: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/36.jpg)
Patching (1/6)Standard Exadata Patching order :
Cells OS Infiniband GI + DB
![Page 37: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/37.jpg)
Patching (2/6)– Use patchmgr or dbnodeupdate.sh for each DomU and on Dom0 :
eg : ./patchmgr -dbnode /root/dbs_dom0_group_db02 -dbnode_precheck -dbnode_loc <FULL_PATH>/p22277124_121221_Linux-x86-64.zip -dbnode_version 12.1.2.2.1.160119 – Be carefull, 12.2 FIRST domU – Firmware/Bios is upgraded when Dom0 is upgraded
![Page 38: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/38.jpg)
Patching (3/6)Collisions on several one-off patches 12.1.2.1.1 image contained a patch on GI to make it work on OVM
![Page 39: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/39.jpg)
Patching (4/6)oplan to generate steps Rollback the conflicting GI and DB patches install the new patches don’t run post patch until GI is patched
![Page 40: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/40.jpg)
Patching(5/6)Patching to 12.2.1.1.1.1 APR 2017
Issues - domU with snapshots still mount by installer - libserver.a corrupt RCA still ongoing - patch mgr bricks IB switch
- going from 2.1.8 -> 2.2.5-3 - major upgrade Centos -> OEL - issue file permission - hwclock issue …
![Page 41: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/41.jpg)
Patching (6/6)- dom0 doesn’t boot
- udev rules gone eth0 -> eth7…8 - no IB coming up - SR still open and ongoing
- rebooting one switch dismounts diskgroup on one node of RAC
All this still ongoing as we speak
![Page 42: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/42.jpg)
to summariseProjectplan! platform / application /version upgrade together ?
young product! bugs but maturing / evolving quickly
have 10vm’s on one exadata means lot of work
Sizing Check certification matrix !! Manage Expectations
![Page 43: OVM on Exadata - vanpupi.stepi.netvanpupi.stepi.net/wp-content/uploads/2017/12/ovm_exa_pieter_DOAG... · Check certification matrix it changes over time mos 888828.1. Things to consider](https://reader033.vdocument.in/reader033/viewer/2022042605/5aa9eab67f8b9a90188d708e/html5/thumbnails/43.jpg)
?@pfierens @vanpupi