Where did my CPU go? Session ID# UGF9790

Karl Arao Enkitec

whoami  

Karl  Arao  •  Senior  Technical  Consultant  @  Enkitec  •  Performance  and  Capacity  Planning  Enthusiast    

 7  years  DBA  experience  

Oracle  ACE,  OCP-­‐DBA,  RHCE,  OakTable  Blog:  karlarao.wordpress.com  Wiki:  karlarao.Kddlyspot.com  TwiLer:  @karlarao    

www.enkitec.com   2  

www.enkitec.com   3  

200+

3

Booth #501 Moscone

South

Agenda  

•  The  different  CPU  events  

•  CPU  Monitoring/Capacity  Planning  on  consolidated  environments    •  HOWTO  compare  CPU  speeds  

 

•  Cores  vs  Threads  

   

www.enkitec.com   4  

www.enkitec.com   5  

12:27:15 SYS@DEMO1> show parameter cpu_count!!NAME TYPE VALUE!------------------------------------ ----------- --------!cpu_count integer 16!

www.enkitec.com   6  

Socket0 Core0

CPU0 CPU8

Core1

CPU1 CPU9

Core2

CPU2 CPU10

Core3

CPU3 CPU11

Socket1 Core0

CPU4 CPU12

Core1

CPU5 CPU13

Core2

CPU6 CPU14

Core3

CPU7 CPU15

Exadata  V2  =>  2s8c16t  

PART1:  Different  CPU  events  CPU  

CPU  Wait  CPU  Scheduler  

 

www.enkitec.com   7  

www.enkitec.com   8  

AAS  CPU  

www.enkitec.com   9  

CPU  Wait  

www.enkitec.com   10  

www.enkitec.com   11  

CPU  Scheduler  

www.enkitec.com   12  

www.enkitec.com   13  

www.enkitec.com   14  

Pu^ng  it  all  together  

Instances  Caged  at  12  CPUs  

SQL  Applied  to  lock  in  good  plan.  

Problem:  A  single  SQL  Stmt.  overwhelming  CPU  resources.  

PART2:  CPU  monitoring  and  Capacity  Planning  

 

www.enkitec.com   15  

OS  Tools  •  The  usual  OperaKng  System  commands  

–  vmstat  –  top  –  mpstat  –P  ALL  1  5  

•  Cool  tools    –  collectl  –sC    (hLp://collectl.sourceforge.net)  –  turbostat.c  –  dcli  (Exadata)  

•  dcli  -­‐l  oracle  -­‐g  /home/oracle/dbs_group  -­‐-­‐vmstat  2  •  dcli  -­‐l  oracle  -­‐g  /home/oracle/dbs_group  upKme  

www.enkitec.com   16  

www.enkitec.com   17  

Load  Map  

www.enkitec.com   18  

Performance  Page  –  Historical  View  

AWR  Toolkit  •  DIY  performance  data  warehouse  

www.enkitec.com   19  

run_awr  run_extract  

Extract  AWR  data  points  as  csv  files  Package  all  the  csv  files  Customer  site  

FRESH_LOAD  CHECK_LOAD  DELTA_LOAD  

Create  new  client  “dimension”  tables  Check  new  data  points  Load  new  data  points  

DIY  DW  server  

awr_topevents_(ClientNameX)  awr_cpuwl_(ClientNameX)  awr_iowl_(ClientNameX)  

1

2

3 Tableau  AnalyKcs  

awr_topevents_(ClientNameY)  awr_cpuwl_(ClientNameY)  awr_iowl_(ClientNameY)  

awr_topevents_(ClientNameZ)  awr_cpuwl_(ClientNameZ)  awr_iowl_(ClientNameZ)  

www.enkitec.com   20  

•  Tableau  auto  creates  a  Kme  dimension  for  the  Kme  column  “MM/DD/YY  HH24:MI:SS”  of  AWR  csv  output  

www.enkitec.com   21  

www.enkitec.com   22  

•  Summary  and  Underlying  data  

1-­‐2AM  

2-­‐3AM  

www.enkitec.com   23  

CPU  usage  across  half  rack  Exadata  

www.enkitec.com   24  

CPU  usage  per  host  

www.enkitec.com   25  

CPU  redistribuKon  across  nodes  

PART3:  compare  CPU  speeds  

www.enkitec.com   26  

Different  methods:  

•  Published  benchmarks  – TPC-­‐C  – SPECint_rate2006  

•  Actual  Benchmarking  – cputoolkit  – SLOB  (lio  test)  

www.enkitec.com   27  

TPC-­‐C  •  TransacKon  Processing  Performance  Council  (TPC)  

•  Throughput  =>  transacKons  per  minute  (tpmC)  •  Price/Performance  =>  USD  /  tpmC  

www.enkitec.com   28  

•  CPU  performance  =>  tpmC  /  core  •  1609186.39  /  16  =  100574  

SPECint_rate2006  •  Standard  Performance  EvaluaKon  CorporaKon  (SPEC)  

•  SPECint_rate2006    •  Integer  performance  •  All  CPUs  are  used  •  Used  by  OEM12c  ConsolidaKon  Planner  (SYSMAN.EMCT_SPEC_RATE_LIB)  

•  CPU  performance  =>  SPECint_rate2006/core  •  702/16  =  43.875  

www.enkitec.com   29  

$ cat spec.txt | grep -i sun | grep -i x3-2 | sort -rnk1!44.0625, 16, 2, 8, 2, 632, 705, Oracle Corporation, Sun Blade X3-2B (Intel Xeon E5-2690 2.9GHz)!44.0625, 16, 2, 8, 2, 630, 705, Oracle Corporation, Sun Server X3-2L (Intel Xeon E5-2690 2.9GHz)!43.875, 16, 2, 8, 2, 628, 702, Oracle Corporation, Sun Server X3-2 (Intel Xeon E5-2690 2.9GHz)!!

2007  vs  2012  

www.enkitec.com   30  

tpmC/core, System, tpmC, Price/Perf, Total System Cost, Currency, Database Software, Server CPU Type, Total Server Cores, Cluster, Date Submitted

Result/# Cores, # Cores, # Chips, # Cores Per Chip, # Threads Per Core, Baseline, Result, Hardware Vendor, System

Actual  Benchmarking  •  cputoolkit  and  SLOB  (lio  test)  •  LIOs/sec  

www.enkitec.com   31  

CPU1  

CPU2  

CPU3  

CPU4  

CPU5  

CPU6  

CPU7  

CPU8  

CPU1  

CPU2  

CPU3  

CPU4  

CPU5  

CPU6  

CPU7  

CPU8  

cputoolkit  ./runcputoolkit-­‐auto  <start  CPU>  <end  CPU>  <db  name>  ./runcputoolkit-­‐auto  1  2  dw  

SLOB  ./runit.sh  <writers>  <readers>  while  :;  do  ./runit.sh  0  2;  done  

Both  at  25%    CPU  u[liza[on  

V2  and  X2  CPU  perf  comparison  

www.enkitec.com   32  

3.6M LIOs/sec

2.1M LIOs/sec

V2  -­‐>  X2  migraKon  

www.enkitec.com   33  

chip  efficiency  factor    =  (source  LIOs/sec)  /  (desKnaKon  LIOs/sec)                                                                                =  2.1M  /  3.6M                                                                                =  .5833    X2  CPU  requirement      =  source  host  CPUs  *  uKlizaKon  *  chip  efficiency  factor                                                                                =  16  *  .46                                                                                  =  7.36    *  .5833                                                                                =  4.29  CPUs    X2  CPU  UKlizaKon              =  CPU  requirement  /  CPU  capacity                                                                                =  4.29  /  24                                                                                =  17.8  %    

V2 X2

PART4:  Cores  vs  Threads  

www.enkitec.com   34  

Socket0 Core0 Core1 Core2 Core3

Socket0 Core0

CPU1 CPU5

Core1

CPU2 CPU6

Core2

CPU3 CPU7

Core3

CPU4 CPU8

www.enkitec.com   35  

~30%  depends  on  the  workload  

www.enkitec.com   36  

cputoolkit   SLOB  

17%   21%  

Intel HT Technology Technical User's Guide http://goo.gl/3Ec5Z

Pu^ng  them  all  together  

•  Enkitec  has  developed  a  tool  called  Provisioning  Worksheet  that  is  mainly  used  for  sizing  and  consolidaKon  of  databases.  

–  AccounKng  Per  database  CPU  usage  not  like  OEM12c  ConsolidaKon  Planner  which  is  just  host  level  CPU  uKlizaKon    

–  Consolidated  database  workload  analysis  across  mulKple  servers  –  CPU,  memory  (SGA/PGA  &  HugePages),  storage  capacity  (storage  &  IOPS)  planning    –  All  the  capacity  planning  math  integrated  with  the  tool  (chip  efficiency  factor  across  mulKple  servers)  –  Layout  instances  across  nodes  according  to  Preferred,  Available,  Failover    –  Create  a  baseline  of  the  instances  node  layout  and  their  current  uKlizaKon  –  Simulate  node  failure  and  different  what-­‐if  scenarios  (add/delete  node,  increase/decrease  capacity,  

etc.)  –  VisualizaKon  module  to  quickly  see  node  level  resource  imbalance  –  Cluster  and  node  level  uKlizaKon  –  Recommended  Hardware  (equivalent  compute  nodes)  

•  See  the  tool  here  hLp://www.slideshare.net/karlarao/pape-­‐rkaraoconsolidaKon-­‐successstory  

 

www.enkitec.com   37  

Wrap  up!  •  The  different  CPU  events  

o  1  AAS  CPU  =  1  CPU  core  o  Oracle  CPU  may  not   correlate  with  Host  CPU   if   you  have  a   lot  of  

CPU  acKvity  outside  of  the  database  

•  CPU  Monitoring/Capacity  Planning  on  consolidated  environments  o  AWR  analyKcs    

•  HOWTO  compare  CPU  speeds  o  SPECint_rate2006,  TPC-­‐C,  Actual  benchmarking    

•  Cores  vs  Threads  o  Always  have  HT  on  o  ~30%  performance  benefit  awer  core  count  

www.enkitec.com   38  

Read  the  WhitePaper  

   

Where  did  my  CPU  go?  hLp://goo.gl/eq9Sn  

www.enkitec.com   39  

Resources  •  Whitepaper:  Where  did  my  CPU  go?  hLp://goo.gl/eq9Sn  •  cputoolkit  -­‐  hLp://karlarao.wordpress.com/scripts-­‐resources/  •  AWR  Tableau  and  R  toolkit  VisualizaKon  Examples  -­‐  hLp://goo.gl/xZHHY  •  AAS  invesKgaKon  -­‐  hLp://goo.gl/5WaAg  •  Cores  vs  Threads  -­‐  hLp://goo.gl/1MLFf  •  Turbostat.c  -­‐  hLp://goo.gl/jDUKg  •  cpu_topology  -­‐  hLp://goo.gl/EUDG7  •  CPU  centric  benchmark  comparisons  -­‐  hLp://goo.gl/nR9Yy  •  SLOB  -­‐  hLp://goo.gl/yKa45  •  Kyle  Hailey  -­‐  hLp://dbopKmizer.com/2011/07/21/oracle-­‐cpu-­‐Kme/  •  The  mindmap  of  this  presentaKon  -­‐  hLp://goo.gl/XeY0e  

www.enkitec.com   40  

karl.arao@enkitec.com  

 –  Network  with  5,000+  database  and    

applicaKon  professionals  –  5  days  of  in-­‐depth  educaKon  built  by    

users,  for  users  –  Complimentary  Pre-­‐Conference    

Workshops  for  IOUG  registrants  

   

   

April 7 – 11, 2014 The Venetian Resort,

Las Vegas, NV

A]end  for  free!  IOUG’s  Call  for  Speakers  is  now  open  collaborate14.ioug.org/call-­‐for-­‐speakers  

Communities

Maximum  Availability  Architecture    with  Oracle’s  Larry  Carpenter  and  Joe  Meeks  October  15:  Milwaukee,  WI  October  16:  Rochester,  NY    Performance  and  Internals  with  Craig  Shallahamer    November  6:  Chicago,  IL  November  8:  Atlanta,  GA    Get  more  informa[on  at  www.ioug.org/masterclass    or  visit  the  IOUG  kiosk  in  Moscone  West,  2nd  floor    

IOUG  Master  Class  Series  Returns  Coming  soon  to  a  city  near  you!    

One  Day  Interac[ve  Training  Events  to  meet  your    educa[onal  needs

Win a Free Registration to COLLABORATE 14!

Stop by the IOUG kiosk in the User Group Pavilion in Moscone West, 2nd floor, to pick up a lucky poker chip and see if you are a winner of a free registration to

COLLABORATE 14 in Las Vegas!    

Complimentary eBook Register Now

www.mhprofessional.com/dbsec  

Use  Code:  db12c  

 

THANK YOU

karl.arao@enkitec.com