aix nfs client performance improvements for databases on · pdf fileaix nfs client performance...
TRANSCRIPT
![Page 1: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/1.jpg)
Slide 12005 NAS Industry Conference
AIX NFS Client Performance Improvements for
Databases on NASOctober 20, 2005
Sanjay GulabaniSr. Performance Engineer
Network Appliance, Inc.
Diane FlemmingAdvisory Software Engineer
IBM Corporation
![Page 2: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/2.jpg)
Slide 22005 NAS Industry Conference
Agenda• Why Databases on NAS?• AIX NFS improvements - CIO• Simple IO Performance Tests• OLTP Performance Comparison• Future work• Conclusion• Q&A
![Page 3: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/3.jpg)
Slide 32005 NAS Industry Conference
Databases on SAN
• Databases do block based I/O and prefer raw blocks
• But, most admins still put a volume manager and a file system: JFS2, VxFS, UFS over HW RAID storage
Reason: Simplicity -- Easier backups and provisioning
![Page 4: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/4.jpg)
Slide 42005 NAS Industry Conference
NAS Myths
• Myth: NFS consumes more CPU for Databases
Reality: Most NFS client performance problems with DB apps are due to kernel locking and are fixed in good NFS clients
![Page 5: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/5.jpg)
Slide 52005 NAS Industry Conference
NAS Myths• Myth: Ethernet is slower than SAN
Reality: Ethernet is 1Gb, FC is 2Gb. 4Gb FC is almost here but so is 10GbE.Ø Ethernet is likely to takeover bandwidth of FC soon;
cost effectivelyØ Its easy to setup multiple wires to match FC
bandwidth with just 1GbE, todayØ Storage latencies at database layer are measured in
msecs, differences between SAN and NAS latencies are in usecs78usec more for 8K blocks < 0.1msec difference!
![Page 6: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/6.jpg)
Slide 62005 NAS Industry Conference
Why NAS?• Networking is simpler• Networking promotes sharing• Sharing = Higher utilization (~grid
storage)• NAS solutions are cheaper than SAN• No one has won against Ethernet!• Seriously, even blocks based storage is
moving to Ethernet (iSCSI)
![Page 7: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/7.jpg)
Slide 72005 NAS Industry Conference
Why DB on NAS?• Complex storage management is
offloaded from the database servers to storage appliances
• Performance vs. manageabilityØ Database requirements different than
traditional NFS workloads: home directories, distributed source code
Ø single writer lock can be a problem
![Page 8: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/8.jpg)
Slide 82005 NAS Industry Conference
Industry Support for NAS• Oracle On-Demand hosts ERP+CRM
1000+ Linux servers on NetApp NFS • Yahoo runs Database on NetApp• Sun 2003 presentation McDougall +
Colaco promoted NFS• NetApp #1 Leader of NAS $1.59b(FY05)• IBM AIX 5.3 ML-03 improvements now• No one got fired for buying from IBM!
(IBM resells NetApp products)
![Page 9: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/9.jpg)
Slide 92005 NAS Industry Conference
Oracle’s Austin Datacenter
• More than 15,000 x86 servers • 3.1 Petabytes of NetApp storage• 100s of mission-critical hosted Oracle apps
Source: Oracle Magazine Mar/Apr 2005
![Page 10: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/10.jpg)
Slide 102005 NAS Industry Conference
I/O, I/O… It’s off to disk I go• Raw logical volumes
– No memory management overhead
• Buffered I/O– Variety of kernel services
• Direct I/O– No memory management overhead– defaults to synchronous accesses over NFS
• Concurrent I/O– DIO plus no inode/rnode contention
![Page 11: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/11.jpg)
Slide 112005 NAS Industry Conference
NFS File Transfer• Metrics:
– CPU utilization and throughput in KB/sec
• Options:– Default (buffered I/O) and CIO
• AIX Client and Server– p630 1.45GHz 4-way– GbE 1500-byte MTU– AIX 5.3 ML-03
• Demonstrates algorithm behavior– Atomic, Single-Threaded
![Page 12: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/12.jpg)
Slide 122005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, 32KB RPCs
NFS File T ransfer Sequential Reads and Writes
0
20000
40000
60000
80000
100000
120000
Seq Write Seq Read
Th
rou
ghp
ut
(KB
/sec
)
0
10
20
30
40
CP
U U
til (
%)
Buff I/O Throughput CIO ThroughputBuff I/O CPU Util CIO CPU Util
![Page 13: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/13.jpg)
Slide 132005 NAS Industry Conference
Simple I/O Load Generator• Metrics
– CPU utilization, IOPS, throughput, etc.• Options
– NFS client mount options to specify I/O mode: Default (buffered I/O), DIO, CIO
• NetApp Filer FAS880– ONTAP 6.5.1
• AIX Client Model p630 1.45 GHz 4-way– AIX 5.3 ML-03– GbE 1500-byte MTU
![Page 14: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/14.jpg)
Slide 142005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
SIO Throughput 50% Writers, 256MB file
01000020000300004000050000600007 000080000
1 2 4 8 16 32 64 128 256
Number of Threads
KB
/sec Buff I/O
CIO
DIO
![Page 15: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/15.jpg)
Slide 152005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
SIO Throughput100% Writers, 256MB File
0
10000
20000
30000
40000
50000
60000
1 2 4 8 16 32 64 128 256
Number of Threads
KB
/sec Buff I/O
CIO
DIO
![Page 16: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/16.jpg)
Slide 162005 NAS Industry Conference
256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
SIO Scaled Throughput using CIO, 256MB file
0
50000
100000
150000
200000
1 2 4 8 16 32 64 128 256
Number of Threads
KB
ps/
CP
U
100% Writers
50% Writers
![Page 17: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/17.jpg)
Slide 172005 NAS Industry Conference
NFS FT and SIO Summary• CIO not appropriate for single threaded
sequential read/write workloads.• CIO aimed at workloads with higher
concurrency such as databases.• Significant gain in raw throughput
performance using CIO vs. Buffered I/O for workloads with higher concurrency.
• CPU utilization illustrates issue with TCP socket lock contention with increased concurrency.
![Page 18: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/18.jpg)
Slide 182005 NAS Industry Conference
OLTP Performance
• AIX 5.2 ML-5 and AIX 5.3 ML-03 (beta)
• PowerPC pSeries 650 (2 * 1.2 GHz, 4 GB RAM)
• Oracle 10.1
• 1GbE for NFS or iSCSI or 2Gb FCP card
• FAS 940c
• 48 Disk Spindles (4 x DS14) 144GB 10K RPM
• ONTAP 6.5.1
![Page 19: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/19.jpg)
Slide 192005 NAS Industry Conference
Transactions on AIX 5.2, 5.3, iSCSI, FCP
OETs = Order Entry Transactions (an Oracle OLTP benchmark, approx 2:1 read to
write ratio)
Chart 1: Transactions for protocols
4710
11817 12171
14139
0
2000
4000
6000
8000
10000
12000
14000
16000
kernel NFS 5.2@60u
NFS 5.3* 60u iSCSI raw 5.3*60u
FCP raw 5.3*60u
tran
sact
ion
s
OETs
![Page 20: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/20.jpg)
Slide 202005 NAS Industry Conference
Host CPU Utilization
Chart 2: Transactions and cpu utilization for protocols
4710
11817 12171
14139
49
9890
88
0
2000
4000
6000
8000
10000
12000
14000
16000
kernel NFS 5.2@60u
NFS 5.3 60u iSCSI raw 5.360u
FCP raw 5.3 60u
tran
sact
ion
s
0
20
40
60
80
100
cpu
OETscpu
![Page 21: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/21.jpg)
Slide 212005 NAS Industry Conference
Protocol EfficienciesComputed using (%CPU used/ OETs) * K:
where K is a constant such that NFS CPU cost per K transactions is 1.0 and relative cost for iSCSI and FCP is computed using the same constant K transactions.
Chart 3: Relative CPU cost per protocol for fixed transactions
1
0.89
0.75
0
0.2
0.4
0.6
0.8
1
1.2
NFS 5.3 60u iSCSI raw 5.3 60u FCP raw 5.3 60u
protocol
no
rmal
ized
cp
u c
ost
(lo
wer
is
bet
ter)
relative cpu cost
![Page 22: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/22.jpg)
Slide 222005 NAS Industry Conference
Oracle Block Read Response TimeNot typical response time. Random workload used. Filer with more
cache improves block response time or more sequential reads improve avg. block response time.
Chart 4: Oracle Block Read Response Time
29
13 1412
0
5
10
15
20
25
30
35
kernel NFS5.2 @60u
NFS 5.3 60u iSCSI raw 5.360u
FCP raw 5.360u
protocol
Blo
ck r
ead
Res
p Ti
me
(low
er is
bet
ter)
block read resp time
![Page 23: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/23.jpg)
Slide 232005 NAS Industry Conference
Future Work
• Testing and analysis on systems with higher numbers of CPUs.
• Further investigation on socket lock scaling issue on AIX
![Page 24: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/24.jpg)
Slide 242005 NAS Industry Conference
Conclusions• AIX 5.3 ML-03 with new Concurrent
I/O ‘CIO’ mount option delivers an OLTP performance on NFS comparable to that of block-based protocols iSCSI and FCP
• Don’t be afraid to deploy databases on AIX NFS, we will support you.
• Proof of AIX NFS performance up to 4 CPU completed.
![Page 25: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/25.jpg)
Slide 252005 NAS Industry Conference
References• AIX Performance with NFS, iSCSI and FCP
Using an Oracle Database on NetApp White Paper @ http://www.netapp.com/library/tr/3408.pdf
• NetApp Best Practices Paper located at: http://www.ibm.com/servers/storage/support/nas/
• Download SIO toolhttp://now.netapp.com/NOW/download/tools/sio_ntap/index.shtml
• Improving Database Performance with AIX Concurrent I/O: http://www-03.ibm.com/servers/aix/whitepapers/db_perf_aix.pdf
![Page 26: AIX NFS Client Performance Improvements for Databases on · PDF fileAIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer](https://reader031.vdocument.in/reader031/viewer/2022021818/5aabd26a7f8b9a59658c5593/html5/thumbnails/26.jpg)
Slide 262005 NAS Industry Conference
Questions/Answers