![Page 2: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/2.jpg)
NLUUG 15 May 2014
My background
• Been a sysadmin since 1990
• Around 1995 started using Linux
• Since 2012 supporting ftp.nluug.nl
• Working @ Proxy in the Managed Services Team
![Page 3: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/3.jpg)
NLUUG 15 May 2014
Hobby
![Page 4: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/4.jpg)
NLUUG 15 May 2014
History of ftp.nluug.nl
• December 1992 – total download 448575834 bytes
![Page 5: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/5.jpg)
NLUUG 15 May 2014
Former hardware
• 2 servers round robin DNS• All SCSI storage attached to 1 server• Storage exported with NFS and GFS
![Page 6: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/6.jpg)
NLUUG 15 May 2014
Current Hardware
• 1 server• Xeon E3-1220 3.1 Ghz• 16 GB Memory• LSI Megaraid SAS 9271-4i• 4 x 1GB ethernet• 13 x 2TB disk, 12 disks in Raid5• 1 hot spare drive• 11 free positions for extra drives
![Page 7: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/7.jpg)
NLUUG 15 May 2014
Performance servers 1 day avg
Max 480 Mb/sAvg 160 Mb/s2 TB/day
Max 800 Mb/sAvg 340 Mb/s4 TB/day
The current server
The 2 servers
![Page 8: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/8.jpg)
NLUUG 15 May 2014
What we serve
• 17 TB of used storage• 13.800.038 files• 12.224.856 files smaller than 1 MB• 681.468 directories• 1.606.561 links (hard and soft links)
![Page 9: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/9.jpg)
NLUUG 15 May 2014
Max of TB/month
Jun 2013 Oct 2013 Apr 2008 May 2013 Nov 2013 Feb 2014 Mar 2014 Dec 2013 Jan 2014 Apr 20140
20
40
60
80
100
120
140
93.67 94.61 94.66 96.43
103.94109.47
117.63 118.05 119.84
128.62
TB/month
![Page 10: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/10.jpg)
NLUUG 15 May 2014
Atoptool with netatop
• Atoptool together with netatop kernel module•
![Page 11: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/11.jpg)
NLUUG 15 May 2014
Mrtg
![Page 12: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/12.jpg)
NLUUG 15 May 2014
Munin
![Page 13: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/13.jpg)
NLUUG 15 May 2014
Nagios
![Page 14: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/14.jpg)
NLUUG 15 May 2014
Design decisions new server
• Machine is I/O based not CPU or Memory• Disk I/O should be as fast as possible• 4 x 1 Gb ethernet cards in bonding mode• Is a public mirror server, downtime is not critical• Costs effective, whole server including disks costed about 5K
![Page 15: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/15.jpg)
NLUUG 15 May 2014
Raid setup
• Machine build was in november 2012• Hitachi Ultrastar 7k3000 2 TB drives
– 2.0 million hours MTBF, 5 years guaranty– 64 MB cache
• LSI MegaRAID SAS 9271-4i Card– 1 Gb cache memory– CacheVault, NV Flash cache and battery
• 256kB Stripe size • 12 disks in Raid5 1 hot spare
![Page 16: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/16.jpg)
NLUUG 15 May 2014
What was changed
![Page 17: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/17.jpg)
NLUUG 15 May 2014
Problem #1
• 4 x 1Gb ethernet interfaces– Bonded as balance-alb
• Send and receive are bonded– Did not work out as we thought.– Maximum speed of all interfaces together did not exceed 960 Mb/s
• In June 2013 together with Surfnet assistance moved to 802.3ad and xmit_hash_policy=layer3+4– With atop we could see that the balancing is working– Did not work out as aspected– Maximum speed at that time was 993 Mb/s
![Page 18: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/18.jpg)
NLUUG 15 May 2014
Solution
• Changes in the bonding parameters • Upgraded / downgraded ethernet driver versions• Tuning kernel parameters• Crap said the Surfnet engineer ;-)• Connection to a Cisco module where 8 ports have only a 1
Gb/s backend port.• We were on ports Gi9/3 Gi9/4 Gi9/5 Gi9/6• After rerouting ports (Gi9/3 Gi9/11 Gi9/17 Gi9/28) peaks of
2.4 Gb/s were seen
![Page 19: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/19.jpg)
NLUUG 15 May 2014
The whole process of tuning
• At first I did not document, only implement– Now I document what I changed, including time stamp
• Buffers were too small• Timeout's were too long• Did not know where to start• So many performance tuning articles• The problem with a public mirror server is that it is public• I even changed parameters while writing this presentation
![Page 20: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/20.jpg)
NLUUG 15 May 2014
Firewall
• Lot's of messages ip_conntrack: table full– net.netfilter.nf_conntrack_max = 1048576
• wc -l /proc/net/nf_conntrack– net.netfilter.nf_conntrack_tcp_timeout_established = 600
• Defaults are 432000 (5 day's)
![Page 21: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/21.jpg)
NLUUG 15 May 2014
We can get troughput
![Page 22: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/22.jpg)
NLUUG 15 May 2014
Networking buffers
• Ethtool -G ethx rx 4096 tx 4096• Ifconfig ethx txqueuelen 20000
– Also for bonding interface (not sure if needed)
• Ethtool -K ethx gso on (generic-segmentation-offload)• Ethtool -K ethx gro on (generic-receive-offload)
![Page 23: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/23.jpg)
NLUUG 15 May 2014
Filesystems
• Limited to 3TB filesystems• Echo “noop” > /sys/block/sda/queue/scheduler
– Also tried deadline
• All are ext4– options noatime,nodiratime,noacl,commit=15
• LSI CacheFlushInterval=10 (default 5)
![Page 24: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/24.jpg)
NLUUG 15 May 2014
OS level
• irqbalance to oneshot• Chkconfig –del cpuspeed; service stop cpuspeed• vm.min_free_kbytes=204800
– To prevent out of memory errors– Prevent deadlocks under high loads
• Ulimit – Max openfiles
![Page 25: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/25.jpg)
NLUUG 15 May 2014
Yum-plugin-fastestmirror
• centos.mirror1.spango.com 1.624 ms • ftp.nluug.nl 1.533 ms• mirror.prolocation.net 1.44 ms• mirror.widexs.nl 1.371 ms
• Add “prefer=ftp.nluug.nl” to
/etc/yum/pluginconf.d/fastestmirror.conf
![Page 26: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/26.jpg)
NLUUG 15 May 2014
Application level
• Rsync just the standards• Vsftpd just the standards• Apache 2.2.15
– Were running 4 instances, 1 for every IPv4 and Ipv6– Reduce now to 1 instance– KeepAlive On– MaxKeepAliveRequests 1000– ServerLimit 1024– MaxClients 1024– MaxRequestsPerChild 800
![Page 27: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/27.jpg)
NLUUG 15 May 2014
Future
• Hardware– Memory from 16GB to 32 GB
• Add more opensource projects– Than we need more disks
• Nginx•• Maybe I should try XFS in the future• We just need more hits !!
![Page 28: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/28.jpg)
NLUUG 15 May 2014
Some of the mirrors
• Most Linux distributions are mirrored (currently 163 different)• BSD (FreeBSD, NetBSD, OpenBSD• Openindiana, opensolaris, illumos• Jenkins• Mariadb• Vim• Blender, gimp, ImageMagick• Apache• Qt , perl, gcc• Vlc, xbmc, openelec
![Page 29: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/29.jpg)
NLUUG 15 May 2014
• Questions......
![Page 30: Performance tuning a public mirror server - Mike Hulsman (Proxy)](https://reader034.vdocument.in/reader034/viewer/2022052522/554dda0bb4c905c70e8b4f1f/html5/thumbnails/30.jpg)
NLUUG 15 May 2014
List of URL's
• https://github.com/jeffmurphy/NetPass/blob/master/doc/netfilter_conntrack_perf.txt• https://gist.github.com/kfox/1942782• http://www.atoptool.nl/• http://www.bufferbloat.net/attachments/9/BufferBloat11.pdf• http://lwn.net/Articles/507065/• http://www.coverfire.com/articles/queueing-in-the-linux-network-stack/