linux on power – hints and tips

1

Daniel MartinNovember 19th [email protected] Find me on developerWorks

mailto:[email protected]

https://www.ibm.com/developerworks/mydeveloperworks/blogs/powermeup

© 2013 IBM Corporation

• Porting from Intel x86 to Linux on Power - Thoughts

• IBM POWER Linux Tools Repository

• IBM POWER Linux Tools Repository – Install

• IBM POWER Linux Performance Recommendations

• IBM POWER Linux Java Install and Performance Recommendations.

• Redhat on IBM POWER – Tips

• Obtaining examples - SPEC.org

• Leveraging IBM – developerWorks, VLP, IIC, Webinars

2


IBM POWER7 and POWER7+ recommended distros -

The following distributions are optimised for POWER7 and POWER7+ processor-based

servers:

SUSE Linux Enterprise Server 11 SP1, any subsequent service packs. SLES 11

SP2 is the latest service pack.

Red Hat Enterprise Linux 6, and subsequent updates. RHEL 6.4 is the latest

update.

PowerLinux Tools –

There are a number of related tools that you should install to get the very best out of your

Linux on Power systems. These are: IBM Linux hardware diagnostic aids and productivity,

IBM Software Development Kit and the IBM Advance Toolchain.

3


The IBM POWER Linux Tools Repository enables the use of standard Linux package management tools to provide access to the following:

• IBM POWER Linux hardware diagnostic aids and productivity tools

• IBM Software Development Toolkit for POWER Linux servers

• IBM Advance Toolchain for POWER Linux servers

The Tools repository supports the following versions of Linux:

• Red Hat Enterprise Linux 5 (RHEL5)

• Red Hat Enterprise Linux 6 (RHEL6)

• SuSE Linux Enterprise Server 10 (SLES10)

• SuSE Linux Enterprise Server 11 (SLES11)

5


Installing the IBM POWER Linux hardware diagnostic aids and productivity tools.

Create your repo information and then update the new repo:

# vi /etc/yum.repos.d/ibm-power.repo

[ibm-power-repo]

name=IBM POWER Tool

baseurl=http://public.dhe.ibm.com/software/server/POWER/Linux/yum/IBM/RHEL/6/

ppc64

enabled=1

gpgcheck=0

# yum update

Once the updates are downloaded, create the related libraries:

# yum install ibm-power-repo.ppc

Confirm what repos are now available:

# yum repolist enabled

6


Update the installed packages to the latest version available, enter:

# yum update

Then choose one of the following to install the recommended packages.

For an HMC or IVM-managed system (in most cases), enter:

# yum install ibm-power-managed-rhel6

OR

For a standalone system, enter:

# yum install ibm-power-nonmanaged-rhel6

I recommend installing these additional packages:Performance Management # yum install ibmPMLinuxNMON– Performance monitoring # yum install nmonPOWER Energy Management # yum install pseries-energyNUMA policy for processes or shared memory # yum install numa*

7


Environmental Settings –

These are a number of recommended environmental settings based on a system with 16 POWER7+

cores and 128GB RAM.

fs.file-max = 4878674

This relates to the maximum number of open files/file descriptors:

# cat /proc/sys/fs/file-max

201561

This will set the limit to be applied to the system after the next reboot:

vi /etc/sysctl.conf

Adding the line:

fs.file-max = 4878674

Run the following to re-read the sysctl.conf file and apply it to the current system:

# sysctl -p

9


Setting the maximum number of open file descriptors.

Displaying your current limits:

# ulimit -n

1024

Changing those limits on your current session:

# ulimit -n 1048576

Preserving the change on system reboot:

# vi /etc/security/limits.conf

Add the lines:* hard nofile 1048576* soft nofile 786432

Then repeat the same principles.

This relates to the maximum number of processes available to a single user:

# ulimit -u unlimited

This relates to the maximum stack size:

# ulimit -s 524288

10


The previous slides cover a few environmental settings, the following are some which relate to the CPU and its features which may help improve performance.

Disabling ‘preempt scheduler’:

# mount -t debugfs debugfs /sys/kernel/debug

# echo NO_WAKEUP_PREEMPT > /sys/kernel/debug/sched_features

# umount /sys/kernel/debug

It can be re-enabled via:

# echo WAKEUP_PREEMPTION > /debug/sched_features

11


Disable Idle Power Saver - Idle Power Saver can be disabled by accessing your browser and utilising the Advanced System Management Interface (ASMI) console.

From the HMC, navigate to System Configuration -> Power Management Mode Setup -> Disable Power Save Mode. Set the Idle Power Saver value to Disabled, then click on the "Save settings" button at the bottom of the page.

12


Disable IPv6

IPv6 support can be disabled in the Linux kernel by adding the following options to the command line in the boot loader configuration:

ipv6.disable_ipv6=1 ipv6.disable=1

Disabling IPv6 support in the Linux kernel guarantees that no IPv6 code will ever be run while the system is booted. Always remember that if you run this command then you cannot enable IPv6 until the system is rebooted. A lighter touch is to let the kernel boot with IPv6 support and then use the sysctl facility to dynamically set a kernel variable to disable IPv6:

sysctl -w net.ipv6.conf.all.disable_ipv6=1

The example above disables IPv6 on all interfaces. You can also opt to just disable IPv6 support on specific interfaces:

sysctl -w net.ipv6.conf.eth0.disable_ipv6=1sysctl -w net.ipv6.conf.eth1.disable_ipv6=1

If your workload only uses IPv4 (in order to allow you to disable IPv6 support in the JVM) add the flag:

-Xaggressive

There is no reason for the JVM to spend any time trying to use IPv6.

13


Red Hat Java install - Firstly here are a couple of binaries that you will need to ensure that the Java ‘installanywhere’ script works:

# yum install libstdc++.ppc64# yum install rpm-build Then you can download the related package (in our case it’s Java7) into the install directory (/usr). Once that is complete, run the *.bin file:

Java 7# ./ibm-java-sdk-7.0-4.0-ppc64-archive.bin

Java 6# ./ibm-java-sdk-6.0-13.0-linux-ppc64.bin

This should enable you to simply follow the prompts from the install script, and as long as you have sufficient space all should configure correctly.

Check the version as follows:

# /usr/<install-path>/bin/java -version

15


Further Recommendations:

Turn off Hardware data prefetch with the Linux command "ppc64_cpu --dscr=1“.

The default page size is 64 KB page size, this will normally provide good performance gains over 4 KB pages. For further potential performance gains, enable 16 MB page size -

# cat /proc/meminfo

Predominantly the best performance is from 32-bit Java, that is unless the memory requirement of the application requires you to be running in 64-bit mode. Make sure libgcc.ppc and glibc.ppc are installed otherwise the 32bit Java will not include the correct libraries.

/root/.bashrc - set the path in your profile to Java 32 or 64 bit.

Most applications benefit from SMT. However, some applications do not scale with an increased number of logical CPUs on an SMT enabled system. One way to address such an application scalability issue is to change to a lower SMT mode with fewer logical CPUs; for example, changing from SMT4 to SMT2. Details of SMT will follow.

16


The server is now installed, below are a few commands to assist you in tailoring your environment.

POWER SMT on Red Hat.

How does Linux handle SMT modes, as IBM POWER7 processors support Simultaneous Multi-

Threading (SMT)?

To toggle the CPUs offline/online consistently across the cores:

# ppc64_cpu --smt=x

Linux numbers all of the CPUs sequentially across the cores:

# ppc64_cpu --smt=1

This will "turn off" cpu1,cpu2, cpu3 of the four CPUs associated with each core, leaving cpu0 running.

0,1,2,3 is the example for the first core. This repeats throughout the CPUs:

# ppc64_cpu --smt=2

This ensures that cpu0, cpu1 are "on", and cpu2, cpu3 are "off" for each core:

# ppc64_cpu --smt=4

This ensures that all cpus are "on“

With SMT=1 mode (aka SMT off), the Linux scheduler should shift to the appropriate scheduling mode.

18


Useful Commands.

If you want to get virtual machine configuration data, then the ’lparcfg’ file contains information relating to

the IBM Power Virtual Machine:[root~]# cat /proc/ppc64/lparcfg

19

The –i flag lists details of the LPAR configuration.

[root ~]# lparstat -iNode Name : server1.iic.hur.cdnPartition Name : server1Partition Number : 6Type : SharedMode : UncappedEntitled Capacity : 0.40Partition Group-ID : 32774Shared Pool ID : 0Online Virtual CPUs : 1Maximum Virtual CPUs : 4Minimum Virtual CPUs : 1Online Memory : 2051328 kBMinimum Memory : 1024Desired Variable Capacity Weight : 128Minimum Capacity : 0.10Maximum Capacity : 4.0Capacity Increment : 0.1Active Physical CPUs in system : 32Active CPUs in Pool : 32Maximum Capacity of Pool : 32.0Entitled Capacity of Pool : 290Unallocated Processor Capacity : 0Physical CPU Percentage : 40Unallocated Weight : 0Memory Mode : SharedTotal I/O Memory Entitlement : 2147483648Variable Memory Capacity Weight : 0Memory Pool ID : 65535Unallocated Variable Memory Capacity Weight : 0Unallocated I/O Memory Entitlement : 0Memory Group ID of LPAR : 32774Desired Variable Capacity Weight : 128

The interactive command version of this is ‘lparstat’ [root~]# # lparstat

System Configuration

type=Shared mode=Uncapped smt=Dedicated lcpu=1

mem=2051328 kB cpus=32 ent=0.40

%user %sys %wait %idle physc %entc lbusy vcsw phint

----- ----- ----- ----- ----- ----- ----- ----- -----

0.13 0.17 0.05 99.65 0.00 0.00 0.30 103475 0

[root]# lparstat 5 2

System Configuration

type=Shared mode=Uncapped smt=Dedicated lcpu=1

mem=2051328 kB cpus=32 ent=0.40

%user %sys %wait %idle physc %entc lbusy vcsw phint

----- ----- ----- ----- ----- ----- ----- ----- -----

0.05 0.00 0.00 99.95 0.00 0.00 0.05 130373 0

0.00 0.00 0.00 100.00 0.00 0.00 0.00 130475 0


Useful Commands continued:

CPU settings utility

Determine cpu frequency:[root ~]# ppc64_cpu --frequencymin: 4.31 GHz (cpu 2)max: 4.31 GHz (cpu 3)avg: 4.31 GHz

Obtain the number of cores installed:

[root ~]# ppc64_cpu --cores-present

Number of cores present = 1

Obtain the number of cores currently online:

[root ~]# ppc64_cpu --cores-on

Number of cores online = 1

[root@eg02ph05 ~]#

20

Linux Performance Customer Profiler Utility (lpcpu)

This script captures potentially interesting performance data (profile information, system information, and some system configuration information) on a single pass approach. Gathering all of this information at once allows the context to be understood and the performance profiling data to be analyzed.The script will check to ensure you have the basic tools installed on your system.

This script takes advantage of all of the "normal" performance tools used on Linux:

iostatmpstatvmstatperfmeminfotopsaroprofileperf

In addition, relevant system information is gathered with the profiler output into a single tarball saved on your system. By default the file is saved in /tmp.


RHEL Kernel image too large?

I have seen this problem a few times whilst trying to install (DVD/ISO or Network) RHEL on IBM POWER:

Welcome to yaboot version 1.3.14 (Red Hat 1.3.14-35.el6)Enter "help" to get some basic usage informationboot: linuxPlease wait, loading kernel... Elf64 kernel loaded...Loading ramdisk...Claim failed for initrd memory at 02000000 rc=ffffffff After some searching on Bugzilla I discovered my problem, and from this I did a little testing to resolve the issue which I have detailed below:

Boot the IBM POWER server and enter the ‘OpenFirmware’ prompt with selection 8 in the IBM menu, and run the following command at the prompt:

0 > printenv real-base

If you see the following the I-------------- Partition: common -------- Signature: 0x70 ---------------real-base 2000000 2000000

21

https://bugzilla.linux.ibm.com/show_bug.cgi?id=68713


Continued: RHEL Kernel image too large?

The issue is that firmware is expecting an image of 32MB, and in our example for RHEL6 it is 16MB (or smaller) so this is how to correct this issue:0 > setenv real-base 1000000

This sets the expected image to 16MB, which we can then verify:

0 > printenv real-base -------------- Partition: common -------- Signature: 0x70 ---------------real-base 1000000 2000000

Now perform a reboot:

0 > reset-all

The system should then boot off of the RHEL image and the installs should now work as expected. In this instance it was a virtual server so until the LPAR is removed from the HMC it will continue to work.

Ref:2000000 is 32MB1800000 is 24MB1000000 is 16MBc00000 is 12MB

22


The previous section covers a few changes that can be made in relation the operating system, and it can be difficult to find reference material and example systems to test comparably too. I would also recommend looking at the www.spec.org website.Searching in the ‘SPECjEnterprise2010’ section will also yield some useful results.

24

https://www.spec.org/


You can view all of the results or search for a direct example:

The website helpfully includes all of the related environmental

information, such as DB2, JVM,

ulimits, and other parameters.

25


Hardware & Facility Access

Special discountedISV Developer Box

Purchase or lease PowerLinuxservers for development or

in-house use at specialdiscounted rates.

Virtual Loaner Program (VLP)Reserve and provision IBM Power

Systems online for migration,porting, testing and solution

demonstrations.ibm.com/systems/vlp

IBM Innovation Centers andLinux Technology Centers

Get remote and on-site accessto IBM Power Systems with

dedicated support at no charge.Build customized environments,

conduct optimization, or host clientworkshops.

IBM Innovation Centersibm.com/partnerworld/wps/servlet/ContentHandler/isv_com_tsp_iic_overview

IBM Linux Technology Centersibm.com/linux/ltc/index.html

SDK & Tools

IBM Installation Toolkitfor PowerLinux

A simple tool to install andconfigure Red Hat Enterprise Linuxand SUSE Linux Enterprise Server,

open source workloads, as wellas IBM value-added software on

Power Systems.ibm.com/webapp/set2/sas/f/lopdiags/installtools

Software Development Toolkitfor PowerLinux

All-in-one solution integratingLinux and IBM tools such asOProfile, Perf, Valgrind, and

Autotools for Linux and FDPR forIBM. You can even use the EclipseIDE directly on the Power Systems

server or in x86_64 clients forremote development.

ibm.com/support/customercare/sas/f/lopdiags/sdklop.html

Technical Assistance

ISV Application Porting AssistanceGet help to port and enable your

applications to IBM Systemsplatforms.

ibm.com/partnerworld/mem/support/trs_develop_rapidport.html

ChiphopperAssistance to port, test, and

support your existing Linux x86applications on other IBM Systemsand middleware platforms—at no

cost.ibm.com/isv/go/chiphopper

IBM Migration FactoryAddress your customers’ concernswhen migrating from Solaris/HPUX/

x86 to PowerLinux withmigration expertise andmethodologies from IBM

ibm.com/systems/migratetoibm/factory

IBM developerWorks – A Technical Community for PowerLinux

Get access to blogs, wiki, andmessage boards for recent

news, discussions, events andassistance.

ibm.com/developerworks/group/tpl27


Technical Support

• Face to face or remote

• Proof of concept

• IBM products consulting

• Prototyping

• Testing

• Porting IBM platforms

• Validation

• Integration

• Performance/scalability testing

• Sizing guides

• SmartCloud

Education and events

• Extensive program of technical education on the latest IBM technologies

• LoB events focused on:– Maximising your Relationship with IBM– Industry focused seminars– New business models

Solutions Laboratories

• Smarter Planet

• Smarter Cities

• Smarter Commerce

• Smarter Energy

• Smarter Government

• Intelligent Operations Centre

Infrastructure (onsite and virtual)

• Latest hardware and software

• Industry and cloud solutions

• Technical expertise

28

© 2013 IBM Corporation29

Technology and ServersSystem x, (incl. Nehalem)System i inc i520, i570,

IBM POWER inc POWER6 570, POWER7 740/50/770/80, &

POWER7+ 760/70/80Storage inc SVC, SFS,Tape, SONAS,

V7000 DS8800 & XIVFibre connected

Blades, & SmartAnalyticsAll Middleware engagements

accommodated, including most SWG acquisitions (e.g. iLog, FileNet,

SPSS, Cognos, etc.) PureSystems(PureApplication, PureFlex

& PureData)

World-class facilitiesLocated in Hursley Park, site of IBM’s largest

non-US Development Laboratory6 project / engagement rooms

3 conference rooms2 Dedicated classrooms, + access to 4 more

Hursley Innovation TheatreSmarterPlanet Lab, Retail Lab,

E&U Lab, Local Govt. Lab, Sustainability Demos

InfrastructureHigh bandwidth infrastructure

Private network separate from internal IBM network

VPN Support to allow remote workingDual SAN Fabric to dedicated Storage

Large number of load generation machinesSoft switching KVM system to allow customers

to work anywhere in centre

Skills and PeopleSkilled “development” knowledge and

experienceWide range of technical expertise across

SWG and STGGreat networks inside IBM’s technical

communitiesTechnical events management and

coordination

29


42 Globally Networked Locations

Results• Build skills• Build solutions• Close deals and improve your bottom line• Execute marketing tactics

Staff• Full-time IBM technical experts for hands-on

assistance and knowledge transfer• Access to IBM marketing and sales teams

Classrooms• Technical workshops and seminars• Business seminars and collaboration events

Meeting rooms • Customer briefings• Sales closure meetings• Business Partner Demo rooms

Technology supportAccess to the latest IBM hardware and software:• Onsite at the Centers• Remotely via secure Virtual Private Network (VPN)

North America (6)• Austin• Chicago• Dallas• San Mateo• Toronto• WalthamLatin America (2)• Sao Paulo• Mexico City

Europe (13)• Copenhagen• Dublin• Stockholm• Hursley• London• Stuttgart• Zurich• Amsterdam• Barcelona• La Gaude• Tel Aviv• Milan• Paris

CEEMEA (12)• Bratislava• Bucharest• Budapest• Casablanca• Istanbul • Johannesburg• Kiev• Ljubljana• Moscow• Prague• Warsaw• Nairobi

Japan (1)• Tokyo

Asia Pacific (8)• Bangalore• Kuala Lumpur• Manila• Seoul• Shanghai• Sydney• Vietnam• Ho Chi Minh

30


The PowerLinux Community(developerWorks)

Power Systems Linux Portal

(Product Information)

www.ibm.com/developerworks/group/tpl/

www.ibm.com/systems/power/software/linux/

@thinkpowerlinux plus.google.com/communities/100156952249293416679


Sessions:– #1: Exploiting Virtualisation on IBM Power Systems with

PowerVM– #2: VIOS - how to get going – #3: Controlling processor resources in virtualised partitions– #4: Deeper dive into Active Memory Sharing– #5: Virtualisation Best Practices– #6: Deeper dive into Shared Storage Pools and Thin

Provisioning – #7: Virtualisation and the world of 10Gbit Ethernet– #8: Deeper dive into Active Memory Expansion– #9: VIOS maintenance– #10: Active Memory Deduplication– #11: IBM i Suspend/Resume– #12: Dual VIOS Upgrade walk-through– #13: Shared Storage Pools ... from Experience– #14: IBM i Live Partition Mobility– #15: Capture and Deploy partitions– #16: Virtual Partition Manager for IBM i

– #17&18: Updating Power Systems, I/O and HMC– #19&20: Power7 Affinity and Performance– #21: Power Advisors (VIOS, LPAR and Java)– #22: PowerSC – What/Why/How– #23: Dynamic Platform Optimizer (DPO)– #24: Removable Disk Cartridge (RDX) on Removable

Mass Storage (RMS)– #25: Electronic Service Agent– #26: Active and Dynamic Systems Optimizer

(ASO/DSO)– #27: Shared Storage Pools Phase 3– #28: Whole POWER Machine Monitoring– #29: Monitoring POWER/AIX/VIO with IBM Tivoli

Monitoring– #30: Tricks of the Power Masters– #31: External Storage and IBM i – sizing/modelling– #32: Power Systems Configuration Best Practices

32

@JyotiDodhia

A series of technical webinars on Power Systems (including virtualisation):

InformalUsually about an hour eachMany include demos

Aimed at:Technical audience - operators, systems administrators and technical specialistsThose using / planning to use IBM's Power based systemsCustomers / Business Partners / IBMersUK & Ireland audience initially, but spread to other countries

More info, replays, and registration at:http://tinyurl.com/PowerSystemsTechnicalWebinarsor send an email to [email protected]

linux on power – hints and tips

Documents

ibm power toolbaseurl

ibm power tips

ibm corporationinstalling

ibm corporationsetting

ibm corporationupdate

ibm corporationibm power7

power systems

leveraging ibm developerworks