jonathan carter, general mills jeremy rempel, interloc solutions … · 2017-10-07 · • use load...

OPTIMIZING IBM MAXIMO AT GENERAL MILLS SESSION 1150

Jonathan Carter, General Mills Jeremy Rempel, Interloc Solutions

www.interlocsolutions.com

General Mills

Located in Minneapolis, Minnesota, General Mills is among the world's largest food companies and markets some of

the world’s best-loved brands



Nourish everyone by making lives healthier, easier, and richer

Healthier by delivering important nutrients and increasing people’s intake of whole grains, by providing foods that contribute to a heart-healthy diet, and by helping people manage their weight Easier with convenient meals and packaging, and foods that provide nourishment on the run Richer with foods that add fun or help to create a special moment with family and friends

Maximo at General Mills Large install base • >3000 active users • Approximately 400 simultaneous users at any given time

IBM Maximo Enterprise Asset Management 6.2.3 IBM Websphere eCommerce Adapter SAP and Mobile Interface


Website (application and reporting) under load stress, causing slow response time Business partners did not want to use the system Business process delays • Reports not completed on time due to heavy system

loads • Delayed month end financial transactions

It’s too slow!

The system failed …. Again!

Maximo challenges


System tightly coupled • If one component failed, the entire

system has to be restarted System restart time consuming • One hour to restart the entire

application. Poor user experience leads to high business impact

Do low impact quick fixes – make business partners happy • Get targeted low impact fixes in quickly • Show benchmarks and metrics • Start with easy wins with low system impact that deliver

high performance

Guiding principle

Fixes

System Stability

Find the right balance – Every fix

can destabilize a

system


The way things were Servers are on physical hardware GZIP performed on User Interface nodes nodes

Small clusters No Actuate cluster

GZIP

F5 Load Balancer

Actuate

UI Admin

UI UI

UI DS

UI UI

UI UI

UI

RM1 RM1

UI

RM1


Where is performance failing?

Network

• Check server response time • Ensure data is being cached • Turn off HTTP compression filter in web.xml

Hardware

• Set up clustering as appropriate (> 25 users) • Clusters can be risky -- plan and test before cutover

• Increase JVM settings only with proper settings • Turn on SQL logging and optimize • Index tables appropriately

Application

• Ensure Maximo is configured appropriately (validate relationships, Java customizations and custom applications)

• Use Firefox rather than IE when possible • Ensure each system has a minimum of 2GB of RAM and at least a 2GHz CPU • Design scripts and use cases to understand what is slow to document and define metrics


Tracking down performance at General Mills

Step 1: Identify and Isolate the issues

Step 2: Resolve the issues

Step 3: Implement the solution


Step 1: Identify and isolate the issues Identify and validate performance issues objectively Turn on Maximo performance logs • Use the application server reports in WebLogic • Use the reports available from the operating system

(Windows) Interview key users • Understand and document use cases, data that is the cause

of the performance issues • Validate the performance issues objectively

Use tools such as Charles/Fiddler • Benchmark specific use cases • Determine if the issue is on the server, network or client

Work with database team • Use database tools such as Oracle to determine frequently

run and slow queries


Implement the new cluster • Performed tuning and load testing using Load Runner

Test and re-implement custom relationships Experiment with Oracle table partitioning • Some Work Order tables were 40m rows

Re-factor Java customizations Move work off of the application server to dedicated appliance • Moved GZIP and caching onto F5 load balancer to reduce

load on application server

Step 2: Resolve the issues


Step 3: Implement the solution Balance risk of change vs. benefit • Refactoring Java and Oracle table partitioning led to

performance gains but were too high risk

The criteria to implement changes in a legacy system is much different than a new implementation Every change to Maximo requires: • Support team and expert users to validate functionality

before any change to a production system • Complete testing/release plan that needs to be rehearsed,

timed and validated

Evaluate the maintenance window procedures • Migrated from a large batch file to a Microsoft Opalis

workflow, reducing downtime window significantly


Identified infrastructure changes that increased throughput by 5x under load • Implemented a 12 node horizontal Web Logic cluster

suitable for 600 concurrent users with no downtime or regressions

• Simplified MEA implementation reducing 8 queues to the standard 3

• Implemented a 3 node Actuate cluster with no downtime or regressions

• Identified and resolved several critical security issues relating to infrastructure

• Moved customization's into an automated build system as opposed to hundreds of individual class files managed manually.

• Identified two cases where the production code was different than version-controlled source code

• Developed, implemented enhanced build system that significantly reduced issues relating to change management

Success at General Mills


Success at General Mills

Network Ensured


Moved servers from physical to virtual hardware • Set up server cluster from scratch with documentation for

clear understanding • Re-sized clusters to handle larger load • Performed load testing and tuning to measure performance • Put in a new production system based on test system • Rebuilt architecture to handle the same users with faster

throughput Started at top and move down • Network

Ensured response time was appropriate • Hardware

Re-sized clusters Moved work from Web Logic to F5 Load Balancer

• Application Analyzed Individual actions Designed scripts and use cases to understand what is slow

Full optimized system


Network Ensured

Revised the system to fully optimize Maximo • Identifying infrastructure changes • Increased system availability • Reduced the number of planned outages • Increased throughput by 4x during high loads, and

improved response time by more than 75%.

New & improved implementation Servers are on virtual hardware GZIP on F5 Load Balancer 4 UI servers Actuate slaves to balance workload

x4

F5 Load Balancer

Actuate Master

Admin

MEA

DS

UI UI

UI UI

UI

RM1 RM1

UI

GZIP Cache

Actuate Slave Actuate Slave


Reproduce the issue and verify that the action is consistently slow • Document the exact steps, fields and actions involved • Work with the local IT group onsite or remote directly

into the users workstation

Identify and isolate the issue • Obtain a detailed list of actions, fields, applications to

reproduce the issue • Understand if the performance issues are related to

server, network or client

4 performance tuning tips


Use tools to track a single request from client to server • Use Fiddler/Charles from within the clients

environment • Determine the root cause to embark on the right

solution

Understand and quantify how the system will perform when under load • Use load testing tools such as HP Loadrunner and

Jmeter • Monitor operating system, application server, database

and network to determine if any of the components is near capacity.

• Analyze the load test script to determine if and how the performance falls as the system is placed under load.

4 performance tuning tips (con’t)


A winning solution!


Maximo Stabilization project nominated for Service Excellence award: • Focused on the stabilization and optimization of the

existing system instead of adding additional functionality • Listened to the customer • Analyzed Remedy tickets to understand the long term

direction the business owners needed to be supported in.

Results: • Added 3x the user load capability • Eliminated system downtime • Reduced system reboot time from 40 minutes to 10

minutes • Fully documented system • Preproduction and production environments are in synch

Both our business partners and Information Systems were winners on this project The business owners have a system that has: • Increased uptime, capacity and stability. • Ability for many users to access the system

simultaneously

Information Systems has a system that has: • High uptime • Full implementation documentation • Scalability (virtualized servers) • Large reduction in remedy incidents

Everybody wins

It’s so easy to do my job!


The team focused on the user Shored up existing system Delivered what the customer wanted – a system that works for them instead of in spite of them Issues that were more than 2 years old were resolved

Realignment with the customer makes everyone happy

Do the right things for your users


Recommendations

Planning an upgrade? Old system in service more than two

years? Performance an issue?

Consider an in-place infrastructure

overhaul – it is doable and beneficial for your users


100% Maximo, AAA Technically Accredited, EAM focused IBM Premier Business Partner

Interloc Solutions

A Maximo Consulting Services Company, with: ─ Proven functional and technical experience ─ Vast experience in Maximo

implementations across industries ─ Experts in implementing custom mobility

solutions ─ Authorized to provide IBM Maximo Level 1

/ Level 2 support ─ Innovative products that extend your

Maximo implementation www.interlocsolutions.com

Check out our innovative tools and products at Booth S214 Attend our ExpoTheater presentation to win an iPad Register for a one on one consulting session with one of our experts here at Pulse Sign up for our monthly mailing list to get our latest blog posts and white papers

Take the next step with Interloc Solutions


Attend or download any of these other sessions:

Take the next step with Interloc Solutions

Maximo Work Management on the iPad Tuesday, March 6 1:00 PM – 1:20 PM Eddy Larson, ATK Aerospace Grahame Low Choy, Interloc ID: 2117

Calgary Airport Authority reaches new heights with IBM Maximo Sunday, May 4, 2012 3:30 PM - 4:15 PM Peter Uy, Calgary Airport Joe Mendoza, Interloc ID: 2096

Perspectives on Asset Management Best Practices in the Rail Industry Sunday, May 4, 2012 5:00 PM – 6:00 PM Ravi Misra, BART ID: 2097

While Nasittuq secures North America; IBM Maximo secures Nasittuq Tuesday, March 6 5:00 PM – 6:00 PM Daniel Gagnon, Nasittuq Joe Mendoza, Interloc ID: 1338

Wednesday, Mar 7, 2012 2:00 PM – 3:00 PM Eddy Larson, ATK Aerospace Sandy Allred, ATK Aerospace Dale Deibert, NEON Grahame Low Choy, Interloc ID: 2066


Perspectives from Two Healthcare Providers Sunday, May 4, 2012 5:15 PM - 6:00 PM Joe Mendoza, Interloc ID: 2100

Thank you! Questions?


[email protected]

jonathan carter, general mills jeremy rempel, interloc solutions … · 2017-10-07 · • use load...

Documents