Enterprise Web Architecture and Performance

Shennon Shen & Scott Carey

--- Plumtree Software Inc.

Who are we

Shennon Shen Ph.D.•Lead Performance and Scalability Engineer

•Plumtree Software

Scott Carey•Sr. Performance Engineer

•Plumtree Software

Agenda

Enterprise Web Architecture

Performance

Tuning and Monitoring

What is the Enterprise Web

The Enterprise Web

The Enterprise Web

The Enterprise Web integrates diverse back-end systems

The Enterprise Web

Enterprise Web applications can automatically include services such as collaboration and content management

Develop New Services In Any Environment

Developers can create components of Enterprise Web applications on any platform and integrate them using Web Services

Manage Applications Through Scalable Framework

Enterprise Web applications are secured, delivered and managed through the portal’s application spaces, tailoring each application to its audience

Enterprise Web Architecture

Enterprise Web Suite Architecture

Cross-platform: deploy with your existing platforms

• Heterogeneous environments: Java, COM, .Net, Perl, Cold Fusion

• HTTP-based Web Services model for integration points

Fault tolerant: deploy distributed applications with confidence

• Distributed components for isolation of network, application faults

• Embedded load balancing, failover on integration tier

Highly scalable: deploy to many users cheaply

• n-Tier architecture serves hundreds of thousands of users

• Proven linear scalability through Web farm architecture

• Parallel orchestration of remote services via Parallel Portal Engine

The Enterprise Web creates user experiences from many components and applications running on many platforms.

Portal Server Must Be 100 Percent Reliable

Portal Server handles end-user requests

Portal Server cannot break with the weakest link

Isolated from faults in back-end systems

Isolated from faults in network

Isolated from bad portlet code

Portal Server scales linearly with additional hardware

Automation Server Manages Scheduled Tasks

Automation Server handles asynchronous processes

Crawling for new content

Synchronizing users from external sources

Publishing content on a schedule

Deleting broken, expired links

Automation Server protects Portal Server CPU

Multiple servers support round-robin load balancing

Database and Search Servers Store Metadata

Databases can be clustered for high availability

Search server stores search index

Search server supports replication for high availability

Applications Scale Individually

Content and Collaboration can run on separate servers in high use scenarios

Uploaded content resides in a scalable repository

Extensibility Through HTTP-Based Web Services

Portlets: build applications by adding services to personal and community pages

Search Services: search external indices in parallel with the portal

Crawler Services: add content from external repositories

Authentication Services: authenticate users against external directories

Profile Services: import user attributes

Performance and the Enterprise Web

Performance Challenges in the Enterprise Web Environment

Loosely Coupled n-tier Architecture

Not just a Web Application

Complex System of Layers and Dependencies

• E.g. Caches, queues, latencies, updates

• Performance appearance vs performance reality

Performance and Enterprise Web Components

Understand how each hardware and software component fits into the architecture

Portal components

Partner Components

Custom Applications

UI Customizations

Hardware and Network Infrastructure

Enterprise Web Uniqueness

Deployments vary tremendously

Centralized vs. Globalized

High Bandwidth vs. Low Bandwidth

Many Users, Many Groups vs. Many Users, Few Groups

Typical User Visits Daily vs. Typical User Visits Weekly

Typical User is an Employee (internal) vs. Typical User is a Customer (external)

No “One Size Fits All” Performance Solution in the Enterprise Web

System Level Tests

Test Database with Users, Documents, and Objects

Randomized User Scenarios that touch many parts of the Portal

LoadRunner Performance, Scale, and Stress tests

Data Dependency

Performance can vary widely depending on:

Number of Documents

Size of Documents

Security on Documents

Number of Users

Complexity of User and Group relationships

Number of Portlets, Communities

Environment

Unique combinations of the above

Data Dependency

Examples:

Document size and security relationships affect Search and Database performance.

User and Group size and complexity affects all security queries.

Test database with 100,000 Documents and 50,000 Users

• Database CPU use can vary by a factor of 6 on Directory Folder page loads depending on the relationship between Cards, Folders, and Security.

• Search CPU use can vary by a factor of 5 depending on the size of the documents and the number of search results returned per average query.

Customer Environments vs. Performance Test Environment

Performance Test Environment

• Controlled Environment

• Static, non-evolving Database

• Not constrained by external factors

Customer Deployments

• UI Customizations

• Various SSO solutions

• Custom applications, portlets

• 3rd party applications, portlets

• Every network is unique

Example of Performance Issues

Examples:

Slow portlets or back end system cause long page load times and starve the portal of available execution threads

Portal Server is tuned incorrectly, limiting server capacity or increasing response times

SSO product on insufficient hardware or configured incorrectly, delaying all logins for several seconds.

Network Bandwidth causes long page load times and document load times. Consider HTTP compression.

Tuning and Monitoring the Enterprise Web

Enterprise Web Tuning

Many components to possibly tune

Each component has tuning possibilities, but not every component needs tuning

Which components need tuning is deployment specific

Enterprise Web Tuning

Synchronous and Asynchronous n-tier distributed architecture

Tuning the Enterprise Web is similar to ordinary Web Application tuning, however determining what needs to be tuned is not easy

Tuning typically adjusts the following at each node:

• Concurrent request capacity (worker threads, connections)

• Resource consumption (RAM, Disk)

• Caching

• Disabling unused infrastructure features

Enterprise Web Tuning Example: Tuning Memory Usage

Java App Servers: Java heap size tuning:

• More Memory is not always better

• Java heap configuration can strongly impact performance.

.NET Web Applications

• Heap Size is configured automatically

• However Web Applications are restarted by .NET if they use too much memory

• .NET machine.config file <system.web> <processModel> memoryLimit attribute controls this behavior

What to Expect from Tuning

Don’t expect much unless the component is a known performance bottleneck

In short, use the “80/20” rule

Enterprise Web Monitoring

Monitoring the health and performance of the Enterprise Web is critical

Occasionally analyze data to locate usage trends that may require attention

Use historical data to rule out causes of performance issues that arise

Challenges in Enterprise Web Monitoring

Monitoring is not always easy

Each application or component may have different monitoring interfaces

Database, application server, and network specialists involved are often on different teams with varied priorities

Every system integrated may have a different expert

How to collate all the data?

Who knows enough about the system as a whole to interpret it?

Monitoring Sources

Plumtree logs

Performance Monitor Counters (Windows)

Top, vmstat, sar, and other UNIX tools

IIS and App Server logs

System Event Logs

Log analysis tools

Search Server logs

HP OpenView, IBM Tivoli, CA Unicenter, others

Summary

Understand the enterprise web architecture and its components

Understand the layers and dependencies of your deployment

Locate problems by identifying what is not the problem first

Monitor Enterprise Web usage and trends

Tune only what needs tuning

Q/A

Questions?

Enterprise Web Applications

Software that defines

• Information

• Processes

• Services

Single user experience

Difficult to tailor to audiences

Software built from

• Components of traditional applications

• Reusable services from different app servers

• Foundation services: content, search, collab

Many experiences, shared platform

Rapidly tailored to audiences’ needs

Traditional Application Enterprise Web Application