1© 2016 Rogue Wave Software, Inc. All Rights Reserved.
1
Top open source lessonsfor every enterpriseEpisode 4:
Top issues in the top enterprise packages
2© 2016 Rogue Wave Software, Inc. All Rights Reserved.
2
Bill CrowellEnterprise architect, OpenLogic supportRogue Wave Software
Vince CoxEnterprise architect, OpenLogic supportRogue Wave Software
Presenters
3© 2016 Rogue Wave Software, Inc. All Rights Reserved.
3
Poll #1What percentage of your mission critical software is open source?
A: 0 to 25%B: 26 to 50%C: 51 to 75%
D: 75%
4© 2016 Rogue Wave Software, Inc. All Rights Reserved.
4
1. Introduction2. Setting the context3. Top issues: middleware + runtimes4. Top issues: databases5. Top issues: security6. Conclusion7. Q&A
Agenda
5© 2016 Rogue Wave Software, Inc. All Rights Reserved.
5
Who wrote LevelDB and what version of ActiveMQ did it debut in?
What percentage of web server market share does Apache HTTP Server hold?
Pop quiz
6© 2016 Rogue Wave Software, Inc. All Rights Reserved.
6
Introduction
What problems do our clients commonly run into?
From our experience…• ActiveMQ High Availability Shared Databases Using NFS• PostgreSQL Database Performance• Request Header Vulnerabilities
7© 2016 Rogue Wave Software, Inc. All Rights Reserved.
7
Poll #2What type of ActiveMQ persistence store do you use?
A: KahaDBB: LevelDB
C: Replicated LevelDBD: We don’t use message persistence with ActiveMQ
E: We don’t use ActiveMQ
8© 2016 Rogue Wave Software, Inc. All Rights Reserved.
8
Top issues:middleware +
runtimes
9© 2016 Rogue Wave Software, Inc. All Rights Reserved.
9
Scenario #1
Observing crashing or unresponsive broker
Typical configuration/requirements• Master/slave network broker setup with NFS mount• NFSv4 (SoftNAS) cloud-based shared file system available
with Amazon Web Services EC2 or GlusterFS• Often a high-throughput requirement (10k messages/second
of 1-3kb message size)
One of the following problems…• Master dies resulting in “no master”
scenario• Slave prematurely claims lock resulting in 2
master brokers leading to message loss• Continuous master/slave re-election• Increasing CPUs/memory doesn’t help• Non-existent prior to production
10© 2016 Rogue Wave Software, Inc. All Rights Reserved.
10
Scenario #1 solution
Use SAN, Replicated LevelDB, or Pluggable Storage Lockers
Three solutions• Use block-level iSCSI driver with Storage Area Network (SAN)• Master-slave for HA and Replicated LevelDB managed by
Zookeeper• Pluggable Storage Lockers
More points• Inherent flaws in OS-level filesystem locking mechanism• Exclusive file locks work great with a SAN but is most
expensive• Replicated LevelDB requires a more configuration and a
quorum of nodes (replicas / 2 + 1)• Pluggable Storage Locker/Lease Database Locker
11© 2016 Rogue Wave Software, Inc. All Rights Reserved.
11
Scenario #1 solution
Lease Database Locker
Points• Master must renew lease before lease expires• The lease period can be configured• If not renewed, then the slave takes ownership of the
lease becoming the new master
More points• Leased locks can survive database replica failovers• Can be used with any JDBC-compliant database• Make sure to uniquely name your brokers• Keep master/slave clocks synchronized with NTP
service• Uniquely name your brokers. Use connection pooling
12© 2016 Rogue Wave Software, Inc. All Rights Reserved.
12
Poll #3Do you use PostgreSQL?
A: We don’t use PostgreSQLB: We don’t use PostgreSQL but plan on using it in the future
C: We use PostgreSQL as a mission-critical application databaseD: We use PostgreSQL as part of another open source project
E: We are migrating away from PostgreSQL
13© 2016 Rogue Wave Software, Inc. All Rights Reserved.
13
Top issues:databases
14© 2016 Rogue Wave Software, Inc. All Rights Reserved.
14
Scenario #2
Database performance is poor
Symptoms• Sorting and querying take a long time• One particular query or web page is hanging the
database• “Sorry, too many clients already” or connection pool
is full
Where do I start?• Localize if possible: Data center, network,
database or application server?• When did it start• What changes took place?
15© 2016 Rogue Wave Software, Inc. All Rights Reserved.
15
Scenario #2 solution
Identify and analyzeQuick checks• top with ‘c’ command shows process ID, CPU, and
memory utilization• ”iostat –x –m 5” reveals disk IO wait times• ELK = Elasticsearch + Logstash + Kibana• “EXPLAIN ANALYZE <SQL>” shows execution time and
table scans
Tools• psql: SELECT pid, datname, usename, query FROM pg_stat_activity;
pid datname usename query42102 jboss jboss SELECT pid, datname, usename,
query FROM pg_stat_activity;
42103 jboss jboss SELECT video FROM news where...;
16© 2016 Rogue Wave Software, Inc. All Rights Reserved.
16
Scenario #2 solution
Identify and analyze
Tools• pg_stat_statmentswww.postgresql.org/docs/current/static/pgstatstatements.html
Note: pg_stat_statements requires more shared memory
postgresql.conf:shared_preload_libraries = ‘pg_stat_statements’
Server restart is required after enabling the shared library
Reset statistics: select pg_stat_reset();
17© 2016 Rogue Wave Software, Inc. All Rights Reserved.
17
Scenario #2 solution
Identify and analyze
ToolsMore on PostgreSQL performance: www.craigkerstiens.com/2013/01/10/more-on-postgres-performance/
SELECT (total_time / 1000 / 60) as total_minutes, (total_time/calls) as average_time, query FROM pg_stat_statements ORDER BY 1 DESC LIMIT 100;
The QueryTotal Query Time (in minutes)Average Time (in milliseconds)
18© 2016 Rogue Wave Software, Inc. All Rights Reserved.
18
Scenario #2 solution
Identify and analyze
Tools• pbBadger: dalibo.github.io/pgbadger/
• Requires Perl
pgBadger: dalibo.github.io/pgbadger/postgresql.conf• log_min_duration_statement = 0• log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d ’• log_checkpoints = on• log_connections = on• log_disconnections = on• log_lock_waits = on• log_temp_files = 0After making changes run: SELECT pg_reload_conf();
19© 2016 Rogue Wave Software, Inc. All Rights Reserved.
19
Scenario #2 solution
Implement and test (…and repeat...)
Supporting points• What if I don’t find the offending SQL or table? • Start looking at the database configuration.• Did you benchmark and tune?
More points• Most parameters are automatically adjusted.• We find that many people make minimal changes to the default
configuration. • As always, remember to test any configuration changes in a non-
prod environment first, and implement changes incrementally. • Do not make numerous configuration changes all at once. • Systematically measure your performance tests. Use a tool like
pgbench.
20© 2016 Rogue Wave Software, Inc. All Rights Reserved.
20
Scenario #2 solution
Implement and test (…and repeat...)
Configuration Settings in postgresql.conf• max_connections: Maximum simultaneous connections to
the database (default is 100).• shared_buffers: database cache size (default is 128MB).
25% of total RAM. Windows-based should be 64-512MB.• effective_cache_size: Tells the query planner how much
RAM there is to execute. 50-75% of total RAM.• work_mem: Used for complex sorts (default is 4MB). Check if
it is uncommented and what the value is.• maintenance_work_mem: Amount of memory for background
processes for pgdump, pgrestore, vacuum, indexing, and bulk loads (default is 64MB). 256MB-1GB for large databases.
• checkpoint_segments: Maxiumum # of log file segments between WAL checkpoints (default is 3).
21© 2016 Rogue Wave Software, Inc. All Rights Reserved.
21
Scenario #2 solution
Implement and test (…and repeat...)
Configuration Settings in postgresql.conf• wal_buffers: Write ahead log buffer used for writing a
transaction to disk.
What should I really set these values to?• pgTune: pgtune.leopard.in.ua/
SELECT name, current_setting(name), SOURCE FROM pg_settings WHERE SOURCE NOT IN ('default', 'override'); PostgreSQL 9.0 High Performancewww.amazon.com/PostgreSQL-High-Performance-Gregory-Smith/dp/184951030X/163-3733534-8577963
22© 2016 Rogue Wave Software, Inc. All Rights Reserved.
22
Poll #4Are you using SSLv3?
A: YesB: No
23© 2016 Rogue Wave Software, Inc. All Rights Reserved.
23
Top issues:security
24© 2016 Rogue Wave Software, Inc. All Rights Reserved.
24
Scenario #3
Pen testing reveals the application server is vulnerable
Supporting points• Cache-control, Pragma, Expires headers are not
set• X-Powered-By reveals the application server
typeMore points• A proxy can inject an intermediate
page and compromise the site• Certain versions of JBoss allow
little to no control of request header manipulation
25© 2016 Rogue Wave Software, Inc. All Rights Reserved.
25
Scenario #3 solution
Request header manipulation strategiesSupporting points• Proxy requests through Apache HTTP Server before
sending them to the application server• Mod_expire could be used to explicitly set them. Does
not guarantee header will be set• Set org.apache.catalina.connector.X_POWERED_BY to
falseMore points• Using Apache HTTP Server’s mod_header module is easiest and
preferred option• mod_headers module allows manipulation before and after the
request• If this is not an option, then a filter can do the same function• There should never be a situation where an application server is
internet facing without the protection of a web server in the DMZ
26© 2016 Rogue Wave Software, Inc. All Rights Reserved.
26
Scenario #3 solution
Request header manipulation strategiesDMZ• The front end should always be in the DMZ• This should never be an application server • Web servers are “far more” capable than
application servers in this role
Best practice• Traditional 3-headed monster, Web/App/DB should always
reside at unique layers in the infrastructure• There should be horizontal and/or vertical separation between
them at all times• This provides greater opportunities for other security gear to
potentially sniff and sort out other possible vulnerabilities. Like HIDS/NIDS, etc.
27© 2016 Rogue Wave Software, Inc. All Rights Reserved.
27
Scenario #3 solution
Request header manipulation strategiesHTTPD Mod_expires• Mod_expires can set the max_age and
expirations• ExpiresByType text/html M604800 (Expires a
week after modification)HTTPD Mod_Header• Much more flexible than mod_expires• Header echo ^KC (copies all request headers starting with KS
to response headers• Header set TESTHEADER “Hi, The request ran in %D uSEC’s”• Env variables can be set as well. Much more capable than
mod_expires
28© 2016 Rogue Wave Software, Inc. All Rights Reserved.
28
Scenario #3 solution
Header manipulation examplesHTTPD Mod_header Example• <ifModule mod)headers.c>• Header set DateTimeHEader “%D %t”• </ifModule>
Header merge Cache-Control no-cacheHeader merge Cache-Control no-store
YieldsCache-Control: no-cache, no-store
29© 2016 Rogue Wave Software, Inc. All Rights Reserved.
29
Scenario #3 solution
Header manipulation examplesHTTPD Mod_header Example• Header set Set-Cookie testcookie “expr=-z %
{req:Cookie}”• Header merge Cache-Control no-cache env=CGI• Header merge Cache-Control no cache
end=NO_CACHE• Head merge Cache-Control not-store
env=NO_STORE
Non-exclusive conditions. If all are set: Cache-Control: no-cache, no-store
Using append instead of merge would cause a duplicate no-cache message
30© 2016 Rogue Wave Software, Inc. All Rights Reserved.
30
Scenario #3 solution
Header manipulation examplesHTTPD Mod_header (Powerful module)• Add, Append, echo, edit, merge, set setifempty,
unset• Allows setting of environment variables to use
are triggers• Expressions can be set as well for more
additional control
Examples prove that web servers are well suited for this type of work.
31© 2016 Rogue Wave Software, Inc. All Rights Reserved.
31
Scenario #3 solution
Request header manipulation strategiesJBoss Wildfly : Header Manipulation• You shouldn’t, but if you absolutely must• You could modify system properties to override
certain values. Or set a filter.• Newer versions of JBoss provide header
modificationExample• <system properties>• <property
name=“org.apache.coyote.http11.Http11Protocol.SERVER” value=“someserver”/>
• <system-properties>• For each version of Jboss, you may need to use the CLI to
determine if the necessary keys are available in that version
32© 2016 Rogue Wave Software, Inc. All Rights Reserved.
32
Scenario #3 solution
Request header manipulation strategies
JBoss Wildfly : Header Manipulation• You can, but apache httpd is still a good accomplice• Remove the following the alter the headers• <filter-ref name=“x-powered-by-header”/>• Performed in the undertow segment of
standalone.xml
Example• As newer versions of Wildfly emerge there is a bit more control
of headers• This does not remove the need to have a web server in front• A reverse proxy is is vital piece of application server security
33© 2016 Rogue Wave Software, Inc. All Rights Reserved.
33
Scenario #3 solution
Request header manipulation strategies
JBoss Wildfly : Header Manipulation• CLI command structure will provide a view of current
header manipulation capabilities Wildfly has• More flexibility in each new release• Wildfly’s capabilities are far behind that of apache’s• Having this level of control at your reverse proxy just
makes the most sense• These features have been requested for quite some
time from the user base.• Not a replacement for a frontend DMZ based web
server
34© 2016 Rogue Wave Software, Inc. All Rights Reserved.
34
Who wrote LevelDB and what version of ActiveMQ did it debut in?Written by Google: Jeff Dean who also has contributed to MapReduce and Google TranslateSanjay Ghemawat who also contributed to MapReduce and iCal which is a popular calendar application in Unix/Linux.
LevelDB debuted in ActiveMQ 5.10.0 but was not production ready.What percentage of web server market share does Apache HTTP Server hold?July 2016Apache HTTP Server: 52%nginx: 30.5%Microsoft: 12%
Pop quiz answers
35© 2016 Rogue Wave Software, Inc. All Rights Reserved.
35
Conclusion
36© 2016 Rogue Wave Software, Inc. All Rights Reserved.
36
JBoss/Wildfly: Newer versions of Wildfly allow administrators to control HTTP headers and older versions of JBoss can be augmented with a filter or Apache HTTP Server's mod_header for equivalent functionality.
So much open source!
PostgreSQL: Use tools like pgBench and pgTune.
PostgreSQL: Troubleshooting tools like pg_stat_statements and pgBadger can help narrow down database performance issues.
ActiveMQ: A Lease Database Locker is a viable alternative to NFS where SAN is cost prohibitive. Consider Replicated LevelDB with Apache Zookeeper for performance.
37© 2016 Rogue Wave Software, Inc. All Rights Reserved.
37
Our support
38© 2016 Rogue Wave Software, Inc. All Rights Reserved.
38
Q & A
39© 2016 Rogue Wave Software, Inc. All Rights Reserved.
39
Watch on demand
• Watch this webinar on demand
• Read the recap blog to see the results of the polls and Q&A session
40© 2016 Rogue Wave Software, Inc. All Rights Reserved.
40
Follow up
For OpenLogic support customers:
OSS Radio
Get a free OSS support ticket to experience our expertise
roguewave.com/freeticket
Free open source newsletter:
roguewave.com/products/open-source-support/openupdate
41© 2016 Rogue Wave Software, Inc. All Rights Reserved.
41