application logging for large systems

©2010 Improving Enterprises, Inc.

Logging For Fun and Profit

Jane Prusakova

Jane.Prusakova@ImprovingEnterprises.comhttp://softwareandotherthings.blogspot.com

Improving Enterprises

Houston JUG

2014

©2010 Improving Enterprises, Inc.

Jane Prusakova

Hard-core developer

Data science geek

Dancer and photographer

©2010 Improving Enterprises, Inc.

Why

How-to

Using the results

So lets talk about logging…

©2010 Improving Enterprises, Inc.

Peek into the soul

Real setup

Real data

Real users

Real workDev environment

Happy path

Minimum load

Limited data

©2010 Improving Enterprises, Inc.

Inside the cloud

No programming-by-coincidence

Replaces debugging

Works on server

©2010 Improving Enterprises, Inc.

Logging [vs Debugging]

Set up once, always ready to use

Minimum disruption to the flow

Works locally and in the cloud

Faster data collection

Collected data is persisted

©2010 Improving Enterprises, Inc.

Logging helps with…

Integration

Multithreading

High load

©2010 Improving Enterprises, Inc.

What else?

Trace what the users do

Features

Sequences

Data flow

©2010 Improving Enterprises, Inc.

When to use logging?

New systems

New features

Bug fixing

©2010 Improving Enterprises, Inc.

Are logs forever?

Permanent

Errors

Extraordinary situations

©2010 Improving Enterprises, Inc.

Some are seasonal

Very temporary

Debugging info

Trace bug fixes

©2010 Improving Enterprises, Inc.

Others last awhile

Medium term

Data flow

Resource utilization

Optimization points

©2010 Improving Enterprises, Inc.

How-to

Using the results

©2010 Improving Enterprises, Inc.

Yes, I want to setup logging!

Pick a framework

Configure logging

Storage and access

Getting value from logs

©2010 Improving Enterprises, Inc.

Log4J SLF4J

Apache MIT license

Logging framework Logging facade

Configuration, log levels and categories,

rotate log files, thread-safe logging.

Support: online documentation, tutorials,

online forums.

©2010 Improving Enterprises, Inc.

Database Files

Easy to parse,

hard to evolve

Very hard to parse,

easy to change

Uses DB

connections

Requires FS

access

Can cause app

crash, loss of logs,

slow down app

Can fill up space

©2010 Improving Enterprises, Inc.

Where to write logs?

Console

File system

DB

Queue mechanisms

Combination

©2010 Improving Enterprises, Inc.

Logging configuration

Java properties-style

XML

Programmatically

Hard-coded

Takes preference

©2010 Improving Enterprises, Inc.

Log Message

What to log with the message

Importance level

Timestamp

Where in the code

Data being processed

System state info

©2010 Improving Enterprises, Inc.

Example layout

log4j.appender.stdout.layout.ConversionPattern=

%d %5p [%t] (%F:%L) - %m%n

- Date

- Log Level

- Thread name

- File and line (slow to retrieve)

©2010 Improving Enterprises, Inc.

Good log messages

Support code

Correctness

Readability

Performance

©2010 Improving Enterprises, Inc.

Better log messages

Relate to generating point

Code

Data

©2010 Improving Enterprises, Inc.

Best log messages

Are formatted for

Readability

Aggregation

©2010 Improving Enterprises, Inc.

Logging objects

.toString()

org.apache.commons.lang.builder.ToStringBuilder

log.info(myDataObject);

©2010 Improving Enterprises, Inc.

What to log

Input and output

Errors and exceptions

Computation results

Integration points

Thread rendezvous

©2010 Improving Enterprises, Inc.

Input and output

public int calculateCost(String user, Flight flight)

{

log.debug(“calculateCost: “ + user + “: “ +

flight);

int cost = … ;

… // cost calculation

log.debug((“calculateCost: cost for “ + user

+ “: “ + flight + “ = “ + cost);

return cost;

}

©2010 Improving Enterprises, Inc.

Errors and exceptions

// do dangerous work

result = DoDangerousWork(user, flight);}

catch (Exception e)

{

log.error(“methodName: exception in

DoDangerousWork for “ + user + “: “ + flight, e);

// recover or re-throw

}

©2010 Improving Enterprises, Inc.

More

Log destination, parameters, outcomesDB calls

SOAP calls

Wait time

Thread joins

Available memory

Memory-intensive operations

Intermediate calculation results

For complicated calculations

©2010 Improving Enterprises, Inc.

Avoid

Little information

… // step 1

logger.info(“did something - XXXX”);

… // step 2

logger.info(“did more work - YYYY”);

foreach (…) {

… // useful work

logger.info(“working hard”);

}

©2010 Improving Enterprises, Inc.

Avoid

Code cluttering

if (logger.isDebugEnabled()) {

logger.debug(“Show Views");

logger.debug("Address:\n" +

view.getAddress());

logger.debug("Email:\n" +

view.getEmail());

}

©2010 Improving Enterprises, Inc.

Watch for logging-caused bugs

Side effects

Logger.info(count++ + “: did something”);

Errors and exceptions

Logger.info(m.GetValue());

©2010 Improving Enterprises, Inc.

DRY principle applies

©2010 Improving Enterprises, Inc.

Growing, growing…

Naming conventions

Rotation

by size

by time

©2010 Improving Enterprises, Inc.

You’ve got logs!

Real-time access

Deal with space limitations

Warning: large datasets

©2010 Improving Enterprises, Inc.

Security

Absolutely

No Sensitive Data

©2010 Improving Enterprises, Inc.

Using the results

©2010 Improving Enterprises, Inc.

Learning from the logs

Never rely on eye-balling

WYS is not WYG

Distinguish common and

rare

©2010 Improving Enterprises, Inc.

Code and logs

©2010 Improving Enterprises, Inc.

Process log

Line by line

No need to load entire file

Aggregate events

Relate to time

©2010 Improving Enterprises, Inc.

Tools

Utilities

grep

sort

uniq

Scripting

Perl

Awk

sed

©2010 Improving Enterprises, Inc.

Learning from the logs

> grep userName myApplication-MMDDYYYY.log

> grep methodName myApplication-MMDDYYYY.log

> grep FATAL myApplication-MMDDYYYY.log

> tail -2000 myApplication-MMDDYYYY.log | grep Exception

©2010 Improving Enterprises, Inc.

Querying logs using Awk

GNU Awk www.gnu.org/software/gawk

Create histograms

Wait times

Number of calls

Exceptions per time period or per user

Compare and relate events

Exception stacks traces

Outcomes by caller or thread

©2010 Improving Enterprises, Inc.

AWK demo

Logs for the demo generously

provided by

http://42graphy.org/

©2010 Improving Enterprises, Inc.

Jane.Prusakova at ImprovingEnterprises.com

SoftwareAndOtherThings.blogspot.com

@jprusakova

Jane Prusakova

http://www.slideshare.net/jprusakova

©2010 Improving Enterprises, Inc.

Logging For Fun and Profit

Jane Prusakova

Jane.Prusakova@ImprovingEnterprises.comhttp://softwareandotherthings.blogspot.com

Improving Enterprises

Houston JUG

2014