Application Diagnosis with Zend Server Tracing

Kevin Schroeder

Zend Technologies

2

About me

• Kevin Schroeder Consultant for Zend

Programmer

Sys Admin

Author

Etc.

Me!

3

"As soon as we started programming, we found to our surprise that it wasn't as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to spent in finding mistakes in my own programs"

- Maurice Wilkes, 1949

4

Why do we need diagnostic tools?

5

Let’s see that again in slow motion

6

The point

• NOT to highlight an individual company

• Why not?

• How many web-based applications look like this when an error occurs?

• Answer

• All of them

• Web requests are intended to be short and quick

• If an error occurs, recover, quit and go on to the next request

7

Debugging the old way

• var_dump($var) (I still use this occasionally)

• Some obvious problems What if you forget to remove it?

Are you doing this in a production environment? If you are …

Very limited scope

8

Debugging the better way – ZS Debugging

• You can see what the variable values are at run time and modify them too

• Good for development

• Not good for production

9

Monitoring

• MUCH better for production systems

• You get a lot of really good information to help replay the issue

10

Monitoring

• You are able to debug or profile the event on a remote machine

11

Events that can be monitored

Function ErrorDatabase ErrorSlow Function ExecutionSlow Query ExecutionSlow Request Execution (Absolute)Slow Request Execution (Relative)High Memory Usage (Absolute)High Memory Usage (Relative)Inconsistent Output Size

Uncaught Java ExceptionCustom EventFatal PHP ErrorPHP ErrorJob Execution ErrorJob Logical FailureJob Execution DelayJob Queue High Concurrency Level

12

Instrumentation on the next level

Zend Code Tracing

13

Code Tracing: Rapid Problem Resolution

• Similar in concept to a black box flight recorder - Eliminates the need to reproduce problems in order to find root cause

• Activated automatically (through rules) or manually

• Records live application execution Function calls

Arguments and return values

Errors

Duration

• Useful in all parts of the software development lifecycle

14

Practical Code Tracing Functionality

• By capturing PHP applications code in real time: Code Tracing enables to capture problems when they occur.

Code Tracing recorded data eliminate the need to reproduce the steps led up to a failure.

Code Tracing enables to analyze PHP applications execution both in production and in test lab environments, by that, it enables to quickly find the root cause of problems.

14

Insert->Header & Footer

15

Supported Workflows

• Manual workflow: Used while performing unit tests during development or

when running functional tests in QA.

Calling URLs as part of the testing process

• Monitoring events workflow Used when running automated tests and load tests and

when running your application in staging or production environments.

Code tracing extends the monitoring rules mechanism by enabling to save trace data when a monitoring event is generated.

Suitable for production in Standby mode

16

Detailed Runtime Instrumentation

Pinpoint root cause by viewing the sequence of events that led up to an error

17

Detailed Runtime Instrumentation

Optimize your code by analyzing application performance at the function level

18

Code Tracing – Monitoring Level Modes

• Always on Collects trace data when monitoring event is kicked off

Collects trace data for manual requests.

• Inactive Collects data only for manual requests.

• Standby

• Off (turn code tracing extension off)

19

Code Tracing – Operating Modes

• Standby mode A monitoring event activates code tracing

Time

Tracing is activated for X

minutes

A monitoring event with an “Activate Trace” action is generated

Tracing switches back for “Standby”

mode

Collect trace data according to the monitoring set of rules

Trace mode:StandbyOn

20

The basics

• Trace a Request

21

Interpreting the results

• Screenshots are boring. Let’s look at live data

22

Setup

• Attaching traces to events Tracing can be attached to any PHP-specific events

23

Tracing Events – Standby Mode

• Set zend_monitor.event_tracing_mode Active, Inactive or Standby

Determines integration with tracer mechanism in case of events that require a trace data dump

• Create an Event Action – “Enable Tracing” will change

24

Tracing Process Signals

• Tracing can also be initiated upon a fatal signal

25

Tracing Log

• Step 1 – Make sure that you have it set to log

• Step 2 – Set the verbosity between 1 and 5

• Restart

26

Example log file[Tracing Log 08.10.2009 14:14:56 DBG1] 5387 Tracer request startup

[Tracing Log 08.10.2009 14:14:56 DBG3] 5387 Setting dump reason to 0

[Tracing Log 08.10.2009 14:14:56 DBG3] 5387 Dump reason is now 0

[Tracing Log 08.10.2009 14:14:56 DBG1] 5387 Remote address is 127.0.0.1

[Tracing Log 08.10.2009 14:14:56 DBG2] 5387 Allow hostmask is 127.0.0.1/32,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12

[Tracing Log 08.10.2009 14:14:56 DBG2] 5387 Deny hostmask is

[Tracing Log 08.10.2009 14:14:56 DBG1] 5387 Remote access allowed

[Tracing Log 08.10.2009 14:14:56 DBG1] 5387 Request parameter requires dump

[Tracing Log 08.10.2009 14:14:56 DBG3] 5387 Setting dump reason to 2

[Tracing Log 08.10.2009 14:14:56 NOTICE] 5387 Installing signal handler

[Tracing Log 08.10.2009 14:14:56 DBG2] 5387 Staring trace generation

[Tracing Log 08.10.2009 14:14:56 DBG2] 5387 Tracer request startup done

[Tracing Log 08.10.2009 14:14:56 DBG3] 5387 Options updated to 7FFF

[Tracing Log 08.10.2009 14:14:57 DBG1] 5387 Tracer request shutdown

[Tracing Log 08.10.2009 14:14:57 WARNING] 5387 Code tracing buffer full

[Tracing Log 08.10.2009 14:14:57 DBG1] Using socket /usr/local/zend/tmp/monitor_events.sock rules /usr/local/zend/etc/events_rules.xml

[Tracing Log 08.10.2009 14:14:58 NOTICE] Event type 1 (0) reported

[Tracing Log 08.10.2009 14:14:58 NOTICE] 5387 Generating trace dump, format 2

[Tracing Log 08.10.2009 14:14:58 DBG2] 5387 Dump name is /usr/local/zend/var/codetracing/dump.5387.1

Monitoring Event was triggered

27

Functional Examples

• Uncaught exceptions

• Destructor being called

• Prepared Statements

28

Where did all my memory go?

• Memory usage

• Large variable going out of scope

29

Impact

0

50

100

150

200

250

Framework sans TracingFramework with Tracing

30

The all-important question…

“Where should I use this?”• Development – Possibly, though profiling and debugging may be more pertinent

• Testing – Absolutely

• Staging – Absolutely

• Production - Absolutely In Standby mode

Activated on a subset of servers

31

Questions?Want to join in the public beta? Go to

http://www.zend.com/en/products/server/zend-server-5-new

for more information

kevin@zend.com

32

Additional Information

33

What happens when you have too much data

• Need to up the zend_codetracing.buffer_size setting

34

Increasing the buffer size to 5MB

• That’s better

35

Settings

Setting Description

zend_codetracing.buffer_size The size of the trace memory buffer

zend_codetracing.dump_fileThe prefix for the dump file names, relative to zend.data_dir.

zend_codetracing.dump_freqMinimum amount of seconds between successive dumps.

zend_codetracing.enable Is tracing functionality enabled?

zend_codetracing.log_fileLog file for the Code Tracing module (relative to the log dir).

zend_codetracing.log_verbosity Logging verbosity level.

zend_codetracing.max_depth The maximal depth of the array preserved zend_codetracing.max_elements

The maximal number of the array elements preserved

36

Settings

Setting Description

zend_codetracing.max_string The maximal length of the string before it is cut

zend_codetracing.trace_always_dump Trace data is always persisted

zend_codetracing.trace_arguments Function call argument collection is enabled

zend_codetracing.trace_arrays Array contents recording is enabled

zend_codetracing.trace_dump_on_segv

Trace data is persisted if fatal signal (like Segmentation Fault) happens

zend_codetracing.trace_enable Tracing data collection is enabled

37

Settings

Setting Description

zend_codetracing.trace_headers Output headers collection is enabled

zend_codetracing.trace_includesInclude/require data collection is enabled

zend_codetracing.trace_internal_functions

Internal functions call collection is enabled

zend_codetracing.trace_memory_usageMemory usage data collection is enabled

zend_codetracing.trace_return_valuesFunction return value collection is enabled

zend_codetracing.trace_source_linesSource file information collection is enabled

zend_codetracing.trace_timeTimestamp/elapsed time collection is enabled

zend_codetracing.trace_user_functions User functions call collection is enabled

zend_codetracing.trace_write Output buffer data collection

38

Settings

Setting Description

zend_codetracing.trace_errors PHP error collection is enabled

zend_codetracing.trace_eventsZend Monitor event recording is enabled

zend_codetracing.trace_exceptionsPHP exception data collection is enabled