Eclipse TPTP

• TPTP = “Test and Performance Tools Platform”,

– Part of the open-source Eclipse project– Project created in August 2004– Successor of the Eclipse “Hyades” tools sub-project

(2002)

– Web site: http://www.eclipse.org/tptp

• Four major components / sub-projects:

– Platform

– Monitoring tools

– Testing tools

– Tracing and Profiling tools

• Most recent release: TPTP 4.4.0.3 (Sept. 2007)

– Java 6 is not yet fully supported

TPTP Objectives

• Aims to bring software test and performance tools into the Eclipse environment in a consistent way that maximizes integration with tools used in the other processes of the software lifecycle

• Reduce the cost and complexity of implementing effective automated software quality control processes

• Share data through an OMG-defined test profile model, Common Base Event (CBE) Model and trace model implemented via the Eclipse Modeling Framework (EMF)

TPTP Project Structure

• TPTP Platform Project– Covers the common infrastructure in the areas of user

interface, EMF based data models, data collection and communications control, remote execution environments and extension points

• TPTP Monitoring Tools Project– Collects, analyzes, aggregates and visualizes data that can

be captured in the log and statistical models.

• TPTP Testing Tools Project– Provides specializations of the platform for testing and

extensible tools for specific testing environments, initially 3 test environments: JUnit, manual and URL testing.

• TPTP Tracing and Profiling Tools Project– Extends the platform with specific data collection for Java

and distributed applications that populate the common trace mode, also viewers and analysis services

TPTP Architecture

Presentation System

Statistical

Trace

EMF Data Models

Lo

g

Test

Te

st

Platform

TPTP Workbench

Standard Widgets

Reference Perspectives

Target System

Data C

ollectio

n

ApplicationD

ata Co

llection

Interface

ExecutionEnvironment

Log Collection

Trace Collection

PerformanceMonitor

JVMPI

Data L

oad

er

DistributedData Collection

Framework

Co

ntro

llerIn

terface

Co

ntro

llerIn

terfa

ce

DistributedControl

Framework Agent Controller

Eclipse Platform

JVMTI

Test Execution

Mo

nito

r

Tra

ce

TPTP Architectural Components (1)

• On the controller system:– User interface: the graphical view from which test

execution is directed.– Data models: based on the Eclipse Modeling

Framework (EMF)– Includes models for:

– test execution data– logs– traces– statistics

– Eclipse platform: used for windows, editors, etc.– Interfaces to the agent controller, and data collection

engine.

TPTP Architectural Components (2)

• On the target system:

– the application to be tested

– the application’s execution environment:

– includes the Java virtual machine and additional monitoring via virtual machine tool interfaces

– the test engine, to start the application and its environment

– data collection

Models implemented by TPTP (1)

• Includes a reference implementation of the UML2 test profile (MOF).

• Test profile: definition model for the creation of:

– test suites

– test cases

– data pools

– invocation

– loops

– synchronization

Models implemented by TPTP (2)

• Definition model for the definition, creation, management and persisting of test executions over time including:

– Tests to be executed.

– Deployments.

– Locations.

– Verdicts.

– Messages and console output from the test execution.

• A collection of test execution traces and results: the test log.

Agent Controller

• A daemon process that resides on each deployment host and enables client applications to launch host processes and interact with agents that coexist within host processes

• Contains a server that has the capability to launch and manage local or remote applications from a local TPTP workbench

• The Java profiling engine included uses the Java Virtual Machine Tool Interface to profile local or remote Java applications from a local TPTP workbench

• Launch and profile local or remote Java applications and import local or remote logs

• Platform support : Windows, Linux, Solaris, HP-UX, AIX, Z/OS, OS/400

Agent Controller

EclipseWorkbench

AgentController

JavaProfilingAgent

Hostprocess

Agent Controller Architecture

Image © 2006 IBM, made available under the EPL

Agent Controller Terminology

• Host process– The process that contains the application under test

• Agent– A reusable binary file that provides services to the host process,

and more importantly, provides a portal by which application data can be forwarded to attached clients

• Client– A local or remote application (e.g. Eclipse Workbench) that is the

terminal destination of host process data that is externalized by an agent

• The Agent Controller– A daemon process that resides on each deployment host and

provides the mechanism by which client applications can either launch new host processes, or attach to agents that coexist within existing host processes. The client can reside on the same host as the Agent Controller, or it can be remote. The Agent Controller can only interact with processes on the same node, and it is required on the same machine the targeted JVM is

Virtual Machine (VM) Interface

• JVMTI, the Java Virtual Machine Tool Interface– http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/index.html

• Provides interface for tools that monitor or control the virtual machine: debuggers, profilers, thread monitors, coverage analysis

• Capabilities: observation / control of– memory– threads– stack frame– memory heap– local variables– break points– classes and object instances; methods– timers

Versions of VM interfaces andthe Agent Controller

• In versions of Java up to 1.4.2, there were two interfaces called JVMPI and JVMDI, whereP = “profiling” and D = “debugging”

– These were deprecated in Java 5 and removed in Java 6.

• Both JVMPI and JVMDI have been replaced by JVMTI.

Testing with TPTP

• JUnit tests can be run using TPTP, as well as the regular JUnit test runner.

• JUnit tests must be imported to TPTP.

– The .java file remains unchanged

– An additional .testsuite file is created. This is the file to be run by TPTP.

• Caveat: Only test case classes that inherit from junit.framework.Testcase will be detected by the import function. At present, only JUnit 3 tests are handled properly by TPTP.

TPTP test import

Opening the .testsuite file

Methods in .java file.

TPTP Testing with JUnit

• This type of JUnit test generates an execution history from which a report can be generated.

• Creating JUnit tests You can create a TPTP JUnit test in one of two ways: manually or from an existing JUnit test case.

• Editing JUnit tests The JUnit Test Suite Editor can create and remove methods on a JUnit test, and control how those methods are invoked.

• Choose an option to run a test:– Running a test There are a number of options available to run a test. – Running a JUnit test remotely If you want to run a TPTP JUnit Test

remotely, you must create and configure a test deployment. At a basic level, test deployments are entities which relate other TPTP entities, artifacts and locations in pairs.

• Analyzing test results When you run a script, the test results for the events are saved in a test log. You can use the Test Log viewer to view the results and events in the test log.

• Generating reports on JUnit tests and JUnit plug-in tests After you have run your tests and generated execution histories, you can generate a report on those test results to summarize the data.

Test Log Overview (1)

Test run details:- Start, end times of run- Which file was run- Host on which run occurred

Test Log Overview 2

Results report summary:- Pie chart- Select verdict type

Test Log Event Detail

Details for this verdict

Test Report Generator

• After you have run your tests and generated execution histories, you can generate a report on those test results to summarize the data.

• To generate the report:

– Right-click on a TPTP JUnit test in the Test Navigator and select Report... from the pop-up menu.

– The New Report dialog launches.

– Select the Time Frame Historic option then click on Next.

– Provide the parent folder and a name for the report, then click on Next and provide the report time frame Start and End dates.

– Click on Finish to generate the report.

Time Frame Historic Report – Overview

Time Frame Historic Report - Test Results (1)

Time Frame Historic Report –Test Results (2)

Profiling Tool

• Broadly useful for performance analysis and for gaining a deeper understanding of a Java program

• Consists of the Profiling and Logging Perspective and a number of graphical and tabular views

• Enables you to profile and interact with your applications, to work with profiling resources, and to examine your applications for performance and memory usage problems

• Help you to visualize and understand your program execution, pinpoint the operations that take the most resources, as well as to explore patterns of program behavior

• Enables you to test your application's performance early in the programming development cycle for improvements

Profiling

• The process of generating a statistical analysis of a program that shows processor time and the percentage of program execution time used by each procedure in the program.

• Analyze the application execution and look for

– Execution bottlenecks

– Memory consumption

– Memory leaks

– High level view of your application execution

TPTP Java Profiling Tool

• A library that attaches to a JVM to capture and record the Java application's behavior

• Is a type of agent managed by the Agent Controller

• Runs in the JVM (Java Virtual Machine) process and receives notifications of JVM events, based on the JVMTI (Java Virtual Machine Tool Interface). This agent is best used to identify performance details such as classes or methods responsible for the poor execution performance, also be used to analyze application heap and find memory leaks

• Output from the profiling agent is in the form of XML fragments

• Can be launched from the TPTP workbench, Applications can be in workbench's workspace or binaries that are on the file system

• Can also be invoked using the -Xrun JVM option in command line

Profiling and Logging Perspective

• The profiling tools available in the Profiling and Logging perspective provide comprehensive information about the performance of an application

• The profiling tool (in this scenario i.e. – Java Profiling) provides information pertaining to

– JVM performance

– Object allocations and references

– Garbage collection

– Object methods performance

– Object->Object interactions

– Thread interactions

to name a few

Profiling and Logging Perspective

Java processassociatedwith a host

Time ofprofile run

Monitors

Profiling and Logging Perspective

• Profiling and Logging perspective provides resources to administering and managing profiling

• Profiling resources

– Project: Make a project of your profiling effort

– Monitor: Aggregate different processes and agents

– Host: The host you are profiling

– Process: The executing program

– Agent : Provides services to a process, a mechanism by which process data can be sent to (attached) clients

– Profiling Type: Group profile data collection

Profiling Monitor View

• Project:– Container of the profiling resources

• Monitor:– A logical container for the profiling information that is

collected from a group of agents. The views at the monitor level show data from these agents. Monitors are useful for aggregating processes and agents from a distributed application

• Host:– Owns the processes that are profiled. A host runs processes.

You can specify a host either by its name or by its IP address

• Process & Agent:– In the Agent Controller architecture model, an agent is a

binary file that provides services to the host process by which application data can be transferred to attached clients

Profiling Views

• Number of views to visualize and organize profiling data

– Memory Statistic view

– Execution Statistic view

– Coverage Statistic view

– Object Reference view

– Execution Flow view and table

– Method Invocation view and table

– UML2 Trace Interactions view

Execution Statistics View