M A N A G E D B L A C K B O X T E S T I N G

A SOFTWARE MONITORING FRAMEWORK FOR QUALITY VERIFICATION

PRESENTED BY:DILEEPA JAYATHILAKE

99X TECHNOLOGY

JCSSE 2012

QUALITY IS PARAMOUNT

Operational Overhead• Constant re-planning• Derailing deadlines• Higher costs

Through-life Costs

• Recalls / Updates• Warranty Claims• Litigation

Business Value Decline

• Loss of market share• Dropped brand equity

Poor Quality Kills

FUNCTIONAL TESTING

Focuses on spotting any observable deviations of the product with respect to the desired features and behavior

Both manual and automated

Covers most feature-related malfunctions

Limited to only the behaviors observable directly

WHITE BOX TESTING

Supplements functional testing

Directly examines code

Can test cyclometric complexity, testability, unit test coverage, potential memory leaks, class and function level complexity, deprecated API usage, code duplication, undocumented code, styling errors

Many tools exist for this

Does not cover operational errors

BLACK BOX TESTING

Certain errors are not captured either by functional testing or by white box testing

They need monitoring application in operation

Black box testing is required for a testing process to be completed

Wide spectrum of tools is available in each platform

Some tools provide a UI while others generate a log

Memory corruptions, memory leaks

Disk access errors Failures in certain low level

operating system calls Pitfalls due to insufficient

user access rights Unjustifiable performance

bottlenecks

BLACK BOX TESTING TOOLSTool Remarks

Apache JMeter

- Used for server load testing in client-server applications- Supports files, servlets, Perl scripts, Java objects, database queries, ftp servers- Works with HTTP, HTTPS, SOAP, Database via JDBC, LDAP, JMS, POP3 and IMAP

protocols- Portable across platforms- Can replay test results- Highly extendable via plugins- Provides simple and intuitive UI- Does not support client-side scripts

Microsoft Application Verifier

- Captures system level problems that occur when a program is run- Can monitor Virtual memory usage, First chance access violation exceptions, Input-

output transfers, Synchronization objects (e.g. locks), Handle usage, Thread pool usage, Thread local storage, Dll handling, Interactive service creation, Dangerous API calls, Driver installations and User privilege issues

- Writes results to an XML log- Optionally a debugger can be attached

BLACK BOX TESTING TOOLS CONTD.

Tool Remarks

LeakDiag & LDGrapher

- A pair of tools that can be used in conjunction to detect memory leaks in an application

- Can monitor Virtual memory allocator, Windows heap allocator, COM allocator, C runtime allocator and TLS slot allocator

- LeakDiag generates an XML log, which can be graphically viewed through LDGrapher

Process Monitor

- Windows logging utility, which collects and records most of the system activity regarding file system, registry and process/thread activity

- Comes with a rich GUI that displays information in real time- Offers advanced filtering options to trace specific activity in the interest- Provides a boot-time logger- Should be run only for a limited time

BLACK BOX TESTING TOOLS CONTD.

Tool Remarks

XPerf - Performance-profiling tool for Windows applications- Can be used for tracking performance bottlenecks in applications as well as for

comparing time taken for various operations in an application- Writes monitored data into an XML log which can be later explored via XPerfView- Comes with a rich collection of preset monitoring profiles

Application Compatibility Toolkit

- This is a Windows application lifecycle management toolset- Toolset comprises Standard User Analyzer, Internet Explorer Test Tool and Setup

Analysis Tool- Can be utilized to

• Analyze portfolio of applications, web sites and computers• Centrally manage compatibility evaluators and settings, Rationalize and

organize applications, web sites and computers• Prioritize application compatibility with filtered reporting• Add and manage issues and solutions for enterprise- computing

environment• Deploy automated mitigations to known compatibility issues• Send and receive compatibility information from the Microsoft

Compatibility Exchange

BLACK BOX TESTING TOOLS CONTD.

Tool Remarks

God - Process-monitoring tool for Linux, BSD and Darwin systems- Provides watchdog functionality on processes with respect to their CPU and

memory usage- Can be configured to perform certain actions on a process depending on its CPU

and memory consumption- Has both a polling mode and an event responder- Generates a line log- Capable of sending alerts as email, chat or Twitter messages

Instruments - Standard tool for profiling processes that run under Mac OS X or iOS- Uses monitoring templates called instruments to collect data on various aspects- Monitoring areas include CPU consumption, memory usage, memory leaks, I/O

operations, power usage and network traffic- Data viewers are built in- Capable of recording a sequence of actions to replay later

Black box testing pitfalls

Tool usage requires platform expertise

Generated logs are bulky

Difficult to correlate results

from multiple tools

Lot of manual work

Difficult to communicate

results to different

stakeholders in a project

Analysis knowledge not

shared

Framework is needed to integrate tools and automate analysis

FRAMEWORK ARCHITECTURE

CONCLUSIONS

LDEL significantly reduces code for log data extraction

LDEL improves readability of the data extraction code

Introduction of SQLite into data manager enabled handling high data volumes and persistence

Need a better database for efficiently handling heterogeneous log data

MML is not appropriate for UI generation