SciDAC SSS Quarterly Report

Sandia Labs

January 25, 2005

William McLendon

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration

under contract DE-AC04-94AL85000.

APItest - Overview

• Current Release : v0.2.5

• Test Driver tool for testing software– Ordered tests

– API Testing

• Available for download:– ftp://ftp.sandia.gov/outgoing/apitest/

Work Since Previous Meeting

• October : Development meeting at ORNL– Imported APItest into OSCAR

– Started developing some APItests

• Fixing bugs

• Finishing up User Guide

APItest - Features

• Scripted Test Driver– XML based tests

• Object Oriented– Easy to define new test

types.• Conditional test ordering

– Batch Scripts• Can call other batches

• Built In Test Types– Shell– Script– SSS

• Fully scripted in Python• Portable

• Can set environment variables for tests.

• SUID Test Capability– CAUTION!

• Timeouts• User Interface Types

– Web Browser– Command Line– Archived Results Browsing

• Test Results Export– Save Test results to an

XML file.• Open Source

– LGPL

APItest - Requirements

• Packages required to run APItest:

– Python 2.3 or greater• lower versions can work but install is not as easy.

– ElementTree (www.effbot.org)

– Twisted (www.twistedmatrix.com)

– SSSlib (ftp://ftp.mcs.anl.gov/pub/sss/)• optional, for SSSlib tests.

APItest - Execution

• There are two ways to execute APItest

• Command Line Only– $> apitest -f test_file.[apt|apb]

• Web browser GUI (Interactive)– $> apitest httpd

• -i [--iroot] : root directory to look for tests• -p [--port] : port number (default = 2112)• -h [--host] : host (default = localhost)

• Running with --help will display command line usage.

APItest - GUI (Top Page)

• Top level index page for APItest

• APItest recurses from the [testroot] to find tests.

• File Extensions:– TEST : .apt– BATCH : .apb

• Batch tests shown in bold

• Organized by directory

APItest - GUI (Test Run Execution)

• Quick status view of test execution.

• This screen loads when the EXECUTE button is selected.

• Color coded for easy viewing.

• Currently running test is indicated.

• Refreshes every 15 seconds until tests complete.

APItest - GUI (Test Run Summary)

• Summary of test results

• Color coding

• Batch Features

– Cumulative summaries• Total # of tests• # that passed (green)• # that failed (red)

• Hyperlinks to see more details about a test or batch.

APItest - GUI (Test Details - PASSED)

• Example of a test that was executed and passed.

• Summary Data

• Dependency Data

– Only shown if test was run as part of a batch.

• Output

– Side-by-side comparison

– Dependent on type of test.

APItest - GUI (Batch Summary)

• Clicking a hyperlink into a batch test gives us a summary view.

• In this example, one test failed a dependency

– FAILDEP in the status column indicates this.

• Clicking on the scriptTest2.apt link will show us more data about this test.

APItest - GUI (Test Details - FAILDEP)

• Detailed view of scriptTest2.apt from previous slide.

• Failed a dependency on cmdTest2.apt– Expected : FAIL– Actual : PASS

APItest - GUI (List of Test Runs)

• Clicking the View All Runs button gives this screen.

• Test Runs are sorted by date and time.

• Click links to view details of a particular run.

SSS - Service Directory Startup Test

PREP TEST SDOFF

SSS START TEST SDON

RM EMNG SDSTAT

SSS STOP TEST SDOFF

APItest - Ongoing & TODO

• User Guide• Validation via file matching• Bug fixing• Configuration file• More SSS tests• Test Developer GUI• Remote test execution• Additional native test types

– HTTP, TCP/IP, XMLRPC• Session management and security (Cookies?)• DB Connectivity

APItest - Test Types

• CMD– Execute a command via the command line.

• SCRIPT– Execute a script.

• SSS– Execute a ssslib communication.

APItest - Example CMD Test

<testDef>

<info> Example test. Runs ls -lr from /tmp, expects 0 exit status and any input.</info>

<test type="cmd” wdir="/tmp">

<command interpreter="/bin/ksh">ls</command>

<input name="argument" format="literal">-l</input> <input name="argument" format="literal">-r</input> <input name="stdin" format="literal" />

<output name="stdout" format="regexp">.*</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output>

</test></testDef>

APItest - Example Script Test

<testDef>

<info>Timeout test</info>

<test name="timestamp1" type="script" timeout="4">

<command interpreter="/bin/bash" wdir="/tmp">for x in 1 2 3 4 5 6 7 8 9 10 do echo "timeout_test :${x}: `date`” sleep 1done </command>

<output name="stdout" format="regexp">.*timeout.*</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output>

</test></testDef>

APItest - Example ENVVAR Test

<testDef> <info> This script executes but adds some stuff to the environment. NOTE: setting envvar OVERRIDES any inherited environment variables. </info> <test type="script">

<command interpreter="/bin/csh">#!/bin/csh env</command>

<input name="envvar" format="literal" key="test_envvar_1">foobar</input> <input name="envvar" format="literal" key="test_envvar_2">foo</input>

<output name="stdout" format="regexp"> .*test_envvar_1=foobar\n.*test_envvar_2=foo\n.*</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output>

</test></testDef>

APItest - Example SUID Test

<testDef>

<test type="script">

<command interpreter="python" uname="wcmclen" gid="500">#!/usr/bin/env python import os print os.getuid() print os.getgid() </command>

<input name="argument">/tmp/testuser/</input> <input name="argument">-l</input>

<output name="stdout" format="regexp">^27030$\n^500$</output> <output name="stderr" format="literal" /> <output name="status" format="literal">0</output> </test></testDef>

APItest - Example SSS Test

<testDef>

<info>Sample sss query of the service directory</info>

<test type="sss" destination="service-directory">

<input name="sendbuf">&lt;get-location>&lt;location component='service-directory' host='*' port='*' protocol='*' schema_version='*' tier='*'/>&lt;/get-location></input>

<output name="recvbuf" format="regexp">.*</output>

</test>

</testDef>