Proprietary and Confidential

Fitnesse and Continuous Integration

Jennifer Wong | Staff SQE Engineer | twitter: @jenlwong

2Proprietary and Confidential

Overview

►Intro► What is FitNesse (Not an exercise program!)► The Details► Integration: putting the pieces together► Lessons Learned

Proprietary and Confidential

• Who: – Jennifer Wong– Scrum Master for Tools Team, Staff SQE Engineer

• Where: – Ingenuity Systems: A leading provider of information

and analytics solutions for life science researchers

• What: – FitNesse as a key element of Continuous Integration

workflow

• Why: – Follow up to last year’s presentation (

http://www.slideshare.net/jenlwong/ingenuity-svcc-ci-presentation-20111007 )

Intro

Proprietary and Confidential 4

FitNesse (Not an exercise program!)

Proprietary and Confidential 5

FitNesse• FitNesse is a wiki-based web server test tool

– Helps abstract test definition from technical implementation

– Provides visual reporting and result history tracking

Proprietary and Confidential 6

Test Types: FitNesse• We use it for:

– Integration tests– Acceptance and Functional tests– UI Tests (com.jbergin.HtmlFixture, webtest)– DB Tests (dbfit)– Backward Compatibility tests

• What is it good for?– Framework and visibility– accessibility to non-technical people

• What is it bad for?– Unit tests– Complicated ui tests– Extensive performance testing

Proprietary and Confidential 7

FitNesse: The Details

Proprietary and Confidential 8

Downloading and Installing FitNesse

• Get the jar file and run it– http://fitnesse.org/FrontPage.FitNesseDevelopment.Down

Load

– java –jar fitnesse.jar

• Or use the demo package: • Get the file: on the svcc web site, attached to the

session, or on Github• unzip the file and cd to lib dir• start the fitnesse server: java -jar fitnesse.jar -p 8080• open web browser and access http://localhost:8080• click the link at the top for "CodeCampDemoPage“• click the test button

Proprietary and Confidential 9

Setting your classpath• Classpath statements

– Fitnesse needs to know where to find your fixture code

• This kind of error :

• Means you need to add this kind of statement: !path– You should usually have this kind of stuff:

!contents !path fitnesse.jar!path C:/eclipse/fit_demo/eclipse-bin!define TEST_SYSTEM {slim}

Proprietary and Confidential 10

Some Basic Test Tables• Script table

– Good for procedural/workflow tests– Flexible– Use syntax: check, reject, ensure, show

• Decision table– Good for data-driven tests– Specific workflow

• Input methods execute() output methods

– Special method name interpretation– Automatically calls reset() and execute() methods

• Query table– Good for validating lists or tables of data

Proprietary and Confidential 11

Variable Usage• Defining a static variable

– !define ROOT_URL {http://myserver.com}– !define TESTUSER {testuser1@something.com}

• Storing a value in a variable on the fly– Store variable: $X=– Use variable: $X

• Using a variable– To use this variable, enclose the variable name in $

{ }– Example:

• this: ${URL_ROOT}/context/index.html• Will resolve to this:

http://myserver.com/context/index.html

Proprietary and Confidential 12

Naming and Parameter passing (Methods)

• (Un) Graceful Naming– Automatically concatenates space-

separated words• isHalloween isHalloween()• Is Halloween isHalloween()• is halloween isHalloween()• Is halloWeen error

– When using methods with multiple parameters, tries to intersperse method name and paremeters

Proprietary and Confidential 13

Parameter passing (cont) • Multi-parameter methods: isHalloween(int,

String)

– |ensure|is|31|Halloween|October|– |ensure|is Halloween|31||October|

• Single parameter: setCostume(String)– |set costume|Clark Kent|– |set|Clark Kent|costume|

• Constructors with parameters– This constructor:

• public Halloween(String month, int day, String costume)

– Translates to this usage in a fitnesse table:

|Halloween|October|31|Cat||isHalloween?|get surprise?|

Proprietary and Confidential 14

UI Test Fixtures• com.jbergin.HtmlFixture

– an adapter between FitNesse and HtmlUnit for use in testing web applications

– Need to use !define TEST_SYSTEM {fit}– http://htmlfixture.sourceforge.net/ – http://uebuild5:8084/FrontPage.UmaFitNesse.IngsecuritySuite.Concurre

ntUserSessionTest.AcceptanceTests

• webtest selenium– an extension to FIT/FitNesse that uses Selenium

Remote Control. WebTest runs inside FitNesse.– http://www.fitnesse.info/webtest – http://uebuild5:8084/FrontPage.ReportsFitNesse.IsoformView.Is

oformViewWebTestSuite.IsoformViewWebTests

Proprietary and Confidential 15

Real world usage is more complex

• What it looks like in the real (ie, complicated)world– Session handling

• http://uebuild5.ingenuity.com:8084/FrontPage.UmaFitNesse.IngsecuritySuite.ConcurrentUserSessionTest.AcceptanceTests

– Static objects to provide data access• http://uebuild5.ingenuity.com:8084/FrontPage.ContentserviceFitNesse.TestSuiteForCurrent

Content.EcsMappingDataProviderTestSuite.P1Tests

– Complex checking of validity • Unmarshaling JSON to check special conditions in a

non order dependent way– http://uebuild5.ingenuity.com:8084/FrontPage.FaFitNess

e.TestSuiteForBaselineContentSpecific.FaProviderTestSuite.ExecuteFaQuery.LfaQueryTestSuite.AcceptanceTests

Proprietary and Confidential 16

Fixture code can get complicated very quickly

public static boolean matchGFAResult(JSONObject jsonResult, GFAResult actualResult, boolean allowSubset,boolean allowPvalueVerification,boolean geneCountVerification ,boolean allowZscoreVerification, boolean

allowGeneEffectVerification) throws JSONException {JSONArray jsonItems = jsonResult.getJSONArray("items");logger.info("expected item size = " + jsonItems.length());logger.info("actual item size = " + actualResult.getFAResultItems().size());if (jsonItems.length() > actualResult.getFAResultItems().size()) {return false;}

Map<String, GFAResultItem> itemMap = buildGFAResult(jsonItems);if (allowSubset) {for (Map.Entry<String, GFAResultItem> entry : itemMap.entrySet()) {logger.info("Look for " + entry.getKey() + " in actual result");GFAResultItem item = entry.getValue();if (!containsItem(item, actualResult.getFAResultItems(), allowSubset, allowPvalueVerification,geneCountVerification,allowZscoreVerification, allowGeneEffectVerification)) {logger.info(item.getId().getAsString() + " is expected but couldn't be found in actual result");return false;}}} else {return equalGFAItems(itemMap, actualResult.getFAResultItems(), allowSubset,allowPvalueVerification,geneCountVerification,allowZscoreVerification, allowGeneEffectVerification);}

return true;}

private static boolean equalGFAItems(Map<String, GFAResultItem> itemMap,Collection<GFAResultItem> actualResultItems, boolean allowSubset,boolean allowPvalueVerification,boolean geneCountVerification ,boolean allowZscoreVerification, boolean

allowGeneEffectVerification) {if (itemMap.size() != actualResultItems.size()) {return false;}

Proprietary and Confidential 17

Tips and tricks• Search in your FitNesse wiki• Use Includes

– Use includes as templates– http://uebuild5.ingenuity.com:8084/FrontPage.IngtestFitNesse.S

tableSuite.FaStableCompat– http://uebuild5.ingenuity.com:8084/FrontPage.IngtestFitNesse.S

tableSuite.ContentserviceStableCompat

• Comments• Escaping special characters

– Start tables with ! to avoid unwanted interpretation of graceful names, etc

– Surround special chars with !- -!• Example: !-gobbledeygook ~!@#$%^&*(){}| as

plain string-!

Proprietary and Confidential 18

Fancy fixtures and other nifty stuff• JSON

– http://uebuild5:8084/FrontPage.MgFitNesse.TestSuiteForBaselineContentSpecific.GraphProviderTestSuite.GetNeighborhoodGraph.P1Tests

• Javascript validation– http://localhost:8080/FrontPage.AutocompFitNesse.FunctionalTests.FitTests.GeneralTests.P1Tests EVAL {

void execute(Parse row, JSONFixture fixture) {Parse textCell = row.parts.more;// row.parts.more;

String evalText = textCell.text();String text = fixture.page.getWebResponse().getContentAsString();try {

jsEngine.eval("result = " + text + ";"); Object evaluationResult = jsEngine.eval(evalText); if (evaluationResult instanceof Boolean) { if ((Boolean)evaluationResult){ fixture.right(textCell);} else {fixture.wrong(textCell); }

• Running tests based on tag• Include: http://<host>:<port>/<suite path and test name>?

responder=suite&suiteFilter=smoke,critical

• Exclude: http://<host>:<port>/<suite path and test name>?responder=suite&excludeSuiteFilter=NotRunningOnHudson

Proprietary and Confidential 19

Test Variations• What we’ve done with it that is

different– Use as execution framework for more

complex tests – Extension of fitnesse server for data-

driven tests– json fixture – pass in javascript– Execution of Selenium tests– Backwards Compatibility tests

Proprietary and Confidential 20

Best practices• Test robustness• Test organization• Test readability• Fixture design

– Tradeoff between flexibility and readability, usability

Proprietary and Confidential 21

Integration: putting the pieces together

Proprietary and Confidential 22

Our Environment• Multiple products (3 external, plus internal tools)• Services-based• Builds:

– Produce multiple artifacts, including a fitnesse package

– Example: A build of contentservice produces:• contentservice-1.2.179886.clover.tar.gz • contentservice-1.2.179886.tar.gz • contentservicedb-1.2.179886.tar.gz • contentservice_fitnesse-1.2.179886.tar.gz

– Fitnesse package contains wiki page tests, libs, config files

Proprietary and Confidential 23

FitNesse as part of our Continuous Integration Workflow

Nightly Build

(Clover)

ApplicationBundle

Run Junit, Javascript

Tests

Hudson Dashboard(JUnit, Fitnesse summary, Code Coverage)

FitnesseBundle

SVN

Deploy Application

Deploy Fitnesse

Run Fitnesse Tests

(Nightly suite)

Fitnesse Wiki(Test history, Details,

Test Case Management)

Link

pu

blish

pu

blish

publish summary

Commit(Test Cases)

Proprietary and Confidential 24

Integration with Hudson/Jenkins servers

• Fitnesse plugin:

Proprietary and Confidential 25

Lessons learned

• Adds a lot of value for our team– Visibility into results and test history– Accessible to non-technical people– FitNesse is very good for visibility and

straightforward verification of data• Not good for everything

– Easy to do it wrong– Requires maintenance – Not as flexible

• To do more, you have to get creative• Fixture and test ownership needs to be a

shared responsibility

Proprietary and Confidential 26

• Demo files will be posted to github under jwong-github

• Slides are on slideshare• Demo and slides are attached to session • Q&A

Proprietary and Confidential 27

The “As Seen By” Matrix