presentation at chep 2006

The LCG SPI project in LCG Phase II

CHEP’06, Mumbai, India

Feb. 14, 2006

Andreas Pfeiffer -- for the SPI team

andreas.pfeiffer@cern.ch

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 2

Overview

• Introduction and Motivation• Services of SPI

• Savannah service• External software and distribution services• Configuration and Build Infrastructure• Software testing and QA services• Documentation service

• Outlook and future plans• Summary

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 3

The SPI project in the LCG Applications Area

• Goal of the project • Provide to the development projects of the LCG Application Area

- Basic environment for physics SW development

- General scientific libraries and class libraries

- Software development tools

- Documentation tools and document templates

- Compiler expertise

• Have consistency and homogeneity in the development of the different packages of the LCG Application Area

- Configuration management

- Central software repositories

- Common policies (naming, building, etc.)

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 4

Software Process and Infrastructure

• Work guided and prioritized by Architects Forum• Providing services to the projects (and experiments)• Channelling the requests from projects (and experiments)

• Providing the following services:• Savannah service

- issue and task tracker

• Software services - central installations of external software, software distribution

• Software development service - configuration mgt, pre- and post-build service, tools for testing

and QA

• Web and Documentation

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 5

Savannah Service

• Web portal for LCG software projects• Single entry point for all projects

- Uniform “look and feel” for all projects

• Totally web based

• Used for various functionalities• Bug/issue tracking• Task management• Download area

• Customized from GNU savannah project • In collaboration with main developer

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 6

0

200

400

600

800

1000

1200

1400

1600

aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'06

Adoption by projects

1575 users

0

50

100

150

200

aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'06

176 projects

0

2000

4000

6000

8000

10000

12000

14000

aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'060

500

1000

1500

2000

2500

3000

aug'02 feb'03 aug'03 feb'04 aug'04 feb'05 aug'05 feb'06

2770 tasks13500 bugs

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 7

Functional enhancements

• Project specific 'cookbooks' • based on the tracker engine• Inherits the tracker functionality while providing a friendly

book-like item display.

• Export facility • Helps extracting data to insert into reports (new releases,

statistics, project monitoring, etc)

• Voting system • Gives registered users the possibility to prioritize the way

items are handled

• Additional browsing criteria• Selection of items modified/unmodified since a certain date

• Extended transition mechanism• Automatic update of multiple fields on specific value

transitions

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 8

External Software Services

• Install software needed by LCG projects and experiments• Open source and public domain software

- Libraries and tools

• Standard structure of directories (central AFS repository)• <packageName>/<version>/<platform>/(lib, src, bin, …)

• Build and dependency information now stored in XML files with scripts to build (semi-)automatically • Very easy to build a new version or new platform• Easy to convert to formats for other tools (e.g. pacman)

• Experiments and projects decide what is installed• Through “Architects Forum”

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 9

External software service -- example

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 10

Software Distribution Service

• Existing solution since 2002• Script to download and install tar-balls of binaries for

software of LCG AA projects with all dependencies• Limited to binaries of platforms build at CERN• Dependency information from configuration management

and build tool (scram)

• Effort started to download source tarballs and (re-) build as needed• Simplifies porting to new platforms (or development work)• Re-using build and dependency information from XML files• Will be extended to download/install binaries

• Work ongoing to fully automate this service

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 11

Configuration and Build Infrastructure

• LCG Librarian• Maintaining configuration information centrally for build

systems in use- XML files + converters to scram and cmt

• Coordinating releases and pre-releases

• Librarians integration group• Discuss with experiments’ librarians/integrators additional

needs on external packages• Bi-weekly meeting (LIM)

• Developed simple tools to automate builds of LCG AA software• Three phases: pre-build, build (on all platforms in parallel),

post-build

• Work ongoing to improve automation• Provide tools for release managers in projects to do this

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 12

Software Testing Service

• Providing Test frameworks, support, policies and documentation• CppUnit, Oval• Qmtest• valgrind• On different compilers/platforms

• Running the tests is an integral part of the development in the projects• Unit tests, integration tests and regression tests

• Also available for experiments• Initial setup• User support

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 13

Quality Assurance Service

• Central QA checklist on each release of LCG AA projects• Code coverage• Providing statistics on tests and from savannah as a

coherent view

• Automated procedure to generate reports• At every release• Published on the SPI web site

• Evaluation and usage of external tools• E.g. RuleChecker, Logiscope

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 14

QA example (ROOT project)

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 15

QA example (POOL project)

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 16

Documentation Service

• Document LCG Policies• Homogenize the projects “look and feel”• CVS and release directory structures• Software testing policies

• Code documentation• Through SPI web site and wiki pages• Tools used: doxygen and LXR• Run for every release of LCG AA projects plus some

external packages (CLHEP, ROOT, HepMC)- Procedure is now semi-automated

• Internal procedures documented• “explicit knowledge” rather than “reading scripts”

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 17

Web and Documentation

• Maintain and improve existing web pages• Update internal and external documentation (wiki pages)• SPI and LCG Workbooks

• Automate content wherever possible• Create more “active/dynamic” web pages

- E.g.: status of external tools, configuration info- New: pages to do pre- and post- build procedures

• Update and clarify existing policies• To be reviewed by AF

• Update web pages for external packages• Responsibles, status, … for each package

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 18

Challenges for LCG phase II

• Servicing LHC experiments more than AA projects• Discussion with experiments to (re-)define scope

- What is needed, what can be improved

• Participation from people in experiments• “On-the-spot” help with requests from experiments• SPI provides a “frame” to do complex work

- Guidelines, AFS space, web space for docs, …

• Coping with strong (50%) reduction of resources • Optimization opportunities • Further optimization and automatization of procedures

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 19

Summary

• SPI provides a large set of stable services that fully use the current resources• Savannah, External Software, Testing Frameworks, Software

Distribution, Build and Release, QA Activities

• SPI work is guided by the Architects Forum• AF represents the users and steers our priorities

• In Phase II we will concentrate more on the experiments’ needs• Who see LCG AA project software as “external”

• Ongoing discussion with AF and experiments• To redefine the scope of the project

- Resources are significantly reduced

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 20

“Thank you!” to the SPI team(s)

• 2005• Alberto Aimar, Johanne Benard, Ian McLaren,

Yannick Patois, Yves Perrin, Andreas Pfeiffer, Eric Poinsignon

• 2006• Full time:

Ian McLaren, Yves Perrin, Andreas Pfeiffer• Part time:

Bertrand Bellenot, Manuel Gallas, Stefan Roiser

14-Feb-2006 CHEP'06, Mumbai, India -- andreas.pfeiffer@cern.ch 21

Additional Information

• SPI project web:• http://spi.cern.ch

• SPI project wiki pages:• https://twiki.cern.ch/twiki/bin/view/SPI/WebHome

• LCG Applications Area web:• http://lcgapp.cern.ch

• LCG web:• http://cern.ch/lcg