glite i/o storm testing in edg-lcg framework elena slabospitskaya, vadim petukhov, (ihep, russia)...
TRANSCRIPT
Glite I/O Storm Testing in EDG-LCG Framework
Elena Slabospitskaya, Vadim Petukhov, (IHEP, Russia)
Gilbert Grosdidier, (CNRC, France)
NEC'2005, Sept 16
Outline
● Framework’ Contributed Authors
● TSTG Suite Dedication
● Design of TSTG Framework S/W
● Up to gLite- port TSTG framework to gLite environment
● Example of gLite plug-in - I/O Stress Test Suite
NEC’05 Elena Slabospitskaya (IHEP) et al.
Framework’Contributed Authors
The mainly contributed authors are● Miguel Barcelo● Frederique Choilett● Gilbert Grosdidier● Andrey Kiryanov● Charles Loomis● Gonzalo Merrino● Rene Metery● Danila Oleynik● Andrea Sciaba● Elena Slabospitskaya● Many other people contributed to the design and ideas leading to the
current suites, among them the INFN Testing Team
NEC’05 Elena Slabospitskaya (IHEP) et al.
New accent of MW testing
"The development of full featured test suites that for the certification of the new MW components"
"The development of testing procedures focused on special details of various software features"
"Porting the exiting test suites and procedures to the TSTG framework"
The coordination of this task will be done in close collaboration with EGEE-II SA3 to address the shortcomings of available tests for gLite middleware. This involves agreement on schedule, provision of access to development test-bed(s), support on issues hindering the development process (e.g. missing or failing test-bed functionality).
NEC’05 Elena Slabospitskaya (IHEP) et al.
Levels of testing required
1. Installation and configuration testing
Targets each machine/service individually
2. Unit testing
Targets each basic functionality of a given service independently from the rest of the TB
3. Functionality testing
For the all TB exercises full functionalities
4. Stress testing
Huge number of jobs launched one by one or in parallel
NEC’05 Elena Slabospitskaya (IHEP) et al.
TSTG suite dedication
1. Site certification• Initial check of the install• Check than changes or misconfiguration occur • Regular daily or weekly re-checking (under cron)2. TB Certification• Basic functional tests of components (services)• Basic grid functionalities with individual tests• Full grid functionalities on a full TB including remote sites with
global/group tests3. MW validation (Functionalities, Robustness, Stress Testing)
NEC’05 Elena Slabospitskaya (IHEP) et al.
Framework S/W design
Configure StepTest LoopPlug-insDefinition FilesPresenter Step
It is split into 2 main parts, dedicated to 3 tasks-the top level driver, written in Perl
-the plug-ins, one of each specific test (Perl, bash, C++)The top level driver is responsible for these 3 tasks----Configure (build environment)----Test Loop (run one of several selected tests)----Presenter : merges the results and builds the web enabled summary page NEC’05 Elena Slabospitskaya (IHEP) et al.
Examples of LCG-2 Plug-ins
Job StormCopy StormReplica StormData StormGFAL StormMatchmaking StromRLSSesMDSChecksum Storm
NEC’05 Elena Slabospitskaya (IHEP) et al.
Structure of CTB SL3
NEC’05 Elena Slabospitskaya (IHEP) et al.
LCG-2 Testing under TSTG Framework
To change: View -> Header and Footer 19
Enabling Grids for E-sciencE
INFSO-RI -508833
Storage Element Interfaces
• SRM interface– Management and control
– SRM (with possible evolution)
• Posix-like File I/O– File Access
– Open, read, write
– Not real posix (like rfio) SRM interface
rfio dcap chirp aio
Castor dCache NeST Disk
POSIXAPIFile I/O
Control
User
•The gLite-I/O server allows posix-like access to files stored in the SRM
gLite Data Access via I/O server
NEC’05 Elena Slabospitskaya (IHEP) et al.
Input/OutputServer
UserInterface
WorkerNodes
WorkerNodes
StorageElement 1
StorageElement 2
Storage Element 3
StorageElement 4
gLite I/O plug-in
User launches multiple (parallel or one by one) jobs from User Interface to Worker Node.Each job on Worker Node launches a script which compiles and linkages C++ program and creates a jdl file (this program is a part of job in input sandbox) The C++ program via API creates files (size is optional) and transfers (read/write/seek) these files between WN and I/O server (there and back). The result returns to user (to User Interface). Sources and results of tests are arranged at web.
NEC’05 Elena Slabospitskaya (IHEP) et al.
Results of I/O Stress Tests
NEC’05 Elena Slabospitskaya (IHEP) et al.
I/O test. XML output
Conclusions
1. Reuse as much as possible existing suites/frameworks/ can be easily incorporated from the LCG-2 certification to gLite•Minimize duplication of effort when creating new tests•Make sure new tests can be easily incorporated into the existing framework
2. The next most urgent is to write new tests or the functionality being released and /or different from LCG-2•Most must one from EGEE JRA1 team•But not exclusively, LCG team will contribute