space science and engineering center university of wisconsin-madison space science and engineering...
TRANSCRIPT
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 1
NPP Atmosphere PEATECritical Design ReviewNPP Atmosphere PEATECritical Design Review
Presented by the Atmosphere PEATE TeamSpace Science and Engineering CenterUniversity of Wisconsin-Madison17 April 2008
Presented by the Atmosphere PEATE TeamSpace Science and Engineering CenterUniversity of Wisconsin-Madison17 April 2008
Part 2: Science Processing SystemScott MindockPart 2: Science Processing SystemScott Mindock
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 2
DesignSystem SubsystemsSoftware Development and VerificationScience Investigation SupportLogistics
Agenda
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 3
UML - Unified Modeling LanguageUse Case - Captures requirements at functional levelActivity - Describes the steps of a use casePackage - Show software structure and dependenciesClass - Show software structure and dependenciesWeb Services - SOAP XML based communications
Definitions
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 4
Sample Activity Diagram
Like FlowchartOvals = ActivityRectangle = DataAction A and B Decoupled Dot = startCircle = end
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 5
Maintainability - System lifetime spans years
Reusability - Subsystems and design patterns
Testability - Subsystem dependencies are managed
Scalability - Design must scale to larger and smaller systems
Design: Goals (1 of 3)
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 6
Learn from others - IDPS, Clusters, Ocean PEATE, Disney Parkwide
Leverage existing proven technologies and standards• Java• Web Services - Provide communications between subsystems • Apache Tomcat - Web Application Server• Apache Axis2 - Web Application providing web services• JAXB - Java XML Binding• ANT - XML Structured make• Eclipse - Java IDE• Subversion - Revision control
Prototype key features - Demonstration Projects
Loosely Coupled System - Manage dependencies, Define interfaces
Design: Methodology (2 of 3)
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 7
All systems use XML configuration.All configurations have a XML schema.System persistence leverages schema based code generation.
All systems can persist to DB or XML (scalable)
Major components are deployed web services. (decouple, scalable)
Test patterns used for scenario based testing / system verification
Design: Patterns ( 3 of 3 )
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 8
External RelationshipsSubsystemsSystem Activity
System
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 9
System: External Relationships
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 10
DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages
Subsystem Relationships
Arrows denote dependency
System design minimizes dependencies
Eases maintenance
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 11
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 12
DMS: Data Management System Stores DataCRG: Computational Resource Grid Processes DataARM: Algorithm Rule Manage Applies Product Rules to DataING : Ingest System Brings Data into System
Subsystem Relationships @ PDR
@PDR@PDR
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 13
System Activity Diagram @ PDR
@PDR@PDR
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 14
ING - Ingest brings data into the Atmosphere PEATE
DMS - Data Management System, stores data in the form of files. Provides a web service interface to locate and retrieve files. CRGALGARMRUL
Subsystems
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 15
DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages
System Components, Java Packages
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 16
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 17
Supports pull ingestsSupports push ingestsVerification functions localizedNotices missing or late data
Dedicated MachineDeploy Multiple Copies
ING
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 18
Schema based persistenceXML ingest type configurationSupport for FTP, RSYNC
ING Configuration
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 19
DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages
System Components, Java Packages
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 20
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 21
Provides well-defined interface deployed as a web service.DMS is autonomousProvides storageProvides catalogSpans multiple file servers on a networkDMS = Linux File system + Postgres database
DMS
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 22
Each file in DMS has associated entry in catalogImportant file characteristics are trackedFile are distributed through system on entry
DMS File Properties
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 23
Schema defines DMS directory structureDMS creates directories on installationDMS uses round robin method to fill file systemsConfiguration not utilizing DB are restricted to one machineMax disk utilization is specified
DMS Configuration
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 24
File system - hold filesDatabase - holds file informationPublic Access - implement DMS interfaceWorker - manages file system
Two flavorsWith DB - spans multiple machinesWithout DB - single machine
DMS Component and Deployment
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 25
DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages
System Components, Java Packages
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 26
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 27
Provides well-defined interface deployed as a web service.Accepts job requestProvides Job StatusMonitor Job StateScalableTestable
CRG
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 28
DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages
System Components, Java Packages
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 29
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 30
One or more per nodeRetrieves data from DMSRetrieves and runs software packagesSaves results to DMSConsumes user jobs by using CRG Web Service
ALG
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 31
DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages
System Components, Java Packages
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 32
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 33
Provides well-defined interface deployed as a web service.Assigns jobs to CRGMonitors data in DMSMonitors the status of jobs in CRGApplies rules to dataGenerates notificationsVolatile logic lives hereProvides extension point for rules
Rules can be added or removed dynamically
ARM
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 34
DMS : Data Management System Stores DataCRG : Computational Resource Grid Job ManagementARM : Algorithm Rule Management Applies Product Rules to DataING : Ingest System Brings Data into SystemRUL : Product Production rules Ties software packages to dataALG : Algorithm Host Runs software packages
System Components, Java Packages
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 35
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 36
Mechanism to deploy algorithms into APSPSRules have namesRules use regular expressionsURL of software package specifiedDestination DMS can be specifiedDays of lease can be specified
RUL
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 37
System Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 38
Used by ARMSpecified by user
Specifies software package to run (SoftwarePackage)Specifies data of interest (FilterRegex)Specifies product output location (DmsUrl)Specifies product lifetime (DaysOfLease)
Rules and Software Packages
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 39
Development processTesting StrategyDefect ReportingDefect Correction
Configuration Management
Nightly BuildUnit and Regression Testing Testing ScenariosRequirements Mapping
Software Development and Verification
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 40
Spiral Method
DesignImplement TestDeploy
Development Process
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 41
We use standard software industry practices• Automate with ANT• Test with JUNIT
Saves Money and Increases Quality• Tests are reproducible• Tests can be run more often than they would be if they were manual• Tests are improved over time
We don’t just build, the process includes testing and verification
Testing Strategy
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 42
Anyone with an account can report a bugReporter chooses software systemReporter enters bug informationOwner of project is notified automatically by emailOwner prioritizes
Defect Reporting
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 43
Verify software before startingVerify problem existsFix and VerifyAdd back to code base
Defect Correction
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 44
Leverages open source softwareProvides versioned access to softwareProvides defect-tracking facilitiesUsed for source codeUsed for test scriptsUsed for software packagesBackup is made on regular basisAccounts are password protected
Configuration Management
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 45
Provides versioned access to softwareProvides defect-tracking facilitiesUsed for source codeUsed for test scriptsUsed for binaries
Configuration Management @ PDR
@PDR@PDR
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 46
Source code management
Software packages distributed to cluster as subversion repositoriesSoftware packages may contain large static data sets
Issue:
Subversion retrieval of large binary datasets slowSubversion stores two copies for diff purposes
Solution:
Use two repositoriesCheckout executing software packageExport static binary dataVerified solution with subversion contributor
Subversion
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 47
Builds systemTests subsystemsTests scenariosUpdates repositoriesLogs results
Scenarios demonstraterequirements
Nightly Build
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 48
May use internal knowledge interfaces for testingTest and exercise public interfacesStress test interfacesEvolve to test and verify bugsTests run in EclipseTest run from command lineFixed defects have specific tests added
Tests run in nightlyTests verify release
Layered approach to testingEverything tested, Every Night
Unit and Regression Testing
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 49
Test ingest functionTest forward and reprocess functions Tests rule executionTest user extension points
Testing Scenarios - Activity Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 50
Single Rule - Simplest test, connects data with software
Double Rule - Chain products togetherDouble Rule Reprocess Half - Chains products, reprocess second half of chain
MODIS L1->CM - real world SingleMODIS L1-> CM and LEOCAT - real world Double
Scenario / Use Case Test Instances
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 51
Level 3 requirements are mapped to level 4 requirements in requirements specification, 3600-0003.080402.doc
Operations Concepts illustrates requirements, 3600-0004.060911.doc
Scenario tests implement Operations Concepts use cases
Scenario tests have several instances
Generic instances exercise system.
Specific instances implement Ops Con use cases
Specific instances currently use MODIS and LEOCAT
Requirements Mapping
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 52
Algorithm LifecycleAlgorithm IngestAlgorithm QualificationLEOCAT
Science Investigation Support
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 53
Algorithm can come from anywhere.Once qualified, the algorithm can be applied from ARM.
Algorithm Lifecycle Use Case Diagram
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 54
Algorithm entered into subversionProduct created in bugzillaAlgorithm is ported and wrappedTests are created
Algorithm Ingest
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 55
Write a script to execute algorithmScript manages execution environmentAlgorithm name, inputs and outputs entered into ARM
Algorithm Qualification
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 56
APSPS executes bash scriptScript is passed arguments
• Binary Directory• Working / Output directory• Static Ancillary Directory• Dynamic Ancillary Directory
• Inputs as specified by rule• Outputs as specified by rule
Binary is executed from line in script
Results are stored by process that started script.
Qualified Algorithm Structure
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 57
Create slide content
LEOCAT
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 58
Create slide content
LEOCAT Structure
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 59
Org ChartStaffingHardwareFloor PlanSecuritySoftware Milestones
Logistics
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 60
Includes sourceIncludes generated persistence code (JAXB) > 600 lines in rulesDoes not include axis generated web service code ~ 30000 lines
ALG increase reflects subversion issueIncreased leveraging of java and third party libraries resulted in code size decrease.
Lines of code
Subsystem Lines Code (Sept 07) Predicted Total ( Sept 07 ) Line @ PDR ING 1200 2500 2515 ALG 600 1500 2101 CRG 2600 5000 1965 ARM 2300 8000 2046 DMS 5000 8000 4802 Rules 989 Total 11700 25000 14418
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 61
Not finished configuration control management plan - need to have backup systems
Write goals
APSPS accomplishments since PDR
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 62
Includes sourceIncludes generated persistence code (JAXB) > 600 lines in rulesDoes not include axis generated web service code ~ 30000 lines
ALG increase reflects subversion issueIncreased leveraging of java and third party libraries resulted in code size decrease.
Prototype system reuse
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 63
Create slide content
Organizational Chart
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 64
Create slide content
Staffing
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 65
Create slide content
Hardware
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 66
PEATE shares space with other projects including GOES archiveOperators located in adjoining roomRoom for expansion
PEATE Operations Floor Plan
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 67
Create slide content
Security
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 68
Create slide content
Build Plan
Space Science and Engineering CenterUniversity of Wisconsin-MadisonSpace Science and Engineering CenterUniversity of Wisconsin-Madison 69
Summary