C L

EarthCARE Level 2 DocumentationEarthCARE Level 2 Documentation

C L

ACM-CAPACM-CAPCloud and Precipitation BestCloud and Precipitation Best

EstimateEstimateSoftware Requirements SpecificationSoftware Requirements Specification

C L

(SRS)(SRS)

VARSY ProjectVARSY Project

C L

Code: L2-ACM-CAP-SRSIssue: 0.1Date: 27/08/2012Reference:

Name Function Signature

C L

Prepared by Robin Hogan Project Scientist

Reviewed by Pavlos Kollias Project Scientist

Approved by Pavlos Kollias Project Manager

Signatures and approvals on original

C L

C L

This page intentionally left blank

C L

C L

C L

Document InformationDocument Information

Contract Data

Contract Number: 4000104528/11/NL/CTContract Issuer: ESA-ESTEC

C L

Internal DistributionName Unit Copies

Pavlos Kollias McGill University 1Aleksandra Tatarevic McGill University 1Wanda Szyrmer McGill University 1

Internal Confidentiality LevelUnclassified Restricted Confidential

C L

External Distribution

Name Organisation Copies

Tobias Wehr ESA-ESTEC 1 Michael Eisinger ESA-ESTEC 1Dulce Lajas ESA-ESTEC 1Robin Hogan University of Reading 1

C L

Julien Delanoë LATMOS 1Gerd-Jan van Zadelhof KNMI 1David Donovan KNMI 1Alessandro Battaglia University of Leicester 1

Archiving

C L

Word Processor: MS Word 2003File Name: VARSY-READING-SRS-001

Document Status LogDocument Status Log

Issue Change description Date Approved

C L

01-draft Draft version. 25/08/2012

C L

Contents1. Purpose and Scope............................................................................................................................6

1.1. Applicable Documents................................................................................................................6

1.2. Reference Documents................................................................................................................7

1.3. List of Abbreviations...................................................................................................................8

2. Compliance Matrix.............................................................................................................................9

C L

C L

1. 1. PURPOSE AND SCOPEPURPOSE AND SCOPEThis document specifies the software requirements associated with the implementation of the ACM-CAP L2b algorithm.

1.1. 1.1. Applicable DocumentsApplicable DocumentsTable 1: Applicable Documents

C L

Reference

Code Title Issue

[SOW] EC-SW-ESA-SY-0310 Statement of Work: VARSY - 1-Dimensional VARiational Retrieval of SYnergistic EarthCARE Products

1.0

[CC] Appendix 2 to AO/1-6823/11/NL/CT

Draft Contract (attachment to SOW) 1.0

[AD 1] EC-SW-ESA-SY-0152 EarthCARE Level 2 Processor Development 1.0

C L

Reference

Code Title Issue

General Requirements Baseline

C L

Reference

Code Title Issue

[AD 2] EC.ICD.ASD.SY.00004 EarthCARE Product Definitions. Vol. 0:

Introduction

[AD 3] EC.ICD.ASD.SY.00005 EarthCARE Product Definitions. Vol. 1:

Common Product Definitions

1.0

C L

Reference

Code Title Issue

[AD 4] EC.ICD.ASD.ATL.00021 EarthCARE Product Definitions. Vol. 2b:

ATLID level 1

1.0

[AD 5] EC.ICD.ASD.BBR.00022 EarthCARE Product Definitions. Vol. 3b:

BBR level 1

1.0

C L

Reference

Code Title Issue

[AD 6] EC.ICD.ASD.MSI.00023 EarthCARE Product Definitions. Vol. 4b:

MSI level 1

1.0

[AD 7] ECSIM-DMS-TEC-ICD01-R ECSIM Simulator Interface Control Document

[AD 8] PE-TN-ESA-GS-0001 Ground Segment: File Format Standard 1.0

C L

Reference

Code Title Issue

[AD 9] EC-TN-ESA-GS-0218 Tailoring of the Earth Explorer File Format Standard for the EarthCARE Ground Segment

2.0

C L

1.2. 1.2. Reference DocumentsReference Documents

Table 2: Reference Documents

C L

Reference

Code Title Issue

[RD1] ECSIM-DMS-TEC-SUM-01-R ECSIM System User Manual

[RD2] ECSIM-KNMI-MAD01-R ECSIM Model and Algorithms Document

[RD3] EE-MA-DMS-GS-0001 Earth Explorer Mission CFI Software:

C L

Reference

Code Title Issue

General Software User Manual

[RD4] EOP-SM/1567/TW EarthCARE Mission Requirements Document

C L

Reference

Code Title Issue

[ATLAS-FR] EC-FR-KNMI-ATL-027 ATLAS Final report 1.0

[ATLAS-ACM-TC]

EC-TN-KNMI-ATL-ACM-TC-024 L2b Classification ATBD1.2

13/03/08

C L

Reference

Code Title Issue

[ATLAS-EBD]

EC-TN-KNMI-ATL-ATBD-A-EBD-021

L2a ATLID Extinction, Backscatter and Depolarization algorithm ATBD

1.1 27/04/09

[ATLAS-FM] EC-TN-KNMI-ATL-ATBD-A-FM-010

L2a ATLID Feature mask ATBD 2.2

C L

Reference

Code Title Issue

[RATEC-FR] RATEC-FR-READING-1 RATEC Final Report 1.0, April 2011

C L

1.3. 1.3. List of AbbreviationsList of Abbreviations

Table 3: List of abbreviations

Abbreviation Name

C L

1D-VAR RS 1-dimensional variational retrieval scheme

ATLID Atmospheric Lidar (The EarthCARE lidar)

CASPER Cloud and Aerosol Synergetic Products from EarthCARE retrievals

CPR Cloud Precipitation Radar (The EarthCARE radar)

EarthCARE The Earth Clouds, Aerosols and Radiation Explorer

C L

ECSIM EarthCARE Simulator

HSRL High-Spectral Resolution Lidar

MSI Multi-spectral Imager (The EarthCARE imager )

C L

2. 2. COMPLIANCE MATRIXCOMPLIANCE MATRIX

The following table lists all the requirements in the General Requirements Baseline [AD1] related to the software and the degree of compliance with each requirement.

*C=compliant, P=partly non-compliant, N=not compliant, NY=not yet compliant but will be, NA=Not applicable

C L

Reference Requirement Implementation Compliance*Design RequirementsDES-1 The level 2 processor shall be implemented as a

set of independent executable entities, referred to below as level 2 modules, each one implementing a specific algorithm generating a specific level 2 data product.

The ACM-CAP is implemented as a stand-alone executable

C

DES-2 No proprietary libraries or code shall be used Libraries used:GNU Scientific

C

C L

Library (GPL), LIDORT (permissive open source license), “Adept” automatic differentiation library (GPL), Multiscatter

C L

library (LGPL); note that Hogan is copyright holder for “Adept” and “Multiscatter”; Mishchenko’s T-matrix code (permissive BSD license);

C L

BHMIE does not specify a license so have emailed Bruce Draine.

DES-3 Only portable libraries or code shall be used.In this context, "portable" means that any libraries or code invoked within a level 2 module, including bespoke libraries and code,

The ACM-CAP currently uses an extension of the GNU C++

N

C L

must not depend on any special features of, e.g., the operating system, the compiler, the specific programming language implementation on the development or target platform.

compiler in permitting variable-length arrays (a C feature) in C++.

Functional RequirementsFUN-1 Level 2 modules shall derive geophysical

quantities (level 2 data products) fromSee associated ATBD

C

C L

EarthCARE level 1b data products. Synergistic (level 2b) modules may use EarthCARE level 2 data products on input as well (see also INT-10).

FUN-2 Level 2 modules shall implement the algorithm specified in the corresponding ATBD.

C

FUN-3 Level 2 modules shall compute and store into products: geophysical parameters, error descriptors and product confidence data (i.e., quality flags).

See associated PDD

C

C L

FUN-4 Level 2 modules shall check whether all required data are available before running.

C

FUN-5 Level 2 modules shall respond in a controlled way to instrument degradation or failure or unexpected values in the input data as per INT-10.

Meaningful errors are reported and program gracefully exits if appropriate in response to

C

C L

missing or corrupt data

FUN-6 Convergence criteria shall be established for each algorithm. These criteria shall be monitored during the execution of the algorithm. Abortion criteria shall be established and applied for deviation from accepted convergence profile. Respective error information shall be produced.

The convergence criterion is that the L2 norm of the gradient of the cost function falls to less than

C

C L

unity. If this is not reached in a specified number of iterations, the algorithm aborts. Error and convergence information is

C L

reportedFUN-7 Level 2 modules shall provide a trace of their

execution in a single Log File for each execution run.

C

FUN-8 The configuration files of a level 2 module shall allow to configure the logging level of the level 2 module.

See associated ICD

C

FUN-9 The following logging levels shall be supported for a level 2 module: Error, Warning, Info,

See associated ICD

C

C L

Debug, Progress.FUN-10 An Error shall be logged when an unexpected

condition was detected and thelevel 2 module cannot perform any error recovery and must exit.

See associated ICD

C

FUN-11 A Warning shall be logged when an unexpected condition was detected but error recovery is possible.

See associated ICD

C

FUN-12 An Info shall be logged when a significant See associated C

C L

condition occurred that directly or indirectly affects the level 2 module outputs. An info shall also be logged at module startup and module termination.

ICD

FUN-13 Debug should be used as needed for the development of the Level 2 module.

See associated ICD

C

FUN-14 Progress shall be used to log the progress of the simulation from 0% to 100%, exploiting the processing steps of level 2 modules.

See associated ICD

C

C L

Interface RequirementsINT-1 The Unix operating system shall be used. The

actual hardware/operating systemconfiguration shall be PC (multi-core/multi-processor system if needed)/Linux. Details(including the Linux distribution and its version) shall be agreed with the agency.

Code developed under Linux

C

INT-2 Coding shall be performed in a high-level programming language such as C/C++

C (1990 standard), C++

C

C L

or Fortran-90. Details (including compiler versions) shall be agreed with the Agency.

(1998 standard) and Fortran (1977 and 1990 standards) are required

INT-3 Compilation options and compiler versions shall be documented and justified.

GNU compiler collection is needed

C

C L

INT-4 Each level 2 module shall be able to run stand-alone.

C

INT-5 Besides being able to run stand-alone, Level 2 modules shall be compatible with and run within the EarthCARE end-to-end simulator ECSIM.

Not yet NY

INT-6 Input to and output from level 2 modules shall be performed via files.

C

INT-7 In addition, level 2 modules shall use Unix standard output for status messages.

Not yet, currently sends

NY

C L

some messages to standard error

INT-8 Level 2 modules shall not use Unix standard input.

C

INT-9 Level 2 modules shall not produce graphical output. Graphical output (e.g. for quicklook results) may be produced from additional tools.

C

INT-10 Level 2 modules shall read the following files on C

C L

input: 1. one EarthCARE level1b data product (in case of synergistic processors: one or more EarthCARE level 1b data products from different instruments, and/or one or more level 2 data products) coveringup to 2 orbits worth of data, 2. a configuration file, 3. support files (e.g., meteorological data from ECMWF, look-up tables, topographic databases, etc.) as required.

C L

INT-11 Level 2 modules shall write the following files on output: 1. one level 2 data product, 2. a log file.

Currently no log file is written but this will be fixed

NY

INT-12 All files (including configuration and ntermediate input/output files) shall be in XML or NetCDF format. The default format shall be XML while NetCDF shall be used for data products and large files only (i.e., where using

Output data is in netcdf but configuration is via an ascii format that is

NY

C L

XML would significantly degrade performance). Details (including the NetCDF version) shall be agreed with the agency.

much more easily edited.

INT-13 The functions of the Earth Explorer mission CFI software shall be used wherever possible.

What are these ?

N

INT-14 The reference systems defined in the Earth Explorer mission CFI software shallbe used throughout, in particular for all outputs.

??? N

INT-15 All processing parameters including the C

C L

environment of level 2 modules (e.g.,paths) shall be configurable in external configuration files without code updates or recompilation.

INT-16 Level 2 modules shall read their configuration file(s) at run-time at start-up.

C

INT-17 Level 2 products shall follow the general EarthCARE file format as specified in [AD1].

Not sure the general baseline document

NY

C L

defines a format?

INT-18 Level 2 modules shall generate one level 2 output data product per level 1 input data product.

C

INT-19 Each level 2 module, complete with reference configuration and support files, shall be delivered as a single tar file, optionally compressed by gzip.

C

C L

INT-20 No unused or otherwise unneeded files shall be delivered with a level 2 module.

C

INT-21 Each level 2 module shall be delivered with a script that plugs it into the ECSIM framework.

NY

INT-22 Only relative path names shall be used in the installation scripts of Level 2 modules.

C

INT-23 The tar file used to deliver a level 2 module shall conform to the naming convention described in Section 6.7 of the General Requirements

NY

C L

Baseline Document [AD?]Testing RequirementsTST-1 Test entry points shall be defined to facilitate

testing and debugging. They shall allow running selected parts of the processing chain by providing breakpoints for reading intermediate input data and writing intermediate output data.

The ACM-CAP processor ins indivisible

N

TST-2 ECSIM shall be used for end-to-end performance testing using appropriate ECSIM models

NA

C L

(including scenes, forward models, instrument and L1b simulators). Furthermore, realistic scenes shall be used, derived from existing measurements.

Configuration management requirementsCFG-1 All elements and sub-elements composing the

level 2 processor shall be put under configuration management.

What is configuration management?

N

CFG-2 At least the following configuration items shall N

C L

be managed under configurationmanagement:

Software modules (source code and object code)

Static configuration and support data Acceptance Test Data Sets Procedures and scripts Any static data, e.g., delivered tar files Software and hardware environment

C L

documentationCFG-3 Widely supported Software Configuration

Management tools shall be used.N

CFG-4 It shall be possible to know, for any discrete point in time, the softwareconfiguration of the level 2 processor.

?

CFG-5 It shall be possible to retrieve, for any discrete point in time, the level 2 processorin this configuration.

?

C L

CFG-6 To maintain software integrity throughout the life cycle, a proper and systematiccontrol of software changes shall be provided.

NY

CFG-7 For all versions of any software module, the modifications with respect to theprevious version shall clearly be identified and documented in the software module itself.

NY

CFG-8 All versions of all software modules shall be archived with a version identifier

NY

C L

and the time from which the software module was in use.

CFG-9 For each delivery of any Configuration Item, a Change Log detailing all changessince the previous delivery shall be provided.

NY

CFG-10 Each version of an Level 2 module shall be uniquely identified through anOverall Version Number in the following format:VERSION = Major Version.Minor Version, each

C

C L

over two digitsCoding RulesCOD-1 Common code should be separately packaged

and reused within a single level 2module as well as across level 2 modules.Code duplication should be eschewed as far as possible.

C

COD-2 Code related to the interfaces (input/output) shall be separated (in different

C

C L

subroutines and different source code files) from code implementing the algorithm.

COD-3 File or path names shall not be used inside the code.

C

COD-4 Redundant calculations in Level 2 modules should be avoided.

C

COD-5 It shall be shown that code does not include any uninitialised variables, unused variables, memory leaks, and dead (i.e., unused) code.

All issues can be verified rigorously using

P

C L

various compiler options, except for memory leaks. The absence of any potential leaks would be difficult to rigorously

C L

demonstrateCOD-6 Standard headers shall be used for subroutines.

Details shall be agreed with the agency.Define “standard header”

?

COD-7 Ample use of comments shall be made. CCOD-8 Comments shall be used to describe the structure

of the code.C

*C=compliant, P=partly non-compliant, N=not compliant, NY=not yet compliant but will be, NA=Not applicable

C L