ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Aprim

eron

netCDFForm

at

G.Giuliani

InternationalCentreforTheorethicalPhysics

-Trieste

Earth

System

Physics

Section

ICTP

-Earth

System

PhysicsSection

AdvancedSchoo

lon

Scientific

SoftwareDevelop

ment

Trieste,20

Feb

-2Mar

2012

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Datain

Science

�Afourth

paradigm

afterexperim

ent,theory,andcompu

tation

�Involves

collecting,

exploring,

visualizing,

combining

,subsetting

,analyzing,

andusinghu

gedata

collections

�Challeng

esinclud

e�

Delug

eof

observationaldata,exafl

oodof

simulationmod

elou

tputs

�Needforcollaborationam

onggrou

ps,disciplines,commun

ities

�Finding

insigh

tsanddiscoveriesin

aSea

ofData

�Data-intensivesciencerequ

ires

�New

tools,techniqu

es,andinfrastructure

�Stand

ards

forinteroperability

�Institutionalsupp

ortfordata

stew

ardship,

curation

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Roles

inDataIntensiveScience

�Datausers:

access,un

derstand

,integrate,

visualize,

analyze,

subset,andcombine

data

�Datascientists:developinfrastructure,standards,

convention

s,fram

eworks,data

mod

els,Web-based

techno

logies

�Scientists/researchers:

acqu

ire,

generate,analyze,

check,

organize,form

at,do

cument,share,

publishresearch

data

�Softwaredevelopers:

developtools,form

ats,interfaces,

libraries,services

�Datacurators:preserve

data

contentandintegrityof

science

data

andmetadatain

archives

�Researchfund

ingagencies,profession

alsocieties,

governments:encouragefree

andop

enaccess

toresearch

data,advocate

elim

inationof

mostaccess

restrictions

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Growth

indata

from

Sensors

According

toScience

article[201

1-02

-11,

Baraniuk]:

�Majorityof

data

generatedeach

year

now

comes

from

sensor

system

s

�Amou

ntgeneratedpassed

storagecapacity

in20

07

�In

2010

theworld

generated12

50billion

gigabytesof

data

�Generated

data

grow

ingat

58%

per

year

�Storage

capacity

grow

ingat

40%

per

year

�Wegenerate

morescientificsensor

data

than

wecanprocess,

commun

icate,

orstore(e.g.LHC)

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

DataMod

el

�Whatisadata

Mod

el?

�Acollectionof

data

objects

�Acollectionof

operations

tobeappliedon

data

objectssuch

asretrieval,up

date,subsetting

,averaging

�Acollectionof

integrityrulesthat

defin

elegalstates

orchange

ofstate

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

DataMod

el

�Whatisadata

Mod

el?

�Acollectionof

data

objects

�Acollectionof

operations

tobeappliedon

data

objectssuch

asretrieval,up

date,subsetting

,averaging

�Acollectionof

integrityrulesthat

defin

elegalstates

orchange

ofstate

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

DataMod

el

�Whatisadata

Mod

el?

�Acollectionof

data

objects

�Acollectionof

operations

tobeappliedon

data

objectssuch

asretrieval,up

date,subsetting

,averaging

�Acollectionof

integrityrulesthat

defin

elegalstates

orchange

ofstate

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

DataMod

el

�Whatisadata

Mod

el?

�Acollectionof

data

objects

�Acollectionof

operations

tobeappliedon

data

objectssuch

asretrieval,up

date,subsetting

,averaging

�Acollectionof

integrityrulesthat

defin

elegalstates

orchange

ofstate

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

DataInfrastructure

�App

lications

dependon

lower

layers

�Sharing

requ

ires

agreem

ents

�form

ats

�protocols

�convention

s

�Dataneedsmetadata

�Isallthisinfrastructure

really

necessary?

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataform

at

�ASCII

�Easyto

read

ifsm

all

�Can

getlarge

�Haveto

know

structureto

makeplots

�Slow

toread,write

�Binary

�Smaller,faster

than

ASCII

�Haveto

know

structure

�Not

necessarily

portable

�Opaqu

efrom

outsideapplication

�netCDF

�Binaryindexedportableform

atwithstandard

access

APIfor

bothdata

andmetadata

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFDataform

at�

Self-Describing:A

file

includesmetadataaswellasdata:descriptionof

variables,unitsofmeasure,etc.

�Portable:Datawrittenononeplatform

canbereadonotherplatform

s.

�Direct-access:A

smallsubsetofalargedatasetmaybeaccessed

efficiently,withoutfirstreadingthroughalltheprecedingdata.

�Appendable:Datamaybeefficiently

addedtoafile

withoutcopyingthe

datasetorredefiningitsstructure.

�Extensible:Addingnew

dim

ensions,variables,orattributestofilesdoes

notrequirechangestoexistingprogramsthatreadthefiles.

�Sharable:Onewriterandmultiple

readers

maysim

ultaneously

accessthe

samefile.WithParallelnetCDF,multiple

writers

mayefficiently

and

concurrently

writeintothesamefile.

�Archivable:Accesstoallearlierform

sofnetCDFdatawillbesupported

bycurrentandfuture

versionsofthesoftware.

�Networkable:Clientaccesstoremoteservers

throughOPeNDAP.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFInfrastructure

�Providesform

atandlibrary

for

netCDFdata

mod

el

�End

orsedby

severalstandards

bod

ies

�Activeconvention

scommun

ities

�OPeN

DAPprotocol

�Several

serversforremotedata

access

�Manyop

ensource

andcommercial

utilities

andapplications ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFHom

e

Unidata

Site

http://w

ww.unidata.ucar.edu/

software/netcdf

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFUsers

�Clim

atemod

elers

�Program

forClim

ateMod

elDiagn

osisandIntercom

parison

(PCMDI)

�Earth

SystemsGrid

�Ocean

andatmosph

eric

sciences

�Forecastmod

els

�Atm

osph

eric

chem

istry

�Neuroim

aging

�MINC-Medical

ImageNetCDF

�NiBab

el

�Fusionresearch

�Culham

CentreforFusionEnergy(C

++

APIfornetCDF-4)

�Molecular

dynamicssimulations

(e.g.AMBER)

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFStand

ardendorsem

ent

�2009-02-05:NASA

Earth

Science

DataSystems(ESDS)

Stand

ards

Process

Group

endorsed

netC

DFclassicand64

-bit

offsetform

atsas

appropriateforNASA

Earth

Science

data.

�2010-03-1:Integrated

Ocean

Observing

System

(IOOS)Data

Managem

entandCom

mun

ications

(DMAC)Sub

system

endorsed

netC

DFwithClim

ateandForecast(C

F)convention

sas

apreferreddata

form

at.

�2010-09-27:SteeringCom

mitteeof

theUSFederal

GeographicDataCom

mittee(FGDC)offi

cially

endorsed

netC

DFas

aCom

mon

Encod

ingStand

ard.

�2011-03-07:Open

GeospatialCon

sortium

(OGC)approved

”OGCNetworkCom

mon

DataForm

(NetCDF)Core

Encod

ingStand

ardversion1.0”

asanew

OGCstandard.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFClassicDataMod

el

AnetC

DF”classic”fileiscomposed

of:

�Dim

ension

s

�Variables

�Attribu

tes

�Data

Afilecanhave

attributes,dimension

s,andvariables.

Dim

ension

sareused

tospecifyshap

esof

variables.

One

dimension

canbeun

limited

(record)

Avariable

canhave

dimension

sandattributes.

Multiplevariablescansharedimension

s(beon

agrid).

Variables

areof

fixed

prim

itivetype(char,int,flo

at)

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dim

ension

Dim

ension

sareused

todefin

eshap

esof

variables.

Eachdimension

musthave:

�Uniqu

enamein

afile

�Aleng

th,i.e.an

integernu

mber

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Attribu

te

Anattributeisused

tostoremetadata,

either

atfileor

variable

level.Eachattributemusthave:

�Uniqu

enamein

level(file

orvariable)

�Atype

�Avalue

Metadataareused

toestablishconvention

sto

sharedata.

For

exam

ple,

fortheClim

ateandForecastCFconvention

,a

variable

MUST

have

someattributes

(for

exam

pleun

its,standard

name,

etc.),

andtheconvention

nameitselfisamandatory

attributeat

filelevel.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Variable

Avariable

istheshap

ed(bydimension

)storageof

data,defin

edby

itsmetadata(attribu

tes).Eachvariable

musthave:

�Uniqu

enamein

afile

�Atype

�Zero(scalarvalue)

ormoredimension

s

�Zeroor

moreattributes

�Asmanydata

values

asspecified

byitsshap

e

Actualscientificdata

arestored

invariables.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFCom

mon

DataLanguage

netcdfsnow{

//exampleofCDLnotation

dimensions:

lon=9;

lat=7;

time=unlimited;

//3currently

variables:

floatIR_flux(lon,lat);

IR_flux:units="Wm-2";

IR_flux:_Fill_value=-999;

IR_flux:standard_name=

"downwelling_longwave_flux_in_air";

floatsnow_cover(time,lon,lat);

snow_cover:units="kgm-2";

//globalattributes

:title="simpleexample,lackssomeconventions";

data:

IR_flux=200,201,...;

snow_cover=0.1,0.2,0.0,...;

}ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

netCDFV4

The

netcdf

data

mod

elisfurtherextend

edwiththenew

V4

form

at,bu

iltup

ontheHDF5data

form

at.

�Multipleun

limited

dimension

s

�Userdefin

edtypes

andop

aque

types

�Datacanbegrou

ped

together

�Com

pression

andchun

king

�NativeParallelandHPCoriented.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Operating

System

choice

Targetof

system

usage

�DataArchive:netC

DFdata

aresing

lefiles

ondisk.Adecent

filesystem

isneeded,go

forUNIX

(orLinux).

�DataCreation:will

dependon

data

source.Ifmod

el,an

HPCsystem

may

berunn

ingit,anditisusually

somesort

ofclusterwithaUNIX

OS.

�DataAnalysis:Usually

perform

edusingothertools,will

be

researcher

choice.Bewarned.

Try

toKISS.

�SoftwareDevelopment:as

netCDFisdistribu

tedalso

asbinary,anyOScanbeused

(evencommercial

desktop

solution

s).Con

siderjust

TCO.

�DataDistribution:theDAPserver

stackisbasedon

Apache

Tom

cat,so

thistask

will

bebestserved

byaLinux

system

.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Installation

type

�Linuxsystem:usually

netC

DFcomes

alreadypacked

inall

know

ndistribu

tion

s,usenorm

alpackagemanager

toinstall.

NOT

OK

forHPC.

�Commercialdesktop:Findprebuiltbinaries

oruseJava

software

�HPC

orUNIX

system:Com

pile

from

source

thelibrary.

�TomcatServer:Just

deploy

theapplicationon

theserver.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Buildingfrom

Source

Normally

just

theCcompilerisneeded,may

also

need

Fortran

orC++

ifusersaredevelopers.

Mostof

thesoftwarebu

iltusing

netC

DFjust

usetheCAPI.Som

epieces

ofsoftwarearerequ

ested.

Follow

checklist.

�Compressionrequested:V4feature.

Needzlib,HDF5.

Ifno

texplicitly

asked,

canavoidszip.

�HPC

system:ProbablyparallelI/O

requ

ested.

NeedHDF5

andavend

oror

opensource

MPIstack.

�Networkeddatasetaccess:Will

need

theclient

OpenDAP,

which

requ

ires

curldevelopm

entlibrary

installed(needxm

lsupp

ort).

Fullfeatureenvironm

entwill

requ

ireto

envelopallrequ

irem

ents.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Normal

Install

The

”normal”installation

procedureisjust

thefour

steps:

�confi

gure

:ascript

which

serchrequ

ired

pieces.Written

tobu

ildthemost,ifneeded

compilers

/libraries

arefoun

don

system

.

�make:binary

distribu

tion

library

iscompiled.

�makecheck:will

build

andrunsometest

prog

ramsto

verify

againstkn

ownresultstheprevious

step

�makeinstall:copy

allneeded

files

tousethesoftwarein

prod

uction

.Ifinstallation

prefixison

system

space,

administrator

credentialsareneeded.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Config

uringbu

ild

The

mostim

portant

step

isconfi

gure.Verifythat

�Allrequ

estedfeatures

arecorrectlyactivated

�Allno

tneeded

features

arecorrectlydeactivated

�You

areusingthesamecompilerforallsoftwarecompilation

�The

installation

path

iscorrectandwritable.

Ifanyof

themakestepsfails,restartfrom

confi

gure.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataaccess

orcreation

Implem

enting

asoftwareto

read

orwrite

data

innetcdf

form

at,

check

�The

BASElibrary

iswritten

inC.Ifperform

ance

needed,use

C.

�ALOT

OFDOCUMENTATIO

Nisavailable.

So,

RTFM.

�The

C++

library

isun

derheavydevelopm

ent.

Giveatryto

thenew

V4C++.

�IfJava

ischosen,useeclipse.

�IfFortran

isneeded,consider

tostartmigrating

toF90

standard.

�Why

notgo

ingforan

high

erlevellang

uage,such

aspython

orperl?

�Dono

tuseanyun

documentedfeature.

Itmay

breakandit

will

break.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Dataanalysis

Analyzing

netC

DFdata,consider

thefollowingpoints:

�Why

write

You

row

ncustom

prog

ram

toaccess

dataset,if

commandlinetoolssuch

asNCO

canbeused?

�Try

toworkat

filelevelusingop

erators:

faster,easier.

�Workby

steps:

dono

ttryto

analyzeterabyte

ofdata

atfirst

try:

subset

data

usingop

erators,analyzesubsets,andif

procedureisOK,go

forbiggo

al.

�Ifaplot

isneeded,useuser

friend

lytoolslikeIDV,GrA

DS,

NCL.Dono

tim

plem

entallback

atlow

level:You

are

supp

osed

tobearesearcher,do

n’tyou?

�Try

asmuchas

possibleto

access

data

throug

hnetwork

servers:

alocalcopy

ofdata

isno

tYou

rtarget.You

need

data

tobu

ildkn

owledg

e,letSysAdm

insdo

theirwellpaid

jobs.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Datadistribu

tion

DistributingnetC

DFdata,consider

thefollowingpoints:

�Ask

for,addby

yourself,integrateas

muchmetadataas

possiblein

distribu

ting

data.Beno

tredu

ndant,bu

tensure

anyinform

ationON

data

isdistribu

tedWITH

data:thisis

oneof

theprim

arygo

alof

netC

DFform

at.

�Toarchivedata,useexstensively

compression

andchun

king

,to

have

smallstoragespaceandfast

data

access.

�You

areno

tjust

distribu

ting

numbers:

youaredistribu

ting

the

base

forthekn

owledg

e.Bereadyto

enhanceon

user

requ

est.

Adeveloper

will

ALW

AYSbeneeded

alon

gadistribu

tion

site,

toim

plem

entnew

features.

�Enforce

from

thebeginning

distribu

tion

policies,bu

thave

wider

aspossibledistribu

tion

asago

al.Ask

forfree

access,

implem

entaccess

asfree

aspossible.

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Program

mingExample

#include<netcdf.h>

...

/*Handleerrorsbyprintinganerrormessageandexiting*/

#defineERR(e){printf("Error:%s\n",nc_strerror(e));exit(ERRCODE);}

... /*netCDFfileIDandvariableID*/

intncid,varid;

/*arrayintowhichwewillreadvaluesof2DnetCDFvariable*/

doublerh_array[NLAT][NLON];

...

/*Openfilewithread-onlyaccess,indicatedbyNC_NOWRITEflag*/

if((retval=nc_open("foo.nc",NC_NOWRITE,&ncid)))ERR(retval);

/*Gettheidofthevariablenamed"rh"*/

if((retval=nc_inq_varid(ncid,"rh",&varid)))ERR(retval);

/*Readvariable"rh"asdoubles,rh_arraymustbebigenough!*/

if((retval=nc_get_var_double(ncid,varid,&rh_array[0][0])))ERR(retval);

/*Closethefile,freeingallresources.*/

if((retval=nc_close(ncid)))ERR(retval);

ICTP

-ESP

netCDF

ictp-log

o

Scien

tificDataFormat

netCDFDataFormat

netCDFDataModel

SysAdmin

netCDF

Usage

ofnetCDF

Thank

You

!

ICTP

-ESP

netCDF