logo prism project 1 lessons of unix/gmt/fortran naota hanasaki iis,ut draft

1

LOGOPRism project

Lessons of UNIX/GMT/FORTRANLessons of UNIX/GMT/FORTRANNaota Hanasaki IIS,UT

DRAFT

2

YOUR SITE HERE

LOGOPRism project

AcknowledgementAcknowledgement

I thank all attendees (2002-2005): T. Yamada, Y. Suga, A. Aslam, C. Manusthiparom, C. Apirumanekul, T. Okazawa, T. Kokubo, K. Takagaki, W. Saita, K. Kubo, Y. Koiwa, Y. Ishizaki, Q. Tang, N.S. Farzin, J. Cho, M. Lin, T. Inuzuka, T. Sakimura, D. Ikari, Y. Arai and N. Utsumi

I thank all advisors (2002-2005):Dr. T. Oki, Dr. S. Kanae, Dr. Y. Agata, Dr. S. Seto, Dr. Y. Hirabayashi, Dr. S. Miyazaki, Dr. D. Komori, Dr. Y. Shen and Dr. S. Islam

3

YOUR SITE HERE

LOGOPRism project

Table of ContentsTable of Contents

Introduction Lesson 1

– Operating UNIX

– Basic UNIX commands (cp, ls, etc.) Lesson 2

– Linking command and files

– Redirect/Pipe/Wildcard/Awk Lesson 3

– Drawing global precipitation distribution

– GMT scripts (xyz2grd, grdimage, pscoast etc.) Lesson 4

– Processing grid data 1

– Arithmetic of grid data using simple programs

4

YOUR SITE HERE

LOGOPRism project

Table of ContentsTable of Contents

Lesson 5 – Reading/Writing binary– Fortran IO

Lesson 6– Processing grid data 2– Masking grid data using simple programs

Lesson 7 – Fortran and Makefile– Useful fortran tips, function/subroutine, makefile

Lesson 8– Bourne shell script– do-loop, for-loop, if-statement

Lesson 9– Setting UNIX environment– .cshrc

5

LOGOPRism project

IntroductionIntroductionWhy UNIX?

6

YOUR SITE HERE

LOGOPRism project

IntroductionIntroduction

An assignment of “Advanced Hydrology” in 2001– Read the following article

Xie and Arkin 1997: Global Precipitation: A 17-year monthly Analysis Based on Gauge Observations, Satellite Estimates and Numerical Model Outputs. BAMS, 78, 2539-2558

– Summarize their methodology– List up advantage/disadvantage– Compare the data with other data

(station observation, global dataset, etc.)– See the site below for original data ftp://

ftp.ncep.noaa.gov/pub/precip/cmap/monthly/

7

YOUR SITE HERE

LOGOPRism project

Introduction (continued)Introduction (continued)

To submit a good report, you need– Powerful computation environment

UNIX

– Skill to process large data FORTRAN (or any other language)

– Skill to create beautiful/complex figures GMT (or any graphic package)

– Skill to manage iterative process Shell script

8

YOUR SITE HERE

LOGOPRism project

Introduction (continued)Introduction (continued)

Target of this seminar– Tutorial of the assignment– Using CMAP dataset, analyze global

precipitation How different global precipitation pattern

between El Nino year, La Nina year? How many people in the world live with less

than 500 mm/year?

– Practical and applicable lessons for your bachelor/master/doctor research

9

LOGOPRism project

Lesson 1Lesson 1Basic UNIX operation (1)

Fundamental UNIX commands

10

YOUR SITE HERE

LOGOPRism project

Today’s goalToday’s goal

Where can we use UNIX? How do we log in? How do we operate UNIX?

11

YOUR SITE HERE

LOGOPRism project

Oki Lab’s UNIX EnvironmentOki Lab’s UNIX Environment

Environment– OS: Solaris (not linux)– Machines: Sun Microsystems

Servers– Rainbow– Hydro– Reservoir– Chaser

Client (PC X server)– Astec X (commercial software)

Please ask your senior for installation and initial setting. Detailed environment setting will be shown in Lesson 9.

12

YOUR SITE HERE

LOGOPRism project

Login/LogoutLogin/Logout

Go to login screen Select language

– [Options]-[languages]-[select] Type user name & password Open terminal window

– [Third button]-[Tools]-[Terminal]

13

YOUR SITE HERE

LOGOPRism project

Managing files (1/2)Managing files (1/2)

Show present directory % pwd

Show file list % ls

Change directory % cd directory

Change directory (to upper) % cd ..

Change directory (to home) % cd ~

Change directory (to current) % cd .

View file % less file% more file

(to quit, type “q”)

Edit file % mule file

Copy file % cp original copy

Move file or Rename file % mv original copy

Remove file % rm file

14

YOUR SITE HERE

LOGOPRism project

Managing files (2/2)Managing files (2/2)

Make new directory % mkdir directory

Remove directory (not empty)

% rm –r directory

Remove directory (empty)

% rmdir directory

Copy directory % cp –r directory

Change permission % chmod 744 fileUser(1st)-Group(2nd)-Other(3rd) Read(+4), Write(+2), Execute(+1)

% chmod u+x

User(u)-Group(g)-Other(o) add(+), remove(-) Read(r), Write(w), Execute(x)

Compress file % gzip file

Uncompress file % gunzip file

15

YOUR SITE HERE

LOGOPRism project

Editing file (1/2)Editing file (1/2)

Mule % mule file

Save Ctrl-x + Ctrl-s

Exit Ctrl-x + Ctrl-c

Delete Ctrl-d

Delete line (kill) Ctrl-k

Cut Ctrl-Space (starting point), Ctrl-w

Paste (yank) Ctrl-y

Forward cursor Ctrl-f

Backward cursor Ctrl-b

Upward cursor (previous) Ctrl-p

Downward cursor (next) Ctrl-n

Page down; End Ctrl-v; Esc->

Page up; Home Esc-v; Esc-<

16

YOUR SITE HERE

LOGOPRism project

Editing files (2/2)Editing files (2/2)

Kanji Ctrl-\

Cancel Ctrl-g

Find Ctrl-s

Replace Esc-Shift-%

Replace all Esc-x replace string

See Emacs/Mule text books for detail

17

YOUR SITE HERE

LOGOPRism project

Other commandsOther commands

Netscape % netscape

Ghost view(PS file viewer)

% gv file

X view(JPEG/GIF viewer)

% xv file

Convert image file % convert file.ps file.gif

% convert file.gif file.ps

Print file % lpr -Pprinter file

See printer job % lpq –Pprinter

Remove printer job % lprm –Pprinter jobID

18

YOUR SITE HERE

LOGOPRism project

In emergency…In emergency…

See manual % man command

Break process Ctrl-c

See process % top

Kill process % kill processID

Exit % exit

19

YOUR SITE HERE

LOGOPRism project

What is permission?What is permission?

Create a text file:

Save as “example1.txt” Change permission and add executable

% chmod 744 example1.txtor% chmod o+x example1.txt

Execute the file (“text” file can be executed)

% example1.txt

pwdlswho

20

YOUR SITE HERE

LOGOPRism project

ExerciseExercise

Check your current directory Go to home directory (/home/user) Make directory “CMAP” Go to directory “CMAP” Copy files from “/home/hanasaki/pub/unix_s

emi/src/”global.gmtgrad.cptdata.xyz

Change permission of file global.gmt to executable

Execute global.gmt View output file “image.ps” Print out the image Convert “image.ps” from Post Script format

to GIF format

21

LOGOPRism project

Lesson 2Lesson 2Basic UNIX operation (2)

Redirect/Pipe/Wildcard/Awk

22

YOUR SITE HERE

LOGOPRism project


Open the file global.gmt. You will find the file consists of numbers of short commands.

If you look more carefully, you will find curious characters such as>, >>, <, <<, | …

These characters are extremely important in UNIX operation. Let’s learn about them.

23

YOUR SITE HERE

LOGOPRism project

Redirect(Write/Read files)

Redirect(Write/Read files)

Redirect is used – to write the output of a command to a file

– to read the input of a command from a file

Write % command > file

Write (add lines) % command >> file

Overwrite % command >! file

Read % command < file

Read until EOF appears

% command << EOF

strings

EOF

24

YOUR SITE HERE

LOGOPRism project

Example/ExerciseExample/Exercise

Example 1

Exercise 1– Make a file listing who are logging in our

system. (Use command “who”)– Make a file listing the calendar for 2004 to

2005. (Use command “cal”)

% date > redirect.txt% more redirect.txt% date >> redirect.txt% more redirect.txt% date >! redirect.txt% more redirect.txt

% cat << EOF> Hello.> My name is Taikan.> EOF% more redirect.txt

% cat << EOF >> redirect.txt> Hello.> My name is Taikan.> EOF% more redirect.txt

25

YOUR SITE HERE

LOGOPRism project

Pipe(Linking commands)

Pipe(Linking commands)

Pipe hands the output of a command to the successive command as a input

% command 1 | command2 Firstly, command1 is executed.The output of command1 is input to command 2.

26

YOUR SITE HERE

LOGOPRism project


Example 2

Exercise 2– Count the number of files and directories in y

our home directory. (Use command “wc”) By using redirect ( > ) By using pipe ( | )

% ls –l% ls –l | more% ls –l | tail% ls –l | head

27

YOUR SITE HERE

LOGOPRism project

Wild cardWild card

Wild card is a special character to indicate any character or string.

* Any character(s) including null

? Any one character

28

YOUR SITE HERE

LOGOPRism project


Example 3

Exercise 3– Copy again the above directory (wildcard2)– Remove files include character “1”– Remove all files in the directory “wildcard”– Copy all files from

/home/hanasaki/pub/unix_semi/lesson2/wildcard/(Do not use –r option for command “cp”)

% cp –r /home/hanasaki/pub/unix_semi/lesson2/wildcard/ .% cd wildcard% ls –l% ls –l *txt% ls –l test*% ls –l te?t1.txt

29

YOUR SITE HERE

LOGOPRism project

AwkAwk

Awk is a kind of program language of UNIX system. Its grammar is quite similar to C language. No need for compilation, directly type to command line. In this lesson, we just use a function of awk.

Print the N th column in file % awk ‘{print $N}’ file

Print the N th column if the M th column is m

% awk ‘($M==m){print $N}’ file

30

YOUR SITE HERE

LOGOPRism project


Example 4

Exercise 4– Let’s make “data.xyz”, we used in Lesson 1.

% cd % ls –l > home.txt% awk ‘{print $9}’ home.txt% awk ‘($5==512){print $9}’ home.txt

31

YOUR SITE HERE

LOGOPRism project

Create data.xyzCreate data.xyz

Visit CMAP web site(ftp://ftp.ncep.noaa.gov/pub/precip/cmap/)

Download a monthly data file 1979 Uncompress the file

% gunzip cmap_mon_v0411_79.txt.gz

Check the file

% more cmap_mon_v0411_79.txt

32

YOUR SITE HERE

LOGOPRism project

Create data.xyzCreate data.xyz

The file should be as follows

Create monthly file (for Jan 1979)

% awk ‘($2==1){print $4, $3, $5}’ cmap_mon_v0411_79.txt > data.xyz

1979 1 -88.75 1.25 0.08 60.00 0.08 60.001979 1 -88.75 3.75 0.07 60.00 0.07 60.001979 1 -88.75 6.25 0.07 60.00 0.07 60.001979 1 -88.75 8.75 0.07 60.00 0.07 60.001979 1 -88.75 11.25 0.07 60.00 0.07 60.001979 1 -88.75 13.75 0.07 60.00 0.07 60.00

yearmonth

lat londata

See the CMAP documents

33

LOGOPRism project

Lesson3Lesson3Drawing figure

<GMT>

34

YOUR SITE HERE

LOGOPRism project


In lesson 1, we drew a nice figure by executing global.gmt.– How was the figure drawn?– How we can modify it?

By the way, what is GMT?– GMT is short for Generic Mapping Tool– Free software developed by Paul Wessel an

d Walter H. F. Smith in Hawaii University.– See: http://gmt.soest.hawaii.edu/

35

YOUR SITE HERE

LOGOPRism project

Reviewing Lesson 1Reviewing Lesson 1

% global.gmt% gv image.ps

36

YOUR SITE HERE

LOGOPRism project

Files needed for GMTFiles needed for GMT

GMT source file (GMT script file)– global.gmt

Data file– data.xyz

CPT file (Color Palette Table file)– grad.cpt

37

YOUR SITE HERE

LOGOPRism project

global.gmt (1/6)global.gmt (1/6)

#!/bin/sh#############################################################ex Input/Output File#on 31/May/2002#by nhanasaki#at IIS,UT############################################################# Define File Names############################################################DATFILE=./data.xyz #data file (input)#DATAFILE=./cmap_mon_v0203_79.txtPSFILE=./image.ps #image file (output)GRDFILE=./grd #temporary fileCPTFILE=./grad.cpt #color paltette table file

Declaration of shell script type (Bourne Shell)

Com

ments

(Starting w

ith #)

Input VALUE into VARIABLEVARIABLE=VALUE

Comments

38

YOUR SITE HERE

LOGOPRism project


############################################################# Define Mapping Area############################################################XMIN=0.00 #Horizontal minimum [degree]XMAX=360.00 #Horizontal maximum [degree]YMIN=-90.00 #Vertical minimum [degree]YMAX=90.00 #Vertical maximum [degree]XWID=21.0 #Width of image [cm]YWID=10.5 #Height of image [cm]DXa=30.0 #a:Horizontal Anotation Interval [degree]DXf=30.0 #f:Horizontal Frame Interval [degree]DXg=10.0 #G:Horizontal Grid Interval [degree]DYa=30.0 #a:Vertical Anotation Interval [degree]DYf=30.0 #f:Vertical Frame Interval [degree]DYg=10.0 #g:Vertical Grid Interval [degree]D=2.5 #grid size############################################################## Short Cuts for Options############################################################RFLAG="-R${XMIN}/${XMAX}/${YMIN}/${YMAX}"JFLAG="-JX${XWID}/${YWID}d"BFLAG=“-Ba${DXa}f${DXf}g${DXg}:Longitude:/a${DYa}f${DYf}g${DYg}:Latitude:neWS

Call variables${VARIABLES}

39

YOUR SITE HERE

LOGOPRism project


XMIN [deg] XMAX [deg]

YMIN [deg]

YMAX [deg]

YW

ID

[cm]

XWID [cm]

Annotation

Frame

Gridcannot see

D

each grid

40

YOUR SITE HERE

LOGOPRism project


############################################################---Jobs#############################################################awk '($2==1){print $4, $3, $5}' $DATAFILE | \#xyz2grd $RFLAG -G$GRDFILE -I${D}/${D} -Fxyz2grd $RFLAG -G$GRDFILE -I${D}/${D} -F $DATFILE#psbasemap $RFLAG $JFLAG $BFLAG -X5.0 -Y5.0 -K > ./$PSFILE grdimage -O $RFLAG $JFLAG $GRDFILE -C$CPTFILE -K >> ./$PSFILE pscoast -O $RFLAG $JFLAG -Dc -W5 -N1 -I1 -K >> ./$PSFILEpsscale -O -C$CPTFILE -D5.0/-1.5/8/0.5h -L -K >> ./$PSFILEpstext -O $RFLAG $JFLAG -N << EOF >> ./$PSFILE180 110 24 0.0 1 6 Jan 1979 Rainfall (CMAP)EOF

comm

ands

data.xyz grd

psbasemap

pscoastline

psscale

pstext (put title)

xyz2grd grdimage

redirects(see Lesson2)

Read until EOF appear

Omit headerOmit trailer

41

YOUR SITE HERE

LOGOPRism project


############################################################# Comments for Options#############################################################-R: Option specifies region of interest#-J: Option for mapping projection (JX: projection type)#-B: Option for frame#-O: Option for "Omit Header" i.e. following GMT command#-K: Option for "Omit Trailer" i.e. GMT commands follow on##--psbasemap##draw basemap#-X: Shift plot origin x-direction [cm]#-Y: Shift plot origin y-direction [cm]##--xyz2grd##convert ASCII-xyz-data into binary-grid-data##--gridimage##conert binary-grid-data into colored mosaic image#

X

Y

42

YOUR SITE HERE

LOGOPRism project


#--pscoast##draw coastline#-D: Resolution#-W: Boldness#-N: National Border#-I: Rivers##--psscale##draw color bar#-D: Place to put#-L: Give Equal size rectangular##--pstext##write text between EOFs#-N: Do not clip#{horizontal[degree]/vertical[degree]/Point/Angle/Font?/Place?/Text}

43

YOUR SITE HERE

LOGOPRism project

CPT fileCPT file

0 255 0 0 1 255 255 01 255 255 0 5 0 255 05 0 255 0 10 0 255 25510 0 255 255 30 0 0 255F 0 0 128B 128 0 0N 255 255 255

Start value End value

Color (RGB)<TAB> spacingDo not use <SPACE> spacing !!

0 1 5 10 30

44

YOUR SITE HERE

LOGOPRism project

Exercise(Midterm report)

Exercise(Midterm report)

Draw a figure (1/3)– Fill ocean with blue color

– Change title of the figure

– Put unit [mm/day] to the figure Draw a figure (2/3)

– Fill land with black color


– Put unit [mm/day] to the figure Draw a figure (3/3)

– Zoom into Iran (horizontal:60deg; Vertical: 30deg)

– Make your original CPT


– Put unit [mm/day] to the figure

45

LOGOPRism project

Lesson 4Lesson 4Processing grid data 1

<HTOOL>

46

YOUR SITE HERE

LOGOPRism project


Let’s analyze global precipitation using CMAP dataset– Global, 2.5ox2.5o

– 1979-2002, monthly See the difference of global

distribution of rainfall in El Nino year and La Nina year– El Nino: 1986(autumn)-1987(winter)– La Nina: 1988(spring)-1989(spring)

Draw anomaly (difference) of annual global precipitation between 1987 and 1988

Whole 1987

Whole 1988

47

YOUR SITE HERE

LOGOPRism project

CMAP dataCMAP data

Binary grid data is stored in:/home/hanasaki/pub/unix_semi/dat_bin

72 grids

144 grids

MonthlyPrecipitation data

CMAP020319870100.cmap

CMAP, ver0203,1987,Jan,monthly data

72 x 144 x 4 = 41472 byte

48

YOUR SITE HERE

LOGOPRism project

Ascii / Binary Ascii / Binary

ASCII Binary

Can we read? Yes No

Machine dependency

No Yes

Access speed Slow Fast

Data size 1 byte/char 4byte/rec

Example 1

1.0 3 byte 4 byte

Example 2

1.000000 8 byte 4 byte

49

YOUR SITE HERE

LOGOPRism project

ExampleExample

1987-Jan

1987-Feb

1987-Mar

1987-Dec

Summation of 12 months

Divide by 12

Annual Average of 1987

12 months

add

add

add

% createcmap CMAP020319870000.cmap% addcmap CMAP020319870000.cmap CMAP 020319870100.cmap CMAP020319870000.cmap% addcmap CMAP020319870000.cmap CMAP 020319870200.cmap CMAP020319870000.cmap…% addcmap CMAP020319870000.cmap CMAP 020319871200.cmap CMAP020319870000.cmap% divcmap CMAP020319870000.cmap 12 CMAP020319870000.cmap

50

YOUR SITE HERE

LOGOPRism project

ExampleExample

El Nino year(1987 average)

La Nina year(1988 average)

Difference

Draw

sub

% mon2yearcmap CMAP 0203 1987 1987 .cmap% mon2yearcmap CMAP 0203 1988 1988 .cmap% subcmap CMAP020319870000.cmap CMAP020319880000.cmap dif.cmap% cmap2ps dif.cmap dif.cpt dif.ps Anomaly 1987-1988

51

YOUR SITE HERE

LOGOPRism project

HTOOL (1)HTOOL (1)

Copy files from /home/hanasaki/pub/unix_semi/src

% cp /home/hanasaki/pub/unix_semi/src/* .

program Arguments Function

createcmap OUT Create binary file

addcmapsubcmapmulcmapdivcmapprocmapratcmap

IN1,IN2,OUTIN1,IN2,OUTIN1,num,OUTIN1,num,OUTIN1,IN2,OUTIN1,IN2,OUT

Add binary filesSubtract binary filesMultiply binary file by valueDivide binary file by valueMultiply binary files (product)Divide binary files (ratio)

asc2cmapcmap2ascxyz2cmapcmap2xyzzuracmap

IN,OUTIN,OUTIN,OUTIN,OUTIN,OUT

Convert text data to binaryConvert binary data to textConvert (lon lat data) to binaryConvert binary to (lon lat data)Shift data by 180o in binary file

cmap2ps IN,CPT,PS,title1,title2,opt Convert data file to figure

mon2yearcmap ORG RUN syear eyear suf Convert monthly data to yearly data

IN: input binary fileOUT: output binary fileCPT: cpt filePS: ps image filenum, miss: real numbertitle1, title2: charactersyear, eyear: 4 digit integerORG: project name RUN: run name

52

YOUR SITE HERE

LOGOPRism project

ExerciseExercise

Create JJA (June-July-August) and DJF (December-January-February) precipitation for 1987 and 1988 and create anomaly figure. (Create your original CPT file for your analysis)

Draw normalized anomaly between 1987 and 1988. (Create your original CPT file for your analysis)

53

LOGOPRism project

Lesson 5Lesson 5Reading/Writing binary files

<FORTRAN IO>

54

YOUR SITE HERE

LOGOPRism project


Master FORTRAN input/output– Change binary and ascii data

FORTRAN 77– In this seminar, we use FORTRAN 77. However,

you can use whatever programming language you like (C, FORTRAN90, C++, C#, Java, Ruby…)

– FORTRAN 77 is “out of date” programming language, however, it is still widely used in science community.

– Especially important programs in our laboratory CCSR/NIES AGCM, MATSIRO, TRIP are written in FORTRAN 77

Reference– Fortran 90 プログラミング　冨田博之　培風館– Numerical recipes in Fortran 77

55

YOUR SITE HERE

LOGOPRism project

FORTRAN 77FORTRAN 77

program helloccc write(*,100) ‘Hello, World’ write(*,*) ‘Very very long sentences such as this $ must be divided in two lines’ 100 format(a11)c end

Start from the 7th column

If line continues, put some character in the 6th column

If line is comment, put c or * in the 1st column

Line number is put in 2nd to 5th column (Usually, no need to write line num)

PRism project

The first 6 columns have special meaning. Maximum: 72 columns

56

YOUR SITE HERE

LOGOPRism project

cmap2asc.f (1/4)cmap2asc.f (1/4)

program cmap2asccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc cto Convert binary (144x72) to asciicon 30th/May/2002cby nhanasakicat IIS, UTcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc implicit nonec integer nx integer ny parameter (nx=144) parameter (ny=72)c integer i,j real rdat(nx,ny) character*128 cifname character*128 cofnamec integer iargc

Declare code type

Do not use implicit variable type

Define integer variable “nx”

Put “144” to the variable “nx” (fixed)

Define real number array “rdat”

Define character variable “cofname”

nx ny i j cifname cofname iargc

144 72

rdat

57

YOUR SITE HERE

LOGOPRism project


cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Get file namecccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc if (iargc().ne.2) then write(6,*) 'Usage: cmap2asc Infile Outfile' stop end ifc call getarg(1,cifname) call getarg(2,cofname)

<In plain English>If the number of arguments is not equal to 2, then write “Usage: cmap2asc Infile Outfile” to the screen (format is default) and stop the programOtherwise, just continue process.

Put the first argument string to variable “cifname” andPut the second argument string to variable cofname

58

YOUR SITE HERE

LOGOPRism project


cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Read Binary Datacccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc open(15, file=cifname, access='direct', recl=nx*4) do j=1, ny read(15,rec=j) (rdat(i,j),i=1,nx) end do close(15)

<In plain English>[Line 1] Allocate file number 15 and open file “cifname”. The file is written in binary (direct access) and the record length is nx *4 (=144*4=576) byte.[Line 2] Change j from 1 to “ny” (or iterate “ny” times)[Line 3] Read the j th record of the file number 15. Put data to array rdat.[Line 4] Process finished, go to the next iteration (or add 1 to j)[Line 5] Close file number 15

Record length

j th

59

YOUR SITE HERE

LOGOPRism project


cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Write Ascii Datacccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc open(16, file=cofname, access='sequential') do j=1, ny write(16,*) (rdat(i,j), i=1,nx) end do close(16)c end

<In plain English>[Line 1] Allocate file number 16 and open file “cofname”. The file is written in ascii (sequential access)[Line 2] Change j from 1 to “ny” (or iterate “ny” times)[Line 3] Write array rdat(i,j) to the file number 16. i is changed from 1 to nx. The format is default.[Line 4] Process finished, go to the next iteration (or add 1 to j)[Line 5] Close file number 16

Record length

j th

60

YOUR SITE HERE

LOGOPRism project

ExcerciseExcercise

Read a FORTRAN source code addcmap.f and translate into plain English or Japanese.(/home/hanasaki/bin/addcmap.f)

Write a program asc2cmap.f that has the opposite function of cmap2asc.f. (After finish editing, type% make asc2cmapon the terminal)

61

LOGOPRism project

Unix Seminar Lesson 6Unix Seminar Lesson 6Grid data processing 2

62

YOUR SITE HERE

LOGOPRism project


How much annual precipitation we have in Tokyo? [E138.75-N36.25]*

How much is the annual average precipitation of the whole world?

How large area in the world has more annual precipitation than Tokyo?

How many people in the world live in the place less than 500 mm/year precipitation?

* Actually, the geographic location for the lon/lat is Takasaki City in Gunma prefecture. Because of large grid size of CMAP, This is the nearest grid from Tokyo!

63

YOUR SITE HERE

LOGOPRism project

MaterialMaterial

Mean Annual rainfall (1979-2001)– rain.cmap

– Unit: mm/year

– Global, 2.5x2.5

Land area– landarea.cmap (land only)

gridarea.cmap (whole globe)

– Unit: km2

– Global, 2.5x2.5

(See /home/hanasaki/pub/unix_semi/lesson6)

64

YOUR SITE HERE

LOGOPRism project

MaterialMaterial

Landmask– landmask.cmap

– Unit: none

– Global, 2.5x2.5

Polulation– pop.cmap

– Unit: capita

– Global, 2.5x2.5

– Source: CIESIN

(See /home/hanasaki/pub/unix_semi/lesson6)

65

YOUR SITE HERE

LOGOPRism project

ExampleExample

How much annual precipitation we have in Tokyo? [E138.75-N36.25]*

How much is the annual average precipitation of the whole world?

Global annual precipitation

138.75

36.25

Global annual precipitation

1. Read the file2. Sum up data for every grids3. Divide by the number of grids

1. Read the file2. Pick up data for the grid

% pointcmap rain.cmap 138.75 36.25% avecmap rain.cmap 0 [Actually, it is not correct]

66

YOUR SITE HERE

LOGOPRism project

ExampleExample

Mask

Precipitation less than 500 mm/year

Sum up

1. How large area in the world has more annual precipitation than Tokyo?

2. How many people in the world live in the place less than 500 mm/year precipitation?

% maskcmap lndara.cmap rain.cmap gt 1530 temp.cmap% sumcmap temp.cmap 0% maskcmap pop.cmap rain.cmap gt 500 temp.cmap% sumcmap temp.cmap 0

67

YOUR SITE HERE

LOGOPRism project

Htools (2)Htools (2)

Program Arguments Function

avecmapmaxcmapmincmapsumcmap

IN,missIN,missIN,missIN,miss

Calc average of binary fileFind max value in binary file Find min value in binary fileCalc summation of binary file

Pointcmap IN,lon,lat Display data at (lon,lat)

findcmaprplccmap maskcmapmaskrplccmap

IN,eq,num,OUTIN,eq,num,num,OUTIN,MASK,eq,num,OUTIN,MASK,eq,num,num,OUT

Find dataReplace dataMask dataMask data and replace

1 2 3 45 1 1 89 1 1 12

13141516

1 2 3 45 6 7 89 101112

13141516

0 0 0 00 1 1 00 1 1 00 0 0 0

0 0 0 00 6 7 00 1011 00 0 0 0

1 2 3 45 6 7 89 101112

13141516

1 2 3 45 6 7 89 101112

13141516

1 2 3 45 6 7 89 101112

13141516

Mask file

0 0 0 00 0 0 09 101112

13141516

Findif (data > 8)then pick up

0 0 0 00 0 0 01 1 1 11 1 1 1

Replaceif (data > 8)replace to 1

Maskif (mask = 1)then pickup

Mask Replaceif (mask = 1)replace to 1

68

YOUR SITE HERE

LOGOPRism project

ExerciseExercise

Calculate correctly the mean annual precipitation of the world– Firstly, think about strategy, how to process data.

– Secondly, combine Htools. (See also Lesson 4) Complete the table below

Precipitation Population Area

0-50

50-100

100-500

500-1000

1000-2000

2000-

Total

69

LOGOPRism project

Lesson 7Lesson 7Fortran and Makefile

70

YOUR SITE HERE

LOGOPRism project


Learn some important techniques for effective handling of temporal varying files

Understand what is “Makefile”(/home/hanasaki/pub/unix_semi/src/Makefile)

71

YOUR SITE HERE

LOGOPRism project

mon2yearcmap.fmon2yearcmap.f

program mon2yearcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc implicit nonec parameter integer nx, ny parameter (nx=144, ny=72)c variable integer ix, iy integer iyear, isyear, ieyear, imon integer ndaymon !! Number of days in a month integer ndayyear !! Number of days in a year real rmondat(nx,ny) !! Monthly data array real ranudat(nx,ny) !! Annual data arrayc function integer iargc !! Default function integer igetday !! My functionc files character*128 corg, crun, csuf, ctmp character*128 cifname, cofnamecccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c Get Argumentscccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc if(iargc().ne.5)then write(*,*) 'mon2year <org><run><isyear><ieyear><suf>' stop end ifc call getarg(1,corg) call getarg(2,crun) call getarg(3,ctmp) read(ctmp,*) isyear call getarg(4,ctmp) read(ctmp,*) ieyear call getarg(5,csuf)

x 31x 28x 31…x 31

Comments. (not standard grammar but valid almost every compiler)

Argument must be character.To get real/integer arguments,write like this.

sum up

divide by 365

72

YOUR SITE HERE

LOGOPRism project

mon2yearcmap.fmon2yearcmap.f do iyear=isyear,ieyearc clear variable ndayyear=0c clear array do iy=1,ny do ix=1,nx ranudat(ix,iy)=0.0 end do end doc do imon=1,12 write(*,*) iyear,imon ndaymon=igetday(iyear,imon) ndayyear=ndayyear+ndaymonc read data write(cifname,101) corg,crun,iyear,imon,0,csuf call read_binmat(rmondat,nx,ny,15,cifname)c calclulation do iy=1,ny do ix=1,nx ranudat(ix,iy)=ranudat(ix,iy)+rmondat(ix,iy)*real(ndaymon) end do end do end doc Get average write(*,*) 'Total num of days:', ndayyear do iy=1,ny do ix=1,nx ranudat(ix,iy)=ranudat(ix,iy)/ndayyear end do end doc Write results write(cofname,101) corg,crun,iyear,0,0,csuf call wrte_binmat(ranudat,nx,ny,16,cofname) end do 101 format(a4,a4,i4.4,i2.2,i2.2,a5)c end

Refreshing arrays/variables

Calling a function from the outside of program

Calling a function from the outside of program

Create file name like:AAAABBBByyyymmdd.cmap

Set file name:AAAABBBByyyymmdd.cmap

Write to display.

Ref

resh

ing

arr

ays/

vari

able

s is

extr

emel

y i

mp

ort

ant

for

accu

mu

lati

ve

pro

ces

s

73

YOUR SITE HERE

LOGOPRism project

Subroutine/FunctionSubroutine/Function

igetday

read_binmat

wrte_binmat

mon2yearcmap

igetday

read_binmat

wrte_binmat

day2moncmap

Commonsubroutine

commonfunction

Differentmain program

subroutine read_binmat(rdat,nx,ny,infile,cfname)cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc open(infile,file=cfname,access='DIRECT',status='old',recl=4*nx) do j = 1,ny read (infile,rec=j)(rdat(i,j),i=1,nx) end do close (infile)c end

call read_binmat(rarray,360,180,15,temp.bin)1) Main programcalls subroutine

2) Arguments aretransferred tosubroutine

/home/hanasaki/unix_semi/src/read_binmat.f

74

YOUR SITE HERE

LOGOPRism project

MakefileMakefile

############################################################# Macro############################################################RM = /bin/rm # path to command rmFC = f77 # path to fortran compilarFCFLAGS = -g -c –C # option for fortran compilarFL = f77 # path to fortran linkerFLFLAGS = # path to fotran linkerOBJS = addcmap subcmap mulcmap divcmap procmap ratcmap \

mon2yearcmap############################################################# Suffix rule############################################################.f.o:

$(FC) $(FCFLAGS) $<############################################################# Dependency############################################################TARGET2 = mon2yearcmapTARGET2.SUB = read_binmat.o wrte_binmat.o \

igetday.o############################################################# Compilation############################################################$(TARGET2) : $(TARGET2).o $(TARGET2.SUB)

$(FL) -o $@ [email protected] $(TARGET2.SUB)clean:

$(RM) -f *.o core *~ *traceall: $(OBJS)

<TAB>

<TAB>

<TAB>

<TAB>

Define variablesSimilar to shell script

Define variables

Set Dependency (A:B)A is dependent on BIf B is newer than A thenexecute the next line

Suffix rule.f is related with fortran

75

YOUR SITE HERE

LOGOPRism project

binary/asciibinary/ascii

1 2 3 45

sample.txt

1 2 3 45

1 byte

4 byte for 1 record

1 <s> 2 <l> 3 <s> 4 5

sample.bin

<s>: space<l>: linefeed

real data(2,2) c open(15, file=‘sample.txt’, access=‘sequential’) do j=1,2 read(15,*) (data(i,j), i=1,2) end do close(15)

real data(2,2) c open(16, file=‘sample.bin’, access=‘direct’,recl=8) do j=1,2 write(16,rec=j) (data(i,j), i=1,2) end do close(16)

8 byte for 1 (unit) record length

4 byte

Ascii file

Binary file

1 2 3 45

Memory data(i,j)

1 2

1

2

76

YOUR SITE HERE

LOGOPRism project

ExerciseExercise

Code a program that gives the number of days of any month.(Put file name “nofday,f”)

% program 1995 331% program 1995 228% program 1996 229% program 2000 229% program 1900 228

77

LOGOPRism project

Lesson 8Lesson 8Shell script

78

YOUR SITE HERE

LOGOPRism project


To complete Exercise 2 of Lesson 6, we need to type 42 commands on the terminal. How can we reduce this troublesome procedure?

79

YOUR SITE HERE

LOGOPRism project

Example (1)Example (1)

Write Jan to Dec on the display and if the month is your birth month, write “Happy birth month!” on the screen

#!/bin/sh##################################################MONS=“01 02 03 04 05 06 07 08 09 10 11 12”BIRTHMON=“08”##################################################for MON in $MONS; do if [ $MON = $BIRTHMON ]; then echo $MON “Happy birth month!” else echo $MON fidone

80

YOUR SITE HERE

LOGOPRism project

Example (2)Example (2)

#!/bin/sh##################################################YEAR=2005MONS=“01 02 03 04 05 06 07 08 09 10 11 12”BIRTHMON=“02”BIRTHDAY=“28”##################################################for MON in $MONS; do DAY=1 # refresh day EDAY=`nofday $YEAR $MON` # set the last day of the month while [$DAY –le $EDAY ]; do if [ $MON = $BIRTHMON –a $DAY = $BIRTHDAY ]; then echo $MON $DAY “Happy birth day!” else echo $MON $DAY fi DAY=`expr $DAY + 1` # day for next iteration donedone

Write 1st Jan to 31st Dec on the display and if the date is your birthday , write “Happy birthday !” on the screen

81

YOUR SITE HERE

LOGOPRism project

Exercise (3)Exercise (3)

Review the Exercise 2 of Lesson 6.Using do-loop, for-loop and if statement, reduce the code as small as possible.Step 1. find the common procedure in the code.

# 0mm/year =< Precipitation < 50mm/yearmaskcmap pop.cmap rain.cmap ge 0 temp.pop.ge0.cmapmaskcmap temp.pop.ge0.cmap rain.cmap lt 50 temp.pop.ge0.lt50.cmapsumcmap temp.pop.ge0.lt50.cmap 0# 50mm/year =< Precipitation < 100mm/yearmaskcmap pop.cmap rain.cmap ge 50 temp.pop.ge50.cmapmaskcmap temp.pop.ge50.cmap rain.cmap lt 100 temp.pop.ge50.lt100.cmapsumcmap temp.pop.ge50.lt100.cmap 0

DAT=pop# 0mm/year =< Precipitation < 50mm/yearLL=0 # lower limitUL=50 # upper limit# 50mm/year =< Precipitation < 100mm/yearLL=50 # lower limitUL=100 # upper limit##############################maskcmap ${DAT}.cmap rain.cmap ge ${LL} temp.${DAT}.ge${LL}.cmapmaskcmap temp.${DAT}.ge${LL}.cmap rain.cmap lt ${UL} temp.${DAT}.ge${LL}.lt${UL}.cmapsumcmap temp.${DAT}.ge${LL}.lt${UL}.cmap 0

# note it doesn’t work

82

YOUR SITE HERE

LOGOPRism project

Exercise (3)Exercise (3)

Review the Exercise 2 of Lesson 6.Using do-loop, for-loop and if statement, reduce the code as small as possible.Step 2. Combining do/for loop and if statement.

#!/bin/sh##################################################for DAT in pop grdara; do JOB=1 while [ $JOB –le 7 ]; do if [ $JOB -eq 1 ]; then LL=0; UL=50 elif [ $JOB -eq 2 ]; then LL=50; UL=100 fi maskcmap ${DAT}.cmap rain.cmap ge ${LL} temp.${DAT}.ge${LL}.cmap maskcmap temp.${DAT}.ge${LL}.cmap rain.cmap lt ${UL} temp.${DAT}.ge${LL}.lt${UL}.cmap sumcmap temp.${DAT}.ge${LL}.lt${UL}.cmap 0 JOB=`expr $JOB + 1` donedone

83

YOUR SITE HERE

LOGOPRism project

SummarySummary

For loop for VAR in VARLIST; do processdone

If statement If [ ]: then process1elif [ ]; then process2else process3fi

While loop while [ ]: do processdone

If file exist If [ -f FILENAME ]; then processfi

If directory exist

If [ -d DIRNAME ]; then processfi

A = B For string:[ “A” = “B” ]For integer:[ A –eq B ]

A ≠ B For string:[ “A” != “B” ]For integer:[ A –ne B ]

A > B [ A –gt B ]

A ≥ B [ A –ge B ]

A < B [ A –lt B ]

A ≤ B [ A –le B ]

And -a

Or -o

84

YOUR SITE HERE

LOGOPRism project

Exercise (Final exam)Exercise (Final exam)

Create a shell script that generates binary data files (144 x 72 x 4 byte) for whole period from the original CMAP ascii data files.ftp://ftp.ncep.noaa.gov/pub/precip/cmap/monthly/

85

LOGOPRism project

Lesson 9Lesson 9Set UNIX environment

86

YOUR SITE HERE

LOGOPRism project


Understanding – What is etc directory in your home directory– Why “rm –f” still ask you for your permissio

n?– Why the command “lpr” knows our default pr

inter – Why we can execute “ls” even the file does n

ot exist in the current directory Understanding our lab’s UNIX enviro

nment.

87

YOUR SITE HERE

LOGOPRism project

$HOME/.cshrc$HOME/.cshrc

See ~/.cshrc– If your login shell, the shell will first check th

e file “~./cshrc”.

#############################################################to set tcsh#by nhanasaki#on 27th June 2005#at IIS,UT############################################################set DEFD=/home/`/usr/ucb/whoami`/etc#source $DEFD/setenv.allsource $DEFD/setdef.all#if ($?prompt) then source $DEFD/setenv.shell source $DEFD/setdef.shell source $DEFD/aliases.shellendif

88

YOUR SITE HERE

LOGOPRism project

$HOME/etc/.aliases$HOME/etc/.aliases

Alias is kind of “short cut”,% alias sayhello ‘echo Hello’% sayhello% which sayhello

############################################################# Replace basic command############################################################alias cd 'cd \!*;echo $cwd; sp'alias cp 'cp -i'alias ls 'ls -bF'alias mv 'mv -i'alias rm 'rm -i'alias gv 'ghostview'alias setgv 'setenv DISPLAY fountain:0.0'############################################################# To raid############################################################alias raida 'cd /export/raidA/hanasaki'alias raid1 'cd /export/raid1/hanasaki'alias raid2 'cd /export/raid2/hanasaki'

Call original “ls”% /bin/ls

89

YOUR SITE HERE

LOGOPRism project

$HOME/etc/setenv.all$HOME/etc/setenv.all

Environmental variable is a special variable used by some programs.

############################################################# BASICS############################################################setenv PRINTER floodsetenv EDITOR /usr/xpg4/bin/visetenv VISUAL /usr/xpg4/bin/visetenv PAGER /usr/local/bin/lesssetenv MANPATH /opt/SUNWspro/SC4.2/man:/usr/share/man############################################################# TEX############################################################setenv BIBINPUTS .:/home/taikan/tex/inputsokisetenv TEXFONTS /usr/local/tex/fontssetenv TEXPOOL /usr/local/texsetenv TEXFORMATS /usr/local/texsetenv TEXINPUTS .:/usr/local/tex/macros:/usr/local/rain/texinputs

Font informationused by TeX

Default value is set here

90

YOUR SITE HERE

LOGOPRism project

Why we can call various commands?% where ls% where f77

$HOME/etc/setdef.all$HOME/etc/setdef.all

set HANAP = ( /home/hanasaki/bin )set BINP= ( /usr/bin /sbin /usr/sbin /usr/ucb)set CMPP = ( /opt/SUNWspro/bin )#if ( -d /opt2/SUNWspro/bin ) then set CMPP2 = ( /opt2/SUNWspro/bin )else set CMPP2 = ( )Endif############################################################# Set path############################################################set path=( $CMPP $CMPP2 $HANAP)############################################################# Unset MACRO############################################################unset CMPP CMPP2 HANAP BINP

91

YOUR SITE HERE

LOGOPRism project

Servers Servers

CPU FortranCompiler(f77)

Attached Raid

Rainbow 400 Mhz x4 Rainbow(ver 4.2)

Raid A-E

Hydro 1280 MHz x2 Rainbow(ver 4.2)

Raid 20-29

Chaser 1593 MHz x4 Mizu(ver 6)(f90 ver 7.1)

Raid 30-39

Reservoir 750 MHz x 6 Mizu(ver 6)(f90 ver 7.1)

Raid 1-9Raid11-18

/usr/platform/`uname -i`/sbin/prtdiag

F77 ver 4.2 and ver 6 aresignificantly different in linking outside functions,subroutines! Be careful!!

92

YOUR SITE HERE

LOGOPRism project

ExerciseExercise

Set alias to “cd” to your export directory

% which raidAraidA aliased to /export/raidA/hanasaki

Make directory bin in your home directory and set path, so as to useany commands in $HOME/bin at whatever directory.

93

LOGOPRism project

After wordAfter word

94

YOUR SITE HERE

LOGOPRism project

After wordAfter word

このテキストは 2002年から 2005年の夏学期に東京大学生産技術研究所沖研究室で行われた初心者向けの UNIX講習会の補助資料として作られました。プログラミングやUNIXの知識がゼロの学生が沖研究室で研究を始めるために最低限必要な知識をまとめて紹介しています。沖研究室は地球規模の水循環に関する研究を行っています。このテキストはその際に必要な膨大な時系列グリッドデータの操作に <偏った >内容になっています。あえて内容を偏らせることで研究にすぐに役立つように工夫したつもりです。このテキストは <なんとか >英語で作りました。沖研究室に毎年やってくる熱意ある留学生に日本語のバリアをはらないためです。下手な英語で分かりにくくなっているかもしれませんが、どうぞご了承ください。またテキストの英語は文法・用法上の誤りが多々あるかもしれません。お気づきの際は、是非私までお知らせください。

2005年 7月 5日花崎　直太

95

YOUR SITE HERE

LOGOPRism project

参考文献参考文献

UNIX– 図解標準最新ＵＮＩＸハンドブック（伊藤和人・秀和システム）

FORTAN– Fortran90 プログラミング（冨田博之・培風館）

Bourne Shell– 入門 UNIX シェルプログラミング（ Bruce Blinn ・ Soft Bank Publishin

g ） GMT

– GMT A Map-Making Tutorial （ Paul Wessel ） VBA

– かんたんプログラミング Excel VBA 　シリーズ（大村あつし・技術評論社）

Solaris– Sun テキスト　 Solaris9 （ Paul A Watters ・ Soft Bank Publishing ）

Make– Make 改訂版（ Andrew Oram & Steve Talbott ・ O’Reilly ）

LaTeX– LaTeX2ε 美文章作成入門（奥村晴彦・技術評論社）

96

YOUR SITE HERE

LOGOPRism project

ReferencesReferences

UNIX– ?

FORTAN– ?

Bourne Shell– Portable Shell Programming （ Bruce Blinn ）

GMT– GMT A Map-Making Tutorial （ Paul Wessel ）

VBA– ?

Solaris– Solaris 10: The Complete Reference （ Paul A Watters; Mcgraw-Hi

ll) Make

– Make （ Andrew Oram & Steve Talbott; O’Reilly ） LaTeX

– ?

logo prism project 1 lessons of unix/gmt/fortran naota hanasaki iis,ut draft

Documents