software carpentry

Sunil Mohan Adapasunil at medhas dot org

Software Carpentry

Some content derived from Software Carpentry Lecture Material http://software-carpentry.org/license/

This work and the original are under Create Commons Attribution 3.0 License http://creativecommons.org/licenses/by/3.0/

About the Tutorial Introductory Hands on Interactive Software Carpentry for academics and

research in any discipline Makes software work easier Enables new kinds of work Gets work done faster

Summary The Unix Shell Regular Expressions Make Version Control Python

Unix Shell

About Shell Why use command line when we have GUI Typical shell: bash Terminal programs: gnome-terminal, Konsole,

xterm, putty

Example Use Cases Mr. A wishes to retrieve all files modified last

week and replace the phrase this week with next week in those files.

Everyday, Mr. B wishes to automatically retrieve all files modified on that day and back them up to different location.

Mr. C likes to rename files so that their extensions are removed

Mr. D likes to combine to merge fives sets of user lists into a single one

File system ls to list the files in the directory

ls -l to list files with extra information pwd to show the current directory cd to switch to a directory cd to switch to home directory / is the top most directory. It is also the path separator . is the current directory .. is the parent directory. /home/user/work/.. is same

as /home/user

File System Structure /root and /home store user data /bin, /usr/bin, /sbin and /usr/sbin store executable commands /usr stores files related to user applications /usr/local contains applications compiled by the user /var contains (variable) files that usually grow over time /lib contains libraries /tmp contains temporary files /proc is a virtual file system containing kernel information /mnt and /mount contain file system mounts

Manipulating Files cp copies one file to another file or directory mv renames a file or moves it to another

directory, overwriting rm deletes files rm -rf deletes files and directories mkdir creates a directory rmdir removes an empty directory

File Permissions ls -l shows ownership and permissions of a file chmod changes the permissions chown changes the ownership su switches the current user by launching a

new shell

Redirection ls > out stores the output of ls into out file cat concatenates files and input given to it cat < out reads the contents of out file and

provides as input to cat sort < out > sorted sorts a contents of out file

and stores it in the sorted file | (a pipe) redirects the output of one command

to another: ls | sort > sorted

Some More Commands du to find the size occupied by file on disk less and more for paginated display find to recursively find files matching a complex criteria xargs to convert input into arguments grep to match a pattern/regular expression in a file head and tail to see part of a file sort to sort data in a file uniq to find items after sorting wc counts number of chars, words and lines in a file

Jobs Control-C terminates a program Backgrounding a program

Control-Z and bg & at the end of the command

jobs list current jobs fg foregrounds a program ps lists processes kill kills a process

References Bash Manual Page: man bash

http://linux.die.net/man/1/bash GNU/Linux Man Pages: man Learning the Shell:

http://linuxcommand.org/learning_the_shell.php

Regular Expressions

What are Regular Expressions? A concise and flexible means for matching

strings of text Like *.txt means all files with .txt extension Parts of matches can be extracted Matched text can be replaced

Example Use Cases Mr. A has list of 1000 phrases in a text file. She

would like to add a full-stop at the end of each line.

Mr. B has a list of percentages of various categories in Wikipedia and their growth in X (Y) format. He would like to covert it X/Y format.

Mr. C would like find out all words in a file containing 3 to 5 alphabets.

Example Use Cases (contd.) Mr. D would like to list all hexadecimal numbers

in a file. Mr. E would like to convert all American

formatted dates in a file to ISO date format. Mr. F would like to retrieve all the sentences

starting with 'Which' from a file. Mr. G would like to retrieve all words in a

document containing two Hindi consonants joined by a halant.

Where are RegExps Used? Editors: Vim, Emacs, Eclipse, Notepad++ etc. Programming Languages

Inbuilt: Perl, Ruby, Javascript etc. As library: C, C++, Java, Php, Python etc.

Unix command line: rename, grep, sed, perl etc. Lot more:

Configuring Apache Web Server Syntax Highlighting in editors Even Google Search (well... not really. Just code search)

Basics A normal alpha-numeric character in regex

matches that character in target string hello matches the text hello

. matches any character * repeats the previous expression zero or more

times

Example Applications Unix command line: grep Editor: Vim Programming: Perl

Metacharacters . matches any character

a. matches as, ab etc. ^ matches the beginning of a line $ matches the end of a line | alternation

H|h matches h or H () grouping

H|hello matches H or hello (H|h)ello matches Hello or hello

\ escapes any metacharacter Mr. matches Mr. and Mrs Mr\. matches Mr. and not Mrs

Character Classes [Hh] means (h|H) [0-9] means (0|1|2|3|4|5|6|7|8|9) [0-9a-z] means ([0-9]|[a-z]) [^ab] matches any characters but not a and b \x{0915} matches devanagari \n matches a new line \r matches a return \t matches a tab

Character Classes (Perl) \w matches a word \W matches a non-word \s matches a whitespace \S matches a non-whitespace \d matches a digit

Quantifiers * matches 0 or more times + matches 1 or more times ? matches 0 or 1 time {7} matches 7 times {5,} matches at least 5 times {2,5} matches at least 2 times but no more than

5 times

Greedy vs. Stingy In text "XYZ" to "PQR" ".*" will match "XYZ" to "PQR" ".*?" will match "XYZ" ? applies to all other quantifiers also

Substitutions s/hello/Hello/ will substitute Hello with hello s/(H|h)ello/Hi/ will substitute Hello or hello with

Hi () will extract a match \1, \2 etc. hold the value of the match s/([0-9])([0-9])/\2\1/ matches two digits and

reverses them

Modifiers i means case-insensitive match

/Hello/i will match hello, Hello or HELLO g means global matching m means multi-line string

References Perl Regular Expressions: man perlre

http://perldoc.perl.org/perlre.html

Build Tools

Building a Project

file3.c file4.c

file3.o file4.o

file1.c file2.c

file1.o file2.o

library1.so library2.somain.c

main.o

program

Make Needs a dependency graph Operates on files and time stamps Executes shell commands Other uses

Any set of tasks with dependency graphs Automated testing Building documentation Even booting an operating system!

Writing Makefileshello: hello.o

gcc hello.o -o hello

hello.o: hello.cgcc hello.c -c -o

hello.o

clean:rm -f hello.o hello

Using Make $ make

$ make clean

Basics

hello: hello.ogcc hello.o -o hello

hello.o: hello.cgcc hello.c -c -o

hello.o

clean:rm -f hello.o hello

Rules

Commands

PrerequisitesTarget

Bigger Projecthello: main.o filel.o file2.o

gcc main.o file1.o file2.o -o hello

main.o: main.c file1.h file2.hgcc main.c -c -o main.o

file1.o: file1.c file1.hgcc file1.c -c -o file1.o

file2.o: file2.c file2.hgcc file2.c -c -o file2.o

clean:rm -f hello main.o file1.o file2.o

Improving: Step 1hello: main.o filel.o file2.o

gcc $^ -o $@

mail.o: file1.h file2.hmain.o: main.c

gcc $^ -c -o $@

file1.o: file1.hfile1.o: file1.c

gcc $^ -c -o $@

file2.o: file2.hfile2.o: file2.c

gcc $^ -c -o $@


Improving: Step 2hello: main.o filel.o file2.o

gcc $^ -o $@

mail.o: file1.h file2.hfile1.o: file1.hfile2.o: file2.h

%.o: %.cgcc $^ -c -o $@


Improving: Step 3TARGET = helloOBJECTS = main.o file1.o file2.o

main.o: file1.h file2.hfile1.o: file1.hfile2.o: file2.h

$(TARGET): $(OBJECTS) gcc $^ -o $@

%.o: %.c gcc $< -c -o $@

clean: rm -f $(TARGET) $(OBJECTS)




$(OBJECTS): %.o: %.c gcc $< -c -o $@


Phony Targets Try this:

$ touch clean

$ make clean

What happened and why? Declaring a target as phony addresses the

problem .PHONY: clean




$(OBJECTS): %.o: %.c gcc $< -c -o $@

.PHONY: clean


Even Better Build System Autoconf

Detect system environment and build accordingly M4

Write macros for Autoconf Automake

Automatically generate makefiles Libtool

Automatically handle different library formats in different OSes

References GNU Make Manual: info make

http://www.gnu.org/software/make/manual/make.html GNU Automake Manual: info automake

http://www.gnu.org/software/automake/manual GNU Autoconf Manual: info autoconf

http://www.gnu.org/software/autoconf/manual

Version Control

Why? Keep track of changes Release management Work as a group Identify regressions easily Maintain personal changes to code elsewhere

RevisionsInitial Version

Added feature 1

Added feature 2

Fixed bug 1

Latest version

Release ManagementInitial Version

Added feature 1

Fixed bug 1

Version 1.1

Added feature 2

Fixed bug 1

Version 2.0

Work as a GroupInitial Version

Added feature 1

A's Feature

Merge

B's Feature

Latest Version

Identify RegressionsBug free version

Latest version contains a bug

Bug introduced

Personal ChangesFree Software Project

on the Internet

Version 2.0

Version 3.0

Version 4.0

My research work

Idea 1

Idea 2

Version 1.0

Idea 3

Getting Started with Git Basic configuration:

$ git config --global user.name "Your Name Comes Here" $ git config --global user.email [email protected]

Creating a repository: $ git init

Adding files to the repository: $ git add file1.c

Committing the changes $ git commit

Editing Edit your file

$ nano file1.c Mark for commit

$ git add file1.c Commit the changes

$ git commit

Reviewing Changes Edit and review changes

$ nano file1.c $ git diff

Current status $ git status

Reviewing Changes (contd.) Changes between two revisions

$ git diff r1..r2 History of changes

$ git log

Exchanging Patches The diff format Patch file Producing a patch file

$ git diff r1..r2 > my_feature.patch Applying a patch

$ patch -p1 < my_feature.patch Better ways

Tagging What are tags? Creating a tag

$ git tag VERSION_1 Deleting a tag

$ git tag -d VERSION_1 Retrieving older versions

$ git checkout -b VERSION_1

More Topics of Interest Branching and Merging Pushing and Pulling from repositories Rebasing Bisecting Stashing changes

Graphical Tools

References Git: http://git-scm.com Official Git Tutorial:

http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html

ProGit Book: http://progit.org Git Manual Pages: man git

Python

Content derived from Official Python Tutorial http://docs.python.org/tutorial/

Why Python? Easy for beginners Yet powerful Rapid development Scalable for large and complex project Object oriented Cross platform Large set of libraries for performing various

tasks

First Python Program$ python>>> 2 + 35>>>

Hello, World!$ python>>> print "Hello, World!"Hello, World!>>>

Hello, World! in a File#!/usr/bin/python

print "Hello, World!"

Python as Calculator>>> 2+24>>> (50-5*6)/45

Variables>>> a = 2>>> b = 3>>> print a * b6

Strings>>> hello = "Hello">>> world = "World">>> print helloHello>>> print worldWorld>>> print hello + worldHelloWorld>>> print hello + ", " + world + "!"Hello, World!

Lists>>> a = ['spam', 'eggs', 100, 1234]>>> a['spam', 'eggs', 100, 1234]>>> a[0]'spam'>>> a[3]1234>>> a[-2]100>>> a[1:-1]['eggs', 100]>>> a[2] = a[2] + 23>>> a['spam', 'eggs', 123, 1234]

More on Lists>>> a = [66.25, 333, 333, 1, 1234.5]>>> print a.count(333), a.count(66.25), a.count('x')2 1 0>>> a.insert(2, -1)>>> a.append(333)>>> a[66.25, 333, -1, 333, 1, 1234.5, 333]>>> a.index(333)1>>> a.remove(333)>>> a[66.25, -1, 333, 1, 1234.5, 333]>>> a.reverse()>>> a[333, 1234.5, 1, 333, -1, 66.25]>>> a.sort()>>> a[-1, 1, 66.25, 333, 333, 1234.5]

More on Lists>>> mat = [... [1, 2, 3],... [4, 5, 6],... [7, 8, 9],... ]

Tuples>>> t = 12345, 54321, 'hello!'>>> t[0]12345>>> t(12345, 54321, 'hello!')>>> # Tuples may be nested:... u = t, (1, 2, 3, 4, 5)>>> u((12345, 54321, 'hello!'), (1, 2, 3, 4, 5))

>>> t = 12345, 54321, 'hello!'>>> x, y, z = t

Dictionaries>>> tel = {'jack': 4098, 'sape': 4139}>>> tel['guido'] = 4127>>> tel{'sape': 4139, 'guido': 4127, 'jack': 4098}>>> tel['jack']4098>>> del tel['sape']>>> tel['irv'] = 4127>>> tel{'guido': 4127, 'irv': 4127, 'jack': 4098}>>> tel.keys()['guido', 'irv', 'jack']>>> 'guido' in telTrue

If .. else>>> x = int(raw_input("Please enter an int: "))Please enter an integer: 42>>> if x < 0:... x = 0... print 'Negative changed to zero'... elif x == 0:... print 'Zero'... elif x == 1:... print 'Single'... else:... print 'More'

For>>> # Measure some strings:... a = ['cat', 'window', 'defenestrate']>>> for x in a:... print x, len(x)...cat 3window 6defenestrate 12

For>>> range(10)[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]>>> a = ['Mary', 'had', 'a', 'little', 'lamb']>>> for i in range(len(a)):... print i, a[i]

Break>>> for i in range(10):... if i > 5:... break... print i... 012345

Continue>>> for i in range(10):... if i == 5:... continue... print i... 012346789

Comments>>> # This is single line comment>>> """ This is a ... multiline... comment"""

Functions>>> def fib(n): # print Fibonacci series... """Print a Fibonacci series up to n."""... a, b = 0, 1... while a < n:... print a,... a, b = b, a+b...>>> # Now call the function we just defined:... fib(1000)0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987

References Python Programming Language Official

Website: http://python.org The Python Tutorial:

http://docs.python.org/tutorial The Python Standard Library:

http://docs.python.org/library The Python Language Reference:

http://docs.python.org/reference

Feedback & Further Assistance:sunil at medhas dot org

Thank you

Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49Slide 50Slide 51Slide 52Slide 53Slide 54Slide 55Slide 56Slide 57Slide 58Slide 59Slide 60Slide 61Slide 62Slide 63Slide 64Slide 65Slide 66Slide 67Slide 68Slide 69Slide 70Slide 71Slide 72Slide 73Slide 74Slide 75Slide 76Slide 77Slide 78Slide 79Slide 80Slide 81Slide 82Slide 83