advanced unix

240-491 Adv. UNIX:Perl/8 1

Advanced UNIXAdvanced UNIX

Objectives of these slides:Objectives of these slides:– introduce Perl (version 4.0-5.0)introduce Perl (version 4.0-5.0)– mostly based on Chapter 1, mostly based on Chapter 1, Learning PerlLearning Perl

240-491 Special Topics in Comp. Eng. 1Semester 2, 2000-2001

8. Introduction to Perl


OverviewOverview

1. Starting with Perl1. Starting with Perl

2. I/O2. I/O

3. 3. If-elseIf-else

4. 4. whilewhile

5. Arrays (lists)5. Arrays (lists)

continued


6. Handling Varying Input6. Handling Varying Input

7. Subroutines7. Subroutines

8. File Processing8. File Processing

9. Reporting9. Reporting

10. Using the UNIX 10. Using the UNIX DBM DBM DatabaseDatabase

11. The Web and Perl11. The Web and Perl


1. Starting with Perl1. Starting with Perl PPractical ractical EExtraction and xtraction and RReport eport LLanguageanguage

– Larry Wall, from 1987Larry Wall, from 1987

Features combine shell, awk, many UNIX Features combine shell, awk, many UNIX tools, C libraries (all coded in Perl).tools, C libraries (all coded in Perl).

Emphasis on string manipulation, table Emphasis on string manipulation, table generation, REs, generation, REs, excellentexcellent libraries libraries– especially good Internet/Web librariesespecially good Internet/Web libraries

continued


On many platforms; freeOn many platforms; free Many Many manman and and infoinfo pages. pages. Many newsgroups (e.g. Many newsgroups (e.g. comp.lang.perlcomp.lang.perl))

Perl v.5 Perl v.5 – OOP, more REs, modulesOOP, more REs, modules– backward compatible with v.4backward compatible with v.4– check your version by typing:check your version by typing:

$ perl -v$ perl -v


1.1. Books1.1. Books

Learning PerlLearning PerlRandal L. Schwartz & Tom Christiansen Randal L. Schwartz & Tom Christiansen O'Reilly, 1997, 2nd ed.O'Reilly, 1997, 2nd ed.

Programming PerlProgramming PerlLarry Wall & Randal L. SchwartzLarry Wall & Randal L. SchwartzO'Reilly, 2000, 3rd ed.O'Reilly, 2000, 3rd ed.– not easy to read, more of a reference textnot easy to read, more of a reference text

In our library


1.2. Web Resources1.2. Web Resources

The main Perl site:The main Perl site:http://www.perl.comhttp://www.perl.com

– current Perl version, documentation, libraries current Perl version, documentation, libraries ((CPANCPAN), FAQs, tools and utilities, features), FAQs, tools and utilities, features

– reference sections on:reference sections on: applications (e.g. games, graphics), Web, applications (e.g. games, graphics), Web,

networking, system admin, etc.networking, system admin, etc.

continued


Resources about PerlResources about Perl– e.g. tutorials, manuals, Web-related, recommended e.g. tutorials, manuals, Web-related, recommended

books, example scriptsbooks, example scripts– http://hem.fyristorg.com/gumby/http://hem.fyristorg.com/gumby/

computing/computing/perl.htmlperl.html

A beginner's Guide to PerlA beginner's Guide to Perl– http://home.bluemarble.net/~scotty/Perl/http://home.bluemarble.net/~scotty/Perl/


1.3. Hello World (hw.pl)1.3. Hello World (hw.pl)

$ cat hw.pl#!/usr/bin/perlprint "Hello, world!\n";

$ chmod u+x hw.pl

$ hw.plHello, world!

Or:$ perl hw.pl


2. I/O2. I/O

#!/usr/bin/perl# hm.pl

print "What is your name? ";$name = <STDIN>;chop($name);print "Hello, $name!\n";

$ perl hm.plWhat is your name? andrewHello, andrew!$


3. if-else3. if-else

#!/usr/bin/perl# ch.pl

print "What is your name? ";$name = <STDIN>;chop($name);if ($name eq "Randal") { print "Hello, Sir Randal!\n";} else { print "Hello, $name!\n"; # ordinary}


4. while4. while

#!/usr/bin/perl# sword.pl

$secretword = "llama";print "What is your name? ";$name = <STDIN>;chop($name);if ($name eq "Randal") { print "Hello, Sir Randal!\n";}

:

continued

Not very secret

Guess the secret word


else { print "Hello, $name!\n"; # ordinary print "Secret word? "; $guess = <STDIN>; chop($guess); while ($guess ne $secretword) { print "Wrong, try again: "; $guess = <STDIN>; chop($guess); }} $ perl sword.pl

What is your name? andrewHello, andrew!Secret word? foobarWrong, try again: llama$


5. Arrays (lists)5. Arrays (lists)

@words = ("camel", "llama", "oyster");@words = ("camel", "llama", "oyster");

$words[0] $words[0] isis "camel""camel"

List variables begin with List variables begin with @@– e.g. e.g. @words@words

– $words[0]$words[0] is a value in the list, so uses is a value in the list, so uses $$


swords.plswords.pl

#!/usr/bin/perl

@words = ("camel", "llama", "oyster"); print "What is your name? ";$name = <STDIN>;chop($name);if ($name eq "Randal") { print "Hello, Sir Randal!\n";}

:

continued

Guess the secret word (v.2)

An array of secret words


else { print "Hello, $name!\n"; # ordinary print "Secret word? "; $guess = <STDIN>; chop($guess); $i = 0; # index into @words $correct = "maybe"; # dummy value

:

continued


while ($correct eq "maybe") { if ($words[$i] eq $guess) { # right? $correct = "yes"; } elsif ($i < 2) { # some words left? $i = $i + 1; } else { print "Wrong, try again: "; $guess = <STDIN>; chop($guess); $i = 0; # start words again }}

}

The user can win by guessing any of the secret words.


UsageUsage

$ perl swords.plWhat is your name? andrewHello, andrew!Secret word? fooWrong, try again: barWrong, try again: llama$

one of thesecret words


5.1. Associative Arrays (tables)5.1. Associative Arrays (tables) PersonPerson Secret WordSecret Word

fredfred camelcamelbarneybarney llamallamabettybetty oysteroysterwilmawilma oysteroyster

%words = ("fred", "camel", "barney","llama",%words = ("fred", "camel", "barney","llama","betty", "oyster", "wilma", "oyster");"betty", "oyster", "wilma", "oyster");

$words{"betty"}$words{"betty"} is is "oyster""oyster"

Associative array variables begin with Associative array variables begin with %%– e.g. e.g. %words%words

– $words{"betty"}$words{"betty"} is a value in the list, so uses is a value in the list, so uses $$


swords2.plswords2.pl

#!/usr/bin/perl

%words = ("fred","camel","barney","llama", "betty", "oyster", "wilma", "oyster");print "What is your name? ";$name = <STDIN>;chop($name);if ($name eq "Randal") { print "Hello, Sir Randal!\n";}

:

continued

Each user has their own secret word


else { print "Hello, $name!\n"; # ordinary $secretword = $words{$name};

# get secret word

print "Secret word? "; $guess = <STDIN>; chop($guess); while ($guess ne $secretword) { print "Wrong, try again: "; $guess = <STDIN>; chop($guess); }}


UsageUsage

$ perl swords2.plWhat is your name? barneyHello, barney!Secret word? oysterWrong, try again: foobar Wrong, try again: llama$

barney'ssecretword!


5.2. Adding a Default Secret Word5.2. Adding a Default Secret Word

... $secretword = $words{$name}; if ($secretword eq "") { # not found! $secretword = "groucho"; }

print "Secret word? ";...


6. Handling Varying Input6. Handling Varying Input

sword.plsword.pl (and others) treats (and others) treats "Randal""Randal" in a in a special way.special way.

But:But: "Randal L. Schwartz""Randal L. Schwartz"

oror "randal""randal"

is treated like other users.is treated like other users.


6.1. Replace eq by =~ and RE6.1. Replace eq by =~ and RE

...if ($name =~ /^Randal/) { # it matches, do something} else { # no match, do something else}...

REs are one of the big advantages of Perl


6.2. Extended REs6.2. Extended REs

Check for a word boundary with Check for a word boundary with \b\b

Ignore case with the Ignore case with the ii option option (after the last (after the last //))


...if ($name =~ /^randal\b/i) { # matches "Randal...", "ranDAL..." print "Hello, Sir Randal!\n";} else { # no match, do something else}...

Part of final1.plPart of final1.pl


6.3. Substitution and Translation6.3. Substitution and Translation

SubstitutionSubstitution: find and replace: find and replace(like (like s/../../ s/../../ in in vivi).).

TranslationTranslation: map characters to others: map characters to others(like (like trtr))


...print "What is your name? ";$name = <STDIN>;chop($name);

$name =~ s/\W.*//; # Get rid of everything # after first word.

$name =~ tr/A-Z/a-z/; # Make lowercase.

if ($name eq "randal") { print "Hello, Sir Randal!\n"; }... e.g. Dr.Andrew Davison

becomes dr


7. Subroutines7. Subroutines

sub good_word { # used in final1.pl local($somename, $someguess) = @_;

# input arguments $somename =~ s/\W.*//; $somename =~ tr/A-Z/a-z/; if ($somename eq "randal") 1 # return true } elsif (($words{$somename} ||

"groucho") eq $someguess) { 1; } else { 0; # return false }} # end of subroutine

Not a good feature


Called in final1.pl:Called in final1.pl:

... print "Secret word? "; $guess = <STDIN>; chop($guess); while (! &good_word($name, $guess) ) { print "Wrong, try again: "; $guess = <STDIN>; chop($guess); }}

Don't forget the &


8. File Processing8. File Processing

Read words from the Read words from the "wordslist""wordslist" file: file:

sub init_words { # used inside final1.pl open(WORDSLIST, "wordslist"); while ($name = <WORDSLIST>) { chop($name); $word = <WORDSLIST>; chop($word); $words{$name} = $word; } close(WORDSLIST);}


wordslist Formatwordslist Format

fredfredcamelcamelbarneybarneyllamallamabettybettyoysteroysterwilmawilmaoysteroyster

The code treats the file as four pairs: name secret-word


Called in final1.pl:Called in final1.pl:

#!/usr/bin/perl

&init_words;print "What is your name? ";$name = <STDIN>;chop($name);...


8.1. File Tests8.1. File Tests

sub init_words { # second version open(WORDSLIST, "wordslist"); if (-M WORDSLIST > 7) { die "Sorry, wordslist is too old"; } while ($name = <WORDSLIST>) { chop($name); $word = <WORDSLIST>; chop($word); $words{$name} = $word; } close(WORDSLIST);}

-M is the file'smodification date


8.2. The open() Command8.2. The open() Command Similar to Similar to popen()popen() in C in C

– the command argument is executed and can be the command argument is executed and can be written to via an output streamwritten to via an output stream

In In good_wordgood_word::...open(MAIL, "| mail [email protected]");print MAIL "bad news $somename guessed

$someguess\n";close(MAIL);...

mailMAIL


8.3. Filename Expansion8.3. Filename Expansion

Assume several files of secret words:Assume several files of secret words:– ad.secretad.secret, , yuk.secretyuk.secret

Modify Modify init_words()init_words() to read in all of these to read in all of these– but only if the files are less than 7 days oldbut only if the files are less than 7 days old

Store in the Store in the %words%words associative array associative array

Also called filename globbing


sub init_words { # third version while ($filename = <*.secret>) { open(WORDSLIST, $filename); if (-M WORDSLIST < 7) { while ($name = <WORDSLIST>) { chop($name); $word = <WORDSLIST>; chop($word); $words($name) = $word; } } close(WORDSLIST); }}

filename expansion


8.4. The Password File8.4. The Password File

A typical password line:A typical password line:ad:x:42497:100:Dr.Andrew DAVISON,,,:ad:x:42497:100:Dr.Andrew DAVISON,,,:

/home/ad:/bin/bash/home/ad:/bin/bash

– the user's details are in the 5th field the user's details are in the 5th field the GECOS fieldthe GECOS field

$<$< always contains the user's ID number (UID) always contains the user's ID number (UID)


#!/usr/bin/perl# final1.pl

&init_words;@password = getpwuid( $< );

# get password data using the UID$name = $password[6];

# get the GECOS field (6th in Perl!)$name =~ s/,.*//;

# throw away everything # after 1st comma

if ($name =~ /^randal\b/i) { print "Hello, Sir Randal!\n";}...

There is a getpwuid() inC which does the same thing


9. Reporting9. Reporting

secret.plsecret.pl: : – list the list the *.secret*.secret files in the format: files in the format:

filename filename user-nameuser-name secret-word secret-word


secret.plsecret.pl

#!/usr/bin/perlwhile ($filename = <*.secret>) { open(WORDSLIST, $filename); if (-M WORDSLIST < 7) { while ($name = <WORDSLIST>) { chop($name); $word = <WORDSLIST>; chop($word); write; # invoke STDOUT format } } close(WORDSLIST);}

continued


STDOUT Format DefinitionSTDOUT Format Definition

format STDOUT =@<<<<<<<< @<<<<<<<<< @<<<<<<<<<<<$filename, $name, $word.

field definition line

field value line

continued


Top of Page FormatTop of Page Format

Used every 60 lines by default:Used every 60 lines by default:

format STDOUT_TOP =Page @<< # field definition line$% # number of pages printedFilename Name Word=========== ======= ========.


OutputOutput

$ secret.plPage 1Filename Name Word============= ======= ========ad.secret andrew f1ad.secret paul f2ad.secret dr foobaryuk.secret jim c1yuk.secret bob c2$

The *.secret files mayneed to be 'touched'.


10. Using the UNIX DBM Database10. Using the UNIX DBM Database

In In final1.plfinal1.pl, the , the timetime of the most recent of the most recent correct guess is stored in correct guess is stored in %last_good%last_good

$last_good{$name} = time;$last_good{$name} = time;

ProblemProblem: : %last_good%last_good will be lost when the will be lost when the script terminates.script terminates.

SolutionSolution: store : store %last_good%last_good in a in a DBM DBM database.database.


...dbmopen(%last_good, "lastdb", 0666);$last_good{$name} = time;dbmclose(%last_good);...

returns time inseconds since 1/1/1970

rw for everyone

Revised final1.plRevised final1.pl


Using final1.plUsing final1.pl

$ final1.pl$ final1.plHello, Dr.Andrew DAVISON!Hello, Dr.Andrew DAVISON!What is the secret word? What is the secret word? bingobingoWrong, try again. What is the secret Wrong, try again. What is the secret word? word? foobarfoobar$ $

foobar is the secret wordfor dr (see slide 45)


List the Last Guessses (ex_last.pl)List the Last Guessses (ex_last.pl)

#!/usr/bin/perl

dbmopen(%last_good,"lastdb",0666);foreach $name (sort keys(%last_good)) { $when = $last_good{$name}; $hours = (time - $when) / 3600;

# compute hours ago write; # use STDOUT format}

format STDOUT =User @<<<<<<<<<<<<: last correct guess

was @<<< hours ago.$name, $hours.


OutputOutput

$ ex_last.plUser Andrew DAVIS: last correct guess

was 2066 hours ago.User Dr.Andrew DA: last correct guess

was 0.07 hours ago.$

last year, whenmy user detailswere different


11. The Web and Perl11. The Web and Perl

Perl supports sockets for building Web Perl supports sockets for building Web clients and servers.clients and servers.

A easier and faster way is to use the A easier and faster way is to use the LWP modulesLWP modules– designed for Web (client) tasksdesigned for Web (client) tasks– hides low-level networking issueshides low-level networking issues


Main LWP FeaturesMain LWP Features

Object oriented notationObject oriented notation Supports all of the HTTP protocolSupports all of the HTTP protocol

– e.g. GET, POST, HEAD, redirectione.g. GET, POST, HEAD, redirection Handles authenticationHandles authentication Deals with proxy serversDeals with proxy servers Can use timed actionCan use timed action Has a framework for writing Web 'robots'Has a framework for writing Web 'robots'


11.1. Retrieve a Web Page11.1. Retrieve a Web Page

!/usr/bin/perl!/usr/bin/perl# geturl.pl# geturl.pl

useuse LWP::Simple; # use module LWP/Simple.pm LWP::Simple; # use module LWP/Simple.pm

print ( print ( getget $ARGV[0] ); $ARGV[0] );

$ geturl.pl http://fivedots.coe.psu.ac.th/~ad<html><head><title>Andrew Davison's Home Page at PSU</title> :


11.2. Parse a Web Page11.2. Parse a Web Page

#!/usr/bin/perl#!/usr/bin/perl# showurl.pl# showurl.pl

use LWP::Simple;use LWP::Simple;use HTML::Parse;use HTML::Parse;

print print parse_htmlparse_html( get $ARGV[0] )( get $ARGV[0] )->format->format;;

$ showurl.pl http://fivedots.coe.psu.ac.th/~adCan't locate HTML/Parse.pm in ...$

call format methodon the result


11.3. Extract Links11.3. Extract Links

#!/usr/bin/perl#!/usr/bin/perl# showlinks.pl# showlinks.pluse LWP::Simple;use LWP::Simple;use HTML::Parse;use HTML::Parse;use HTML::Element;use HTML::Element;

$phtml = parse_html( get $ARGV[0] );$phtml = parse_html( get $ARGV[0] );

for ( @{ $phtml->extract_links() }) {for ( @{ $phtml->extract_links() }) { $link = $_->[0]; # access array $link = $_->[0]; # access array print "$link\n"; print "$link\n";}}

$ showlinks.pl http://fivedots.coe.psu.ac.th/~ad

call extract_links methodon the $phtml object


11.4. Deal with Proxies11.4. Deal with Proxies #!/usr/bin/perl#!/usr/bin/perl

# hcat_proxy.pl# hcat_proxy.pluse LWP::UserAgent;use LWP::UserAgent;use HTTP::Request;use HTTP::Request;use HTTP::Response;use HTTP::Response;

mymy $ua = $ua = newnew LWP::UserAgent; # local object LWP::UserAgent; # local object $ua->$ua->proxyproxy('http', 'http://cache.psu.ac.th:8080/');('http', 'http://cache.psu.ac.th:8080/');$ua->$ua->no_proxyno_proxy('coe.p-net');('coe.p-net');

my $request = new HTTP::Request('GET', $ARGV[0]);my $request = new HTTP::Request('GET', $ARGV[0]);my $response = $ua->my $response = $ua->requestrequest($request);($request);if ($response->if ($response->is_successis_success) {) { print $response-> print $response->contentcontent;;} else {} else { print $response-> print $response->error_as_HTMLerror_as_HTML;;}}


Use (on fivedots)Use (on fivedots)

$ perl hcat_proxy.pl http://www.perl.com$ perl hcat_proxy.pl http://www.perl.com<html><html><head><title>www.perl.com - Beginners Intro to <head><title>www.perl.com - Beginners Intro to Perl - Part 3</title> <meta name="robots"Perl - Part 3</title> <meta name="robots" : :


11.5. More Information on LWP11.5. More Information on LWP

On On calvincalvin::– man LWPman LWP– pod2text < /usr/lib/perl5/lwpcook.podpod2text < /usr/lib/perl5/lwpcook.pod

Perl modules (including LWP) are in:Perl modules (including LWP) are in: /usr/lib/perl5/usr/lib/perl5

Related modules:Related modules:– HTML, HTTP, MIME, URI, WWWHTML, HTTP, MIME, URI, WWW

continued


Web Client Programming with PerlWeb Client Programming with PerlClinton WongClinton WongO'Reilly, 1997O'Reilly, 1997I have a copy

advanced unix

Documents