procmail recipes

7/27/2019 Procmail Recipes

http://slidepdf.com/reader/full/procmail-recipes 1/184

Table of Contents1.0 Document id

1.1 General1.2 What is Procmail?1.3 Abbreviations and thanks1.4 Version information

1.5 Document layout and maintenance1.6 About presented recipes1.7 Variables used in recipes1.8 About "useless use of cat award"

2.0 Procmail pointers

2.1 Where is procmail developed2.2 Procmail resources2.3 Procmail mode for Emacs2.4 Procmail module library project2.5 Procmail code to filter UBE

3.0 Dry run testing

3.1 What is dry run testing?3.2 Why the From field is not okay after dry run?3.3 Getting default value of a procmail variable

4.0 Things to remember

4.1 Get the newest procmail4.2 Csh's tilde is not supported4.3 Be sure to write the recipe starting right4.4 Always set SHELL4.5 Check and set PATH4.6 Keep the log on all the time4.7 Never add a trailing slash for directories4.8 Remember what term DELIVERED means4.9 Beware putting comment in wrong places4.10 Brace placement4.11 Local lockfile usage4.12 Global lockfile4.13 Gee, where do I put all those ! * $ ??4.14 If you Send an automatic reply, use X-loop header4.15 Avoid extra shell layer and check command for SHELLMETAS4.16 Think what shell commands you use

4.17 Using absolute paths when calling a shell program4.18 Disabling a recipe temporarily4.19 Keep message backup, no matter what4.20 Order of the procmail recipes

5.0 Procmail flags

5.1 The order of the flags5.2 Flags HB at top of recipe (warning)5.3 Flag w and recipe with pipe(|)5.4 Flag w, lock file and recipe with pipe(|)5.5 Flag f and w together5.6 Flags h and b5.7 Flag h and sinking to /dev/null5.8 Flag i and pipe flag f 5.9 Flag r5.10 Flag c's background5.11 Flag c before nested block forks a child



9.3 Emergency stop for your .procmailrc

10.0 Scoring

10.1 Using scores by an example10.2 Brief Score tutorial10.3 Score's scope10.4 Counting length of a string10.5 Counting lines in a message (Adding Lines: header)

10.6 Determining if body is longer than header10.7 Matching last Received header10.8 Testing value range with scoring (bogofilter)10.9 How to add Content-Length header10.10 Testing message size or number of lines10.11 Counting commas with recursive includerc

11.0 Formail usage

11.1 Fetching fields with formail -x11.2 Always use formail's -r switch11.3 Rewriting the From address11.4 Formail -r and Resent-From header11.5 Quoting the message11.6 Without quoting the message11.7 How to include headers and body to the reply message11.8 Adding text to the beginning of message11.9 Adding text to the end of message11.10 Adding text before quoted message11.11 How to truncate headers (save filing space)11.12 Adding extra headers from file11.13 Splitting digest11.14 Mailbox: Splitting to individual files11.15 Mailbox: Extracting all From addresses from mailbox

11.16 Mailbox: Applying procmail recipe on whole mailbox11.17 Mailbox: run series of commands for each mail (split mailbox)11.18 Option -D and cache11.19 Option -D and message-id in the body11.20 Reducing formail calls (conditionally adding fields)11.21 Formail -A -a options11.22 Formail -e -s options

12.0 Saving mailing list messages

12.1 Using subroutine pm-jalist.rc to detect mailing lists12.2 Using plus addressing [email protected] Using RFC comment trick for additional information12.4 Simple mailing list handling12.5 Archiving according to TO12.6 Using Return-Path to detect mailing lists

13.0 Procmail, MIME and HTML

13.1 Mime content type application/ms-tnef 13.2 Trapping HTML mime messages13.3 Complaining about HTML messages13.4 Converting HTML body to plain text13.5 Getting rid of unwanted mime attachments (HTML, vcard)13.6 Sending contents of a HTML page in plain text to someone

14.0 Simple recipe examples

14.1 Saving: MH folders -- numbered messages14.2 Saving: to monthly folders14.3 Modifying: Filtering basics

22.5 RFCs and messages signature22.6 RFC and using MIME in Usenet newsgroups22.7 Some RFC Pointers

23.0 Introduction to E-mail Headers

23.1 To find out more about mail23.2 Lecture by Alan Stebbens23.3 Applied to received messages

23.4 Bcc lecture by Alan Stebbens23.5 Bcc lecture by Philip Guenther

24.0 Message headers

24.1 What is correct From address syntax24.2 What's that X-UIDL header?24.3 What is that first From_ header?24.4 Message-Id header24.5 Received header24.6 Return-Path24.7 Errors-To24.8 X-Subscription-Info24.9 Reply-To header24.10 Mail-Copies-To header24.11 Mail-Followup-To and Reply-To-Personal headers24.12 Content-Length header and From_ specification24.13 Moral about CC copies in Usenet

1.0 Document id1.1 GeneralCopyright © 1997-2012 Jari AaltoHomepage http://pm-doc.sourceforge.netURL links last checked: 2010-12-05

License: This material may be distributed only subject to the terms and conditions set forth in GNUGeneral Public License v2 or, at your option, any later version.

Procmail is powerful mail handling tool and a lot of space here has been devoted to discuss aboutUBE (aka Spam) and its es sence.

This is a Procmail Tips page: a collection of procmail recipes, instructions, howtos. You will also findmany other interesting subjects that discuss about Internet mail in general: mail headers, MIME andRFCs. Another part of this document is dedicated to Emacs and its maili handlling capabilities. Emacs

is pow erful tool that can be used for both mail and news reading; available in W indows platform aswe ll. The tips have bee n compiled from the procmail discussion list, from comp.mail.misc and from theauthor's ow n experiences with procmail.

This document does not intend to teach the basics of procmail, instead you should be familiar withthe procmail manual pages already. If you're using Windows operating system, procmail is availablein Cygwin < http://www.cygwin.com/ > distribution.

See also Nancy's and Era's procmail FAQ pages.

If you find errors or things to improve in this document, please send mail to this document'smaintainer (see project page). If some URL is not alive any more, you may still be able to find it by

using a WW W search such as Google.There is never too much to learn about procmail and the best source is the rc files that people havedone. If you have some time, please place your .procmailrc with good comments to your homepage.



2008-09-21 510 Links checked.2008-03-10 510 Add Gmane URL. Links checked.2007-10-02 519 New HTML/CSS layout. Links checked.2006-02-15 519 Sanitized all email addresses.2004-10-10 516 Spam related things removed.2002-08-31 596 Removed old UBE pointers.2002-08-13 596 Removed old UBE pointers.2002-02-01 608 Spelling checked with Emacs ispell2002-01-28 608 URL links checked and updated2001-08-09 608 http://pm-doc.sourceforge.net opened.1999-12-27 603 Netscape spam filters added1999-10-01 602 Mark Seiden's patch applied. Now under CVS.1999-04-26 599 document moved to www.procmail.org1999-04-21 597 Links corrected1999-03-29 597 Ricochet -- Perl script to fight UBE1999-02-26 592 procmail's Y2K compliance1999-02-23 590 RFC and using MIME in Usenet postings1998-01-29 587 Added "Lua" language pointer1998-01-07 579 Eli's procmail recipes in module section1998-12-14 578 Philip took care of bugs/patches listing1998-11-26 602 More Richard's comments integrated1998-10-30 595 Richard's english correction patch1998-10-21 591 UMASK, .forward if procmail already is LDA1998-10-12 583 SmartList and other MLM software discussed1998-10-06 575 PLUS addr. Convert HTML body to text1998-08-29 565 Fetching fields with formail -x1998-08-24 554 Procmail doesn't pass 8bit characters1998-08-24 553 Flag c forking study, procmail wish list1998-08-18 541 Small changes. MIME notes1998-08-10 529 Guido.Van.Hoeck's 55k patch applied1998-06-24 526 Added live urls to procmail archive1998-06-23 521 All recipes checked by eye. Many fixes.

1998-06-19 516 Detecting mailing lists with pm-jalist.rc1998-06-17 510 How to disable recipe quickly with1998-04-03 493 Includerc rewritten, plus addressing1998-04-02 488 ORing and supreme scoring added1998-03-23 471 All recipes checked (by eye)1998-03-10 469 Better ordering: ORing rules discussed1998-01-30 429 "regexp" section rewrite.1997-12-30 415 up till 1996-12 is now included1997-12-09 343 up till archive 1996-07 now included1997-11-25 2601997-11-08 218 Era's correction suggestions.1997-10-13 181 archive file 1995-10's tips included1997-10-11 1421997-10-01 1271997-09-18 941997-09-16 761997-09-14 531997-09-13 46 (k)

1.5 Document layout and maintenanceThe base version o f this document is kept in plain text f ormat, which requires no special editors or

learning a markup language. The tools to help maintaining this document include:

Emacs and technical formatting package (TOC, indentation control, fontifying) tinytf.el fromhttp://freecode.com/projects/emacs-tiny-too lsText-to-HTML conversion program t2html.pl from http://freecode.com/projects/perl-text2html



Text version of this file is converted to HTML with:

perl -S t2html --Auto-detect --Out pm-tips.txt

The t2html converter project is at http://freecode.com/projects/perl-text2html

SENDING IMPROVEMENT S

If you have any spare moment, a glimpse to f ind some spelling mistakes or misuse of the verbs,please go ahead and send a patch to maintainer of this page. The pref erred way to send correctionsto this document is as diff(1) output. Here's how to make corrections send them f orward. Pleasetry to use unif ied dif f -u option. The source is available at version control repository of http://sourcef orge.net/projects/pm-doc

cp pm-tips.txt pm-tips.txt.orig

... load the pm-tips.txt to a text editor / edit / save

... Generate the difference

diff -bwu pm-tips.txt.orig pm-tips.txt > pm-tips.txt.patch

...Send content of pm-tips.txt.diff by mail to maintainer

If you do not know what a diff f ormat is, then simply send your comments in email. Use "Linux: pm-doc" as subject to bypass spam f iltering.

1.6 About presented recipes

The recipes have been kept as original as possible, but a generalization of the ideas have beendone when necessary. If some recipe doesn't work as announced, please a) send note to[maintainer] b) send mail to procmail mailing list and ask how to correct it. Sometimes a simple dot(.)has been used in regular expressions, where the right, pedantic way would have been to use anescaped dot. If you want to be very strict, you should use the escaped dot w here applicable.

# free hand version # pedantic version:0 :0* match.this.site * match\.this\.site

Procmail also accepts ass ignments without quo tes, like this:

var = valuenum = 1dir = /var/mail

But in this document a s trict style ha s been adopted, where literal strings are ass igned with doublequotes:

var = "value"

That's because the procmail code checker (Emacs package tinyprocmail.el ) then won't warn about



missing dollar-sign, which might have very we ll been f orgotten. Emacs package f ont-lock.el , a syntaxhighlighting assistant, also displays double quo ted s tring in color.

# If you do this...

var = value

# then you might have made a typo. It is in fact not clear# what was intended:

var = "value" # Did you mean: literal assignment?var = $value # Did you mean: variable assignment?

Recipe f lags are also not stuck together, because the visual distinction of :0 and flags is a valuableone. Reasoning f or which flags are kept together and in which order is explained later in details.

# Erm, all stuck] # This may be visually more clear:0ABDc: :0 A BD c:

1.7 Variables used in recipesThese are pa rt of the procmail module pm-javar.rc and are us ed in recipes.

# Pure newline; typical usage if you want to write# Something directly to procmail's active logfile:#

# LOG = "$NL message $NL"

NL = ""

Ref er to "improving Space-Tab syndrome" section f or more deta ils

WSPC = " " # whitespace: space + tab

SPC = "[$WSPC]" # Regexp: space + tabSPCL = "($SPC|$)" # whitespace + linefeed: spc/tab/nlNSPC = "[^$WSPC]" # negation

s = $SPC # shortname: like perl -- \sd = "[0-9]" # A digit -- Perl \dw = "[0-9a-z_A-Z]" # A word -- Perl \wW = "[^0-9a-z_A-Z]" # A word -- Perl \Wa = "[a-zA-Z]" # A word, only alphabetic chars

Writing recipes is now a little easier and may look more clear at least to people that haveaccustomed reading Perl regular expression s hort names:

:0*$ Header-Name:$s+$d+$s+$d # Matches "Header: 11 12"



{# Matched "whitespace" + "digit" + "whitespace" + "digit"# Do something

}

SUPREME = 9876543210, is the highest score value tha t causes procmail to bail out. [david] Actuallythe maximum is 2147483647, but 9876543210 is easier to remember/type and will f unction just aswell.

PMSRC = Procmail module source code directory. Location where *.rc f iles reside. A nywhere youwant it to be. Usually $HOME/pm or $HOME/procmail/lib . Here you can keep the procmail f iles, log filesand includerc scripts. A nother common used synonym is PMDIR .

SPOOL = Directory where your procmail delivers the categorized messages . Like mailing lists:

list.procmail, list.lynx-users, list.emacs, list.elm

and w ork mail:

work.announcements, work.lab, work.doc, work.customer

and your private message:

mail.usenet, mail.private, mail.default, mail.perl

and unimportant messages

junk.daemon, junk.cron, junk.ube

If you read the procmail-delivered f iles directly, this directory is usually $HOME/Mail or $HOME/mail. If you use some other software that reads these files as mail spool files (like Emacs Gnus), then thisdirectory is typically $HOME/Mail/spool or similar.

MYXLOOP = Used to prevent re-sending messages that have already been handled. Typically$LOGNAME@$HOST, but this can be any user chosen string. Make it it unique to your address. In thisdocument the def inition is:

MYXLOOP = "X-Loop: $LOGNAME@$HOST"

SENDMAIL = Program to deliver composed mail. Usually standard Unix sendmail(1) , but it must havesome sw itches with it. See man page f or more. We use following def inition in scripts:

SENDMAIL = "sendmail -oi -t"

NICE = In a Unix environment you can lower the scheduling priority with nice(1) . If you areconscious of how many external processes you launch f or each piece of mail it would be polite to

lower the priority of such processes. You may see in this document that external processes arecalled with NICE enabled:

:0 w # Same as "nice -10 script.pl"| $NICE script.pl

IS f unctions; Functions to tes t f ile or directory attributes . E.g. IS_EXIST is defined as "tes t -e" and soon. The definitions of IS f unctions are system-dependent. E.g. On Irix the "-e" option is notrecognized and the nearest equivalent is "test -r". A ll IS f unctions are def ined in the pm-javar.rcmodule.

1.8 About "useless use of cat award"Randal Schwartz, a we ll-known Perl programmer and Perl boo k writer, started giving rewards f or the"useless use of cat command" w henever someone w rote examples without token "<". Like this:

$ cat file.name.this | wc -l

Instead he writes that the call shou ld have been w ritten like this, which saves the pipe (never mindthat wc can read the f ile directly; this is an example).

$ wc -l < file.name.this

[Paul David Fardy <pdf A T morgan.ucs.mun.ca>] There is we ight in the pipeline, but the true cost isin process startup. Try running wc 100 times on /etc/motd or on this message. My tests show theuseless use of cat doubles the real and processing time (real, user, and system time are eachroughly doubled):

$ cat > /tmp/randal << 'EOF'COUNT=100i=1while :do

wc < /etc/motd > /dev/nulli=$(expr $i + 1)

[ "$i" = "$COUNT" ] && breakdoneEOF

$ cat > /tmp/useless << 'EOF'COUNT=100i=1while :do

cat /etc/motd | wc > /dev/nulli=$(expr $i + 1)[ "$i" = "$COUNT" ] && break

doneEOF

# NOTE: The timing values should be read as absolute, but# examine the relative differencies.

$ time sh /tmp/randallreal 0m0.568suser 0m0.208ssys 0m0.348s

$ time sh /tmp/uselessreal 0m0.825suser 0m0.348ssys 0m0.476s

This becomes important, f or example, when you decide to filter all your mail with procmail – lookingf or virus signatures f or example. I might well decide to look only at the f irst 3 or 4 kilobytes. It's notthe size of messages --most are small anyway – but the number of messages that cause a problem.Do you want to double the processing cost of all our mail? I'm looking at a system-wide filter for allmy users' mail. I'm considering Sendmail's mail f ilter versus procmail filtering. I'll likely be using a bitof both. A nd given that a ll of the f iltering really just getting in the way of legitimate traff ic, it'd reallypiss me of f if I naively doubled the cost.

2.0 Procmail pointers2.1 Where is procmail developedPhilip Guenther <guenther A T gac.edu> is currently taking care of and coordinating procmail bugf ixes. Please send any procmail bugs to the mailing list or to [email protected] . The developmentmailing list is running SmarList at [email protected] . Newest Procmail code it at< http://www.procmail.org/ > and ready packages are a vailable a t Linux distributions' repositories.

2.2 Procmail resourcesProcmail is discussed in Usenet newsgroup comp.mail.misc and mailing list accessible at NNTP server

< http://news.gmane.org/gmane .mail.procmail >.

Nancy McGough <nm A T noadsplease.ii.com - Prcmail Quick start>Era Eriksson's Procmail FAQ and link collectionProfessor Timo Salmis's P rocmail pageEli the Bearded's address ing tipsConcordia University's procmail page <webdoc A T alcor.concordia.ca>: "...People often askhow to avoid receiving "spam" mail, or how to bounce mail from someone who is annoyingthem. These pages tell you ho w to install procmail; you can then ta ilor it to do all those things,or whatever else you want."

2.3 Procmail mode for EmacsIf you use Emacs , See Procmail mode tinypm.el at < http://freecode.com/projects/emacs-tiny-tools >.It can also be used to sta tically syntax check recipes. Here is an example o f its o utput:

*** 1997-11-24 22:13 (pm.lint) 3.11pre7 tinypm.el 1.80cd /users/jaalto/junk/pm.lint:010: Warning, no right hand variable found. ([$`']pm.lint:055: Pedantic, flag orer style is not standard `hW:'pm.lint:060: Warning, message dropped to folder, you need lock.pm.lint:062: Warning, recipe with "|" may need `w' flag.pm.lint:073: Warning, Formail used but no `f' flag found.

2.4 Procmail module library project2.4.1 Where to get various modules

[email protected], net=com, account=foo... pm-jastore.rc – Subroutine for general mailbox delivery. Define MBOX as the folder where todrop message and this subroutine w ill store it appropriately. Supports s ingle mboxes, ".gz"mbox files, directory files and MH folders w ith rcvstore.

2.4.6 Spam modules

Read "Thoughts about increasing spam annoyance" at < http://pm-lib.sourceforge.net/README.html >which explains these modules better in context "2.0 A lightweight UBE block system with pure

procmail". pm-jaube.rc – Subroutine to investigate the message for know spam pattern like numericaddress , invalid address, Pe gasus bulk mail, advertising s logans etc. This is the generic Spamdetection module. Needs only one external program: nslokup1(1) to verify the sender'sdomain. The results of classification appears in returned variables that the caller can use fordeciding what to do. Optional headers can be added to the message to anno unce the results.

pm-jaube-keywords.rc – Subroutine to scrutinize the message against known spam keywords.This is the "bare bones" and very simplistic (but fast) way to check if message is Spam. Theresults of classification appe ars in returned variables that the caller can use for deciding whatto do.

pm-jaube-prg-runall.rc – An Interface module to call external statistical bayesian s pam classifier

programs. This subroutine will call other modules , like pm-jaube-prg-bogofilter.rc (forbogofilter), pm-jaube-prg-bsfilter.rc (for bsfilter) and many many more that help fighting spam.It is possible to activate specific bayesian programs available in current host.

2.4.7 Mime m odules

pm-jamime.rc – Subroutine to read MIME headers and put the mime version, boundary string,content-type information to variables.

pm-jamime-decode.rc – recipe to decode quoted-printable or base64 encoding in the body. pm-jamime-kill.rc – Recipe for attachment killing: wipes out the e xtra mime cruft leaving onlythe plain text. Applications for killing: ms-tnef a ttachment (MS Explorer 7k), HTML attachments(Netscape, MS Express) vcard (Netscape), PCX attachment (Lotus Notes).

pm-jamime-save.rc – Recipe for saving simple file attachment. When you receive ONE fileattachment in a message, this recipe can save it in a sepa rate directory. The content is alsodecoded (base64,qp) while saving.

2.4.8 Filtering message body or headers

pm-jadaemon.rc – Handle DAEMON messages by changing subject to reflect a) the error reasonb) to whom the message w as o riginally sent c) original subject sent and what was thesubject. Store the DAEMON messages to separate folder.

pm-jasubject.rc – Standardize Subject "Re 32 : FW: Sv: message" or any o ther derivate to defacto "Re: message"

pm-janetmind.rc – [obsolete] Reformat minder.netmind.com messages (no longer exists 2005).

The default 4k message is shortened to a few important lines.2.4.9 Mailing list modules

pm-jalist.rc – Subroutine to extract mailing list name from message. Do you need to add a newrecipe to your .procmailrc every time you subs cribe to new mailing list? If you do , take a loo k atthis module, which examines the message and defines variable LIST to ho ld the mailing listname. You can use it directly to save the messages adaptively to correct folders. No morehand work and manual storing of mailing list messages .

2.4.10 Miscellaneous modules

pm-jaempty.rc – check if message body is empty (nothing relevan t). Define variableBODY_EMPTY to "yes" o r "no" if message is empty.

pm-janslookup.rc – Run nslookup on given address . If you compose return address w ith"formail -r -x To:" you can verify if domain is registe red before sending reply. Uses cache foralready looked up domains. This module is alos used by the pm-jaube.rc to verify the sender'sdomain.

uess-mua.rc – Guess the Mail User A ent and set MUA: MH,PINE,MAIL



2.4.11 Low-level Date and time handling

For these, you get the date string f rom somewhere, then f eed it to some of these subroutines:

pm-jatime.rc – a low -level subroutine. Parse time "hh:mm:ss" from variable INPUT pm-jadate1.rc – a low-level subroutine. Parse da te "Tue, 31 Dec 1997 19:32:57" from variableINPUT

pm-jadate2.rc – a low-level subroutine. Parse ISO standard date "1997-11-01 19:32:57" from

variable INPUT pm-jadate3.rc – a low-level subroutine. Parse da te Tue Nov 25 19:32:57 from variable INPUT pm-jadate4.rc – Call shell command "date " once to construct RFC "Tue, 31 Dec 1997 19:32:57"and parse the YY MM HH and other values. You usua lly use this subroutine if you can't get thedate anywhere else.

2.4.12 Higher-level Date and time handling

You use these recipes to get the date directly from the message:

pm-jadate.rc – higher-level recipe. Read date from message's headers: From_ Received, or callshell date if none s ucceeds .date.rc – higher-level recipe. From Alan's procmail-lib: parse da te or from headers Resent-Date:, Date , and From

2.4.13 Forwarding and account modules

pm-japop3.rc – Pop3 movemail implemented with procmail. You can send a "pop3" request tomove your messages from account X to account Y. Each message is send separately. Thisrecipe listens to "pop3" reques ts.

pm-jafwd.rc – control forwarding remotely. You can change the forward address w ith a "controlmessage" or turn forwarding on/off with a "control message"

pm-japing.rc – Send short reply when subject contains the w ord "ping" to show that theaccount is up a nd mail address is valid.correct-addr.rc – From alan's procmail lib. To he lp forward mail from an OLD address to a NEW

address , and do some mailing list mail management. This recipe file is intended to make it easyfor users to forward their mail from their old address to a new address , and, at the same time,educate the ir correspondents about it by CC'ing them with the mail.

2.4.14 Vacation modules

pm-javac.rc – A framewo rk for your vacation rep lies. This recipe will handle the vacation cacheand compose an initial reply; which you only need to fill in. (Like putting vacation message tothe body)ackmail.rc – From Alan's procmail lib. procmail rc to acknowledge mail (with either a vacationmessage, or an acknowledgment)

2.4.15 Message-id based modules pm-jadup.rc – Handle duplicate messages by Message-Id. Store duplicate message in separatefolder.dupcheck.rc – From Alan's procmail-lib. If the current mail has a "Message-Id:" header, run themail through "formail -D", causing duplicate messages to be dropped. Can use MD5 hash incache.

2.4.16 Cron m odules

pm-jacron.rc – A framewo rk for your daily cron tasks. This recipe conta ins all the needed checksto ensure tha t your includerc is called w henever a day changes . (Day change is subject tomessages you receive). Your ow n cron includerc is run once a day.

2.4.17 Backup modules

pm-jabup.rc – Save messages to backup directory and keep only N messages per day. Idea byJohn Gianni. Note: The implementation w ill always call she ll for each message you receive; sousing this module is no t recommended if you ge t many messages pe r day. Instead, use the

cron module to clean the messages' backup directory only once a day, and no t every time amessage arrives.

2.4.18 Conf irmation modules

pm-jacookie.rc – Handle cookie (unique id) confirmations . Also know n as Procmailauthentication service (PAS). This simple procmail module will accept messages only from userswho have returned a "cookie" key. You can use this to to protect some services before access .Uses subroutine pm-jacookie1.rc, which generates the unique cookie; CRC 32 by default.

NOTE: Please read page < http://pm-lib.sf.net/README.html > before you may start thinking touse this module as a generic Challenge-Respo nse module to reduce spam.

2.5 Procmail code to filter UBE Sys adms remember : S pam filtering is much more efficie ntly done in the MTA, e spe cially if you are just looking at From and Tolines . For example , you can se tup in Exim a rule that block s \d.*@aol\.c om (that is any aol.co m local part that begins w ith a digit).

AOL guarantees that none of their addres se s be gin with a digit. Exim re jects such bogus addre sse s at the S MTP leve l before themessage is received.

pm-jaube.rc -Procmail module library's UBE filter After Daniel Smith poste d his spam recipes toprocmail mailing list, the code w as adopted and more generalized to handle lot more UBE.Module needs no special setup and can be installed via simple INCLUDERC. All UBE detectionhappens using procmail rules with no external files needed. The module is available in Procmail

module library at < http://freecode.com/projects/procmail-lib >. 2.5.1 o Catherine A. Hampton'sSpambouncer" . The attached set o f procmail recipes/filters, which I call The Spam Bouncer, arefor users who are sick of spam (unsolicited junk mail) and wa nt to filter it out of their mail aseas ily as possible. These recipes can be used as sha red recipes for a w hole system, or by anindividual for the ir own mailbox only.Junkfilter . by Gregory Sutter. Junkfilter is a user-configurable procmail-based filter system forelectronic mail. Recipes include checks for forged headers, key words, common spam domains,relay servers and many others.Nonplussed Spambouncer Procmail module for bouncing spam. Requires sendmail with plussedusers.

3.0 Dry run testing3.1 What is dry run testing?It means that you call your procmail test s cript directly with sample tes t mail

% procmail $HOME/pm/pm-test.rc < $HOME/tmp/test-mail.txt

The script pm-tes t.rc has the procmail recipe you're te sting or improving. The test-mail.txt is any valid

mail message containing the headers and body. You can make one with any text editor, e.g. vi ,pico , nano , emacs or xemacs . Here's a simple test mail skeleton. Copy verbatim:

From: [email protected]: [email protected] (self test)X-info: I'm just testing

BODY OF MESSAGE SEPARATED BY EMPTY LINEtxt txt txt txt txt txt txt txt txt txt

Remember that you can define environment variables as well in the dry run call. Here's an examplewhere procmail just executes the s cript and does no thing fancy.

% procmail VERBOSE=on DEFAULT=/dev/null \

Beware writing it 0: as it happens easily. A lways put a zero af ter the colon that begins the recipe. Inthe f irst versions of procmail, you would put the number of conditions, with a def ault of 1. That wasannoying, and the computer can do the counting easier, so Stephen made it so that a count of 0indicates that the conditions are all the lines beginning with a * . The def ault is one, unless the a , A ,e , or E f lags is given, in which case the de f ault is ze ro. ALW AY S ST ART a RECI PE WIT H :0 .

4.4 Always set SHELLIf your login shell is a C shell (csh or tcsh), avoid havoc: as a precaution, always put f ollowing at thetop of your $HOME/.procmailrc.

SHELL = /bin/sh

4.4.1 I f system has no /bin/sh and you're f orced to use csh/tcsh

[<kuhlmav A T elec.canterbury.ac.nz>] Csh and tcsh execute the .cshrc f irst, THEN if , and only if it isthe login shell (not a sub shell) it executes the .login, which should contain basic important systemsetting like stty commands. Likewise, bash and ksh users are taught to def ine and export PATH in

prof ile, so our per-shell startup f iles w ould not haveclobbered the PA TH set in .procmailrc the way your .cshrc did.

[philip] ...I have been told by other sysadmins that there are systems on which csh was hacked tosource the .login before the cshrc. For various reasons I suspect these to be systems bas ed onolder versions of BSD (say, 2.3 BSD).

As f or tcsh, the o rder in which the .login and .cshrc is sourced is a compile-time option w hich defaultsto the .cshrc (or .tcshrc) before the .login. There may be some wackos out there who change thedef ault in memory of the system(s) that they were raised on. I suggest electroshock as the propertreatment.

...done sys admin on Crays, Conve xes , Suns, S G Is, De cs , PC running BS DI, Linux and Fre e BS D, and I have ne ve r run into a

system whe re the .cshrc is so urce d AFTER the . login. I f someone goes to the trouble to change the order, I would love to know avalid reason f or it.

4.4.2 Procmail won't work well with SHELL set to csh derivate

[1998-08-17 PM-L <kuhlmav A T elec.canterbury.ac.nz> Volker Kuhlmann] ...The blame lies withprocmail and its documentation. Obviously, procmail is programmed with the assumption that thelogin shell is a sh derivative. This assumption is a) not very nice, and b) not sta ted in the otherwisevery good documentation. Of course a user can se t SHELL to tcsh. If then procmail is too stupid tohack it, it ought to say so clearly, and the above-mentioned questions of people using tcsh willdisappear f rom this list. One could also be nice and point out pitf all (3) mentioned above in theprocmail docs. It is customary to have terminal conf iguration in .login. If it is shifted to .cshrc it shouldbe properly surrounded by if .. endif. Perhaps it is not customary to configure the terminal in bashrc

(where e lse then? - only a rhetorical question), but thatis no reaso n to blame it on tcsh.

My .cshrc only setenvs the environment when it is a login she ll (shell level 1). Obvious ly procmail runsa login shell. A s I said earlier, there are good reasons f or setting a f ull PA TH independently whetherthe shell is interactive or not. So, when procmail executes programs with SHELL=tcsh, PA TH is set tothe tcsh def aults. That may or may not be desirable, depending on the individual case. No problemwith that and avoidable (run tcsh w ith -f). Nice if it was in the procmail docs.

But then, the PA TH getting clobbered is not the point here (just a side-effect I didn't realize until 2people po inted it out).

4.5 Check and set PATHIt is very likely that the default PA TH environment variable that your $HOME/.procmailrc sees it notenough. To play safe, so that all the needed binaries can be f ound when escaping to shell in.procmailrc, set the PATH variable as a very f irst statement. A dding paths that don't exist in anothersystem but does exists in the other makes it possible to use the same $HOME/.procmail on multiple



servers (Like HP, SUN, IBM, Linux)

PATH = \$HOME/bin:\/usr/local/gnu/bin:\/usr/contrib/bin:\/usr/local/bin:\/opt/local/bin:\/bin:\/usr/bin:\/usr/lib:\/usr/ucb:\/usr/sbin:\/vol/bin:\/vol/lib:\/vol/local/bin:\${PATH}

4.6 Keep the log on all the timeIt's best that you put these variables at the very start of your .procmailrc . When you start usingprocmail, you also want to know all the time what's happening there and why your recipes didn'two rk as expected. The answer to almost a ll your questions can be f ound in the log file. A s the log f ilewill grow to be quite big, remember to se t up a cron job to keep it moderate size.

LOGFILE = $PMSRC/pm.logLOGABSTRACT = "all"VERBOSE = "on"

4.7 Never add a trailing slash for directoriesDrop the trailing slas h: it'll choke if you ever end up on A pollo's DomainOS where double s lashes arenetwork ref erences. If the directory has a trailing slash, it will choke on most OSes (they treat it like"/.").

DIR = /full/path/to/www/directo ry/ # Wait...FILE = $ARCHIVEDIR/file # Ouch !

4.8 Remember what term DELIVERED meansWhen procmail delivers a piece of mail, whether to a f ile or a pipe-command, if the write succeeds,then the mail is considered to have been delivered, and processing stops with that recipe f ile. Hereis the relevant text from man page:

...There are two kinds o f recipes: delivering and non-delivering recipes. If a delivering recipe is f ound to match, pro cmail considers the mail (you guesse d it) delivere d and will cease processing the rc f ile af ter having succes s f ully exe cuted the actionline o f the re cipe. I f a non-delive ring rec ipe is f ound to match, proces sing of the rcf ile w ill continue a f te r the action line of thisrecipe has been exe cuted.

4.9 Beware putting comment in wrong placesYou like commenting a lot, sticking them everywhere possible? Yes, I do that too, and got intotrouble because one is not that f ree to comment code in procmail. Pay attention to the f ollowingexample



:0 # comment ok* condition # OUCH, ouch. This comment must not be here.# Hm, Old procmail versions don't understand this# Are you sure you want to put comments inside# condition line?* condition{ # comment ok

# comment ok:0 # comment ok/dev/null # comment ok

} # comment ok

So, the place to watch is the condition line. Later procmail versions may understand those, but if youintend to share your recipe, play it safe and think about backward po rtability.

4.10 Brace placementBe caref ul with your braces and remember that old procmail versions aren't as f orgiving as newerversions. Below you see class ical "Test OK condition f irst, and if that f ails then do something else".

See the side comments.

:0* condition

# No space allowed here!{} # Wrong, at least _one_ empty space:0 E{do_something } # Again mistake, must have surrounding spaces

4.11 Local lockfile usageLock f iles are only needed when procmail is doing something that should be serialized, i.e., whenonly one process at a time should be do ing it.

This generally means that any time you write to a file, you should have a local lock, pref erably basedon the name of the file being w ritten to. Forwarding actions ('!'), and 99% of all f ilters don't need lockf iles. However, if a f ilter action writes to a file while filtering, then you may need a lock. Procmailalways do es kernel locking when it writes mail to files via simple f ile actions. So even if you f orgot thelock colon, procmail tries to play saf e if kernel locking has been compiled in.

Beware misplacing the lock colon(:)

:0: a # Ouch! Wrong unless you want a lock file named a:0 a: # Okay.

Note that in delivering recipes where you manually write the content, you must use local lock file with> token, because procmail can't determine lock by itself. It can only determine the lock file f rom the >>token. However, putting a lock f ile on a recipe like this is, of course, utterly useless. So you might aswe ll omit the locking entirely.

# Save last body of message to file mail.body

:0 b: mail.body$LOCKEXT| cat > mail.body



* ? $FORMAIL -D $MID_CACHE_LEN $MID_CACHE_FILE{

LOG = "dupecheck: discarded $MESSAGEID from $FROM $NL"

:0 # no lockfile !$DUPLICATE_MBOX

}

LOCKFILE # kill variable

You cannot use local lockf ile a s below:

:0 : $MID_CACHE_FILE$LOCKEXT* ^Message-ID:* ? $FORMAIL -D $MID_CACHE_LEN $MID_CACHE_FILE

because the local lock file named on the flag line will be created only if the conditions have matchedand the action is a ttempted.

One more note: watch carefully, that there is no : lock when delivering to DUPLICATE_MBOX becausethe outer global lock file already prevents all other procmail instances f rom executing this part of therecipe.

4.13 Gee, where do I put all those ! * $ ??Ahem. I can't tell you exactly what to do or how to write your own procmail recipes, but I can showyou an example. Here is one poss ible s tyle for condition line token o rder:

*$ ! ? BH VAR ?? test

That won't sa y much unless you see something to compare with. Here is one perf ectly valid rule, notf ollowing the above s tyle:

:0*$ ^Subject:.*$VAR*! ^From:.*some*B ! ?? match-the-string-in-body

*$? $IS_EXIST $FILE*VARIABLE ?? set

It might be better to line up things in multiline condition lines. The f irst place is reserved f or dollarsign, the second f or not operator and so on. T he key here is to be able to comprehend the recipeeasily. E.g. with the dollar put to the lef t, one can tell at a glance if variable expansion is happeningon the who le line. The same with small f ormatting changes :

:0

*$ ^Subject:.*$VAR* ! ^From:.*some* ! B ?? match-the-string-in-body*$ ? $IS_EXIST $FILE* VARIABLE ?? set

| | |

| | || | What is matched: (H)eader portion, (B)ody or (HB) both.| | The (??) associative operator is required.| || Not operator (!) or shell call (?)|Variable expansion (important)

4.14 If you Send an automatic reply, use X-loop headerDo not send automatic reply without checking "! ^FROM_DA EMON" condition and always include X-Loop heade r and check its existence to prevent mail loops

:0* conditions-for-auto-reply*$ ! ^$MYXLOOP* ! ^FROM_DAEMON| $FORMAIL -A "$MYXLOOP" ...other-headers...

4.15 Avoid extra shell layer and check command forSHELLMETAS[dan] It is very important to study your shell command calls and try to save the overload of the extralayer of shell. It may be extra w ork once when you w rite your rcfile but it saves e f f ort on each pieceof arriving mail. When procmail sees a character f rom SHELLMETAS, it runs

# Default SHELLMETAS: &|<>~;?*[

# Default $SHELLFLAGS: -c

% $SHELL $SHELLFLAGS "command -opts args"

instead o f

% command -opts args

That is because procmail's ability to invoke other programs does not include f ilename globbing ([, *,?), backgrounding (&), piping (|), succession (;), nor conditional succession (&&, ||). If it sees any of those characters (bef ore expanding variables), it hands the job over to a shell.

Sometimes those characters appear in arguments to a command without having their shell metameaning and procmail really could invoke the command directly without the shell. You can see thedistinction in a verbose log file: if procmail runs the command itself , it logs

Executing "command,-opts,args"

with a comma betw een each positional parameter, but if it calls a shell, the original spacing from thercf ile appears unchanged in the logfile:

Executing "command -opts args"



Hm. Can w e draw some conclusion? Not anything def initive, but at least something:

While sed(1) and grep(1) may be bigger than awk(1) in some systems, this is an exception.They are usua lly much smaller. It's more e ffective to use one awk process instead o f manycombined filtering commands .Complex commands that w ould require many processes to be chained together, like ̀ grep -v |grep | sed' could be usually accomplished with one awk(1) call. Ask somewhere how to do itwith awk(1) if you don't know the language, it's quite alike perl(1)Try to use s tandard awk(1) . gawk(1) and nawk(1) are bigger and may not be found o n allsystems.Avoid perl(1) at a ll costs; it's many times (6) bigger than awk(1) . Perl is s low-to start up, dueto intermediate compilation process at sta rtup and hogs o odles o f memory.Remember that if procmail is running in a dedicated mail host, it probably doesn't even haveany goodies installed, just the boring standard versions; which may not be even the same aswhat you see on current host.

Here are some more programs. Don't even think of extracting fields with grep or awk , like "grepSubject", because formail is much smaller and more optimized for tasks like that. Better yet, manytimes you can do a ll with procmail's regexp matches .

37007 Sep 5 15:53 /usr/local/bin/formail # 3.11pre728672 Jun 10 1996 /usr/bin/tr20480 Jun 10 1996 /usr/bin/tail20480 Jun 10 1996 /usr/bin/cat20480 Sep 26 1996 /usr/bin/expr16384 Jun 10 1996 /usr/bin/head16384 Jun 10 1996 /usr/bin/cut16384 Jun 10 1996 /usr/bin/date16384 Jun 10 1996 /usr/bin/uniq16384 Jun 10 1996 /usr/bin/wc12288 Jun 10 1996 /usr/bin/echo

4.17 Using absolute paths when calling a shell programShell programmers know that if absolute pa th is used for calling the executable, shell doesn't haveto search through long list of directories in $PATH. This may speed up shell scripts remarkably. Thebest w ay to use such an optimization is to define variables to those programs.

Should you use such optimization in your procmail code? That is a two folded question. Examine howmany shell calls do you use? Do you use grep or formail a lot? Then you could optimize these calls.To be portable, define variables for executables:

# perhaps defined in separate INCLUDERC## INCLUDERC = $PMSRC/pm-mydefaults.rc

FORMAIL = /usr/local/bin/formailGREP = /bin/grepDATE = /bin/date

:0 fhw| $FORMAIL -r

When you port your .procmailrc to different environment which has different paths, you could usethis recipe in addition to one just mentioned above:



FORMAIL = ...as above

:0* HOST ?? second-host{

# In this host the paths are different. Reset.

$FORMAIL = "formail"$GREP = "grep"$DATE = "date"

}

4.18 Disabling a recipe temporarilyIf you have a recipe that you would like to disable for a while, there is an easy way. Just add the"f alse" condition line bef ore any other conditions. The "!" also nicely visually flags that "this recipe isNOT used".

# This recipe stops at "!" and doesn't get past it.

:0* !* condition* condition{

...}

4.19 Keep message backup, no matter whatIt's good to have a saf ety measure in your .procmailrc . A lthough you are an expert and havechecked your recipes 10 times, there is still a chance that something breaks. One morning, when youbrowse your BIFF reminder log; you notice "Hm, there is that interesting message but it was notf iled, where is it?". A nd when you go to study the procmail logs (you do keep the log going all thetime) and it hits you: "Gosh; a mistake in my script! Message was f ed to malicious pipe and I hadthat i f lag there... snif f ". A nd you greatly regret you didn't back up the message in the f irst place.

So, bef ore your procmail does anything to your message, put the message into some f older which isregularly expired. Emacs Gnus can do mailbox's expiring, but one could also use a cron(1) to do thecleaning. A f ter that, you can relax know ing your mail is saf e.

# Your incoming messages are stored here, filtered by procmail

SPOOL = $HOME/Mail/spool

# Backup storage## - This could be directory too. In that case you could use# cron job to expire old messages at regular intervals# - For once a day expiration, see procmail module list# and pm-jacron.rc

BUP_SPOOL = $SPOOL/junk.bup.spool

:0 c:$BUP_SPOOL

w i t h " M y a p o l o g i e s , t h e s c r i p t h a d a n e r r o r , i t w o n ' t h a p p e n a g i n " t o a l l t h e v a l i d a n g r y m a i l st h a t a r e n o w a d d r e s s e d t o y o u .

Drop the UBE to a f older, manually select the messages that need actions and send message topostmasters in the Received chain explaining that their mail relay has been hijacked.

5.0 Procmail flags5.1 The order of the flagsThe order of the f lags does not matter in practice, but here is one s tylistic suggestion. The idea hereis that the most important flags are put to the left, like giving priority 1 f or aAeE , which affect therecipe immediate ly. Priority 2 is given to f lag f , which tells if a recipe filters something. A lso (h)eaderand (b)ody should immediately f ollow f , this is considered priority 3. In the middle there are otherf lags, and last flag is c , which ends the recipe, or allows it to continue. In addition according to[david] : "...I'm quite sure that putting anything othe r than the opening colon and the number to thelef t of AaEe will cause an error."

:0 aAeE HBD fhb wWir c: LOCKFILE| | | | || | | | (c)ontinue or (c)lone flag last.| | | (w)ait and other flags| | (f)ilter flag and to filter what: (h)ead or (b)ody| (H)eader and (B)ody match, possibly case sensitive (D)| Note: Procmail 3.22 bug| < http://mailman.rwth-aachen.de/pipermail/procmail/2002-February/008355.html >The `process' flags first. (A)nd or (E)lse recipe

You can write the f lags side by side

:0Afhw:$MYLOCK$LOCKEXT

Or, as suggested, leave f lags in their own slot f or more distinctive separation. Note that procmailvariable $LOCKEXT must be next to $MYLOCK, because it conta ins string ".lock".

:0 A fhw: $MYLOCK$LOCKEXT

5.2 Flags HB at top of recipe (warning)[Philip] Version 3.22 has a bug that keeps the 'H' flag f rom being cleared, such that once you use it,it never gets cleared. Using the 'H' flag will theref ore cause p roblems with latter recipes that use justthe 'B' but not the 'H' f lag. Either way, the only time you should use the 'H' flag is on recipes thatneeds to match against both the header and the body. If you want a recipe to match only againstthe body and you're using 3.22, use the "B ??" modif ier on the conditions. See Procmail-L messagetitled "Canno t get recipies to work properly" . So to be most pportable possible, convert all previouslyused condition lines from:

:0 B* body-check-here



to use this format:

:0* B ?? body-check-here

5.3 Flag w and recipe with pipe(|)[alan] If the f ilter program exits with a 0 status (0 == okay), then procmail will replace the originalinput body with the output of the f ilter program. If the f ilter program exits with anything but zero,procmail will report an "error" to the log, and " recover" the input (not f ilter it)

[david] I am very sure that that's the case only if you have the w or W f lag on the f iltering recipe.Without w or W, procmail won't care about a bad exit status from the filter and will replace the filteredportion with whatever standard output the filter produced. It may still report an error to the log butit won't recover the previous text. This, for example, will destroy the body of a message, evenwithout i :

:0 fb| false

With this, how ever, procmail will recover the o riginal body:

:0 fbW # same results even if we add `i'| false

[stephen] No, not on all occasions. Procmail will not care about the exit code here. However, if procmail detects a write error, it will recover (because of the missing i f lag). Procmail will only detecta write error in such a case if the mail is long enough and does not f it in the pipe buf fer that's in thekerne l (typically 10KB).

5.4 Flag w, lock file and recipe with pipe(|)[manual] In order to make sure the lock f ile is no t removed until the pipe has f inished, you have tospecify option w otherwise the lock file wo uld be removed as soo n as the pipe has accepted the mail.So if you see anything that looks like ">" or ">" in your recipe, then that should immediately ringyour bells. immediate ly check that you have included the w flag and the lock f ile : .

:0 hwc: headc$LOCKEXT* !^FROM_MAILER| uncompress headc.Z; cat >> headc; compress headc

5.5 Flag f and w togetherT he w tells Pro cmail to hang around and wait f or the script to f inish. Hm, Wouldn't you think this ough t to be implied by the f f lagalready?

[david] Of course the f f lag is enough to make procmail wait f or the f ilter to f inish, but the w meanssomething more: to wait to learn the exit code of the f iltering command. If sed f ails with a syntaxerror and gives no output, without Wor w procmail would happily accept the null output as the resultsof the filter and go on reading recipes for the now body-less message. On the other hand, with Worw sed will respond to a non-zero exit code by recovering the unf iltered text.



5.6 Flags h and b[david] hb is the def ault; you need to use h only when you don't want b or vice versa. You can thinkof it this way: h means "lose the body" and b means "lose the header," but the two together canceleach other out.

[philip] hb (feeding w hole message) is the default f or actions. You need to specif y h without b if youwant the action applied only to the head. H is the def ault for conditions. You need to specif y HB or BHif you want to test a condition against the entire message.

5.7 Flag h and sinking to /dev/nullWhen you drop something to /dev/null, use the h flag so that procmail does not unnecessarily try tof eed w hole message there.

:0 h* condition/dev/null

[philip] Procmail knows that it shouldn't create a local lock on /dev/null and that it shouldn't kernellock /dev/null, and it knows to write it "raw" (no "From " escaping or appended newline). This meansthat procmail simply opens /dev/null, does its write with one system call, and closes it. I'm not sure if adding the h f lag makes a real dif ference on modern UNIX kernels. I suppose it depends on howoptimized the write() data is and in particular, whether a user-space to kernel-space copy isrequired, or whether it's delayed. If it's delayed then the code for handling /dev/null wouldpresumably not do it, and the size of the write wouldn't actually matter.

5.8 Flag i and pipe flag f Flag i is usele ss in mailbox deliveries.

[FAQ] The f ollowing will work some of the time, when the message is short enough, but that's acoincidence. With a longer message, though, Unix starts paying attention to what is happening,because it will have to buffer some of the data, and then when the buf f ered data is never read, anerror occurs. The error is passed back to Procmail, and Procmail tries to be nice and give you backyour original message as it was bef ore this malicious program truncated it. Never mind that in thiscase you wanted to truncate the data. A nyway, the fix is easy: Just add an :i flag to the recipe (:0fbwi instead o f :0fbw ) to make P rocmail ignore the error.

:0 fbw* condition

| malicious-pipe

[dan] here's why the i f lag is needed (courtesy of Stephan): Y ou told procmail to f ilter the entire mail(header and body), so it does and it attempts to write out header and body to the f ilter. Thenprocmail notices that not the entire body is being consumed. Procmail, being rather paranoid when itcomes to de livery of mail assumes something went wrong and considers this a f ailure of the f ilter.

:0 fbwi| head -2

5.9 Flag r[philip] Procmail auto matically turns on the r (raw mode) flag f or deliveries to /dev/null, so there's no



need to do it yourself .

:0 r # you can leave out the `r'* condition/dev/null

[david] You can use the r f lag (f or raw mode) on every recipe where you do not want a From_ lineadded. I'm assuming that there isn't one already there; the r flag keeps procmail f rom making surethat there are a From_ line at the top and a blank line at the bottom, but it will not make procmailremove them if they are already present. A lso, be careful to use the -f option on all calls to f ormailso that f ormail won't add a From_ line.

Someone who didn't need From_ lines – I forget w ho – found it annoying to put r onto every recipeand altered the source to prevent procmail f rom adding From_ lines a t a ll, ever. I think a better ideawo uld be a procmailrc Boolean to enable or disable them f or all recipes witho ut aff ecting other users.(Then perhaps we 'd need a reverse r flag to undo raw mode f or one recipe a t a time?)

5.10 Flag c's background...Intere sting. My vision o f c is to think of CONT INUE with mess age pro ce ss ing af terwards eve n i f conditions matched.

[david] Precisely: when you have braces, thinking "continue" instead of "copy" or "clone" can get youinto trouble.

Early versions of procmail, bef ore braces and bef ore cloning, called the c f lag "continue" in theirdocumentation; I think it is s till called that in the source.

When Stephen introduced braces (but not cloning at this point), it was of course implicit that anaction line of "{" was non-delivering, and a c was extraneous. People put c's there because theywanted procmail to continue to the recipes inside the braces on a match, and procmail brushed it of f with an "extraneous c-flag" w arning. No harm done.

When Stephen introduced cloning, though, I was rather upset that he was giving double duty to cinstead o f introducing something new like C f or it, especially because people who abso lutely wantedno clone but intended the recipes inside the braces to run in the same invocation of procmail aseverything else were mistakenly putting c's on their braces to make sure procmail would "continue".Peo ple would (and did) get double deliveries.

Roman Czyborra, though, said that if you consider c to stand f or "copy", that covers both uses of c :provide a copy to a simple recipe or, if there are braces, to a clone procmail that will handle therecipes inside the braces. Stephen agreed and changed the documentation accordingly.

Longtime users of procmail and people who read old docs may still think of it as "continue", but since

the introduction o f clones , that is no t a goo d way to look at it. "Copy" is much saf er.

5.11 Flag c before nested block forks a child[alan] The combination of a nes ted block and the c flag causes procmail to f ork a child process f or thenested block, while the parent skips over it and continues on. The child process doesn't necessarilystop unless a delivering recipe (without the c flag) action succeeds.

[david] When Stephen van den Berg added the use of c on a recipe that launces a braced nestingblock to fork a clone procmail, I objected that it should have a dif ferent f lag, such as C, becausepeople were always putting c on recipes that open braces because they thought it was necessary tomake procmail continue into the braced area. Until then, it had been a harmless error f or anextraneous f lag. Roman Czyborra came up with the idea of changing the meaning of c f rom"continue" to "copy": read c as "send a c opy to the action" and then it would cover both simplerecipes with c flags and cloning blocks.

5.12 Flag c and understanding possible forking penalty



... I run she ll commands that nee d not to be se rialized, so instead of doing the standard way:

:0 hic # nbr.1 / standard way| command

I as sume I can avo id the e xtra f ork caused by (c)lone f lag altoge ther by using thes e. Any di f f ere nce between these two?

:0 # nbr.2 / alternative* ? command{ } # ...No-op, Procmail syntax requires this

dummy = `command` # nbr.3 / alternative

[philip] There is a misunderstanding here. Let me clarif y:

P r o c m a i l o n l y f o r k s a f u l l - b l o w n c l o n e o n a r e c i p e w i t h t h e ' c ' f l a g w h o s e a c t i o n i s a n e s t e d b l o c k .

If it's a simple mailbox deliver, pipe, or forward action then procmail does not f ork a 'clone ' (for pipeand f orward actions procmail does have to fork, but only so it can execute the action). nbr.1 andnbr.2 take the same number of f orks to execute. They also take the same ef f ective number of writes(in case you 're concerned about tha t). The latter also requires tha t procmail wait f or the command tof inish. nbr.3 is worse than the above two, as procmail has to not only wait f or the command tocomplete but also save the output into the named variable.

5.13 Flags before nested blockGiven the f ollowing recipe, let's examine the f lag part

:0 $FLAGS{

do-something}

[david] HB AaEe and D af f ect the conditions and thus are meaningf ul when the action is to open abrace. HB and D would be meaningless, of course, on any unconditional recipe, but they should notcause error messages . Generally, f lags that aff ect actions are invalid there, and bhfi and r alwaysare, but the others are partial exceptions: if you are using c to launch a clone, then w Wand a locallock file can be meaningf ul. If there is no c , then w Wand a local lock file are invalid at the opening of a braced block.

5.14 Flags aAeE tutorial[david] AaEe are mutually exclusive and no more than one should ever appear on a single recipe.[philip] Actually, this is no t true. e does no t wo rk with E or a (and procmail gives a warning if you try),and A is redundant if a is given, but at least so me of the other combination make sense and wo rk.

A = try this recipe if the conditions succeeded on the most recent recipe at that nesting levelthat did not itself have an A nor an a

a = same as A, but moreover the action must have succeeded on the most recently tried recipeat that nesting levele = Almost like A, try this recipe if the conditions matched but the action failed on the mostrecently tried (not skipped) recipe at this nesting level. universe, e is the opposite of a . e onlylooks backwards past E recipes tha t were skipped because o f their E. It doesn't care whethera previous recipe had a n A or a flag.



E = try this recipe if the conditions have failed on the most recent recipe at that nesting levelthat did not have an E and on since then every recipe a t that level that did have an E;essentially opposite of A

These mnemonics might he lp:

A: if you did the recipe at the s tart of the chain, try this one (A)lsoa: if the last action at that nesting level was (a)ccomplished)e: if the last action at that nesting level (e)rred

E: (E)lse because the conditions down the chain so far have not matched. Or "try this recipeunless the las t tried recipe matched".

# [philip] demonstrates `e'

:0 : # match, but action fails/etc/hosts/foo

:0 A # no match* -1^0/dev/null

:0 e # this is skipped because the last tried recipe didn't match{

...whatever}

How they interact with one another when used consecutively has not been fully tested to myknowledge. Consider this:

:0* conditionsnon-delivering-action1

:0 aaction2

:0 eaction3

Is action3 done if action2 failed or if action1 failed (or perhaps in both situations)? [philip] Action 3 isonly done if action2 failed.

If the answ er is action2, does this work to get action3 do ne if action1 failed? I think it does, but doesit also run action3 if the conditions didn't match on the first recipe? [philip] Yes, and yes.

:0 # [david]* conditionsnon-delivering action1

:0aaction2

:0Eaction3



[philip] If that's not w hat you w ant, combine so me f lags:

:0* conditionsnon-delivering action1

:0 Aeaction3

:0 aaction2

If the conditions match, action1 will be executed. action3 will then execute if action1 f ailed, otherwiseaction2 w ill be e xecuted [if action1 succeeded] .

[david] I know wha t this structure does because I use it:


:0Aaction2

:0Enon-delivering action3

:0Aaction 4

If the conditions match, action1 and action2 are perf ormed and action4 is not (of course action3 isnot either), even if action2 is non-delivering; if they f ail, action3 and action4 are performed. The A onthe f ourth recipe ref ers back to the third and no f arther. But I don't know about this:


:0A* more conditions

action2

:0Enon-delivering action3

:0Aaction 4

Now, suppose the conditions on the f irst recipe match but those on the second recipe do not match.Wo uld the third recipe (and thus the f ourth one) be a ttempted? I w ould expect so. [philip] Yes. Thelast tried recipe didn't match, therefore the E f lag will be triggered.

If that isn't what you w ant, you can prevent it this way:

:0* conditions



{:0non-delivering-action1

:0* more-conditionsaction2

}

:0 E # ignores mismatch inside braces, looks only at same levelnon-delivering action3

:0 Aaction4

If that is what you want, you can be positive this way:

# if action2 is non-delivering or vulnerable to error that

# would cause fall-through

DID2 # Kill variable

:0* conditionsnon-delivering-action1

:0 Aaction3

:0* ! DID2 ?? (.)non-delivering-action3

:0 Aaction4

# if action2 is delivering and sure to succeed:0* conditionsnon-delivering-action1

:0 A* more-conditionsaction2

:0non-delivering-action3

:0 Aaction4

[philip] or those who a re interested, I'll note that there a re only 3 combinations of the a , A, e , and Ef lags that aren't either illegal or redundant. They are Ae, aE , and AE. I've shown a use f or Ae upabove. Here's an example o f AE:

:0

* condition1non-delivering action1

:0 A* condition2non-delivering action2

:0 AEaction3

action3 will only be executed if condition1 matched but condition2 didn't match. Without the A f lag,action3 would be executed if either of them failed. This can also be done with a instead of A withanalogous results.

Procmail's "f low-control" f lags may not be particularly easy to describe in straight terms (and this canall be made more complicated by throwing in a more varied mix of delivering vs non-deliveringrecipes), but I've f ound that it usually does what I expect it to do, and when it doesn't or I'm indoubt or I want to be pa rticularly clear, I can always f all-back to doing it explicitly via nesting blocks.Pick your poison...

6.0 Matching and regexps (regularexpressions)6.1 Philosophy of abstraction in regexps

Here are two ways to vie w or w rite re ge x ps . Make up your ow n mind. More on re gular e xpre ss ions at < http://regexlib.com/ >.

People who are in favor of writing pure native regexps in the recipes:

[ ]<[ ]*("([^"\]|\\.)*"|[-!# -'*+/-9=?A-Z^-~]+)... # "

They think:

I'm not planning on "maintaining" tha t code, as the syntax for XXX will not ever change, it'sRFC or something.I somehow doubt that a nyone e lse will change that regexp more than triviallyIf none of your othe r regexps use the categorical variables, and you're not changing theregexp, then what's the point? The variablized version will be slow er, and will clutter theenvironment with subprocesses.

Where someone that immediately wants to abstract things says (this is from philip's greatMessage-Id matching recipe)

dq = '"' # (literal) double-quotebw = "\\" # (literal) backwhackatom = "[-!#-'*+/-9=?A-Z^-~]+"word = "($atom|$dq([^$dq\]|$bw.)*$dq)'local_part = "$word($s\.$s$word)*"

$s<$s$local_part... # ignore comment here

...abstraction: It makes code clearer w hen you break it to manage able parts, which possibly surfacesreusable parts. It also makes thing look simpler, and enables even novices to understand what'sgoing on there. After we're not connected to the net anymore, others could possibly understand ittoo. So, naturally we can't agree with any of the previously mentioned arguments presented forkeeping regexp "in pure native format".



Although you won't maintain it, it's an example for others. What you post first, people will saveit to their mailboxes and circulate elsewhere in the net: "Hey, I've saved this, try it"You can write cryptic regexps or break them into pa rts where the who le looks much s impler.Consider novice's welfare :-) This has nothing to do with the "It never changes in my lifetime".The speed penalty imposed by additional variables is not so mething we can measure inpractice. CPU wo n't even hiccup. An extra formail call in your recipes is 10x as expensive as100 variables. (I don't know how to measure tha t, but launching a shell and creating aprocess is a much more expensive task).Cluttering the env process? C'm on. That won't matter either. No o utside process uselowercase environment variable names, or then it must be real special program. So called"cluttering" of environment space is a lso no-issue. CPU w on't even ge t a hiccup for that.

6.2 Matches are not case-sensitiveOkay, okay; if you read the manual you knew that already. But sometimes someone with years of experience w ith Unix may take it for granted that procmail would be case -sensitive as the rest of theUnix tools a re. Use the D flag to turn on case-sensitivity.

6.3 Procmail uses multi line matchesProcmail uses multi line matches by default. This means that ^ and $ match a newline, even in the

middle of a regexp. Now you know this, you can easily interpret e.g. $^> as: `a newline followed bya line not starting with a >.

If you put a $ after the \/ match token then procmail will include the matched newline if there's onethere. Solution? Don't put a dollar sign there unless you really want a newline, use period thatmatches all but newline:

:0* B ?? ^Search-string: \/.+

6.4 Headers are unfolded before matchingIf you have a heade r that continues on separate lines, you don't have to wo rry abo ut the line feeds.Procmail silently unfolds the header onto one line, before matching it

Received: from unknown (HELO Desktop01) (208.11.179.72) bypalm.bythehand.net with SMTP; 4 Dec 1997 23:29:09 -0000

:0 # note, match on continuation line

* ^Received:.*bythehand\.{

# Do something}

6.5 Improving Space-Tab syndromeProcmail doesn't know abo ut standard escape codes like \t and \n or [\0x00-\0x133]:

# Not what you think # You have to write: space + tab[ \t] [ ]

But using the space+tab is not very readable and it's a very error prone construct. Here is asuggestion to use variables to improve the readability:



WSPC = " " # whitespace = space + tabSPC = "[$WSPC]" # regexp whitespace, the short name

# SPC was chosen because you use this# a lot in condition lines.

NSPC = "[^$WSPC]" # negation of whitespace

:0

*$ var ?? $NSPC{

# match anything except space and tab}

:0*$ ! var ?? ($SPC|$){

# match anything ecxept space and tab and newline}

But you cannot use ne wline inside brackets.

WSPCL = " "' # Whitespace with line feed'

# Won't work although WSPCL definition is correct.

*$ var ?? [$WSPCL]

Instead use variable syntax:

SPCNL = "($SPC|$)" # space + tab + newline

If you absolutely need a range of characters, see if you have echo command in your syste m to def inevariables like this:

NUL_CHAR = `echo \\00`

DEL_CHAR = `echo \\0177`REGEXP_NON_7BIT = "[^$NUL_CHAR-$DEL_CHAR]"

6.6 Handling exclamation character[philip] you do need the first backslash, to keep procmail from considering the backslash as arequest to invert the sense o f the match. For example, these two conditions are equivalent:

* ! 200^1 foo

* 200^1 ! foo

Theref ore, a leading '!' must either be backslashed, enclosed in either pa rens or brackets (I suspectthat parens would be more eff icient), or prefaced with an empty pair of parens. I would recommendwriting the condition with one o f these:



* 200^1 \!!!!* 200^1 ()!!!!* 200^1 (!!!!)

6.7 Rules for generating a character class

In a "character class" (things betw een "[" and "]"), metacharacters don't need to be escaped. W ell,a backslash is an exception. e.g. [$ ^\\ would match any one of the literal characters dollar, openingbracket, caret, and backslash.

To match "])" use [])]

To match "[(" use [ )

To include a literal ^ must not be firstTo include a literal - must be first, last or \-To include a literal \ you must use \\ To include a literal ] must be firstTo include a literal [ ( ) or $ just use it anyw here

[elijah] If you are inverting a character class "first" means just after the(^). So the character classthat contains everything but ] ^ and - must look like this:

[^]^-]

[david] What if I want literal $ inside bracket? A $ inside brackets, unless it begins a variable nameand the "$" modifier is on, a lways means a literal dollar sign. It cannot mean a newline if it appearsinside brackets. A good way to keep it exempt from "$" interpretation is to put it last inside thebrackets (unless one also need to include a literal hyphen and one can't put the hyphen first; thenyou'll need to escape the dollar sign with a backslash and put the hyphen last – well, you couldalternatively escape the hyphen, I guess), because procmail knows that "$]" cannot possibly be areference to a variable.

General guideline:

($) always matches a newline, with or w ithout "$" interpretation;[$] always matches a do llar sign, with or w /o "$" interpretation;

6.8 Matching space at the end of condition[david] If you need to have tab o r space at the end of condition line you can use these:

* rest of string .** rest of string[ ]* (rest of string )* rest of string ()* rest of string( ) # This may be the best

[philip] From my looking at the source, the last two should be equal in efficiency, and except for atrace difference in regcomp time, should match at the same speed as a solitary trailing blank. Thecharacter class version [ ] will be slower. Of course , I suspect that neither you nor your sysadmin willever notice the difference in speed, and given that 99% of all systems are I/O bound and not CPUbound, the system is incredibly unlikely to notice either. I can't complain though, as I also go tovarious extremes to seek out every last bit of poss ible performance. Ah well. The first one would beslower yet, though perhaps no slow er than the bracket form.

6.9 Beware leading backslash I am trying to come up with a proc mail re cipe that among othe r things should have the condition 'body doe s not contain a particu lar word '. Here is what I tried :

* ! B ?? \<word\>

[david] You have f allen into the leading backslash problem, If the f irst character of a regexp is abackslash, procmail takes it as "end of leading whitespace" and s trips it. What you coded means "aless-than sign, then the word, then any non-word character." (It also prevents the less-than signf rom being taken as a size operator.) Unless the non-word character immediately to the lef t of thewo rd was a less-than sign, that regexp would f ail (and thus the condition would pass). Try this:

* ! B ?? ()\<word\>

This would work too:

* ! B ?? \\<word\>

but in a casual reading it would look like "literal backslash, less-than sign, the wo rd, word boundarycharacter," so we on the list generally recommend the empty parentheses.

Do note that the dif f erence in meaning of \< and \> in procmail (where they must match a non-wordcharacter) f rom their meaning in perl and egrep (where they match the zero-width transition into andout of a word respectively) does not come into play here. Because procmail's \< and \> can matchnewlines (both real and putative), it rarely is a f actor. It's a problem only when a single character

has to serve both as the ending boundary of one word an also the opening boundary of another.Well, it's also a problem when you have o ne as the last character to the right of \/, but that's easilysolved.

6.10 Correct use of TO MacroTO is not a normal regular expression; it is a special procmail expression that is des igned tocatch any des tination specification. For deta ils, see the miscellaneous section of theprocmailrc(5) man pages .Prefer TO_ instead o f TO if you have new procmail. TO_ is better because TO used to be tooloosePlease remember to w rite ^TO, with the anchor in it.Do not put a space betw een the caret ( )̂ and the word TO in ^TO.Do not put a space between the ^TO and the text that you a re matching on; it must be^TOtext If this bo thers you, you can use TO()text instead to get better separation of text.Both letters in TO must be capitalized.

6.11 Procmail's regexp engine[philip] procmail's regexp engine has no s pecial optimization for anchoring aga inst the beginning of the line. Most program that have such an optimization ha ve it because they ne ed the line distinctionfor other reasons (for example, grep by default prints the entire line containing a match). Procmailhas no such other reason, so it treats newline like any other plain character in the regexp. There

should be no speed difference as long as procmail can sa y: "the first character I see must be a 'foo'".Note that case insensitivity is handled by making everything lowe rcase , so a letter being first doesn'tbring in the spectre of character-classes or anything like that.

> re cipe may have just changed the size o f the he ad, procmail > cannot ke ep a byte-count pointer nor a line -count pointe r to> where the body be gins but must scan through the he ad to find the> blank line at the neck be fore it begins a body se arch.

Procmail does this when it reads in the head, not when it goes to sea rch the body, so that cost can'tbe avoided. Let me repeat; that searching the body is no slower than searching the header, if wef orget the minimum impact of the size of these two.

6.12 Procmail and egrep differences[By david]

^ and $ are non-zero-width and anchor to real or putative newlines (rather than to the zero-

width start and end o f a line);An initial ^^ or a final ^^ anchors to the opening or closing putative newline respectively;^ and $ in the middle of a procmailrc regexp match to an embedded newline (and must beescaped to match to a caret or a do llar sign);\< and \> are no n-zero-width and match to a character that w ouldn't be in a w ord (or to a realor putative new line) [rather than to the zero-width transition into o r out of a word]; it alwaysmatches one non-word character. It will fail when there is no w hitespace after the colon. Thisis rather patho logical but still perfectly compliant with RFC822. For this reason, you should use(.*\<)? instead of just .*\< after the colon that terminates a header field name:

^Subject:.*\<humor\> # Wrong^Subject:(.*\<)?humor\> # Right, notice ?

*, ?, and + in the absence of \/ are stingy rather than greedy, and tha t generally won't matter,but in the presence of \/ they are stingy to the left of \/ and greedy to the right of \/, while inmost applications the leftmost w ildcard on a line is the greedies t and greed decreases fromleft to right.

6.13 Understanding procmail's minimal matching (stingy vs.greedy)

...I w ant to have a procmail rec ipe that will save ce rtain mail to folders w he re the folde r name (always a number) is s pec ified inthe subjec t.

:0 :* ^Subject: *\/[0-9]*$HOME/Mail/$MATCH

[philip] ...and this won't quite work. For a subject w ith a space after the tab, the '*' on the left handside w ill be matched minimally (zero times), and then the stuff on the right hand side will be matchedmaximally, but starting at the space still, which will match nothing. This is a case were procmail's

minimal matching can cause massive confusion and frustration. The solution is usua lly the following:

FORCE THE RIGHT HAND SIDE TO MATCH AT LEAST ONE CHARACTER

By Changing the recipe to :

:0 :* ^Subject: *\/[0-9]+$HOME/folders/$MATCH

it'll work, because then the left hand side will have to match a ll the way up to the first digit (but notthe digit itself). If you follow the rule in caps then you'll almost always be able to ignore procmail'swe irdness in this area.

[david] And examine how procmail matches "Subject: Keywords 9999"

* ^Subject:.*Keywords.*\/[0-9]*

procmail: Match on "^Subject:.*Keywords.*\/[0-9]*"procmail: Matched ""

The right side was as greedy as it could be; the problem is that we seem to expect greed on the lef tas well. MA TCH is set to null, in contrary to our expectation. It is not a bug but rather a frequentlymisunderstood e f fect of the way extraction is advertised to operate.

Reme mbe r that only the right side is gre edy; the le f t side is stingy, and lef t-side stinginess takes precedence over right-sidegreed.

Extraction is implemented this way: the entire expression, lef t and right, is pinned to the shortestpossible match; then the division mark is placed and the right side is repinned to the longestpossible match starting at the division. The tricky part is to remember that the division is markedduring the s tingy stage.

If the expression is

^Subject:.*Keywords.*\/[0-9]*

and the text is

<newline>Subject:<space>Keywords<space>9999<newline>

then the shortes t poss ible match to the entirety is

<newline>Subject:<space>Keywords

because ".*" and "[0-9]*" both match to null. Then the division mark is placed on the space af ter"Keywords" and procmail looks for the longest possible match to [0-9]* starting with that space.That, again, is null, so MA TCH is set to null.

We see that it works as expected if regexp is changed to this:

^Subject:.*Keywords.*\/[0-9]+

That is a who le other ball of wax. Now the sho rtest match to the entirety is

<newline>Subject:<space>Keywords<space>9

and the division mark is placed at the 9. Then procmail refigures the longest match to the right sidestarting at the division mark and sets MA TCH=9999. However here

^Subject:.*Keywords\/.*[0-9]*

the second ".*" wo uld have reached not just up to the digits but through them to the end of the line.MATCH would contain the rest of all of it matched to ".*" plus null match "[0-9]*".

[f or curious reader]

Given line

Subject: Keywords 9999

the second, which diff ers only by inserting the extraction marker, would not match and w ould not se t$MATCH:

^Subject: Keywords *9999 # matches ok^Subject: Keywords *\/9999 # won't !

because the lef t side would be matched to "<newline>Subject: Keywords" and the immediatelyf ollowing text, " 9999", did not match the right side. It would actually make the condition f ail andkeep the recipe f rom executing. It took a lot of circuitous coding to allow f or not know ing in advanceexactly how many spaces there w ould be bef ore the digits.

C all it counte rintuitive , but it's not a bug. General advice : always mak e s ure that the right side cannot match null or that the last ele ment of the le f t side cannot match null. Or in other wo rds: f orce the right-hand side of the \/ to match at least one character.

6.14 Explaining \/ and ()\/M ATCH strips all leading blank lines in 3.11pre7

[david] \/ with nothing to the lef t of it means "one f oreslash". To start a condition w ith the extractionoperator, use ()\/ or \\/; the latte r looks counter intuitively like "literal backslash and literal foreslash"(as it would mean if it appeared f arther along in the regexp), so most o f us prefer the f ormer.

*$ var ?? $s+\/$d+ # ok, \/ in the middle*$ var ?? \/$d+ # Wrong, when \/ is at the beginning*$ var ?? ()\/$d+ # No ok, () at the beginning

6.15 Explaining ^^ and ^[philip] Procmail doesn't think lines when it matches; but it concatenates all lines together and then

runs the regexp engine. This may be a bit su rprising, but consider the following where w e want todiscard any message that is likely a HTML advertisement

# Body consists entirely of HTML code# something which'll match any message which has "<HTML>"# in the body

:0 :*$ B ?? $s*<HTML>HTML.mbox

The condition test is applied to the entire body. If you want to limit it to match only against thebeginning of the body, you have to say so using the ^^ token, as you discovered. A simple lineanchor (^ or $) just says that there must be a newline (or the beginning or end of the area beingsea rched) at that particular point in the text being matched. notice the leading anchors below .

# trap spam where the *very* first line of the body started with# <HTML>

:0 :*$ B ?? ^^$s*<HTML>HTML.mbox

W hat, exactly, doe s " Anch or the expre ss ion at the ve ry star t of the se arch area..." i.e. the ̂ ^ ?

[dan] Technically, an opening ^^ anchors to the putative newline that procmail sees before the f irstcharacter o f the search area (and a closing ^^ anchors to the putative newline that procmail seesafter the end of the search area). When the search area is B, that is a point equivalent to thesecond of the two adjacent newlines that enclose the empty line that marks the end of the head.

The reason I'm bringing that up is this: if there are multiple empty or blank lines betwee n the headand the body, ^^ will mark the start of the second of those lines, not the start of the f irst line of thebody that contains so me text.

So if you want to test whether <pattern> is the f irst printing text in the body, even if it is not

necessarily f lush left on the very f irst line, you might need a condition like the f ollowing, where thereis space/pipe/tab/pipe/dollar.

*$ B ?? ^^$SPCNL*<pattern>

6.16 ANDing traditionallyErm, you knew this already if you read the man pages. Stacking condition lines one after anotherdoes the A ND operation, where a ll of the conditions must be present:

* condition1* condition2

6.17 ORing traditionallyHere is simple OR case. There are some cases where it's imposs ible to OR conditions with this style.[philip] knows more about those cases.

* condition1|condition2

Likewise, two exit code tests can often be ORed like this

* ? command1 || command2

But there are many situations where two tests cannot be ORed by combining them into one

condition:a regexp search of one area ORed with a regexp search of a different areaa positive regexp search [i.e., for a match to its pattern] ORed with a negative regexp search[i.e., for the absence of any match to its patte rn]an exit code condition ORed with a regexp search condition

an exit code condition seeking success ORed with an exit code condition seeking failurea size tes t ORed with anything else (even anothe r size test)

How can I mak e OR conditions that all use the S AME action? I want to be ab le to te st for a numbe r of variants on ce rtain reque sts,all in one blo ck.

[hal] Yes, this can be eas ily done

CASE = ""

:0* case 1 tests{

CASE = 1}

:0 E* case 2 tests{

CASE = 2}

:0* ! CASE ?? ^^^^{

# real work, perhaps with explicit tests on CASE}

Case study: Finding text from header and body

[david] In addition to the standard ways of coding OR, here's a spe cial one for sea rching the subjectand the body for a given word in either:

* HB ?? ^^(.+$)*(Subject:(.*[^a-z0-9])?|$(.*\<)*)remove\>

If the string doe sn't have to be preceded by a w ord border, it gets a little simpler:

* HB ?? ^^(.+$)*(Subject:.*|$(.|$))*string

6.18 ORing and score recipeOnce any of the conditions match, the score gets a po sitive value and the recipe s ucceeds . Idea byErik Selke <selke A T tcimet.net>

[era comments] ...allegedly the scoring system is going to cost you more than plain old regexmatching. Floating-point math and all that, even if you use extremely simple scoring. Thus, it wouldprobably be slightly more efficient to do it the De Morgan way.

* 1^0 condition1* 1^0 condition2

We can now write the previous case stydy (HB ORing traditionally) with scores. I was tempted towrite it like this, when [david] told me the following.

* 1^0 H ?? match-it* 1^0 B ?? match-it

[david] That will work, but it isn't the best w ay to do ORing, because if a match is f ound to the f irstcondition procmail still takes the trouble to test the second one . Better, use the supremum score oneach condition:

$SUPREME = 9876543210

*$ $SUPREME^0 first_condition_to_be_ORed*$ $SUPREME^0 second_condition_to_be_ORed* ... etc. ...*$ $SUPREME^0 last_condition_to_be_ORed

Upon reaching the supreme score, procmail will skip all remaining weighted conditions on the recipe,deeming them matched. Since all conditions on this recipe are weighted, once procmail f inds onematched condition it will skip the res t and execute the action.

6.19 ORing by using De Morgan rules[Tim Pickett <tbp A T cs.monash.edu.au>] I thought I'd point out that there are a few ways to do alogical OR of conditions. Someone posted a solution here that involved using procmail's scoringsystem, but I f igured you could do it without scoring by taking advantage o f De Morgan's rule:

a or b is same as not(not a and not b)

or mathematically:

a || b <=> !( !a && !b )

Here's a way to do ORing

:0* ! condition1

* ! condition2{ } # official procmail no-op. MUST LEAVE SPACE:0 Eaction_on_condition1_or_condition2

7.0 Variables7.1 Setting and unsetting variablesYou have already set variables with the "=" syntax. Variable names are case sensitive: var is

dif f erent from VAR

VAR = /var/tmp # directoryVAR = "this" # literal



VAR = 1VAR = $FOO # another.VAR = "$VAR at" # combined with previous value

Unsetting a variable is done like this

VAR # kill variable.VAR= # same, but with old styleVAR = "" # Variable is said to be "null" now

And you can put multiple a ssignments on the same line, although no t recommended:

VAR=1 VAR=2 VAR=3

Examine the f ollowing, which are all equivalent. The back ticks will not require a shell in the absenceof any SHELLMETAS so neither of these will spawn a shell

# case1: We Don't care if file exists this time...

VAR = `cat file`

# case2: The use of {} is considered "modern"

:0* condition

{VAR = `cat file`

}

# case3: oldish, and procmail specific and errors have# been reported if you use this construct.# Note: There must be no space in "VAR=|"

:0* conditionVAR=| cat file

7.2 Variable initialization and sh syntaxProcmail borrows some sh syntax f or variable initialization. Note that sh's ${var:=def ault} and${var=defaultvalue} syntaxes are not available in a procmail rcfile.

VAR1 = ${VAR2:-value} sets VAR1 to VAR2 if VAR2 is s et and non-null, and sets VAR1 todefault "value" otherwiseVAR1 = ${VAR2-value} sets VAR1 to VAR2 if VAR2 is set, and sets VAR1 to default othe rwiseVAR1 = ${VAR2:+value} sets VAR1 to "value" if VAR2 is set and no n-null, and sets VAR1 toVAR2 otherwise.

VAR1 =${VAR2+value} Sets VAR1 to "value" if VAR2 is set and sets VAR1 to VAR2 otherwise.

And here are the class ic usage examples

VAR = ${VAR:-"yes"} # set VAR to default value "yes"



VAR = ${VAR+"yes"} # If VAR contains value, set "yes"

Ever wondered if this calls `date` in a ll cases?

VAR = ${VAR:-`date`}

No, procmail is smart enough to skip calling date if VAR already had value. It doesn't evaluate thewho le line. Below you see w hat each initialising ope rator does . Study it caref ully

VAR = "" # Define variableVAR = ${VAR:-"value1"} # VAR = "value1"VAR = ""VAR = ${VAR-"value2"} # VAR = ""

VAR = ""VAR = ${VAR:+"value3"} # VAR = ""VAR = ""VAR = ${VAR+"value4"} # VAR = "value4"

# Note these:VAR = "val"VAR = ${VAR:+"value3"} # VAR = "value3"VAR = "val"VAR = ${VAR+"value4"} # VAR = "value4"

VAR # kill the variable

VAR = ${VAR:-"value1"} # VAR = "value1"VARVAR = ${VAR-"value2"} # VAR = "value2"

VARVAR = ${VAR:+"value3"} # nothing is assignedVARVAR = ${VAR+"value4"} # nothing is assigned

And if you want to choose from several initial values, you might use the recipe below instead of thestandard var = ${var:-"value"}.

:0* VAR ?? ^^^^{

# no value (or was empty), set default value here based on# some guesses

VAR = "base-default"

:0

* condition{

VAR = "another-default"}

...more conditions..



}

You could also use equivalent, but less readable condition line in previous recipe:

*$ ${VAR:+!}

It works, because if variable contains a value the line expands to

* !

Where "!" is the procmail "false" operation. One more way to do the same would be, that we requireat least one character to be present. You could use also regexp (.), which wo uld require at least onecharacter to be present, but you might not like matching pure spaces.

* ! VAR ?? [a-z]

7.3 Testing variablesIf poss ible, perf orm positive tes ts, rather than negative, like below:

* ! TEST_FLAG ?? yes

With negative test, this would be:

* TEST_FLAG ?? no

Using literal strings like "yes" and "no" might present more clear though what is going that atraditional "!" negation of a tes t. Note, that the following f ails if the variable is unset or null.

* variable ?? (.)

That was w hy it would be better to test:

*$ variable ?? $NSPC

Or

* variable ?? (.|$)

to require that variable contain at least one character. But neither is a way to check whether avariable is set or not, because e ach treats a null variable the same as an unset one. T his is the best

way to check whether a variable is set or not:

*$ ! ${VAR+!}

[<gsutter A T pobox.com>] Here is yet another way to test if variable is set and if it isn't, sets it to adef ault value.

:0*$ ! VAR^0{

VAR = "value"}

7.4 What does $\VAR mean?[era and david] Procmail 3.11, $\VA R w ill escape regexp metacharacters. It should produce a s uitablybackslash-escaped expression f or Procmail's own use. In addition $\VA R will always begin withleading empty parentheses.

You can't pass the $\VA R construct to shell programs, because there is that leading parenthesis.Here's a recipe to s tandardize the regexp. You can pass SA FE_REGEXP to an external programs likesed .

PROCMAIL_REGEXP = "$\VAR"

:0

* PROCMAIL_REGEXP ?? ^^\/.*{

SAFE_REGEXP = "$MATCH"}

[era] Note that this is slightly inexact; Procmail will backslash-escape according to Procmail's needs,not sed's. For example, Procmail doesn't think braces are magic (although that would be nice to havein Procmail as w ell) whereas many modern variants of sed do.

7.5 Common pitfalls when using variables

Procmail is p icky and f orgives nothing. Here are some of the favorite mistakes one can make:

$EMAIL = "[email protected]" # Done Perl lately? Remove that $

# Erm, this is ok, but many procmail recipe writers want to# take extra precautions and include the regexps in parentheses.# So, maybe (yabba|dabba|doo) would be more safe

REGEXP = "yabba|dabba|doo"

* Subject:.*$REGEXP # Hey, you need the "*$ Subject..."

*$ $REGEXP ?? hello # surely you meant '* REGEXP ?? hello'

7.6 Quoting: Using single or double quotesPay attention to this:

VAR = "you"NEW = 'hey "$VAR"' # won't extrapolate $VAR; you get literalNEW = "hey '$VAR'" # extrapolates to: hey 'you'

You can even combine separate words togethe r

VAR = "1 ""and"" 2" # same as "1 and 2"

Don't let these many quotes disturb you, just count the beginning and ending quotes. Superf luoushere, but you may nee d so me s imilar construct somewhere else.

VAR = '1 '"'"'and'"'"' 2' # same as: 1 'and' 2

[david] Beware forgetting quotes , like w hen you'd do

SENDMAILFLAGS = -oQ/var/mqueue.incoming -odq

Procmail translates ! into | "$SENDMAIL" "$SENDMA ILFLAGS" as the procmailrc(5) man page warns

us. By the rules of sh quoting, that means that shell sees only the f irst switch

% sendmail -oQ/var/mqueue.incoming

My sugges tion: since you need a so f t space inside $SENDMAILFLAGS, use the quotes when you def ine$SENDMAILFLAGS but do this instead of using the ! operator f or forwarding:

SENDMAILFLAGS = "-oQ/var/mqueue.incoming -odq"

[Walter Haidinger <walter.haidinger A T gmx.net>] Here's yet another approach: deliver messagesf rom procmail directly to mailboxes in all those users' homes. No sendmail involved, much lowerloads.

:0:* <condition>/var/spool/mail/someuser

[philip] Assuming that "someuser" is an actual user in the password f ile (I haven't been following thisthread, some maybe that isn't true he re), then the following is probably better:

W alter Haidinger comme nts on this re cipe: I 'm happy to announce that this w orks really we ll. No harm is done to the s ystem-load a nymore. W hat a relie f !



:0 w* conditions|procmail -d someuser

That lets procmail's very tricky "screenmailbox()" routine take care of bogus mailboxes in a securef ashion.

I s that as sa f e as f orwarding? Doe s anothe r sendmail delivering to /var/spool/mail/some use r use the same lo cking mechanismand notice that m ailbox is already loc ke d? I don't want to risk a corrupt mailbox.

[philip] Sendmail only delivers directly to f iles through aliases that say things like:

whatever: /some/local/file

Under normal circumstances, sendmail calls the local mailer to actually store mail in a file, and sincethat's procmail (right?), there shouldn't be a problem. A lso, sendmail 8 does kernel-level lockingwhen it delivers directly.

7.7 Quoting: Passing values to an external programRemember to include the double quotes when you send variables' values to the shell programs.Below you see a mistake, because the content o f the SUBJECT is not quoted and thus no t availablef rom perl variable $ARGV1 .

:0 # Use procmail match feature* ^Subject:\/.*{

SUBJECT = "$MATCH"}

:0* condition| perl-script $SUBJECT # mistake; use "$SUBJECT"

There is also ano ther way. If your script can access environment variables (almost all programs can),then you do not need to pass the variables on the command line. A bove, the SUBJECT is already inthe environment and in Perl you can get it with:

$SUBJECT = $ENV{SUBJECT};

Next, do you know wha t is the dif ference between these two recipes?

:0| "command arg1 arg2 arg3"

:0| command "arg1" "arg2" "arg3"

You guessed it. The f irst one quotes the entire command and does not do the right thing, the latteris correct and depending on the content of argN variables. A nyway, play saf e and always addquotes.



Sometimes you need trickier quoting to to get single quotes around the arg . Pay attention to this,because this may be the reason w hy your grep command doe sn't seem to succeed as you expect.

# If $GREP "$arg" doesn't seem to work

:0* ? $GREP "'"$arg"'" $DATABASE{

# Do something}

7.8 Passing values from an external programExternal programs cannot set procmail variables directly. Programs must write the values to externalf iles and then read the values from these files. Capturing only one value is easy:

var = `command` # capture STDOUT

But if a program modif ies the body and exports some status information it is trickier. We assumehere that the script is controlled by you and that you have added the switch --export-status optionwhich causes the program to print information to a separate f ile.

LOCKFILE = $HOME/.run$LOCKEXT # protect external file writingvalueFile = $HOME/tmp/values

# modify body, and export status values to external file: one

# value in every line## VALUE1# VALUE2# VALUE3

:0 fb| $NICE script.pl --export-status $valueFile

values = `cat $valueFile`

# Derive values from each line

:0 # line 1*$ values ?? ^^\/[^$NL]+{

var1 = $MATCH}

:0 # line 2*$ values ?? ^^.*$\/[^$NL]+{

var2 = $MATCH

}

:0 # line 3*$ values ?? ^^.*$.*$\/[^$NL]+{

var3 = $MATCH

}

LOCKFILE # Release lock

[richard] Alternatively write valueFile f rom your rc or external program with lines like

PARAM1="value for param 1"PARAM2="value for param 2"PARAM3="value for param 3"

and read it with

INCLUDERC $valueFile

Now there is no need to worry about synchronizing the read with the lines, or about adding newparameters, since each is labeled in valueFile.

7.9 Incrementing a variable by a value N[dan, phil and Richard] Here's a recipe f or incrementing a variable by a value N. If $VAR is not anumber, we get an e rror. Note tha t if $VA R + $N is not greater than 0, this recipe will not change thevalue of VAR if the assignment happens inside braces. You must place the assignment after theclosing curly brace.

:0

*$ $VAR ^0*$ $N ^0{ } # procmail no-opVAR = $=

7.10 Comparing valuesIt's too expensive to call the shell's test function to do [-lt|-eq|-gt] because you can do the samewith procmail. The do-something below is run if SCORE <= MAXIMUM. The recipe simply subtractsSCORE from MA XIMUM and determines if the result is po sitive.

:0*$ -$SCORE ^0*$ $MAXIMUM ^0{

.. do-something}

[idea by era] it's getting slightly cumberso me if it's be tween MIN and MA X:

:0*$ $SCORE ^0*$ -$MIN ^0{



dummy # no-op, just for the LOG

:0*$ -$SCORE ^0*$ $MAX ^0{

suitable}

}

Eg. Whe n values are MIN=1, MA X=5, SCORE=4

procmail: Assigning "SCORE=4"procmail: Score: 4 4 ""procmail: Score: -1 3 ""procmail: Assigning "dummy"procmail: Score: -4 -4 ""procmail: Score: 5 1 ""

procmail: Assigning "suitable"

7.11 Strings: How many characters are there in a givenstring?

:0* 1^1 VAR ?? .{ }LENGTH = $

7.12 Strings: How to strip trailing newline.Suppose you have used regexp, w hich left new line($) in the MATCH. If you wonder why the recipewo rks, remind yourself that regexp operator "." never matches a newline.

:0* VAR ?? ^^\/.+{

VAR = $MATCH}

7.13 Strings: deriving the last N characters of a string.

# 1998-06-23 PM-L [walter] Note the use of# the $ sign below to anchor to end-of-string...## For last 2 characters use * VAR ?? ()\/..$

# For last 5 characters use * VAR ?? ()\/.....$

:0 # Last character* VAR ?? ()\/.${

TAIL = $MATCH



}

7.14 Strings: Getting partial matches from a string.[dan] Getting a match to the right is quite easy with procmail's match operator.

VAR = "1234567890"

:0* VAR ?? ()\/3.*{

result = $MATCH # now 34567890}

but deleting 2 characters f rom the end is nearly impossible without f orking an outside process. Thecheapest might be expr because it doesn't need a shell to pipe echo to it (as sed wo uld and I believeperl would):

# by resetting the shellmetas, this will only call# èxpr'. If we wouldn't have fiddled with shellmetas,# this would have called two processes: sh + expr

saved = $SHELLMETASSHELLMETASresult = èxpr "$VAR" : '$.*$..'` # now 12345678SHELLMETAS = $saved

ksh or bash could do it as well:

# semicolon to force invoking a shell, actually# first question mark will force a shell already.

saved = $SHELLSHELL = /bins/shresult = ècho ${VAR%??} ;`SHELL = $saved

Now, if you know that the last two characters will be "90", that's dif f erent. Of course, this totallyscrews up if the third-to-last character is a 9.

:0* VAR ?? ()\/.*[^0]* MATCH ?? ()\/.*[^9]{

result = $MATCH # now 12345678}

[jari] Comments: If a shell must be used, then awk is a good tool f or simple string manipulation. Itsstartup time is f aster that perl's whose overhead is due to internal compilation. awk also consumesless recourses overall than perl . Following will only work if VAR is a string of continuous block of



characters. (A RGV1 can be used)

saved = $SHELLMETASSHELLMETAS

VAR = ` awk 'BEGIN{ v = ARGV[1]; \print substr(v,1,length(v)-2); exit }' \

"$VAR" \`

SHELLMETAS = $saved

This version requires some f ile, any f ile, so that we get awk started. In the previous code all thewo rk was done in the BEGIN block and no f ile w as ever opened.

saved = $SHELLMETASSHELLMETAS

VAR = ` awk '{print substr(v,1,length(v)-2) ; exit }' \v="$VAR" /etc/passwd \

`

SHELLMETAS = $saved

[dan] comments awk: expr is sure to be a smaller binary than awk for procmail to fork, and it needsmuch less command-line code to do this job. Note also that one still has to diddle with SHELLMETA Sto avoid a shell, because the aw k code contains brackets; thus it doesn't replace all.

There is also a way to remove words f rom the end of string by procmail means if the strings areseparated by same separator. Let's use the word this-mailing-list-request which we would like tosho rten to this-mailing-list. [david] presented the recipe 1998-06-16 in PM-L.

VAR = "this-mailing-list"

# 1) if there is match at the end ending to these words# 2) Get everything up till last match and store it to MATCH# 3) Read MATCH, but exclude last dash "-"

:0* VAR ?? -(owner|request|help)^^* VAR ?? ^^\/.*-* MATCH ?? ^^\/.*[^-]{

VAR = $MATCH}

7.15 Strings: Procmail string manipulation example

[1998-06-23 PM-L walter] ... Now we get to apply these f ormulas to strip the last character of f astring. It gets a bit ugly f or special cases. I've de liberately chosen a wo rst-case scenario.

VAR = "Testing 012301230111"RC_APPEND = $PMSRC/pm-myappend.rc



:0* VAR ?? ()\/.${

TAIL = $MATCH # last character of VAR "1"

# Get the longest match that does not end in the TAIL character

:0*$ VAR ?? ()\/.*[^$TAIL]{

HEAD = $MATCH # now "Testing 012301230"

# if the last two or more characters in VAR are# identical, they all get chopped, oops

:0* -1^0* 1^1 VAR ?? (.)* -1^1 HEAD ?? (.){

dummy = "tooshort"INCLUDERC = $RC_APPEND

}}

}

result = $HEAD # "Testing 01230123011"

# ........................................ pm-myappend.rc# LENGTH(HEAD) plus 1 SHOULD equal LENGTH(VAR). That is# not the case when the last 2 (or more) ending

# characters are identical. in that case, call appendrc# recursively to stick back an appropriate number of# TAIL characters.

:0* -1^0* 1^1 VAR ?? (.)* -1^1 HEAD ?? (.){

HEAD = "$HEAD$TAIL"INCLUDERC = $RC_APPEND

}

7.16 How to raise a flag if the message was filed

FILED = ! # ! is procmail "false"

:0 c: # We process the message more* conditionfoo

:0 a{

FILED # Kill variable}



...

:0 # Stop if previous cases filed the message*$ $FILED{

HOST = "_done_"}

Or alternatively: procmail auto matically sets LASTFOLDER if it delivers message to mailbox.

LASTFOLDER # kill variable

:0 c:* conditionfoo

:0 c:* condition

bar

... et cetera ...

:0* ! LASTFOLDER ?? ^^^^ # Or ${LASTFOLDER+!}!{

HOST = "_done_" # Force procmail to stop}

7.17 Dollar sign in condition lines.#todo , check this recipe

This doesn't seem to work for me...

* ^TO()$\[email protected]

[david] An unescaped dollar sign later in the line represents a newline, so what you have there issea rching f or the f ollowing:

1. A n expression that matches the expansion of the ^TO token (which is anchored to the start of a line by its def inition), followed by

2. A newline, f ollowed at the start of the next line by3. "f oo@bar" [the backslash escapes the f , which didn't need escaping], f ollowed by4. any character that is not a new line (the period is unescaped), and f inally5. "com".

Try this instead:

*$ ^TO()$\foo@example\.com

#todo: the dollar seems exactly the same in the above two #todo Examples: are you sure that thisis correct?



In f act, to avoid matches to things like f [email protected], you might want to do it thisway:

*$ ^TO()$\foo@example\.com\>

7.18 Finding mysterious foo variable I have my f ellow work er's proc mail code and he us es a variable FOO that I c an't f ind in his code anyw here . It's not a she ll variableeither, because it' s literal. Where does it come f rom?

Your procmail runs /etc/procmailrc when it starts, please check that. It may define some commonvariables already f or all users.

7.19 Storing code to variableOne way to run complex code in a procmail recipe is first to store it in a variable. Idea by [era] . Youcould do this in a separate shell script too. The f ollowing example reads URLs f rom the body of amessage: the URLs have been put to separate lines and some special Subject is used to trigger thedumping of the HTML pages:

# Code by [era]#COMMAND='while read url; do

case "$url" in*://*)

lynx -traversal -realm -crawl -number_links "$url" |$SENDMAIL $LOGNAME;;

esacdone'

# Notice the trailing semicolon after `eval' !:0 bw* ^Subject: xxxxx| eval "$COMMAND" ;

If you want to run the code inside the nested block, then look carefully, there are double quotesaround the command in back ticks. If you leave double quotes out, then ea ch word in SH_CMD wouldbe interpreted separately:

$SH_CMD = '$echo "$VAR" >> $HOME/test.tmp'

:0* condition{

# condition satisfied; run the given shell command# and do something more.

dummy = `"$SH_CMD"`

..rest of the code..}

A similar construct works f or message echoing too:



MESSAGE='Thank you so much for your message.Unfortunately, the volume of mail I receive .... (blah blah blah).If your matter is urgent, try calling +358-50-524-0965.'

:0 hw* ! ^X-Loop: moo$

| ($FORMAIL -r -A "$MYXLOOP"; echo "$MESSAGE") | $SENDMAIL

7.20 Getting headers into a variable[david] Here are several ways to get the entire header into a variable:

HEADER = `$FORMAIL -X ""` # The space after the X is vital.HEADER = `sed /^$/q` # also writable as HEADER=`sed /./!q`

:0 hHEADER=|cat -

will save the entire header into one variable. It has to be smaller than LINEEBUF , though. This waymight work as we ll, and will require no outside process es if it does:

:0* ^^\/(.+$)*${

HEADER = $MATCH}

7.21 Converting value to lowercaseIf you know that a w ord belongs to set o f choices, you can do this inside procmail

LIST = ":word1:word2:word3:word4" # Colon to separate wordsWORD = "WORD1"

:0*$ LIST ?? :\/$WORD{

WORD = $MATCH}

But if you don't know the word or string bef orehand, then this is the generalized way: [idea by eraand david]

:0 D* WORD ?? [A-Z]{

WORD = `echo "$MATCH" | tr A-Z a-z`}

8.0 Suggestions and miscellaneous8.1 Speeding up procmail

Use absolute paths to take the burden o f searching binary along path from shell: Use$FORMAIL variable abs traction.

$FORMAIL = "/usr/local/bin/formail"

:0 fhw| $FORMAIL -I "X-My-Header: value"

Multiple echo commands that sprea d many lines can be converted to s ingle echo command if \nescape is suppo rted. You usually see these in auto responde rs

echo "........."; \echo "........."; \echo ".........";

-->

echo ".........\n" \".........\n" \".........\n";

You can avoid multiple and possible expensive FROM_DAEMON tests by caching the result atthe to p of your .procmailrc. You can now use variable $from_daemon like the big brotherFROM_DAEMON. The same idea can be applied to FROM_MAILER regexp. If you have pm-

javar.rc , it already defines variables $from_daemon and from_mailer exactly like here:

from_daemon = "!"

:0* ^FROM_DAEMON{

from_daemon = "!!" # double !! means "OK"

}

:0*$ ! $from_daemon{

..do-it..}

Count the back ticks and you know how many shell calls procmail has to launch. See if you canminimize them and use some procmail code instead.^TO and o ther macros are expensive, see if you can use s imple Header:.*\<match-it\>instead. Well, it's not clear if this gives you much speed advantage.Don't call "$FORMAIL -xHeader:" every time you nee d a header value, cons ider if it suffices touse match operator \/.You can minimize the calls to only one formail if you add many headers along the way: Seeformail usage tips in this document



Searching body is expensive, simply because it contains more text. There isn't much to doabout this, because you use B anyway when you need it.See if you can move some tasks to your .cron file. procmailrc is not meant for those purposes.Instead of calculation da ily values every time in procmail, let cron do that a t 04:00 or 21:00.Don't run cron a t midnight if you can, because everybody else is running their crons at thesame time. If "logical" date change time can be used (when you arrive to w ork, when youleave the work), use it in cron jobs .[philip] Setting LINEBUF permanently to a big value s lows procmail down.Remove all calls to perl and use programs that are nicer to the system (If you just callcommand line perl, there is probably an equivalent alternative with awk tr sed cut )Examine each shell command and see if you do need SHELLMETAS. If you can set SHELLMETAS toempty, this saves calling "sh" for each invocation of the external command.

8.2 See the procmail installation's examplesDid you remember to look at the examples that come with procmail? If not, it's time to give them achance to educate you. Here is one possible directory you could take a look. Ask from your sysadm if you can't find the directory where to look into.

% ls /usr/local/lib/procmail-3.11pre7/examples/

Or if you're really anxious to get on your own, try this. The directory /opt/local is for HP-UX 10machines and the forward contains example how to define your .forward for procmail.

% find /opt/local/ -name "forward" -print

If the find succeeded and found the file, then you know where the procmail files installation directory

is.

8.3 Printing statistics of your incoming mailIf you keep the procmail log crunching, it will record to which folder the messages wa s filed. There isprogram mailstat which can process the procmail.log file and print nice summary out of it. If yougenerate the summary at midnight and clear the log, you get pretty nice per day/per folder trafficanalysis.

# -m merges all error messages into a single line

% mailstat -km procmail.log

8.4 Storing UBE mailboxes outside of quota I w ant to stor e s pam outs ide d isk space . Proble m: if I tell pro cmail to deliv e r to, say , /tmp/spam.b ox, it doe s so just fine (ac cordingto the log). Unfortunately, it delive rs to /tmp on the mail host which I cannot access . spam.box doe sn't appear in the /tmp directoryof the she ll machine w he n procmail is invoke d for incoming mail.

[philip] Under the most likely configuration of sendmail in this situation, it is impossible to haveprocmail invoked by sendmail on the shell machine: sendmail is probably set to just forward all mailto the designated mail delivery machine.

There are other options: you could temporarily store the mail in your account, then have a cron jobon the shell machine that reprocesses the message. That would probably be more efficient thanhaving each message trigger an rsh to the shell machine. If you actually get enough spam that it'spushing against your quota, then the rsh is too expensive – use a cron job that invokes somethinglike:



cd your-maildir &&lockfile spam.lock &&test -s spam &&{

cat spam >> /tmp/spam.box && rm -f spam spam.lock || \rm -f spam.lock;

}

WARNING: the above assumes the f ollowing:

everything in your-maildir/spam is spam and belongs in /tmp/spam.boxno further filtering of the messages is necessary: they just need to be moved (it actuallytreats everything in the your-maildir/spam as a s ingle message and uses procmail as a reliablecopy command, thus the DEFAULT assignment as the use of /dev/null as a empty procmailrc)

/tmp/spam.box is a not a directory

If the latter two of those conditions isn't true OR IF THEY MIGHT CHANGE then you should useformail -s to break the message apart and invoke procmail on each one separately.

[era] Many s ites cross-mount directories for various reasons. /tmp is always local but /var/tmp mightbe cross-mounted between the login hos t and the mail host; ano ther one to try is /scratch – and if all else fails, ask your admin to se t up an NFS share for this purpose.

8.5 Using first 5-30 lines from the message[era] The regexp to grab few lines (or all of them, if there are less than fifty) is not going to be verypretty, but it saves launching an extra process.

:0

*$ B ?? ^^$SPCNL*\/$NSPC.*$(.*$)?(.*$)?{toplines = $MATCH

}

The skipping of whitespace at the beginning of the message is of course not necessary. You shouldprobably set LINEBUF reasonably high if you grab many lines, say 30: 80*30 = 2400 bytes; probablysetting it to 8192 or 16384 is a good idea, depending how much you want to match. The abo ve getsugly quickly, so

# But if N=30, sed ${N}q if you don't have head

:0 i{

toplines = `head -$N`}

:0 a* toplines ?? pattern{

...do-it

}

8.6 Using cat or echo in scripts? I have se e n a lot of example s that use 'echo', i.e .,



:0* condition| echo "first line of message" \

"second ..." \"et cetera"

I s tarted o ut with spam.rc f rom "ariel" which got me into the habit o f

:0* condition| cat file_containing_message

although I note that spam.rc did have one re cipe using the e cho me thod. What are the re asons f or choosing each me thod ove r theother?

Here is a comparison table. Choose the one you think is best f or you

Echos don't have dependency on an external file: everything is contained in the .procmailrcfile. Echos keep all the relevant s tuff in one file. Cat's make you maintain multiple files. That'sthe main reason I lean tow ard echo's; you may have accounts on several machines. It iseas ier to be able to copy just one generic .procmailrc between them without having to copy abunch of messages also. Mostly, though, there's no real difference between the two methods .Echo is easier to use with variables.Echo starts many processes, cat only starts one , but this is not a lways true: In most currentBourne shell implementations, echo is a bu ilt-in. This holds true with tcsh too.The main problem I see w ith the use of cat is "what happens when you forget the file ordestroy it ?". I suggest to, at least, test that the file is readable before catting it.[richard] An argument against echo is that it is not we ll standardized, and different versionsmay exist on the same machine. Some recognize -n, some don't; some recognize embedded

metacharacters , some don't.This is an argument in favor of print . Print, however, is not abuilt-in on all systems. The comment on built-ins is pe rtinent to situations when a s hell isspawned. When procmail handles the call directly, it will always look for a stand-aloneexecutable. I guess echo may be better, as long as w e are aware of any differences inbehavior between built-in and stand-alone versions.

8.7 How to run an extra shell command as a side effect? [jari] I w as once wonde ring what would be the wise st way to se nd mess ages to my daily "biff" log file about the ev ents that happene d during my .procmailrc e xecution. This is ho w [david] comme nted on my ideas

# case 1: print to BiffLog

dummy = `echo "message: $FROM $SUBJECT" >> $biff`

[david] Problems you get no locking on the destination file, and unless you put it inside braces youhave to run it on every message unconditionally. (Also procmail tries to feed the who le message to acommand that won't read it, but the remedies for that don't help very much.)

# case 2: We consume delivering recipe and therefor have to use# `c' flag.

:0 whic:| echo "message: $FROM $SUBJECT" >> $biff



Here it locks the destination f ile and you can add conditions to it, so it's probably the best. If thehead o r the body is less than one buf f erf ul, you can limit the unnecessarily written data with h or b ,but I think that in most OSes a partial buf f er and a full one a re the same amount of ef fort.

# case 3: We use side effect of "?" here. Cool, but this# doesn't do $biff file locking thus message order may# not be what you expect.

:0* condition* ? echo message: $FROM $SUBJECT >> $biff{ } # procmail no-op

We have conditions poss ible, but there is no locking on the des tination f ile. I'd go with metho d #2 ora variation thereo f :

:0 hic: # we don't necessarily need `w'* condition| echo message: $FROM $SUBJECT >> $biff

:0 hi: # Or you could use this* conditiondummy=| echo message: $FROM $SUBJECT >> $biff

[ jari] Now, when [ david] has ex plaine d how v ario us ways d i f f er f rom each other, I prese nt the re cipe where I use d the case 3.W hen I was dropping a message to a f older, I wanted to se nd a message to my bi f f log too. T he idea is that the drop-conditionshave already matched and then w e run e xtra command by using side ef f ect of "?" token. As f ar as the re cipe is conce rned, the "?" isa no-op. The pedantic way would have be en to add the LOC KFILE aro und to the re cipe , but imagine 5 0 similar re cipe s likethis...and you understand why the LOCKFILE was le f t out. It's only nece ssary i f you wo rry about se quential writing to the bi f f f ile.

:0 :* drop-condition* ? echo message: $FROM $SUBJECT >> $biff$MBOX

8.8 Forcing "ok" return status from shell script...the "?" trick only allow s running some additional shell c ommands ( true command always succ ee ds) while conditions abovehave already de te rmined that drop will take place. And you can always make condition to succe ed i f a misbe having shell script always re turns a f ailure exit code .

* ? misbehaving-shell-script || true

[david] If the script always returns a failure code, just do this:

* ! ? misbe having-she ll-script

The more complex case is a script that can return either success or failure but you don 't care w hich; if the drop conditions passed, you want to run the action line. echo can also f ail if the process lackspermission or opportunity to w rite to stdo ut. A more reliable choice is true(1); its purpose in life is todo nothing but exit with status 0.

The command : is a shell built-in which a lways returns true status . Not exactly more readable thantrue(1) "|| :" will save the invocation of true (unless true is built into $SHELL), but procmail will stillrun a shell. On the o ther hand, as long as the command itself has no characters f rom SHELLMETAS a

weight of 1^1 and no "|| anything" will avoid the shell process as well.

However, there is yet a better way to make sure that a failure by the script doesn't make procmailabort the recipe:

:0 flags* other conditions* 1^1 ? shell-script

action

Regardless of the exit status o f the script, the condition will score 1 and not interf ere with procmail'sdecision about the action line of the recipe. Weighted e xit code conditions behave like this (see theprocmailsc(5) man page):

* w^x ? command

scores w on success o r x on f ailure.

* w^x ! ? command

scores the same as this:

* w^x pattern_that_appears_in_the_search_area_$?_times

8.9 Using grep with file lists to mach messagesIf you want to implement blacklisting or whitelisting, here is the idea how to call grep program to domatches. First, suppose you wa nt to match against bad words. The following example supposes GNUtools f or sed, egrep. The regexp variable is read f rom f ile by f irst calling tr to produce "this |that|\ <get this!|" which is a combined string f rom f ile's lines. It is f urther post processed with sed by a)deleting any trailing whitespaces bef ore (|) and by b) deleting unnecessa ry trailing or(|) token(s)added by tr .

EGREP = "/bin/egrep"SED = "/bin/sed"TR = "/usr/bin/tr"file = $HOME/procmail/spam-regex p.lstregexp = `$TR '\n' '|' < $kwdfile | $SED -e "s/[ \t]+|/|/ ; s/|+$//" `

:0 HBw* ! regexp ?? ^^^^*$ ? $EGREP --quiet --ignore-case --regexp='$regexp'{

# Matches, do something}

It is a little easier to check sender's address aga inst a w hitelist, because it is possible to use "wo rd"based checking in contrast to regular expression checking aboce. Supposing that file containsknown email addresses listed one at a time, the recipe recipe would be:



file = $HOME/procmail/spam.keywordssearchFields = "-xSender: -xFrom -xFrom: -xReturn-Path: -xReply-To:"

:0 w*$ $FORMAIL $searchFields | $EGREP --quiet --ignore-case --file='$file'{

# This sender is known

}

A word of caution: white list or black list based sender matching does not work 100%. Thespammers hijack large amount of other people's email addresses which they ruthlessly use inidentif ying the message's sender. It is no surprise to receive a Unsolicited Bulk Email f rom f riend – heis not the real sender, but his address was drif ted to spammers email database. 9.0 x1 8.11 Usingdates ef ficiently

Note : Se e module list, where you will f ind date and time parsing modules . You can also parse the date f rom the f irst Received or From_ header i f it is the same each time in your sys tem. T hat would be orders of magnitude f aster and decrease s your systemload i f you rece ive lot of mail.

Calling date in your procmail script many times is not a good idea. Use the MATCHas much as poss ibleto be eff icient in procmail, like below where we call date only once. If you are no t in the same timezone as your server, and you want an accurate report of the date, you might amend the invocationto the following:

date = `TZ="KDT9:30KST10:00,64/5:00,303/20:00";date "+%Y %m %d"`

The bas ic recipe is here

# By [richard] add %H:%M%S if you want these as well

:0* date ?? ^^()\/....{

YYYY = $MATCH}

:0* date ?? ^^..\/..{

YY = $MATCH}

:0* date ?? ^^.....\/..{

MM = $MATCH}

:0* date ?? ()\/..^^{

DD = $MATCH}

TODAY = "$YYYY-$MM-$DD" # ISO std date: like 1997-12-01

9.1 Keep simple header logHere is a simple strategy: record all what comes in and record all what happened to that message .See how brief inf o is constantly recorded to BIFF f older. You can now check the BIFF log every day tosee if the messages were sunk to right f olders: Remember to add BI FF rule to every recipe, so thatthe sink message [sunk-somewhere] is recorded af ter incoming message headers.

Emacs can display the f ile in one buf fer window and keep it updated with M-x auto-revert-mode . Itgives a nice overview of arriving mail messages and is equivalent to biff(1) .

# this requires that HH and MM have been setup before,# see pm-jadate.rc

NOW = "$HH:$MM" # the time onlyTODAY = "$YY-$MM-$DD $NOW" # ISO 8601: date and time

$NULL = $SPOOL/junk.null.spool # /dev/null is dangerousBIFF = $PMSRC/pm-biff.log

# If you prefer a log per day (easy for cleanup):# BIFF = $PMSRC/pm-biff.log.$YYYY$MM$DD

# .............................................. headers ...

# DON'T USE THESE: they call shell## FROM = `$FORMAIL -zxFrom:`# SUBJECT = `$FORMAIL -zxSubject:`

:0 # Use procmail match feature* ^From:\/.*

{FROM = "$MATCH"

}

:0 # Use procmail match feature* ^Subject:\/.*{

SUBJECT = "$MATCH"}

# ............................................. incoming ...# record log of incoming mail

:0 hwic:| echo "$TODAY $FROM $SUBJECT" >> $BIFF

# ......................................... null recipe ...# Now, this is how you add the "message" what happened# to that mail. See "?" shell call in the recipe

:0 :* From:.*(remove|delete|free|friend@)* ? echo " [null-AddrReject]" >> $BIFF

$NULL

9.2 Gzipping messages[Sean B. Straw <PSE-L A T mail.professional.org>] On the recipe delivery line where you'd normally



be tossing it into a f older do this instead:

:0 c:|gzip -9fc >> $MAILDIR/mail.mbox.gz

This will compress each messa ge as it comes in (and since most are T EXT, it does a f ine job - MIME,

OTOH is o ne of the bes t ways to mailbomb someone since it doesn't compress well - but the indirectbombing via mailing lists doesn't do this), reducing the disk space required, usually dramatically.Done in conjunction with so mething like the following at the end of your .procmailrc, you could have aheader file you could quickly rummage through looking f or valid messages to add to a procmailrecipe, then run:

gzip -d -c mail.mbox.gz | formail -s procmail -m recipe.rc

(note that if the recipe delivers into the mail.mbox.gz file on any condition, then you should look toMOVE the file bef ore running this process, and use the moved version. In f act, this would be a goodidea anyway, as newly delivered mail may appear in the end of the gzip file while you're doing this -and since your ultimate goal is to be able to eliminate junk, you'll want to know that after you'veprocessed a gzipped mail file, you can delete it without accidentally whacking new mail).

:0* LASTFOLDER ?? ^^^^{

# Save the message in case we need to retrieve it.

:0 c:

|gzip -9fc >> $MAILDIR/mail.mbox.gz

# copy headers for easy browsing - including being able to# identify lists you're being subscribed to.

:0 h:header.log

}

9.3 Emergency stop for your .procmailrc[jari] If I have a bad luck while I am testing a new recipe, it may run in a loop and and it may sendme continuously mail messages. I then have to quickly recall .procmailrc and start disabling myindividual "control" recipe files. Yet I f igure, in situations like this where every second is important,there must be a better way. [alan] This is quite easy already; put this at the to p of your procmailrc:

# instead of leading dot file, you may prefer# stopFile = $HOME/procmailrc.stop which shows up in default ls.# In the other hand you can do ls ~/.procmail* to see both...

stopFile = $HOME/.procmailrc.stop

:0*$ $IS_EXIST $stopFile{

EXITCODE = $EX_TEMPFAIL # Means: retry later; requeueHOST = "_stopped_by_external_request_"

}

Then, when testing your procmailrc and disaster happens, you can simply do f ollowing to disableyour procmailrc f iltering.

% touch $HOME/.procmailrc.stop

[richard] This is a lso a candidate recipe f or including in an INCLUDERC. Combining the two ideas, wehave a file procmailrc.stop which contains the recipe and is included near the top of .procmailrc,When you don 't want it, mv it to procmailrc.go. Procmail complains abo ut missing INCLUDERCs, but itdoes not complain about them if they exist and are empty. A nother reason to not use dotted f ilenames, but to use cp instead of mv.

10.0 Scoring10.1 Using scores by an exampleFirst make all the needed matches and let the SCORE value to be set. Examine the score after thef inal value has been calculated. The condition lines say:

Start with some threshold: -250.Read the subject into MATCHAdd 50 for each match of !. Notice the "^1": if it read "^0", only one 50 w ould be a dded for"!!!!", now that counts a s 4 x 50 = 200. See procmailsc(1) for "^N" syntax.Any dollar sign is likely spam.find uninteresting subject wordsAnd a negative count for replies.Usua lly spam doesn't seem to have Re: in subject field. (but don't rely on this, spammers havestarted to use "re:")letters such as !!! frequently found in the body are usually indication of spam. Add 100 foreach match.

# Idea by 26 Sep 97 Stephane Bortzmeyer <bortzmeyer A T pasteur.fr>

:0* -250 ^0* ^Subject:\/.+$* 50 ^1 MATCH ?? [!]

* 50 ^1 MATCH ?? [$]* 100 ^1 MATCH ?? ()\<(free|sex|opportunity| money|great)\>* -250 ^0 ^Subject: *(Fwd|Fw|re):* B ?? 100 ^0 ()!!!{ } # official procmail no-op

SCORE = $= # Score has been calculated

:0 fhw| $FORMAIL -i "X-Spam-Score: scored $SCORE"

:0: # If score had positive value, sink message*$ $SCORE^0junk.spam.mbox

Given the f ollowing subject:

"Great opportunity for free sex; no money required!!!!"

procmail scores it this way: ! was f ound 4 times (200/weight 50), "free|sex..." regexp matched 4times (400/weight 100).

condition score Total sum so far---- ----------------

procmail: Score: -250 -250 ""procmail: Score: 200 -50 "[!]"procmail: Score: 0 -50 "[$]"procmail: Score: 400 350 "^Subject:.*\<free|sex|. ..>"procmail: Score: 0 350 "^Subject: *(Fwd|Fw|re):"procmail: Score: 0 350 ! ""procmail: Assigning "SCORE=350"

[david] Some notes on po ssible regexps and their dif f erences:

* 100^1 ^Subject:.*\<(free|sex|opportunity|money|great)\>

That condition says to score 100 f or every subject line that contains any of those f ive words ... not toscore 100 f or every one of those words in the subject, but 100 f or every subject line that containsany of those words. So it will never score more than 100 unless there are multiple subject lines. Yousee , it of f ers f ive alternative regexps:

^Subject:.*\<free\>^Subject:.*\<sex\>^Subject:.*\<opportunity\>^Subject:.*\<money\>^Subject:.*\<great\>

Of f hand, I think regexp below would score 400: 100 f or "Subject.*f ree" and 100 f or "sex" etc. Of

course, the score might be higher if other lines in the header included the strings "sex","opportunity", "money", or "great<wo rd border>", but appearances of "<word border>free" outsidethe subject wouldn't be counted.

* 100^1 ^Subject:.*\<free|sex|opportunity|money|great\>

[translates to]

^Subject:.*\<freesexopportunitymoneygreat\>

And this one wo uld score 400 too. How? MATCHwould contain whole subject and there would be non-

overlapping matches to " great ", " oppo rtunity ", and " f ree ". If we got rid of either or both of theword-border marks, it would score 500.

Subject: Great opportunity for free sex; no money required!!!!* 100^1 MATCH ?? ()\<(free|sex|money|opportunity|great)\>

10.2 Brief Score tutorial#todo: test

[elijah] If you're serious about using scores, please spend a minute reading this short example.

VERBOSE = "yes"

:0* 1^1 foo* -2^2 bar

{ }a = $=

:0* 1^1 foo* -2^2 bar{

:0 f| echo Whee: fun ; cat -

}b = $=

:0* 1^1 foo* -2^2 bar{

whee = "fun"}c = $=

:0 h/dev/null

Then if you would send a message

From foo FooofTo: barSubject foobar

body-something-here

The log f ile w ill tell you what happened.

procmail: [20175] Fri Sep 26 10:25:23 1997procmail: Score: 3 3 "foo"procmail: Score: -6 -3 "bar"



procmail: Assigning "a=-3"procmail: Score: 3 3 "foo"procmail: Score: -6 -3 "bar"procmail: Assigning "b=0"procmail: Score: 3 3 "foo"procmail: Score: -6 -3 "bar"procmail: Assigning "c=-3"procmail: Assigning "LASTFOLDER=/dev/null"procmail: Opening "/dev/null"From foo Fooof

Folder: /dev/null 46

10.3 Score's scopeIf you have a delivering recipe and the score is positive, the action lines are executed. If the score isless or equal to 0, then the $= inf ormation is lost, but also at the next recipe de f inition, even if therecipe is never executed. Study following example:

:0* 10^0{

dummy = "Score for condition xxxx was: $= $NL"

:0{

dummy = "Next recipe, Score no longer available: $= $NL"}

}

# Wont' work. $= is getting set back to 0 outside of# the delivering recipe.

dummy = "Score outside of all recipes: $= $NL"

Here is interesting anomaly which [richard] discovered. It is presented here only as a curiosity. DONOT USE IT IN YOUR RECIPES. (this not "clean programming", but a hack)

[david] If you want to save the score f or later use (even if it is zero or negative):

:0* 10^0{ } # procmail no-op

SCORE = $=

:0 Aaction_if_positive

If other recipes that clobber the ref erences f or the A flag intervene, this will work:

:0* 10^0{ } # procmail no-op

SCORE = $=

... more stuff ...

:0*$ $SCORE^0action_if_positive

10.4 Counting length of a stringSupposing VAR contains some text, we can count the characters by using dot to match everycharacter and increasing score for every match.

:0* 1^1 VAR ?? .{ }

LENGTH = $=

10.5 Counting lines in a message (Adding Lines: header)[1995-10-03 PM-L Idea by David Karr <dkarr A T nmo.gtegsc.com>] [david] later corrected 1998-01-02: For one thing, the second condition always counts one too many (the final newline plus theclosing putative newline create the extra match); second, af ter making that correction, an emptybody wo uld score ze ro and leave the variable undefined.

:0

* 1^1 .* 1^1 ^.*$* -1^0{ }lines = $=

:0 fhw* ! ^Lines:| $FORMAIL -a "Lines: $lines"

The reason we used it at all was that size conditions worked only on the entire text regardless of Hor B or HB flags at the top o f the recipe. Nowadays we can do this and get the accurate figure in onecondition:

# leave `B ??' out to measure the entire message:0* 1^1 B ?? > 1{ }size = $=

If you want to be s illy about it (as some of us very of ten do),

:0* -1^1 B ?? > -1

{ }size = $=

gives the same result, and as long as the search area is non-empty, so do these, which are evensillier:

:0* 1^-1 B ?? < 1{ }size = $=

:0* -1^-1 B ?? < -1{ }size = $=

[Karr] This recipe counts bytes in the message, you could use this Content-length replacement,pref er using the next recipe. The first score counts every character, and the second score sums upevery line (that is: new lines are added).

:0 H # use B to measure body only* 1^1 B ?? .* 1^1 B ?? ^.*${

textsize = $=

:0 fhw

* ! ^Content-length:| $FORMAIL -a "Content-length: $textsize"}

10.6 Determining if body is longer than header

:0* 1^1 B ?? > 1* -1^1 H ?? > 1

{..body was longer

}

10.7 Matching last Received header[david] Here is way to use scores to hit the bottommost Received header.

:0

*$ 1^1 ^Received:.*by$s+\/.*action

10.8 Testing value range with scoring (bogofilter)

Bogofilter adds headers to the message that contains the propbability scode of the message beingspam in range 0.0 - 1.0:

X-Bogosity: No, tests=bogofilter, spamicity=0.365761 ...

If the filter runs a t MTA , the values that aff ects the word "No" at canoot necessarily be conf igured. To

test directly the result score to catch messages in range 0.2 - 0.9 as "Unsure" can be done withscoring. If the spamicity value was 0.92, the first score would return: 1.90 - 0.92 = 0.98, which islower than 1 the score OK value.

:0* ^X-Bogosity:.*spamicity=\/0\.[0-9][0-9][0-9]{

# check for maximum:0* $ -$MATCH^0* 1.90^0

{# check for minimum:0:* $ $MATCH^0* 0.8^0{

# VAlue is betweeb A .. B}

}

10.9 How to add Content-Length headerW e use procmail f or local de livery, and would like to get it to generate the co ntent-length he ader, i f one doe sn't exist. SUN-OS mailtool at le ast gets co nf used and merge s mess ages together i f there is no message body.

[stephen] All you need to do is: a) Make sure that procmail is s tarted without the -Y f lag. b) Either, inyour sendmail.cf , inse rt:

H?l?Content-Length: 0000000000

Or (slightly less ef ficient), inse rt the following recipe in your /etc/procmailrc f ile and P rocmail will takecare of any necessary magic.

:0 hfw* !^Content-Length:| /usr/bin/formail -a "Content-Length: 0000000000"

10.10 Testing message size or number of linesSize conditions ignore H and B on the flag line and always work on HB unless another search area isspecified on the condition's ow n line. To test only the body,

:0 # Note: this is in BYTES*$ B ?? < $NBR



{...whatever when fewer bytes

}

This syntax would obey a B f lag on the f lag line:

:0 # Note: this counts LINES* -1^1 B ?? .* -1^1 B ?? ^.*$*$ $NBR^0{

...whatever when fewer lines}

10.11 Counting commas with recursive includerc[jari] Foreword: David and Phil really are experts with procmail, and let this section serve as anexample to "what on Earth is recursive procmailrc and ho w it is used?". I would not persona lly userecursive includerc, simply because I would not trade clarity: I f ind this easier to understand andmaintain. split just explodes input according to comma and the print return how many elementswere exploded to array a . The perf ormance hit is not bigger than forked procmail binaries inrecursive version.

:0* ^CC:\/.*{

field = $MATCH

saved = $SHELLMETASSHELLMETAScommaCount = `echo $field | awk '{print split($0,a,",")}' `SHELLMETAS = $saved

}

[richard] Here is recipe that needs no recursion. MAX_RECIP is set to 9, but you may pref er someother value. This counts each comma. It allowed in addresses.Some folks sum Resent-xx or non-Resent-xx headers. I sum all.

:0* 1^1 ^(resent|apparently-)?(to |b?cc):\/.** 1^1 MATCH ??,*$ -$MAX_RECIP^0{

:0*$ $=^0*$ $MAX_RECIP^0{

RESULT = "Count of commas is $="

}}



11.0 Formail usage11.1 Fetching fields with formail -xIf you're new to procmail your f irst though to read a header content f rom the message wo uld mightbe call:

SUBJECT = `$FORMAIL -xSubject:`

That's not good. DON'T Do THA T. You just created expensive shell subprocess where procmail callsformail and feeds full message to it. We can do the same with minimum eff orts:

:0* ^Subject:\/.*{

SUBJECT = $MATCH}

No shell subprocess called. This is much f aster and consumes f ewer resources, while it may needmore typing. Use it and your your sysadm is happy with your well behaving procmail recipes thatdon't load the CPU unnecessarily. The equivalent with formail might be more secure, because itcontains full RFC-compliant parser. The traditional way of deriving the address with formail is:

FROM = `$FORMAIL -rzxFrom:`

But you can still make this more eff icient. Here is one example where you actually want to use "old"=| style variable assignment, make sure there are no extra spaces:

:0 hwFROM=|$FORMAIL -rzxFrom:

That way only the header gets fed into formail, whereas the previous back tick f ed the wholemessage. Another benefit is, that you can then check the return code of f ormail with a or A recipeafter this one .

11.2 Always use formail's -r switch[Philip] The -r option has been chnage in 3.14. In older procmail version there was need f or -rt , butnow o n use only -t . To quote the f ormail manpage :

By de f ault, when gene rating an auto-re ply header procmail sele cts the e nvelope se nder f rom the input mes sage. This is corre ct f or vacation mes sage s and other automatic re plies re garding the routing or delive ry o f the original message . I f the s ender isex pe cting a reply or the re ply is be ing gene rated in res pons e to the c onte nts of the original message then the -t option should beused.

11.2.1 For procmail versions prior 3.14

[FAQ] -r breaks RFC822, so always use -rt if you don't know what this means. Perhaps you shouldalways use it anyway.

[david] There is f ormail -r t rank bar graph in the source code of 3.11pre4. It might be easier to f ollowas a top-to-bottom listing (and again, Tom Zeltwanger appea rs to be using one of the o lder versionswhere From_ was mistakenly over promoted). These are the rankings in version 3.11pre4:

formail -r: formail -rt:

Resent-Reply-To: Resent-Reply-To:Resent-Sender: Resent-From:Resent-From: Resent-Sender:Return-Receipt-To: Reply-To:Errors-To: From:Reply-To: Sender:Sender: Return-Receipt-To:From_ Errors-To:Return-Path: Return-Path:Path: From_From: Path:

[Stephane Bortzmeyer <bortzmeyer A T pasteur.fr>] Always use -rt and never -r . Because suchprecedence (Sender over From) is an important violation of RFC 822. There is one canonical order,described in the RFC and nothing e lse should be used, like f uzzy ranking or, worse, reordering. Thisis a se rious problem with f ormail.

The proper order is:

Reply-To, else From, else Sender, else <error>

And, ho w would yo u deal with re se nt mail?? Ie : Res ent-Reply-To , Rese nt-From, and Res ent-Sende r?

It treats Resent-X as X (" Whenever the string Resent- begins a f ield name, the f ield has the samesemantics as a field whose name does not have the prefix. "). So you have to choose an orderbetween them, the RFC does not specify it.

[david] I think that the idea is that -r is intended to determine the origination address , not the placeto reply; -rt is f or determining the place to send replies. For addressing a response, yes, -rt willinvert the header in a way more in line with the rules; f or f iguring out the origination point,

formail -r -zxTo:

might be better than

formail -rt -zxTo:

And here's an additional problem: formail -rD always uses the -r precedences; you can't make ituse the -rt precedences and the -D cache checking f unction at the same time.

4.4.4. AUTOMATIC USE OF FROM / SENDER / REPLY-TO (RFC 822 excerpt)

For systems which automatically generate address lists for replies to messages, the f ollowingrecommendations are made:

The Sender field mailbox should be sent notices of any problems in transport or delivery of theoriginal messages. If there is no Sender field, then the From field mailbox should be used.The Sender field mailbox should NEVER be used automatically, in a recipient's reply message.If the Reply-To field exists, then the reply should go to the addresses indicated in that fieldand no t to the address(es) indicated in the From field.If there is a "From" field, but no Reply-To field, the reply should be sent to the address (es)

indicated in the From field.

Sometimes, a recipient may actually wish to communicate with the person that initiated the messagetransfer. In such cases, it is reasonable to use the Sender address.

This recommendation is intended only f or automated use of originator-f ields and is not intended tosuggest that replies may not also be sent to other recipients o f messages . It is up to the respectivemail-handling programs to decide what additional f acilities w ill be provided.

11.3 Rewriting the From addressSendmail adds the From header which points to your account. But in some cases you may wish torewrite the From .

You respond to spammer and you want to hide in some extents your address . ( The headerswill still be there, but at leas t hitting r in most MUA's pick up the From )You want to rewrite From to show your virtual address [email protected] are in some other account currently, but you want to send message to some Net service(e.g Mailing list) that expects to see the same address you first time used in subscription.

You could also use Reply-To to signify where you want further responses to go, but that doesn'thide your true From address. And there are still MUAs that don't obey Reply-to . Whatever reasonyou have to rew rite From header, here is the command.

:0 fhw| $FORMAIL -r -I "From: [email protected]"

11.4 Formail -r and Resent-From header

Here is something that made me scratch my head a lot. Let's examine scenario first which explainshow the mail travels.

account --> virtual-address --> Local-address

In this chain I was sending message from my one account to another address, the virtual-addressdelivers the mail to right local domain. There is only one problem with this picture. When a responseis generated from Local-address with formail -r , the generated address pointed back to virtual-address , which pointed back to Local-address of course. A loop back was ready, you could not get the

route to travel to original address: account What was happening here was that the mail server that handled the virtual-address, didn't forward the message, but instead resent the message. In this process a set of new headers weregenerated:

Resent-From: <virtual-address>X-From-Line: <account>Received: from <the virtual-address mailserver>Resent-Message-Id: <199710151903.WAA28670@virtual-address>Resent-Date: <date>

Resent-To: <local-address>Received: ...<account domain>Message-Id: <199710151904.WAA05050@account-domain>From: <account-domain>

And now when the formail -r command was used, it picked up the Resent-From added destinationwhere the message should be returned. Surprising, but according to procmail, 100% correct. Resent-From has higher priority than From .

The Resent-* headers are considered inf ormative , and should never be used when automaticallygenerating a response. The problem here is the middleman, it should not resend a message, butrather forward it. So I put this into my .procmailrc to handle the broken middleman in our s ite.

# Remove that misleading Resent-From if it was added by our# "middleman"

:0 fhw* Resent-From: <our-domain>| $FORMAIL -IResent-From:

[edward] adds to this that: A s you know, formail -r is for composing a response to the addressf rom which an e-mail was sent. Let's say you are on vacation and have se t up a p rocmail recipe toauto respond to all e-mail you receive. Furthermore, let's say Joe sends me an e-mail and I re-send it

to you. If you wanted to respond to the sender of the e-mail that you received, w ould you e-mail meor Joe? You better e-mail me because I was the one who sent it to you. Joe may not e ven know you.Imagine if you did send your respo nse to Joe. It would probably cause him considerable conf usion asto why you are sending him e-mail inf orming him that you are vacation.

f ormail -r uses a heuristic algorithm to de termine who it should respond to , based on the presenceof various headers and their contents. If you look at the f ormail.c source code, you'll see a graphicalrepresentation of this a lgorithm.

Resent-Reply-To has the highest relative importance/reliability of all header fields. Next is Resent-From and Resent-Sender, f ollowed by Reply-To, From, Sender, et a l.

11.5 Quoting the messageUse formail -rk

11.6 Without quoting the messageUse formail -rkb or formail -rk -p ''

11.7 How to include headers and body to the reply messageThe idea is that you f irst capture whole header in a variable, then add it to the body of message.Here a custom message is added to the beginning and the headers next. Notice that the orginal

body is already adde d by rtk . Be sure to have that space inside braces; they are important.

#todo:

:0* ^^\/(.+$)+${

header="$MATCH"}

:0 fhw| $FORMAIL -r; ... now generate reply ...

11.8 Adding text to the beginning of message

We don't actually f ilter anything here. It's just a trick to reprint headers and add some text af terthem: text appears at the beginning of body.

:0 fhw| cat - ; echo "This text comes after the headers."

11.9 Adding text to the end of message

:0 fb| cat -; echo "added text after body"

11.10 Adding text before quoted messageIf you are generating an auto-reply message where you want to place the notif ication to thebeginning of body f ollowed by the quoted original message, here is recipe f or it. Substitute condition

to trigger the reply condition.

:0* condition{

:0 fhb| $FORMAIL -rk -p '>' \

-I "From: [email protected]" \-I "$MYXLOOP"

:0 fhw| cat -; echo "added message at the start of body"

}

11.11 How to truncate headers (save filing space)[Idea by Rodger A nderson <rodger A T hpbs2245.boi.hp.com>] As a last recipe, if you're tight of space, you could remove extraneous headers. But make sure you want to that, because headersmay contain use f ul inf ormation about URLs and other things like mail server address es. Some peoplekeep signature inf ormation in separate X-header (say: X-My-Info ) instead of at bottom of messageso that it won't bother people and disturb reply quoting.

# Strip header to bare minimum# If this is MIME multipart, then skip recipe

:0 fhw* ! multipart| $FORMAIL -k \

-X Date: \-X Subject: \-X Message-Id: \-X From \-X To: \-X Cc: \-X Reply-To: \-X Mime-Version: \-X Content-type:



:0 :mail.default.mbox

[david] comments the f inal recipe

You should keep the Reply-To heade r if there is one. If the sender wanted replies directed toa different address than that in the From header, you a re losing that information and, whenyou respond, writing to the wrong place.You ought to keep To and Cc so that you can tell when you read your mail who else was sentit. If your mail user agent has a group-reply or reply-all function, keeping To and Cc will allowthat feature to continue w orking. This way you are cheating yourself out o f it.'-X From' is enough to keep bo th the From_ line and the From header. You don't need to specify-X From: again after it. (To keep From_ without From: you need to say -X "From " or so methingsimilar, with a quoted space.)All mail is going to have a line (usually two) beginning 'From'.

Another slightly different approach is to kill the headers that take the most of the space. If you're notinterested in tracking down the original sender of possible UBE message , then you can remove the

Received headers. You may want to fill out the condition line to simplify only your work or campusmessages , and let other messages retain their full headers.

:0 fhw* possible-condition-to-handle-only-certain-messages| $FORMAIL -I Received:

11.12 Adding extra headers from file

[stephen] Notice that the o bvious solution won't do here:

:0 fhw* condition| $FORMAIL -r | cat - $HOME/newHeaders

The problem here is that there will be a newline in the middle, which causes the header to beshortened (procmail determines the new header/body bo undary after having processed each filter).Use the following instead.

:0 fhw* condition| $FORMAIL -r -X "" ; cat $HOME/pm-newHeaders.txt ; echo

[david] If $HOME/newHeaders ends in a blank line, you don't need the "; echo". Under somecircumstances procmail puts back the blank separating line if it gets lost, but I'm not sure exactlywhat those are, and you have a SHELLMETAS character in there already (the first semicolon), so ashell is forked anyway.

But this is my favorite way (it assumes that formail -r will never generate a continuation line forFrom:); if you use it, make sure that the new Headers file does NOT conta in a trailing blank line:

:0 fhw* whatever

| $FORMAIL -rn

:0 A fhw| sed "/^From:/r $HOME/newHeaders"

11.13 Splitting digest

[Idea by David Hunt] One interesting idea to handle digests automatically as single messages if thatwe call procmail recursively. First Ca ll f ormail to split the mail when headerfields are contained in thebody, calling procmail again as the output-program of f ormail. Insertion of X-Loop makes it possibleto reuse ~/.procmailrc f or the separate messages.

# If it more than one mail, send to formail for# splitting, then send back to procmail for sorting again.

:0* B ?? ^From [-_+.@a-z0-9]+ (Sun|Mon|Tue|Wed|Thu|Fri| Sat)* B ?? ^From:

* B ?? ^TO*$ ! H ?? ^$MYXLOOP| $FORMAIL -A "$MYXLOOP" -m4s procmail

11.14 Mailbox: Splitting to individual files[david] To split some old mail archives into individual files while stripping unimportant header f ields,use f ollowing. The keys are to use procmail's -p option, to strong-quote $FILENO in the setting of DEFAULT, and to use /dev/null or a known empty f ile as the rcf ile.

% setenv FILENO 0000% formail -kXDate: -XFrom: -XTo: -XSubject: -XIn-Reply-To: \

-XX-Mailer +1ds \procmail -p DEFAULT=`pwd`/'$FILENO.txt' \/dev/null < inputfile

11.15 Mailbox: Extracting all From addresses from mailboxThe -ns option causes formail to split the mailbox and feed ea ch mail separately to next process .

% formail -ns formail -xFrom: < mailbox | sort -u

11.16 Mailbox: Applying procmail recipe on whole mailbox

% formail -ns procmail pm-experiments.rc < mailbox

11.17 Mailbox: run series of commands for each mail (splitmailbox)

...Maybe the he at has melted my brain, but I can't se em to ge t f ormail to pe r f orm a se ries of commands on eac h mail that it has split f rom a f older. Here ' s an example of a simple de bugging attempt: I've tried pare nthe se s, putting the commands into a she ll f unction, and othe r f lailings too numerous to reme mber, all to naught.



When the file size grows to equal-to or greater-than the size specif ied on the command line, f ormailstarts over at the beginning, using a double-null to mark where it stopped. However, entries af terthe double-null, except f or the partially overwritten one, are still valid and checked, so that the f ile isthen in the f ormat:

entry\0entry\0entry\0\0partial-entry\0entry\0entry\0\0

New entries will be written after the first double-null, so that it implements a circular cache. Checkout lines 319-322 of f ormail.c

11.19 Option -D and message-id in the body S ome o f my mes sages contain the original Message -ID in the b ody of the letter and not the Heade r. Is there an option f or Formail to ove r come this problem?

[david] This is strictly untested; I don't know where in the body the Message-ID's appear, but if they're at the top of the bo dy, this might help:

:0 hW # Message-Id: in the head,*$ ^Message-Id:.*$NSPC| $FORMAIL -D $cache_size $cache_name

:0 E bW # If not but there's one the body, try body.*$ B ^Message-Id:.*$NSPC| $FORMAIL -D $cache_size $cache_name

You might want to copy a Message-Id from the body to the head in any case (if there's none alreadyin the head) just to have it in the right place, so w e could do that f irst and then formail -D will worknormally. This form will run formail twice if the Message-Id header is in the body instead of the head,but it w ill look for Message-Id on any line of the body, not just at the top:

:0 fhw*$ ! H ?? ^Message-Id:.*$NSPC*$ B ?? ^\/Message-Id:.*$NSPC| $FORMAIL -A "$MATCH"

:0 hW| $FORMAIL -D $cache_size $cache_name

11.20 Reducing formail calls (conditionally adding fields)#todo: url

Suppose you want add f ields to the message when some condition is met:

:0 # compose initial reply| $FORMAIL -r

:0* condition1| $FORMAIL -A "X-Header1: value1"

:0* condition2



| $FORMAIL -A "X-Header2: value2"

Hm, we have three processes called here, can we minimize the calls? Yes, this is idea f rom [philip]and [david] . Notice that the re is only ONE process needed.

:0

* condition1{hdr1 = "-AX-Header1:value"

}

:0* condition2{

hdr2 = "-AX-Header2: value"}

:0 fhw

| $FORMAIL -r ${hdr1+"$hdr1"} ${hdr2+"$hdr2"}

And if you want to stack all headers to only one variable, it is a bit of extra w ork. Below we use shortvariable names on ly because of the line space: the calls f it on one line.

field = all (f)ields stacked to one string.nl = continuation new line terminator of previous field

The recipe says: if field has previous value, set nl to newline separator, later concat previouscontents of field with possible newline and new header field.

field # kill variable:0{

nlnl = ${field+"$NL"}field = "$field${nl}X-Header1: value"

}

:0{

nlnl = ${field+"$NL"}field = "$field${nl}X-Header2: value"

}

:0 fhw # If we have something in *field** ! field ?? ^^^^| $FORMAIL ${field+-A"$f"}

The above recipe was the most general one, each recipe determined by itself if the f existedpreviously or not . But if you know that f is already set, you can write simpler recipe:

:0 # We know f has value before our module{

field = "$field${NL}X-Header1: value"}

11.21 Formail -A -a optionsYou can't use option -A with -a or -I if the header name is the same. Like below where you try tokeep only the last def inition of X-1, but the first -A isn't seen whe n -a is app lied.

formail -A "X-1: 1" -a "X-1: 2"-->X-1: 1X-1: 2

Whereas; separate pipes give you the des ired results.

formail -A "X-1: 1" | formail -a "X-1: 2"-->X-1: 1

formail -A "X-1: 1" | formail -I "X-1: 2"-->X-1: 2

11.22 Formail -e -s options[david] I had a f ile of alternating From and Date lines and w anted to convert it into an mbox.

formail -dem2 -s < input > mailbox

should have done it, right? Nope; formail -s took it all as one message, even with -m1. When Iedited in blank lines, the command worked. My f irst reaction was that the -e option wasn't workingas advertised and that the blank lines were necessary af ter all.

Then I realized the real problem: there was no interruption in the succession o f valid header lines inthe input for anything that could look like a body. I could have put something other than blank lines

between each pair of header f ields and then -e would have done its job, but as long as everyadditional line looked like a valid RFC822 header f ield, even if its name was the same as one thathad appeared earlier, formail -s assumed that it was s till the same message 's head.

12.0 Saving mailing list messages12.1 Using subroutine pm-jalist.rc to detect mailing listsBecause I didn't have sendmail plus addressing capabilities (explained in next section) I wrotemodule pm- jalist.rc . It is included in the pm-code.zip

The subroutine tries to detect and derive the mailing list name directly f rom the message. ManyMailing daemons: ezlm, smarlist, listserv, majordomo use standardized headers f rom where the listname can be picked. A fter this subroutine has been applied to message, the variable LIST containsthe mailing list name. You no longer have to manually insert separate recipes f or each new mailinglist you subscribe to , because this s ubroutine adaptively f inds new new mailing lists.

Once the mailing list name has been grabbed, you can easily "map" or convert the name to anysuitable f older name before saving it:

LIST LIST name Description of mailing list(as grabbed) you want--------------------------------------------------------------jde java.jde Java Development Envjava java.prog Java programmingFLAMENCO flamenco Flamenco musictango-l tango Argentine Tango dancingtm-en-help tm-en Emacs TM mime package mailing listw3-beta w3 Emacs WWW mailing list

You set then conver grabbed LIST to new f older name with conversion table:

JA_LIST_CONVERSION = "\jde java.jde,\java java.prog,\FLAMENCO flamenco,\"

And to detect all mailing lists, you only need one recipe, like below:

INCLUDERC = $PMSRC/pm-jalist.rc

:0 : # if list name was grabbed

* ! LIST ?? ^^^^$LIST_SPOOL_DIR/list.$LIST

12.2 Using plus addressing [email protected] you have a recent enough (8.8.8+) sendmail , please ask your sysadm to activate the plusaddressing. Procmail gets bar in $1 automatically.

http://www.f aqs.org/faqs/mail/addressing/

[Bennett Todd <bet A T mordor.net>] The PLUS feature has also been Implemented in qmail andPostf ix (nee VMailer). By def ault qmail uses "-" rather than "+", but it can be conf igured to usedif f erent rules; Pos tf ix doesn't come with either enabled, but its example main.cf has a commented-out line to enable "+"-based support.

[Roy S. Rapoport <rsr A T macromedia.com>] Plus addressing is implemented using sendmail (well,I'm sure the o ther MTA s can a lso do it, but my experience is with sendmail). The last few releases of sendmail (8.8.6, 8.8.7, 8.8.8) all seem to automatically def ault to allowing it. Basically, f or anyaddress of the form foo+baz , sendmail ignores the +baz part and just delivers it to foo .

If you w ant the easiest method to handle mailing list mails, then subscribe to list by using ded icatedplus address:

[email protected][email protected][email protected]



When you receive message f rom any of these mailing lists to your login account, the list.procmailis already in variable $1 and the recipe to sink all mailing lists to their individual f olders is very s imple:

# Note: The $1 contains value only _IF_ procmail# is invoked with option -m or -a (with an argument).# Be sure procmail is invoked with that oprion either as from# LDA or ~/.forward.## $1 is pseudo variable and it can't be used in condition line,# so we copy the value to ARG.

ARG = $1

:0 :* ARG ?? list$ARG

[david] Here is what I have configured to sendmail.cf to support plus addressing:

Mprocmail, P=/usr/bin/procmail, F=DFMmShu, \S=11/31, R=21/31, \T=DNS/RFC822/X-Unix, \A=procmail -m $h $f $u

Well, this is def inition of the procmail mailer, not the local mailer. Furthermore, there's more to plus-address ing support than the def inition of the local mailer. Ruleset 0 or 5 needs to be set up to moveeverything af ter the + into the 'host' variable ($h). Unless you have a strong understanding of

sendmail rule sets and rewriting rules, you should not attempt to add plus-addressing to yoursendmail.cf, but instead just install the latest version of sendmail and use the m4 sendmail.cf generation tools with a .mc f ile tha t contains:

FEATURE(local_procmail, `/usr/local/bin/procmail')

plus whatever else your site requires.

...Ok, I corrected it. Well, here's what that looks like. I didlook into the part about Ruleset 5 while trying it onoriginally. But all I could do was make sure that theplus-addressing section was there.

Mlocal, P=/usr/bin/procmail, \F=lsDFMAw5:/|@qSPfhn9, S=10/30,

R/40,T=DNS/RFC822/X-Unix,A=procmail -Y -a $h -d $u

Mprog, P=/bin/sh, F=lsDFMoqeu9, S=10/30, R/40, D=$z:/,T=X-Unix,

A=sh -c $u

12.3 Using RFC comment trick for additional information



Recall f rom [rf c1036] that the pref erred Usenet mail address f ormats a re f ollowing

From: [email protected]: [email protected] (First Surname)From: First Surname [email protected]

I invented this idea after reading Eli's excellent FAQ about mail addressing. Please read it (especiallysection 19.) bef ore you continue in order to understand what I'm going to present.

I have an account which does no t support plus address ing and I was kinda jealous to everyone thatcould use this neat sendmail addressing scheme. The plus addressing helps so much better to dealwith mailing list messages.

But as it turns out, we can simulate in some extent plus addressing with pure RFC compliantaddress . We exploit RFC comment syntax, where comment is any text inside parentheses. A ccordingto Eli's paper, comments sho uld be preserved during trans it. T hey may no t appear in the exact placewhere originally put, but that shouldn't be a problem. So, we send out message with f ollowing Fromor Reply-To line:

first.surname@domain (First Surname+list.procmail)

Now, when someone replies to you, the MUA usually copies that address as is and you can read inthe receiving end the PLUS inf ormation and drop the mail to appropriate folder: mail.procmail .

[About subscribing to mailing lists with RFC comm ent-plus address]

It's very unf ortunate that when you subscribe to lists, the comment is not preserved when you'readded to the list database. Only the address part is preserved. I even put the comment inside

angles to f ool program to pick up everything between angles.

first.surname(+list.procmail)@example.com

But I had no luck. T hey have too goo d RFC pa rsers, which throw away and clean comments like this.Eg. procmail based mailing lists , the f amous Smartlist , use formail to de rive the return address andformail does not prese rve comments. The above ge ts truncated to

[email protected]

Also many mailing lists send ou t messages as Bcc , so your address is no t even available in headersanywhere, neither is this nice RFC comment. Ah well, but this RFC comment trick works very well inprivate communication, virtually all MUA s copy whole contents of a From or Reply-To header to Toheader, preserving comments and you get the benefit of plus addressing. Here is procmail code todemonstrate reading the PLUS inf ormation f rom RFC comment-plus field:

RC_EMAIL = $PMSRC/pm-jaaddr.rc # Address explode module

:0*$ To:\/.*{

INPUT = $MATCHINCLUDERC = $RC_EMAIL # Explore grabbed To address

# If COMMENT_PLUS was defined, module found "+"# address which contained, say, "mail.procmail".# Save it to folder.

:0 :* COMMENT_PLUS ?? [a-z]$COMMENT_PLUS

}

Pretty simple. A nd you can put anything inside RFC comment and do whatever you w ant with theseplus addresses. NOTE : there are no guarantees that the RFC comment is preserved every time.Well, the standard RFC822 says is must be passed untouched, but I'd say it is 90% of the caseswhere mail is delivered from one server to another, it is kept.

Example: if you discuss in Usene t groups, you could use address

[email protected] (First Surname+Usenet.default)

[email protected] (First Surname+Usenet.games)[email protected] (First Surname+Usenet.emacs)[email protected] (First Surname+Usenet.linux)

12.4 Simple mailing list handling[Peter S Galbraith <galbraith A T mixing.qc.dfo.ca>] I have used this in the past (by simply looking atthe spool f ile and seeing the From_ line of the message):

:0 :* ^From debianlist.debian.mbox

:0 :* ^From procmaillist.procmail.mbox

Now, I collect specif ic high-volume mailing lists (like Debian) into their o wn spool f iles like above, andlet othe r recipes catch all other mailing lists (like procmail and f vwm) into a single spool file with laterrules:

:0 : # Majordomo lists* ^Sender: owner-\/[-a-zA-Z0-9_.]*list.$MATCH.mbox

:0 :* ^X-Mailing-List: <\/[-a-zA-Z0-9_.]* # SmartList listslist.$MATCH.mbox

So Debian mailing list mail goes to Debian, procmail and fvwm mail go to mail lists and mailaddressed to me yet CC'ed to a list go to my main spoo l f ile.

12.5 Archiving according to TO

Traditional way to detect and save mailing list messages is:

:0 :* ^TO()procmaillist.procmail

[and so on...]

The f ollowing code will save the message to f olders list.foo, list.bar, list.procmail when the name is inthe TO address .

# generalised version By David W. Tamkin. Cases desired# for foldernames

LISTS = "(foo|bar|procmail)"

:0:

*$ ^TO_()\/$LISTS*$ LISTS ?? ()\/$\MATCHlist.$MATCH

12.6 Using Return-Path to detect mailing lists[philip] For most mailing lists, a more accurate way to determine whether it came f rom the list is toexamine the Return-Path:, From_ or Resent-From: header. This catches messages f rom the list,regardless of whether they were To: the list, Cc: the list, or even Bcc: the list, something whichdoesn't show in the message a t all.

For instance, I ref ile message f rom the procmail mailing list using the f ollowing recipe:

:0* ^Return-Path: +<procmail-request@informatik~/Lists/procmail/.

There's one tricky thing to note: if someone sends a message to both me and the list (say,responding to a message I sent to the list), then the copy that go t to me through the list will end upin my procmail folder, while the copy that went directly won't. I like this behavior, but some people,poss ibly yourself , may pref er it if both messages end up re-f iled. If so, your best bet is to combinethe above w ith matching aga inst the To: and Cc: headers via the ^TO_ token:

:0* ^Return-Path: +<procmail-request@informatik|\^TO()_procmail@informatik~/Lists/procmail/.

(If you have a version of procmail bef ore 3.11pre4, then you'll need to use "^TOprocmail" instead of

"^TO_procmail".). If you're subscribed to many mailing lists, here is one general recipe

Notice : you don't want to include < in the recipe like: ^TO_\<\/$LISTS because The ^TO_ tokencontains something similar to \< but better, so that the \< can only cause problems. A trailing \> isnot a bad idea, though because it's not a zero-width as sertion but rather an actual character class ,you have to s trip it f rom the match

LISTS = "(foo-list|bar-list)"

# 1) to get the match# 2) rematch sans the trailing \># 3) Note: preserves capitalization of the string

:0

*$ ^TO_()\/$LISTS\>*$ MATCH ?? \/$LISTS*$ LISTS ?? ()\/$\MATCH{

M = $MATCH<action>

}

[Era] gives this sample example to describe what happens above:

VAR = "MOO"what = "(moo|bar|baz)"

:0 # Search what from VAR*$ VAR ?? ()\/$what{

# Now; what is was that really matched, there were several# choices: moo,bar,bar# Beware: $MATCH must not contain regexp characters

:0

*$ what ?? ()\/$MATCH{ } # no-op

# Fine, New MATCH contains moo}

13.0 Procmail, MIME and HTML13.1 Mime content type application/ms-tnef

...A member of one o f my mailing lists appears to be using Microso f t Mail. His me ssage s to the list are usually accompanied my anenco ded attachme nt like this one: "c:\eudora\use rs\stev en@idm a.com\attach\W INMAI L11.DAT" T he me ssage he aders includethe f ollowing clause : Content-T ype: multipart/mixed; boundary="openmail-part-058c9 f 3d-00000001" T his is driving peoplecrazy. What is causing this and is there any way to make it stop?

Most likely the sender is using Exchange (or Windows Messaging or Outlook97) and sent themessages in Rich Text Format. It puts the RTF message in an attachment called WINMA IL.DAT(application/ms-tnef ). But this a ttachment is useless unless the recipient is also using Exchange.

The sender can turn off the RTF option f or messages to you. For more inf ormation, see: "XCLN:Sending Messages In Rich-Text Format" at http://support.microsof t.com/kb/136204

13.2 Trapping HTML mime messages

[era] Here's a simple f ilter to throw out unwanted HTML that is sent by using mime. [jari] This recipedetects if the message is classif ied as mime text/HT ML and junks it to separate f older. It does notchange the message content. If you want to actually remove HTML or other attachments f rom themessage , see pm-jamime-kill.rc in the module list.

:0:*$ ^Content-Type:$s*multipart/(mixed|alternative);\

$SPCNL*boundary="?\/[^;"]+*$ B ?? ^--$\MATCH\$([-a-z]+:.*)*Content-type:$s*text/HTMLjunk.html.mbox

Some more examples can be f ound f rom section: 'Explaning ^^ and ^'

13.3 Complaining about HTML messages[Marek Jedlinski <eristic A T gryzmak.lodz.pdi.net>] . This how I respond to HTML messages. In mynoHTML.txt I politely explain why I don't appreciate receiving HT ML mail, and ask to resend themessage as plain text. What happens in the majority of cases is that the sender resends the samemessage again ("oh, it bounced, let's try again") and I assume they don't actually read myexplanation since they just happily resend the HTML cr*p. It bounces again at which point they giveup... Tough luck, I say ;)

BTW, the above recipe is placed af ter mailing list mail gets sorted. When someone sends HTML mailto a mailing list I read, I just flame them in person

TXT_NO_HTML = $HOME/noHTML.txt

:0* ! H ?? ^FROM_DAEMON*$ ! H ?? ^$XLOOP* HB ?? ^Content.Type.+multipar t.alternative* HB ?? ^Content.Type.+text.htm l{

LOG = "$NL --TRASH: multi-part HTML $NL"

:0| ($FORMAIL \

-rk \-A "X-Mailer: Procmail Autoreply" \-A "$XLOOP" ; \

cat $TXT_NO_HTML \) | $SENDMAIL

}

13.4 Converting HTML body to plain textThe most popular solution to convert HTML body into plain text is to use lynx . Another morestraightf orward method is to use a perl one liner: it's quicker, easier to use with procmail but itdoesn't pretend to know about HTML DTD. The recipe below should be taken with grains of salt:seeing HTML tag is no guarantee that the body "only" has HTML. A cautious recipe writer alsowatches f or MIME multi part messages. (See pm-jamime.rc to draw some mime characteristics f rommessage)

This recipe has been written so that you can add more alternative HTML conversion scripts. You mayeven w ant to select the appropriate conversion for a message: e.g perl f or unimportant ones.

Note : This is oversimplified method of checking if body contains HT ML. It would be probably a good

idea to check mime headers which indicate HTML encoding here as well.

:0* B ?? ()<HTML>* B ?? ()</HTML>

message.Lotus Notes sends similar extra attachment.Microsoft Express sends a copy of message in HTML format in the attachment.Netscape's Mozilla sends a copy of message in HTML. See example. It Also sends annoyingvcards .

The module is called pm-jamime-kill.rc and included in Jari's pm-code.zip . (Note: Procmail modulelist )

13.6 Sending contents of a HTML page in plain text tosomeone[timothy] Send an mail with the subject: "GetPage: some.url.here/". And it comes back. Kurt Thams<thams A T thams.com> also pointed out that lynx allows file:// protocol and since procmail isrunning as you, this w ould be a security risk.

GetFile: ~user/.login

We make the script safe here by forcing http://$MATCH and not s imply using "$MATCH"

:0*$ ^Subject:$s+GetPage:()\/.**$ ! ^$MYXLOOP| ($FORMAIL \

-r \-I "Precedence: junk" \-I "Subject: Requested page: $MATCH" \-I "$MYXLOOP" ; \

lynx -dump " http://$MATCH&quot ; \)| $SENDMAIL

[era] If all you need is to create a suitable MIME package, there are various MIME command-lineutilities such as metase nd (which is for interactive use , and so doesn't work very well with Procmail)and mpack you can try. If your needs are simple, you could even read up a bit on the MIME spec andgenerate the necessary headers and separators yourself (echo Content-Type: multipart/mixed etcetc etc). Conversely, if your needs are complex, get the Perl MIME package from CPAN and cook upyour own tool. The MIME FAQ (especially part 6) is a good place to look for info.http://www.faqs.org/faqs/by-newsgroup/comp/comp.mail.mime.html

14.0 Simple recipe examples14.1 Saving: MH folders – numbered messagesHm. This is explained in the procmail man pages, but not very well. There are just one or twooccasions where the man page tells how to create individual files instead of catenating messages toa folder. Notice the /. at the e nd of folder name

:0* condition

dir-folder/.

[manual] When delivering to directories (or to MH folders) you don't nee d to use lockfiles to preventseveral concurrently running procmail programs from messing up.



On a save to a direc tory, how doe s procmail de termine what to put a f ter $MS G PRE FIX to co mple te the name o f the f ile?

[philip] It's the inode number of the f ile encoded in base-64 with the set of characters A -Za-z0-9-_, inreverse order. So, f or example, the inode numbered 59699 would be encoded a s f ollows:

59699 = 51 + 64 * ( 36 + 64 * 14 )A=0, B=1, ..., N=13, O=14, ..., a=26, ..., k=36, ..., z=51,0=52, ...

--> zkO

14.2 Saving: to monthly folders

# Use any date method mentioned previously to define variables# YYYY YY MM DD. Archive digests monthly

:0 c:* ^From:.*\/[email protected]

{# Get the "mailing-list-digest" string, do not use following## MBOX = `echo $MATCH | sed -e 's/@.*//' `## Because we really don't need those extra shell processes.# Procmail can derive the word 10x more efficiently

:0* MATCH ?? ()\/[^@]+{

MBOX = $MATCH}

:0 :$YYYY-$MM-$MBOX

}

14.3 Modifying: Filtering basicsPay attention to the cat command position in each recipe.

:0 fbw| echo "This is a line of text _before_ the body"; \

cat -

:0 fbw| cat - ; \

echo "This is a line of text _after_ the body"

:0 fbw # prepend text before the body| cat msg.txt -

:0 fbw # append text at the end of body| cat - msg.txt

:0 fbwi # replace the body with text from file| cat msg.txt



14.4 Modifying: Squeezing empty lines around message body[david] Anything that replaces the body is going to require an outside process, even if it's only

/bin/echo. In order to trim empty lines from the beginning of message and f rom the end of message,you can do this, if the entire body fits into LINEBUF

:0 fbw* B ?? ^^$*\/.(.|$)*.$| echo "$MATCH" # trailing extra newline intended

If your version o f cat is BSD-ish,

# SysV's cat has a different meaning for -s and cannot do this

:0 fbw

* B ?? $$$| cat -s

otherwise , it can be do ne w ith a very simple sed f ilter:

:0 fbw* B ?? ^^($)|$$$| sed /./,/^$/!d

Note that cat -s has slightly diff erent results f rom the others: if there a re any empty lines a t the topof the body, cat -s will keep one. The echo and sed suggestion will remove all empty lines from thetop and, like cat -s, keep one a t the bottom.

14.5 Modifying: shuffling headers always to same order[phil] To sort the headers in the message into predictable order, you can use following recipe. Thespaces have been eliminated between the -I and its argument in the above. The shell may or maynot allow unquoted spaces in the second part of the ${variable:+blah}. For example, under solaris2.6, /bin/sh barf s on ${FROM:+-I "From: $FROM"}, while /bin/ksh handles it just f ine. I think thePOSIX shell standard requires that it be allowed, but, well, will your next system be POSIXcompliant?

:0* ()\/^From: +\/.*{

FROM = $MATCH}

:0* ()\/^Reply-To: +\/.*{

RT = $MATCH}

:0* ()\/^X-Mailer: +\/.*



{XM = $MATCH

}

:0* ()\/^Message-Id: +\/.*{

MID = $MATCH}

:0* ()\/^Date: +\/.*{

DATE = $MATCH}

:0* ()\/^To: +\/.*{

TT = $MATCH}

:0* ()\/^CC: +\/.*{

CC = $MATCH}

:0* ()\/^Subject: +\/.*{

SUBJ = $MATCH

}

:0 fh w| $FORMAIL \

${XM:+-I"X-Mailer: $XM"} \${TT:+-I"To: $TT"} \${FROM:+-I"From: $FROM"} \${RT:+-I"Reply-to: $RT"} \${CC:+-I"Cc: $CC"} \${MID:+-I"Message-Id: $MID"} \${DATE:+-I"Date: $DATE"} \${SUBJ:+-I"Subject: $SUBJ"}

14.6 Service: Auto answerer to empty messages[elijah] Here is piece o f code that responds to empty messages .

:0* ! B ?? ...| (echo "From: [email protected]" ; \

$FORMAIL -r -A"Precedence: junk" \

-A"X-Loop: [email protected]" ; \echo "Your blank message was received.\n" \

"Did you mean to say something?\n" \"\n" \"-- \n" \"My Signature\n" \



"this has been an automated response\n" \) | $SENDMAIL

14.7 Service: Ping responderSometimes I'm on the road and I don't s eem to get access to the site w here my messages are. Thetelnet connection f ails and standard Unix "ping" plays dead for me. "What's happening in that site?"

I wo nder. Here is a recipe that I have added to a ll of my accounts. It sends an immediate reply if atleast the mailhost is up and gives so me s tatus information.

:0* ^Subject: ping${

:0 fh| $FORMAIL -r

# Remember, Don't send back anything that would be vital to# attacker. It doesn't matter if the `uptime` or other

# scripts fail, the reply is sent anyway.

:0 c # Record this ping request| ( cat -; \

echo `uptime`; \echo "$HOST User count: " `who | wc -l`; \

) | $SENDMAIL

:0 : # or sink to $DEFAULT$PING_SPOOL

}

14.8 Service: simple vacation with procmailDon't f orget to look into procmailex(5) man pages which also has vacation example. The onespresented below may not work for you. Here is a very simple vacation recipe. Whenever the f ile~/.vac exists, the vacation program is called. Be sure that you have the ~/.vacation.msg file readytoo . Remember that vacation does not save you messages ; so we need c f lag here.

# Some prefer the non-dotted file which shows up in ls listing

vacationFlagFile = $HOME/.vac

:0 wc*$ ? $IS_EXIST $vacationFlagFile| vacation $LOGNAME

Some people like to raise a flag in .procmailrc instead of creating a file. If you like the variableapproach better, here is the equivalent implementation of the above

VACATION = "yes" # Comment this when not in vacation

:0 wc* VACATION ?? yes| vacation $LOGNAME

[philip] and [era] Since vacation only sends replies – it never sends the original # messages, oneway to do two things with your .f orward f ile. Substitute "abc" with your login name.

|/usr/ucb/vacation","exec /usr/local/bin/procmail -f- ||exit 75 #abc

14.9 Service: vacation code example[By Eric Black <eric A T Mirador.COM>] Here is the procmail part

OFFSITE = "[email protected]"

# Forward urgent mail to me at my off site address; afterward,# continue processing it as normal The procmail pattern match# may be case-insensitive, in which case this rule could be# simplified...

:0 c* ^Subject: .*urgent| $SENDMAIL $OFFSITE

# Use "vacation" to tell other people I'm not here To enable,# un-comment the next two lines; to disable, comment them out## The -a Identifies another name that can legitimately# appear in the To: line of the mail header instead# of your login name

:0 wc| vacation -a ericb eric

And he re the ~/.vacation.msg file

Subject: I'm out of town for a whileFrom: eric (via the vacation program)

I'm out of town until <return-date>. Your mail regarding"$SUBJECT"

will be read when I return, or possibly at some unknowntime before then if I get a chance to check for mail.

If your message must be seen by me before I return,you can send it with the word "URGENT" in the subject header.Such mail will be automatically forwarded to me so thatI see it sooner.--Eric

14.10 Service: Auto-forwarding[timothy] I have my .procmailrc se tup to f orward mail to another (mail only) account. When I am notgoing to be a t the account, I want to turn f orwarding off

# look for the file to tell us whether or not to forward mail# if the file exists, forward the mail# or not

ELSWHERE = "[email protected]"FILE = "$HOME/.forwardmail"

:0 c*$ ? $IS_EXIST $FILE! $ELSWHERE

# if a message arrives from the other account# with the Subject 'forward-off' then remove the# file, efectively turning off forwarding

:0 hwic*$ ^From:.*$ELSWHERE* ^Subject: forward-off| $NICE mv -f $FILE $FILE.off

# if a message arrives from the other account# with the Subject 'forward-on' then remove the# file, efectively turning off forwarding on

:0 hwic*$ ^From:.*$ELSWHERE* ^Subject: forward-on| $NICE mv -f $FILE.off $FILE

14.11 Service: forward only specific messagesHere is piece of code that triggers f orwarding according to addresses. If you have lot of these kind of f orwarding, you should use simple awk database which you would grep.

# By Jim Hribnak <hribnak A T nucleus.com># [email protected] goes to [email protected]# [email protected] foes to [email protected]

:0* ^TO_()[email protected]\>{

FORWARDTO = "$FORWARDTO [email protected]"}

:0* ^TO_()[email protected]\>{

FORWARDTO = "$FORWARDTO [email protected]"}

:0 fhw* FORWARDTO ?? @* ! ^$MYXLOOP

| $FORMAIL -A "$MYXLOOP"

:0 a! $FORWARDTO

14.12 Service: Making digests

# By <jimo A T eskimo.com># Add this message to the digest accumulator

:0 c:| $FORMAIL -k -X From: -X Message-Id -X Date -X Subject >> $DIGEST

# Check size of digest, and send it off if it's big enough

:0*$ -$DIGSIZE ^0*$ `wc -l <$DIGEST` ^0| $NICE send-digest $DIGEST

14.13 Kill: killing advertisement headers and footers A mailing lis t that I subs crib e re ce ntly began adding a blo ck o f "boiler plate" text to the beginning and end of eve ry message that

goes through the list (groan). The text is always the same , and is always at the beginning and end of the message .

[david] sed could do both at once, but the problem is that sed never knows when it is N lines f romthe end if N>0; it know s the last line w hen it reads it, but when it is looking a t the next-to-last line itdoesn't know that there is only more one line to come. It does, however, know how many lines of input it has a lready read.

So I have three suggestions: if you know that the header is X lines long [let's say 5 f or this example]and that the f irst line of the footer contains some string or pattern that will not occur in thesignif icant part of the post,

:0 fbwi* conditions| sed -ne 1,5d -e '/pattern/q' -e p

If you recognize the end by the last line that you want to keep instead of the f irst line that you wantto delete, omit the n option and the p instruction:

| sed -e 1,5d -e '/pattern/q'

Finally, if the only reliable w ay to spo t the footer is by reaching so many lines f rom the end (becauseany search pattern might occur in the real text as well), we can score as you've been doing to getthe number of the last significant line. Let's say the f ooter is three lines long; because ^.*$ alwayscounts one line too many (long story), we subtract four instead of three:

:0 fbwi* conditions* 1^1 B ?? ^.*$* -4^0| sed -e 1,5d -e "$="q

14.14 Kill: simple kill file recipe with procmailKill f iles are widely used with news readers to delete uninteresting posts when you enter a

newsgroup. A kill file usually contains one single entry per line to match the message content andthis can be easily done with procmail. Remember however that f or every message procmail forks aprocess, so before you apply the kill file rules to the messages, be sure your recipes are in thisorder: the kill f ile rules are applied only to unknown messages

SINK MAILING-LISTSSINK ANNOUNCEMENTSSINK WORK MESSAGESOTHER DELIVERIESapply kill file rules and UBE recipes to the rest

Recipe w ill drop the message (i.e. cons ider it 'delivered ') if one of its headers matches a pattern in killf ile.

:0 hW: $HOME/.kill file$LOCKEXT| egrep -i -f $HOME/.kill file

The reason why there is explicit lock f ile is that you must be able to update the kill f ile while yourprocmail is running. A n example edit script is presented below.

#!/bin/sh# program: kill file.sh#file=$HOME/.kill filelock=$file.lockcp $file $file.tmp

emacs -q $file # or use whatever you prefer: vi, picolockfile $lockmv $file.tmp $filerm -f $lock

14.15 Kill: duplicate messages[Lars Kellogg-Stedman <lars A T bu.edu>] Put this a s a f irst entry in your .procmailrc and you won'tsee any duplicates as long as the 8K cache doesn't get full. The duplicates f older is cleaned outweekly via a cron job. While it may be tempting to simply sink duplicates to /dev/null, I have come

across broken mail clients the stick the same value in the Message-id header of all outgo ing mail.

:0* ^Subject:\/.*{

SUBJECT = $MATCH}

MID_CACHE_LEN = 8192MID_CACHE_FILE = $PMSRC/msgid.cacheMID_CACHE_LOCK = $PMSRC/msgid.cache$LOCKEXT

LOCKFILE = $MID_CACHE_LOCK

# IF the message has a message-id header# AND formail -D is successful (exit status=0)# THEN

# log a message to the procmail log# sink the message

:0* ^Message-Id:* ? $FORMAIL -D $MID_CACHE_LEN $MID_CACHE_FILE{

LOG="dupecheck: discarded message, $SUBJECT $NL"

:0 # Store duplicates, notice no lock!duplicate.mbox

}

LOCKFILE # Release lock by killing variable

And here is a bit simpler recipe, a slightly modif ied version f rom the [manual] . Procmail noticesf ormail's success, considers the message delivered and does not stop processing the rcf ile due to cf lag, which let's a message to f all into safety copy inbox.

:0 hWc: $PMSRC/pm-msgid.cache$LOCKEXT* ^Message-id:| $FORMAIL -D 8192 $PMSRC/pm-msgid.cache

:0 a:duplicate.mbox

There was a pretty heavy thread around September 1997 about duplicate detection, where somepromising stuf f was posted. One item you should def initely have in your collection is Eli's hashd [in

Procmail mailing list 1997-09]

14.16 Kill: spam filter with simple recipes[Ed McGuire <emcguire A T i2.com>] Seeing several junk mail f ilters posted recently, varying f rom thesimple to the co mplex, I thought I would a lso share my o wn. I junk whatever comes f rom my ISP butis not addressed to my domain or to one of the mailing lists I subscribe to.

# 1. mail to my domain# 2. NOT addressed to me directly# 3. NOT coming from mailing lists I'm subscribed to.

0:* ^(received):.*psi\.com* ! ^((apparently-)?to|cc):.*(i2|intellection)\.com* ! ^(to|cc):.*(pdp-?8-lovers|procmail|sunshine|info-pdp11)junk.ube.mbox

[Gordon Matzigkeit <gord A T m-tech.ab.ca>] I have just discovered an ef f ective rule f or separatingSPA M from the rest o f my e-mail. Just substitute your username f or gord in the line below

# Anything which is not addressed to me is probably SPAM.:0:* !^TO().*\<gord\>junk.ube.mbox

This only works because I handle all mailing list address es abo ve that point in my .procmailrc (i.e. alltraff ic that arrives f rom mailing lists that I am subscribed to goes into other folders). Most SPAMmersseem to do it nowadays by sending mail via mailing lists, rather than creating huge To lists of users

Many times sysadm install a list of know addresses that send spam and then they check theincoming mail against the "black list". Keep in mind that that some fgrep implementations have aproblem with the -w wo rd switch. Note that the above recipe scans the FULL HEA DER, so use it withsome caution, i.e., be caref ul what you add to your list of spam domains.

# by [philip]; egrep would do here too, if it is posix# compliant, it may have -f switch that makes it behave# like fgrep.## Note: option -F would make [ef]grep to search fixed string# instead of regexps.#

BLOCK_FILE = $HOME/Mail/DeniedNames.ls tUBE_MBOX = $HOME/Mail/junk-ube.mbox

# To filter out the Subject lines, so that mails sent# with the subject "Have you received a message from# blah-blah@spam" don't get filtered.# [era] suggested we use formail## Edsel Adap <edsel.adap A T Canada.Sun.COM> agrees there is a# likely bug in Solaris 2.5.1 "/usr/bin/fgrep -i" and# suggested the use of /usr/xpg4/bin/fgrep instead.## <edsel.adap A T canada.sun.com> Sun Microsystems Developer Support

# Files in /usr/xpg4 are available via the SUNWxcu4 package,# which is part of the user, developer, all, or Xall Solaris# clusters.## Solaris 2.4 doesn't have /usr/xpg4/bin/fgrep :-(, you# must use `tr A-Z a-z' before piping the message to fgrep.

:0 hw:*$ ? $FORMAIL -ISubject: |fgrep -i -f $BLOCK_FILE$UBE_MBOX

The f ile DeniedNames.lst is s imply a list of addresses

[email protected]@[email protected]@dm1.example.com

14.17 Kill: (un)subscribe messages I ' m ge tting tire d o f those pe sky (un)subsc ribe me ssage s that ce rtain "other" mailing lists se em to pass through to the list at largeinste ad of capturing them at the list serve r, like S martList does .

[Adam Shostack <adam A T bwh.harvard.edu>] The f ollowing do help, although they're of ten toobroad. (I use a .saf e rule to cover those cases) The < 1000 is a useful hueristic. It's rare thatunsubscribe messages a re long.

:0 :* (Delete|u*n*Sub(s| )*| add | leave | help )* < 1000junk.misc.mbox

[Rodger A nderson <rodger A T hpbs2245.boi.hp.com>] I've been working on a recipe to f ilter outthose pesky s*bscribe and uns*bscribe messages from mailing lists, and I'm posting what I have sof ar. As an aside, it also filters out very short messages, which I've f ound are usually some sortmessage meant for list owner/request address.

I give heavy weight to Subjects starting with (un)?s*bscribe, with also pretty heavy weight toSubjects containing either of those words. I then give heavy weight to the body of messagesstarting with those w ords, and a lighter weight to lines s tarting w ith them. Then multiple occurrencesget some weight too, up to a point. Then I count the wo rds in the message against all that.

:0* 1^0

* 30^0 H ?? ^Subject: +(un)?subscribe\>* 20^0 H ?? ^Subject:.*\<(un)?subscribe\>*$ 20^0 B ?? ^^$SPCNL*(un)?subscribe\>*$ 10^0 B ?? ^$SPC*(un)?subscribe\>* 8^.4 B ?? \\<(un)?subscribe\>* -.4^1 B ?? \\<$a+\>junk.misc.mbox

[Adam Shostack <adam A T bwh.harvard.edu>] How about looking f or sub & unsub, as well as aperennial misspelling 'unsuscribe me'? I also find filtering on add, leave and help to be useful. Thismay well be the o nly word on the line. I think it has to do w ith broken list management packages.

| :0| * 1^0| * 30^0 H ?? ^Subject: +(un)?subscribe\>

* 20^0 H ?? ^Subject: +(un)?sub?(scribe)?\>

(The B is often missing, as is the word fragment 'scribe')

| * 20^0 H ?? ^Subject:.*\<(un)?subscribe\>

* 20^0 H ?? ^Subject: +(add|leave|help)$

# fewer points if more words

* 15^0 H ?? ^Subject: +(add|leave|help)

[david 1998-10-20] You want to match on messages w here the f irst non-blank thing in the body is"unsubscribe" at the end of a line, where there are f ive lines or f ewer in the body?

:0*$ B ?? ^^$SPCNL*unsubscribe$* 7^0* B ?? -1^1 ^.*$junk.misc.mbox

And if your cron do esn't know the HOME variable (that'd be an exception)

0 23 * * * /bin/csh -c 'touch ~LOGNAME/.mail.relay.on'0 7 * * * /bin/csh -c 'rm -f ~LOGNAME/.mail.relay.on'

Then, in your .procmailrc do:

:0 c* ^From.*some-address*$ $IS_FILE $HOME/.mail.relay.on| command

the script will run_my_program only if both the subject matches and the f ile tes t succeeds. The f iletest w ill succeed o nly between 11pm and 7am.

In all honesty, if system gives usable From_ lines, I like f ollowing suggestion better. I use it all thetime to turn blocks of procmail code on and of f at given times or dates, and it works likes a charm. Ituses many f ewer processes and is less likely to get the status wrong if f or any reason one of thecron jobs fails to run o r doesn't do its job.

This pages only at day time

:0 c* ^From .*some-address.* (0[789]|1.|2[012]):[0-5][0-9]:| command

This pages at night

:0 c* ^From .*some-address.* (0[0-6]|23):[0-5][0-9]:| command

14.21 Decoding: Uudecode[philip] here is piece of code to do uudecode match when certain condition is matched. The magicstring here is "begin ...file", the body is then fed to my_uudecode_program whatever it does to it.

:0 b* ^From:.*someone@example\.com* ^Subject: Subject* B ?? ^begin 644 file.tar.gz| my_uudecode_program

14.22 Decoding: MIME

# by Peter Galbraith <galbraith A T mixing.qc.dfo.ca># MIME filtering of accented characters and split lines.#

:0* ^Content-Type: *text/plain{

:0 fbw* ^Content-Transfer-Encoding: *quoted-printable| mimencode -u -q

:0 A fhw| $FORMAIL -I "Content-Transfer-Encoding: 8bit"

:0 fbw* ^Content-Transfer-Encoding: *base64| mimencode -u -b

:0 A fhw| $FORMAIL -I "Content-Transfer-Encoding: 8bit"

}

# 1995-10-18 Tim Pickett <tbp A T cs.monash.edu.au>## Decode MIME quoted-printable Content-Transfer-Encodi ng## Conditions## Mail has a MIME-Version header with a number in it.# Header saying "Content-Transfer-Encoding : quoted-printable"# exists

:0*$ ^MIME-Version:$s*$d*(\.$d*)*$ ^Content-Transfer-Encoding:$s*quoted-printable

{:0 fhw # Remove header| $FORMAIL -I"Content-Transfer-Encoding:"

:0 fbw # Decode the body.| mmencode -u -q

}

14.23 How to send commands in the message's body

:0 b* ^Subject: ARCHIVE| sed -e '/$s*[^a-zA-Z]/,$ d' | sh

14.24 Matching two words on a line, but not oneHow does one write a recipe that will do this: Put mail in mailbox which has a line with two string(one and two) like:

one two

but save mail in error- f older i f the line as only the f irst string like: one (string two is missing)

[philip] I presume these lines would be located in the body of the message, and that by "spacebetween one and two" you mean "whitespace between one and two". If those assumptions arewrong then you'll need to tw eak the f ollowing recipes:

# The 'B' tells procmail to look in the body instead of the header.# The second colon tells procmail to lock the mailbox with a# local lock file -- if mailbox is a directory then you don't need# it. The brackets in the condition contain a space and a tab.

:0 :*$ B ?? one$s*twodefault.mbox

:0 :* B ?? oneerror.mbox

Now, the above will match even if "one" or "two " is part of another word (at the end in the case of

"one" and at the beginning in the case of "two"). If you don't want that then you'll need to changethe recipes to read:

:0 :*$ B ?? ()\<one$s*two\>default.mbox

:0 :* B ?? ()\<one\>error.mbox

14.25 How to define personal XX macros?By macro, I'm referring to the procmail's FROM_DAEMON, TO and TO_ that you can use in matches.Here is one way to make one 's own macro

[alan] Def ine HEA DERS to include those headers you care about. Pick one of the definitions below(and remove or comment out the others). Here are three ways to def ine user to_ macro

1. use only To:2. use either To: or Cc:

3. To:, Cc:, or A pparently-To:

to_ = '^To:(.*\<)?'to_ = '^(To|Cc):(.*\<)?'to_ = '^((Apparently-)?To|Cc):(.*\<)?'

And you use it like this

:0 :*$ $to_()[email protected]

[jari] and he re are so me more examples

cc_ = "(^((Original-)?(Resent -)?(Cc|Bcc)):(.*[^a-zA-Z])?)"from_ = "(^(Apparently-|Resent- )*\(From|Reply-To|Sender):(.*\<)?|\^From $NSPC+)"}

14.26 How to change subject by body matchSuppose you to change the mail's subject when there is a match in the body. The desired outcomewo uld be this:

From: [email protected]: Fault: NNNN in program block YYY << changed

Fault: NNNN in program block YYY

Here is the answer

:0 fhw* ^Subject: NOK case report*$ B ?? ^$s*\/Fault: [0-9a-f]+ in program block.*| $FORMAIL -I "Subject: $MATCH"

14.27 How to change Subject according to some other header

Suppose you want to change the subject when mail comes to some particular address; or whensome other header f ield. Here is one way to do it, we suppose that mail comes to various internalmail addresses. See the HEADERS macro in previous section.

# By [alan]# Examine headers, create a subject tag if we recognize a list

TAG = ""

:0*$ ${HEADERS}[email protected]

{TAG = "info"

}

:0 E*$ ${HEADERS}[email protected]{

TAG = "check"}

# ...and so on...# now, if TAG is set, insert it into the subject

MATCH # kill this

:0 fhw* ! TAG ?? ^^^^

* ^Subject: *\/[^ ].*| $FORMAIL -I "Subject: $TAG - ${MATCH:-<no subject>}"

Or you could use the command line arguments, add following line to your .forward . (alias file syntax)

foo: "|/usr/local/bin/procmail -m /usr/local/etc/pm-tagit.rc foo"

Then in tagit.rc you would instead say:

ARG = $1

:0* ARG ?? ^^foo^^{

TAG = "foo@go"}

:0* ARG ?? ^^somethingelse^^{

TAG = "somethingelse@go"}

This method w ill work even if someone Bcc:s a message to f [email protected].

14.28 How to call program with parameters...now, suppose I w ant to call program with parameter $FOUND, and get the res ult back in RES ULT, how do I do it ?

The stdout of myprogram will be captured at stored in the variable RESULT. A lso consider whatshould happen if there are spaces or tabs in the value of $FOUND. Perhaps it should be better of f enclosed with quoted.

# Make sure FOUND is not empty before passed to program

:0* ! FOUND ?? ^^^^

{ RESULT = `program "$FOUND"`}

15.0 Miscellaneous recipes15.1 Matching valid Message-Id header[philip] wrote full RFC compliant matcher. Follow the link < http://www.xray.mpe.mpg.de/mailing-lists/procmail/1998-03/msg00375.html >

dq = '"' # (literal) double-quotebw = "\\" # (literal) backwhackws = "[ ]*" # whitespaceatom = "[-!#-'*+/-9=?A-Z^-~]+"

word = "($atom|$dq([^$dq\]|$bw.)* $dq)'local_part = "$word($ws\.$ws$word)*"domain = "(\[$ws([^][\]|$bw.)*$ws\] |$atom($ws\.$ws$atom)*)"

:0* ! $ ^Message-Id:$ws $ws$local_part$ws@$ws$domain$ws thats-non-valid-message-id

15.2 Sending two files in a messageIf you plan to send multiple files in a message, be sure that every f ile has extra blank line at the endso that they can be cat d together. Instead o f doing

(cat THIS; echo " "; cat THAT ) | $SENDMAIL

You do

(cat THIS THAT ) | $SENDMAIL

But sometimes you don't have control over the files, then you can do this to make sure there is blankline. Notice, only two processes used compared to f irst choice.

(echo '' | cat THIS - THAT ) | $SENDMAIL

[David] And an sed expert would do it this way

(sed -e '$ !b' -e '/./G' -e "r THIS" THAT ) | $SENDMAIL

$: the last line!: everywhere except the range (in this case, everywhere except the last line)b: branch to a labe l. No label: branch to the end (and, since -n is no t in effect, print thepattern space)

Now remember that everywhere except the last line, we've skipped ahead, so the rest of the codewill be executed only for the last line of the input.

/./: on lines that conta in a character (but w e ge t here only for the last line, so o n the last line if it contains a character)

G: append a new line and the contents o f the hold space to the pattern space (the hold spaceis empty, so bas ically, if the las t line w as already empty, do no thing, but if the las t line w as notempty, append a newline and thus add a blank line after it).r file: After finishing with this run through the se d instructions, read the named file and copy itto the output.

This side o f sed comes out only after sed has had a few drinks...

15.3 Excessive quoting of message [25 N ov 1997 buck A T Compact.COM] I administer a LIS TSERV mailing list and our host has aske d us to reduce exce ss quoting of pre vio usly pos ted mate rial. .. .S ubje ct: ask ing if this was e xce ss ive quo ting. With the we ights be low , this extra copy w ill activate at 66% quote d lines of all body lines.

[era] I would def initely tolerate 75% quotes . A nd in the end, you will of course always have to facethe kinds of people who would rather change their quoting style to evade such constraints thanquote less . A n idealized quo te parser should perhaps realize that a non-blank pref ix that recurs on alot of lines is probably a customized quote string.

This w ill preserve the correspo ndent's o riginal subject (with a Re: added if it didn't already have o ne)and thus the template text should indicate the nature of the problem.

I'm not sure wha t would be appropriate to generate be havior more like I sugge st below, any takers?

Perhaps no score at all f or empty lines, neutralize .signatures (hope sender obeys "-- " convention)and add 10^0.5 f or each quoted line and dish out -15^0.3 f or non-quoted? (I haven't really exploredthis – could be completely up the creek.) [A lso, perhaps long runs of quoted material should bepenalized harder than quoted snippet – reply text – quoted snippet – reply text alternations?]

COPY_ADDRESS = "[email protected]"

:0* ^Sender: <mailing list tag>{

# - quoted lines# - non-blank, non-quoted lines# - completely blank lines

:0*$ 10^1 B ?? ^$s*>*$ -15^1 B ?? ^$s*[^>$WSPC]*$ -15^1 B ?? ^$s*${

# You don't need to repeat the original condition here# You also don't really need to extract SENDER# Generate a reply with appropriate headers and the

# body quoted

:0 fhw| $FORMAIL -rk -A "Bcc: $COPY_ADDRESS"

# Now "replace" the body with template text + body (In# other words, add the template before the quoted body)

:0 fbw| cat $HOME/template.txt -

# Now send it off to recipients mentioned in generated

# header

! -t}

# Wasn't excessively quoted; save it:0 :$SOME_MBOX

15.4 Sending message to pager in chunks I have a 200 characte r limit on my page r. But I have wo rdy co ntacts who go ov er that limit. W hat I w ould like to do is have arec ipe split up mess ages addre sse d to my pager into 200 character (max) mes sages (Procma il mailing list 1997 -12).

[era] This stuff about f orwarding to pagers is a recurring topic on this list. I've tried to find a goodsummary of all the issues bu t there always seems to be some tiny twist to what peo ple would like tohave implemented. A s a general comment for f uture generations, the Procmail part is usually trivial

and the problem reduces to writing a go od program (shell script or otherwise) for f ormatting the textprecisely the w ay you w ant it, and spitting it out in suitable chunks.

Here's something to split up the body of the message into smaller chunks and do a shell script oneach chunk. The -s option to fold says to o nly wrap lines on w hitespace if possible

# Create a duplicate of the message to forward to the pager.# This will be reformatted and have most headers stripped off.

:0 c{

# Construct header with only From: and Subject: retained

HEADER = `$FORMAIL -XFrom: -XSubject:`

# Reformat body as 200-character lines and send each# as a separate message with the preconstructed minimal# header

:0 bw| tr '\012' ' ' | fold -s -w 200 | while read line; do

echo -e "$HEADER\n\n$line" | \$SENDMAIL [email protected] ; done

}

If your version o f echo doesn't understand \n to mean newline (and/or the -e option to enable thisescape processing), you need to tweak this. (You might need to anyway – this is mostly untested. Inmy limited testing, I f ound the messages would arrive in more or less random order. Insertingpause s in the script s hould help to some extent, but could lead to other problems and is no t an idealsolution anyhow.)

I don't know if the header trimming is required; some pa ger gateways appear to count the headersas part of the message, while others don't. A gain, f or future generations, details like this arerelevant to include when you ask about how to do this.

15.5 Playing particular sound when message arrives[Peter S Galbraith <galbraith A T mixing.qc.df o.ca>] Here is the command in shell to produce thesound:

% cat anyfile | /usr/X11R6/bin/auplay /usr/lib/exmh/drip.au

However, it won't work directly in the recipe

procmail: Executing "/usr/X11R6/bin/auplay /usr/lib/exmh/drip.au"Can't connect to audio server

Strange. The command works from the shell if I su to user mail . A nyway, I got it to work by fullyspecifying the audio server (which is my workstation, where I receive mail)

AU = /usr/X11R6/bin/auplayTUNE = /usr/lib/exmh/drip.au

16.2 Getkeys from key server

# by Adam Shostack <adam A T bwh.harvard.edu> 1996-02## This first ruleset protects me from mailbombs from an automated# service that I often send incorrect commands to, generating 5mb# of reply. It also sorts based on success of the command.

## swissnet.ai.mit.edu is fast key server

:0* From [email protected]{

:0 h* >10000/dev/null

:0 h*^Subject:.*no keys match

/dev/null

:0 E| pgp +batchmode -fka

}

16.3 Auto grab incoming pgp keys

# [Opher Kahn <kahn A T dg-rtp.dg.com>] This first# ruleset protects me from mailbombs from an automated# service that I often send incorrect commands to,# generating 5mb of reply. It also sorts based on success# of the command.## swissnet.ai.mit.edu is PGP key server

:0* From [email protected]{

:0 h* >10000/dev/null

:0 h*^Subject:.*no keys match/dev/null

:0 E| pgp +batchmode -fka

}

# auto key retrieval

## I have an elm alias, pgp, points to a key server The log file# gets unset briefly to keep the elm lines out of my log file.

:0 W* B ?? -----BEGIN PGP



somewhere that I don't know beforehand (or I have just forgot to tweak my .procmailrc)

ME = "([email protected])"LISTS = "(procmail|list-a|list-b )"

:0 # Idea by Bill Moseley*$ ! ^TO_()$ME*$ ! $LISTS{

# Could be UBE or I might be on a unknown distribution list.INCLUDERC = $PMSRC/pm-ubecheck.rc

}

[dan] That wo uld w ork; common practice, however, is to put recipes for f iling mail f rom lists (and, perBill's pref erences, anything mentioning procmail in the head gets treated the same as mail f rom thislist) f irst; then the only remaining condition to consider there would be unexpected blind carbons: * !^TO_moseley. This method is good if you get much more spam than legitimate mail (including mailf rom list subscriptions as legitimate) and you want procmail to deal with spam right away. I belong to

several very active mailing lists, so I actually receive more pieces of legitimate mail than pieces of spam.

One way to get the best of both worlds is this:

*$ ! ()\/(^TO_$LOGNAME|procmail|list-(ABC|123|XYZ))

because then, if the regexp matches (and thus the negated condition f ails and you don't detour into$PMSRC/checkspam.rc), MA TCH is already set to the name of the mailing list, and you can do f urthertests by just examining MA TCH (or a variable you copy it into) instead of a repeating a completehead search. [I pref er to use the variable $LOGNA ME rather than hard-coding my name becausethen others can use the code, and I can use it unchanged on sites where my logname is dif f erent,and if my logname is changed my procmailrc will keep up with it.] For example (I've separated theconditions into two lines so that, per Bill's preferences, a mention of procmail in the head will get themessage into the Procmail List folder, even if a match to $^TO_$LOGNA ME is also present andappears sooner):

:0* ! ()\/(procmail|list-(ABC|123|XYZ))*$ ! ^TO_$LOGNAME

{INCLUDERC=$PMSRC/pm-ubecheck.rc

}

# The next recipe has an `E' flag, so it will be examined# only if the preceding one didn't match; thus if $MATCH was# set inside pm-ubecheck.rc, it won't hurt anything here, and a# value for $MATCH set in pm-ubecheck.rc# won't be mistaken for a list name:

:0 E: # MATCH is non-null only if it matched a list name* MATCH ?? (.)$MATCH

# Remaining recipes will be read only for two types of mail:# those that met $^TO_$LOGNAME but not any expected list# name, and those that went through pm-ubecheck.rc but came out

# undelivered.

17.3 Using: autoloading an rc fileNow when you know that includerc can be called conditionally, let's discuss about "autoloading o f amodule". For example you may see f ollowing statement modules which import predefined variables:

:0* ! WSPC ?? ( ){

INCLUDERC = $PMSRC/pm-javar.rc}

It says that "If variable WSPC does no t contain space, then load module". If the module has alreadybeen loaded by some other rc f ile, the WSPC would exist. If it does not exist yet, then the module isloaded. This is classical example of conditionally loading f unctions or variables into current module:

Check if feature is present, No? Then load module module.

Justin Lloyd <jlloyd A T harris.com> sugges t a general w ay of caching the included rc f iles. Use top-level script that records every module that was included. The module is loaded only if it it not yetincluded:

# pm-xximport.rc

:0* ! INCLUDE_CACHE ?? ()\<$RC\>{

# Module was not there yet, add it to the listINCLUDE_CACHE = "$INCLUDE_CACHE$RCFILE$ NL"INCLUDERC = $RC

}

This is dif f erent approach then the previous one. Instead of checking features, the presense of module is checked. Two sides of the coin which can be used for the sa me thing. You can pick either

solution but here are some thoughts:Adding extra top level INCLUDE_CACHE is extra work. Procmail must open a separate top-levelrc file every time with call

RC="pm-xxscript.rc" INCLUDERC=pm-xximport.rc

If feature already existed, you wo uld still have to open the pm-xximport.rc file for every call tofind it out. E. g. here you pm-xximport.rc is called 3 times no matter if 1, 2, 3 were alreadypresent

RC="pm-xxscript1.rc" INCLUDERC=pm-xximport.rcRC="pm-xxscript2.rc" INCLUDERC=pm-xximport.rcRC="pm-xxscript3.rc" INCLUDERC=pm-xximport.rc



restore-x-1

17.6 Making: Public and private variables in rc fileAs you learned above, the variables should be put to RC f ile's name space. The user interfacevariables (public) should be all caps and private variable should start with lowercase letter. Whether

you use "theVarStyle" o r "the_var_style" is up to you.

[script pm-xxscript.rc]

# ........................... public

XX_SCRIPT_FLAG = ${XX_SCRIPT_FLAG:-"default"}XX_SCRIPT_VAR = ${XX_SCRIPT_VAR:-"default"}

# ........................... private

charset = "a-z1-2"regexp = "something-that-matches"

Whether you need to stick pref ix xx_script to the private variables depends on whether you callanother includerc which may happen to use sa me names as you:

[pm-xxscript.rc]charset = ... # watch this...

INCLUDERC = .. # call another subroutine

charset = .. # holy cow, it used same variable

..back in the pm-script.rc

:0* $charset # BOOM, not what you think.

In this case it would be wise a) not to def ine charset at the top of the f ile but to move the def initionto just before the recipe where it is used or b) make the name unique, with xxScriptCharset .

17.7 The rules of thumb for constructing general purpose rcfile

Write goo d documentation a t the beginning of file: how to set up the includerc and e xplainwhat it does. If you don't include docs, people may skip your extraordinary useful script. Also,remember that the script lives in the Net and passes through many hands long after you havebeen disconnected.

Keep the layout like this: the us er interface variables must all be in capital letters. Familiarizeyourse lf with what(1) tags too . Notice the first and las t lines : if you keep the format like this,then any universal too l can rip your code from any file (or mail), because it's de limited by "pm-xxScript.rc – " and "end of pm-xxScript.rc". See Unix wha t(1) for first line's syntax.

# pm-xxScript.rc -- procmail script for ...# DOCS



USER VARIABLES

private variables

CODE

# end of pm-xxScript.rc

Always include version number or last modification date somewhere. Prefer some versioncontrol tool, like RCS, VCS, ClearCase, whatever you have at hand.

Use a variable name like dummy in appropriate places to tell what's happening in the code.Remember that the VERBOSE setting isn't much help if you can't tell by looking at the LOGwhere on earth the code is executing.

dummy = "start of pm-xxScript.rc"...

dummy = "Now testing if we have control message XXX":0* condition{

dummy = "Now testing if the command is YYY":0* condition...

}...dummy = "end of pm-xxScript.rc"

If you need the value o f some common headers, don't just call formail like this because thevalue may already be available prior your includerc. For example the user may already haveneeded the Subject value and s tored it in a variable

[in pm-xxScript.rc]

XX_SCRIPT_SUBJECT = `$FORMAIL -xSubject:'

[User may have already read the content to SUBJECT]

SUBJECT = `$FORMAIL -xSubject:'INCLUDERC = $PMSRC/pm-xxScript.rc

Your pm-xxScript.rc launches an unnecessary formail call. Instead,use the existing SUBJECT.

[user]:0* ^Subject:\/.*{

SUBJECT = $MATCH

}

...

XX_SCRIPT_SUBJECT = $SUBJECT # Note this!

## o pm-xxScriptA.rc# o pm-xxScriptB.rc## Call arguments (variables to set before calling)## o INPUT, the string from where to parse...# o VAR1, description, default is ...# o VAR2, description, default is ...## Returned values## ERROR will have value "yes" if couldn't parse INPUT# OUTPUT will have result after successful parse## Example usage## :0# * condition\/.*# {# INPUT = $MATCH# INCLUDERC = $PMSRC/pm-xxscript.rc# # OUTPUT has the result# }## Change Log: (none)

# ..................................................... &init ...

dummy = "init: pm-xxscript.rc start"

# Read the standard variable definitions if they are not

# yet defined: that's "if WSPC variable does not contains space,# as it should, then global variables haven't been read yet"

:0* ! WSPC ?? ( ){

INCLUDERC = $PMSRC/pm-javar.rc}

# .................................................... &input ...# - User configurable variables with reasonable defaults# - But parameters like "INPUT" that must be set beforehand# are not mentioned here.

VAR1 = $VAR1{VAR1:-"default1"}VAR2 = $VAR2{VAR2:-"default2"}

# .................................................... &do-it ...

dummy = "subroutine: pm-xxscript.rc parses now that and that"

<the code>

dummy = "subroutine: pm-xxscript.rc end."

# end of pm-xxscript.rc

usually used by the MTA to deliver mail to users, and indeed, procmail will return failure if it is givenan invalid user name. In delivery mode, procmail reads /etc/procmailrc bef ore the use r's .procmailrc.

Note : Procmail will work in delivery mode only if it is setuid root, if it is invoked with the ruid of therecipient named in -d, or, under certain OSes where the build routines have determined that it issaf e, if the euid is that of the recipient and the egid is the recipient's login group.

Mailf ilter mode is invoked using the -m f lag. It accepts only one rcf ile as an argument – otherarguments are either variable a ssignments or arguments that are made availible to the rcf ile itself as

$1, $2, etc. If the spe cif ied rcfile is located under /etc/procmailrcs/ then procmail will take on the uidof the owner of that file. Otherwise, it will run as the user who invoked it. /etc/procmailrc, thatprocmail -d reads, is ignored. In mail filter mode, procmail unsets ORGMAIL and DEFAULT to suppressnormal delivery – reaching the end of the rcf ile results in the mail bouncing. If the rcfile se ts either of them then procmail will attempt delivery to that mailbox if it f alls off the end of the rcf ile; how ever,the mailbox will have to be writable by the uid/user that procmail is running as.

Note : Only one rcf ile can be na med on the command line, but names of other rcfiles can be passed inthe positional parameters to be used later in INCLUDERC ass ignments.

Normal mode is invoked by not using the -m or -d f lags. It accepts any number of rcfiles and variableassignments as arguments. Procmail runs as the invoking user in this mode. /etc/procmailrc is

ignored.So, to answer your questions: if procmail reaches the end of the specif ied rcf ile, it bounces the mail(/etc/procmailrc is ignored). Everything is up to the rcfile – how to determine whether the address isvalid and w here to put the message if it is.

19.2 Procmail as sendmail Mlocal mail filtering device...I'm a new sys admin at my company, and I've be en trying to se t up Procmail as the mail f iltering de vice (still using mail as the

Mlocal) I 've tried se tting up the sendmail. c f to use Procmail as a f ilter (we want to use the curre nt mailer as the local mailer) withone loc al procmail rc f ile. Procmail see ms to work just f ine i f se t up as the local maile r, but I'm still having proble ms s etting it as the

f ilter.

[John M Vinopal <banshee A T abatto ir.com> answe rs se ndmail.cf ]

R$+ < @ $=a . > $*$#procmail $@ /etc/mail/procmailrc $: $1 < @ procmail > $3

R$+ <@ procmail > $* $1 < @ example.com .> $2

so this sends anything of the f orm [email protected] through procmail and rew rites it as [email protected] procmail script reinjects it and it bypasses the call to procmail and then is rewritten back tof [email protected].

/etc/mail/procmailrc::0! -oi -f "$@"

19.3 Procmail doesn't pass 8bit charactersYou've mistaken . Procmail does not do that to your mail. Frank Gadegas t <phade A T powerweb.de>tells you:

procmail wasnt the problem, it was sendmail

I uncommented this line in sendmail.cf and no w I ge t all nice German Umlauts.

# strip message body to 7 bits on input?# O SevenBitInput



anyway) def ault mailbox protection of 660, gid=6 (mail). I think that's an OS-dependent bug, withthe `chmod u+x ...' as the w orkaround.

19.7 Changing mbox permission during compilation to 660...it appears that mail that procmail delive rs back into the spool it is w riting out with ow ner.group use r.mail and rights 600. T o methis is re asonable. Mail delive red to the spool by /bin/mail is written out owne r use r, group mail 660.

W he n procmail delivers mail 600 late r attempts at delive ry with procmail remove d f rom the .f orward f ile f ail: /bin/mail doe sn't have permissions (or ref uses to use s its permissions).

S ince we hav e f ickle and unruly users w ho will be moving their f orw ards in and out o f place this is a prob le m. I s the c orr e ct solu tion to f orce procmail to write 660? I f so, how is this done? I as sume in the se ction of con f ig.h just below thewarning about only mes sing with a se ction i f you think you k now what you are doing. I don't like f ee l like I k now well enoughwhat I'm doing to walk into that te rritory without some guidance .

[alan] I used to be the manager of the system support in the College of Engineering, at theUniversity of California, Santa Barbara.

We supported about 1500 users from two HP 9000 G30's, using one of them as the centralizedmailer. Mail was available via NFS exported /usr/spool/mail to over 200 workstations, of all kinds:SGI, HP, Sun, etc.

We replaced /bin/mail with procmail as the local mailer (Mlocal) because procmail correctly avoidedNFS-locking problems, and it supported user-conf igurable mail filtering, without compromising systemsecurity.

In over two years subsequent to the change, we had no loss of mail due to procmail being used asthe local mailer. If you wish further comment f rom the current system managers, send mail to"postmaster A T eci.ucsb.edu".

To answ er your specific questions:

* you can conf igure the pe rmissions directly, by changing one of the f ollowing def ines in config.h:

/* bit set on mailboxes when mail arrived */#define UPDATE_MASK S_IXOTH/* if found set */#define OVERRIDE_MASK (S_IXUSR|S_ISUID|S_ISGID |S_ISVTX)/* the permissions on the mailbox will be left untouched */#define INIT_UMASK (S_IRWXG|S_IRWXO) /* == 077 */#define GROUPW_UMASK (INIT_UMASK&~S_IRWXG) /* == 007 */

We did not f ind it necessary, however:

We did disable all locking except dot-locking, since the kernel locks were the source of the

NFS-locking problems. There have continued to be occasional locking problems, but these are"victim"-induced problems caused by using non-supported and discouraged mailers, such as"mailtoo l" from older Suns. These locking problems have nothing to do w ith mail delivery, butfrom the mail client using kernel-advisory locks, and then orphaning them or, leaving themlocked all day long.An alternative to ha ving users use .forward files, is to create a file o f users w ho w ould like touse procmail as the ir local delivery agent, and use this file to initialize a class variable.

Write a special rule in sendmail.cf which delivers mail using Mprocmail instead of Mlocal when thedestination user is in the special procmail user class.

This allows users who want procmail-direct delivery in spite of management worrying.

I set this up to test procmail delivery on our system before changing Mlocal to use procmail. Weplaced some "volunteer" users in the procmail class file, and they never had any problems (I wasone of them).

19.8 The .forward file must be real file

...I tried to make a sof tlink to ~/.f orward, but then my procmail wo uldn' t run. W hen I made a real ~/.f orward f ile, then it work ed again. My ques tion is – why w ould proc mail treat a link to a f ile any di f f e re ntly than the actual f ile itse l f ?

ln -s ~/.procmail/forward ~/.forward

[Werner Reisberger <wr A T tribe.ping.de>] That's not a problem with procmail, this is an MTA issue.Due to security reasons sendmail will not deliver mail to f iles whicharesymlinks.

[david] procmail has restrictions on what permissions it will tolerate on an rcf ile. For example (I'm just guess ing here) it can te ll whether it can read the target f ile but it cannot tell who might be ableto write to it. This prevents a major security hole

You can make hard link to the file, since A hard link is completely indistinguishable f rom the originalf ile. But note: a file hard-linked to two or more names is very distinguishable f rom a file with only one(hard) link, and procmail, f or example, will not deliver to a plain f older that has two or more hardlinks.

You can also put the real f ile at ~/.f orward and let ~/.procmail/forward be a symlink to

[<mikk0022 A T maroon.tc.umn.edu>] I suppose , the reasoning behind procmail's f older policy is thatprocmail locks the f ile by name, not inode. Hence it cannot guarantee mutual exclusion f or access toa f ile which has multiple names.

My understanding of the .forward policy is that a symlink need not share the permissions of itstarget. T heref ore somebody's .forward symlink could have proper permissions, while its target couldbe writable by others. T his would allow anybody with the w rite permissions to execute any program(potentially) f rom the user's f orward f ile.

Two hard links sha re the same permission, so this argument doesn't hold.

19.9 Using .forward if procmail already is LDA [ Elie Ros e nblum < f nord A T jurai.net> ] I f you have a .f orward, it is use d by se ndmail to replace a call to the LDA f or the use r inquestion. So i f you have a .f orward that does n't call procmail, procmail is ...

[david] Elie sent the a nswer to me with a carbon to the list, but since reading my persona l copy myinbox got trashed. A s of this writing the list copy hasn't reached me, but the rest of that sentence(as I recall f rom reading it bef ore it got hosed) was to the ef f ect that procmail is then never invokedat all on your incoming mail; a .f orward takes precedence over the LDA . That scenario never occurredto me. Thank you f or explaining.

[ Philip] S cratch the bit about /e tc/ procmailrcs/$LOG NAME. You're mixing up procmail -d with proc mail -m.

Ah, got it ... af ter rereading the man page. The part about /etc/procmailrcs really can apply onlywhen procmail is setuid root, so again it's something I've no experience with and never quitef ollowed or retained. So no f ile in /etc/procmailrcs is ever used implicitly, but /etc/procmailrc can be.

[ Philip] $HOME /.f orward is handled by se ndmail. If you have a f orward, then se ndmail re writes attempts to delive r to you intoattempts to delive r to the address es listed in the .f orward f ile.

Or in other words, the .f orward takes precedence over the LDA . Thank you both.

19.10 Mail should be put in the mailqueue if write fails...We want to de liver dire ctly to a use r's home directory. But this can o f course be temporarily f ull. T he n the mail should not bounce, but instead be put back in the mailqueue and tried again until either it succe eds or se ndmai l bounces it af ter 5 days (asusual). The RE ADME f ile says this is my choice (to bounce or not ), but I canno t f ind any place w he re I can set this. What is thecorre ct place to se t this be havior

[1998-06-24 PM-L phil] The -t f lag causes procmail to return EX_TEMPFA IL where it normally wouldhave returned EX_CA NTCREAT. If you've made procmail the local delivery agent then you should add-t to the A = def ine, before the -d f lag.

19.11 Qmail: how to make it work with procmail[1998-11-10 PM-L John Conover <conover A T inow.com>] All you do is install f astf orward and dot-f orward, (they are optional, and are not required.) Then cp /var/qmail/boot/proc or

/var/qmail/boot/proc+df , to /var/qmail/rc.

[1998-11-10 PM-L Greg Boes <gboes A T ashf ordtech.com>] From the qmail FAQ (4.4 How do I useprocmail with qmail?) Put

| preline procmail

into ~/.qmail. You'll have to use a full path f or procmail unless procmail in in the system's startupPATH. Note that procmail will try to deliver to /var/spoo l/mail/$USER by def ault; to change this, seeINSTA LL.mbox.

19.12 Qmail: Procmail looks file from /var/spool/mail only...Procmail see ms to want to do something in /var/spool/mail. But since I use qmail, I don't have a /var/spool/mail. Is the re away to have procmail not to create temp s tu f f there?

[philip] Get procmail 3.11pre7 and uncomment and and correct for your local setup theMAILSPOOLHOME="/.mail" def ine in src/authenticate.c. Compile and install. t's relative to the user'shome directory. Thus the name MAILSPOOLHOME.

[Ekkehard Knopp <knopp A T rz-online.de] at the qmail-home-page you can f ind a pa tch for procmail-3.11.pre7 called procmail-maildir-patch. When you can't f ind it, I can send you a netmail. Have noproblems with procmail and qmail. Wo rks good.

19.13 Qmail: patch to procmail 3.11pre7 to work with Maildirs[Jaye Mathisen <mrcpu A T cdsnet.net>] On the www.qmail.org page is a patch that lets procmail3.11pre7 work with Maildir's, (qmail's NFS safe delivery f ormat), and not must Mailbox's.

Very usef ul. Really slows down delivery though. On my test box, just adding procmail to the deliverywhere all it did was deliver to the def ault mailbox, and no other rules whacked my speed test f romsomething like 600,000 messages /day to about 180,000.

Killer. I suspect Procmail's locking of the Maildir 8 ways from Sunday is probably partially to blame.

19.14 AFS: How to use Procmail when HOME is in AFS cell...I 've viewed some of the archived pos ts concerning AFS and procmail, but e ach se ems to have a di f f ere nt perspe ctive on thesubject. Bes ides the f act that AFS isn't the gre ates t product in the world, doe s e ve ryone agree that it is not possible to use procmail whe n your $HOME l ies in an AFS ce ll? Mail se nt locally se ems to work w ith procmail, but mail f rom users w/o a toke n or AFS id

just ge ts de live re d to /var/spo ol/mail/so me one .

[Christopher Lindsey <lindsey A T ncsa.uiuc.edu> 1998-03-09 PM-L] AFS is awesome! You just haveto treat it nicely. :) The only viable solution that we've been able to come up with involves patchingthe procmail-3.11pre7 sources to "f ake" user home directories out o f another directory.

For example, my home directory in A FS is

/afs/ncsa.uiuc.edu/.u1/lindsey/

It is kept as such on the mail server in /etc/passwd as well. However, we have some space set upvia NFS in /var/forward with space f or each individual use r (so /var/f orward/lindsey in my case).

The procmail patch intercepts requests f or the use r's home directory and replaces it with the "f ake"directory (the /var/f orward one). So f or all practical purposes, procmail things that my home directoryis /var/f orward/lindsey, and everything w orks f ine.

19.15 Help, some idiot sent my address to 30 mailing listsYou can make a procmail recipe to junk incoming mail f rom the lists until you get the unsubscribemessages delivered to cancel your participation. You should complain to the list's maintainer thatsuch things was even possible: The mailing list should have sent you a conf irmation message with



unique "participate ID number" that you need to send back in order f or the subscription to take ineff ect.

KILL_FILE = $PMSRC/.kill-immediately

:0*$ ? $IS_READABLE $KILL_FILE{

KILL = `cat $KILL_FILE`}

# 1) Make sure KILL has value# 2) if match is found from header.# 3) /dev/null does not need lockfile

:0* KILL ?? [a-z]*$ $KILL/dev/null

[sean] ...In the long haul, your best bet w ith dealing with this problem is to stamp out the offender -bring this harassment to the attention of their ISP and get their account closed. Repeat asnecessary. Most of the mailing lists should have some record of the submission request. Even if f orged, the abuser probably has their IP address in the headers somewhere (and if the person isactively subscribing your f riend to so many lists and actually WORKING at covering their tracks,apparently you've REA LLY crossed them). Most people who stoop to these immature harassmenttactics aren't bright eno ugh to f ully cover their tracks.

Another a lternative to having to manually deal with unsubs on certain lists is once you've identif iedf ilterable characteristics of the lists , BOUNCE them. Most semi-intelligent listserv implementations willunsub you if they get repeated bounces. Yea, not nice to the listserv maintainer - but then, if perhaps they'd implement a subscription verif ication system, it wouldn't have been a problem tobegin with.

:0* condition{

# may expose your .forward - but if you're bouncing lists,# it probably doesn't matter much.EXITCODE = 67

# save header for examination.:0 h:bounce.log

}

You've got a sticky situation. You can't simply ditch all unrecognized mail - you need to be able toreview potential ref use f irst, and take action on anything which doesn't belong (because youcertainly don't want to continue getting the non-wanted lists till the end of eternity - you shouldwant to unsubscribe f rom them to simplif y your mail).

19.16 Help, Procmail beeps and prints to my console...when messages get f iltere d through procmail I ge t a be ep and then f irst 10 lines or so are also se nt to the conso le. I get a lot o f messages so the bee ps, and stu f f on my scre en is ge tting very annoying.

[sean] One or the other should do the trick (or both even): Go to your login f ile (what it is nameddepends on the shell you're using), and add:

I've seen other posting about corruption of the From line. Perhaps you have the same problem.

[Christopher B. Smith <cbsmith A T envise.com>] I had the exact same problem with my upgradedOpenLinux system. For the record, if you are running the imapd that comes with it, you should reallyset your permissions for the directory is a s f ollows:

chmod 1777 /var/mail/spool

I got that f eedback from the guy w ho w rote imapd, and it works very well.

19.19 Directing user's mail to HOME instead of /var/spool/...I h ave a nee d to direct all a single us er's mail to a mailbox in his home dire ctory, to $HOME /mailbox ,

# One possible solution, not perfect

UHOME = /tmp_mnt/usersUHOME_LIST = "(login1|login2|login3)"

*$ ^TO\/$UHOME_LIST@* MATCH ?? ()\/[^@]+$UHOME/$MATCH

[era] Perhaps pref erably use ^TO_ if you have Procmail 3.11pre7 or newer. This is the classical caseof using Procmail where you really need the envelope recipient inf ormation. The headers are notenough to determine who a message is f or. If Procmail is your MDA , you can have this, but I'd stillthink something involving Sendmail would be more appropriate. For one thing, what if this userwould suddenly really want to use Procmail? You can set DEFAULT and ORGMAIL f or this one user in

/etc/procmailrc to come around tha t, but the bottom line, as so many times bef ore, is that Procmailmight not be the right too l f or this.

19.20 NFS mounting /var/mail is a good way to get badperformanceProcmail mailing list 1998-06

> /var/mail stays at a Solaris 2.5 machine. Cucipop is working> at the same machine. It's fine there. But, I want to have> more than one machine with cucipop and when I put cucipop at> another machines, NFS clients, it is delaying more 30 or 40> seconds to close the session.

[1998-06-23 PM-L Brad Knowles <brad A T colltech.com>] NFS mounting /var/mail is a good way toget bad perf ormance, especially when you're doing any NFS writes. Even if you're not doing a ny NFSwrites, just having to deal with local f ile locking and trying to translate that into NFS f ile locking is anightmare (in general, file locking is one o f the s ingle biggest problems lef t with NFS).

> Procmail is working good on NFS, it finishes quickly. But when

> cucipop is put on a NFS client, procmails starts to delay too.

Procmail probably isn't writing to NFS, or if it is, it's probably not using the same locking mechanismas cucipop. Unf ortunately, each vendor and each program have their own ideas on how to best do



that.

[ philip ] cucipop was written b y the author o f procmail. I de ally , whe n you co mpile cuc ipop yo u edit its co n f ig.h to use the lockingtechniques that procmail's autoconf pro ce ss de termine d f or your system(s). However, e ven i f you didn't do that, cucipop us es thesame dotlocking algorithm as procmail.

Also, keep in mind that any POP3 server will have to copy the mailbox in order to work on it, andmany of them copy the mailbox to /var/mail/.username (you got it – creating lots of NFS writes).When they're done, they copy the mailbox back to /var/mail/username (after they copy any new mailmessages that have come in to the end of /var/mail/.username and locked then truncated theoriginal /var/mail/username file).

[ philip ] cucipop does n't use a temporary f ile: it kee ps it all in memory . On dele tes it update s the mailspool in place which s hould never lose data, though i f the s erve r crashes in the middle of this you can end up with one or more bogus me ssages.

This is a real nightmare when you start talking about users who select "Leave mail on server" andhave multi-megabyte mailboxes .

[ philip ] Assuming you have e nough memory, cucipop should be pre tty f ast.

I think maybe now you're s tarting to understand why POP3 really doesn't scale w ell at all in multi-machine environments (unless you've cooked up a custom mail store that uses a real database back-end, like Oracle P arallel Server), with /bin/mail (or procmail) as a writable interf ace to this messagestore and POP3 and/or IMA P as a readable (and writable) interf ace to this same message store.Then you can let the database vendors deal with the hard data replication and distribution problems.

Otherwise, it's a pain-in-the-ass.

> Is there another good pop server?

Have you tried QPopper f rom Qualcomm? It's the single best POP3 server I've ever run across,although I wouldn't put even it in an NFS write environment.

BTW, I used to be the Mail Systems A dministrator f or GNN (Global Network Navigator), the website/National ISP co-operative between O'Reilly & Assoc. and A OL. At our peak, we had hundreds of thousands of registered users, of which up to f ive to six thousand were logged in at any one time,with their MUA set to check their mail every minute.

We had a single primary Mail/POP3 server machine (Dec A lpha 2100 w/ four 250Mhz processors, 4GBRAM, 28GB hardware mirrored/striped mail spool), and one warm spare (same CPU/RA Mconf iguration, physically hooked up to the same d isks, but through DECsaf e A SE not mounting themunless the primary died).

19.21 I can't see the sendmail's response in LOGFILE...As the man page s ays, this should've written to my LOGFILE. It didn't. But it DID activate the pipe in the rec ipe. S o w hat's uphere?

:0 hc*$ ? $IS_EXIST $HOME/.vacation| LOG=| ($FORMAIL -r; echo $IM_NOT_HERE) | $SENDMAIL -t

[david] The man page says that a variable capture recipe assigns the standard output of thecommand to the variable. Since you are repiping the output of f ormail and echo to sendmail,sendmail sucks up the standard output of f ormail and sendmail. Sendmail itself does not write tostandard output, so the stdout of ( $FORMA IL -r ; echo $IM_NOT_HERE ) | $SENDMA IL -t is no thing.

Thus you're a ssigning a null string to $LOG, and w hen procmail writes $LOG to the logf ile you can'tsee a dif f erence.

19.22 Compiling procmail and choosing locking schemeGeneral advice: Everything except dot locking is usually broken.

[stephe n, <199607292139.XA [email protected]>] . Remove f cntl() and lockf (), only allow f lock() (oromit it completely) Kernel locks don't work. But that's all some programs use. A cross a networkedf ilesystem, lockf() doesn't work, fcntl() and f lock() should, but they don't either because the lockd isbuggy. Mailtool uses fcntl() but does it wrong, so that's another problem. The only thing that workson a ll platforms, all netwo rks, all the time a re .lock f iles.

Makefile ref ers to:

# Uncomment (and change) if you think you know#LOCKINGTEST=100# it better than the autoconf lockingtests.# This will cause the lockingtests to be hotwired.# 100 to enable fcntl()# 010 to enable lockf()# 001 to enable flock()# Or them together to get the desired combination.

conf ig.h refers to :

/*#define NO_fcntl_LOCK uncomment any of these three if you *//*#define NO_lockf_LOCK definitely do not want procmail to make *//*#define NO_flock_LOCK use of those kernel-locking methods */

19.23 Forwarding lot of mail causes heavy load...There are se veral f orward (e.g. ! walter @localhost) recipe s For eve ry f orwarde d mail, a distinct se ndmail proce ss is c reated.T his leads to a he avy (IMHO unbearable ) system load. How can I stop procmail f rom running a se ndmail proce ss f or eve ry mail

f orwarded?

SUMMARY: Look at qmail, it's better than sendmail.

[era 1998-08-15 PM-L] (Blows dust of f old underutilized Bat Book/ORA sendmail book) Yeah, settingQueueFactor (q) and QueueLA (x) to suitable values should do what you want. You need to haveload-balancing support compiled in, though; according to the Bat Book, sendmail -d3.1 tells whetheryou have it o r not. (Mine just sa ys getla:0 w hich I would imagine means I have the support but theload average was below the cutoff level.

AFAI K using load ave raging w ould have the f irst message s de livered and the res t queued. Howeve r, also not being a sendmail guru, I do not know how to e mpty a se ndmai l queue f or incoming mail only. Moreove r, ev en i f I k new ho w to do this, it would have to be done af ter procmail f inishes.

[Liviu Daia <daia A T stoilow.imar.ro>] Instruct sendmail to queue messages when called f romprocmail:

SENDMAILFLAGS="-oi -od d"

then disable the normal sendmail daemon from your system init scripts, and run it in f lush queuemode only, that is, replace

/usr/sbin/sendmail -bd -q 15m

in your init scripts with

/usr/sbin/sendmail -q 15m



message, ala

:0* > 10000000/dev/null

would require the body to be read... It really is just better to simply have sendmail enf orce the limit.You should be doing it there anyway to cut down on the totally trivial denial-of -service attacks andbecause it's more ef f icient.

...I am running procmail ver 3.11pre7 and I ke ep ge tting "out of memory as i tried to allocate 8xxxxxx bytes.". I have ove r 100 megavailable swap space s o i have a di f f icult time unde rstanding this. I s this a know n e rror?

Procmail's memory allocation technique appears to non-optimal for some OS/libc combos, namelyimplementation of the libc system f unction realloc() (FreeBSD has been reported). It's conceivablethat the conf iguration process could be enhanced to de tect this system limitation to use a s trategymore e f f icient on them. Don't hold your breath.

19.26 Procmail signaled out of memory in my verbose log...I notice in my procmail verbo se log the f ollow ing 'transac tion':

procmail: [10239] Sat Jan 9 08:49:02 1999procmail: Out of memorybuffer 0: "formail"buffer 1: " formail -A "X-Check: List""

Folder: **Bounced** 5744procmail: Notified comsat: "bhoule@:**Bounced**"

I f I ac t quick e nough whe n this happe ns, I can look in spoo l/mq ue ue and f ind a mes sage with a gazillion addres se s in the T o: line.

S o it seems that f ormail is having trouble adding my X-Chec k he ader to an already large se t of headers.

[philip] No, it's procmail that's unable to allocate enough memory. The buf f er dumps indicate thatprocmail was unable to get enough memory somewhere between parsing the action line andreaching the next recipe – buf f er 0 would not contain the string "formail" if procmail had gotten toanother recipe or variable assignment. What's weird is that the message is so small (only 5744bytes according to procmail). Do you only see this error on this recipe, or at random places in your.procmailrc? If the later, then I would guess that your mailserver is running out of memory f or someother reason and that procmail happens to be an innocent bystander. If the former, then, well, I'mnot sure.

T he message is neve r delivere d to me. Is there anything I can do so that procmail/ f ormail will act as i f it was neve r there so theincoming dumps into my inbox rathe r than re turning an erro r to the mailer ? T his "* Bounce d *" busine ss is not a ve ry help f ul action.

Giving procmail the -t flag will cause fatal internal errors that are normally returned as permanenterrors to be returned as temporary failures instead. Otherwise there's no way to control that.(Setting EXITCODE won't work because procmail needs to malloc memory to handle T RAP andEXITCODE, and it'll refuse to try that when it was malloc that caused the exit.)

19.27 Variables DEFAULT and ORGMAIL...According to the man pages , DEFAULT is def ined as ORGM AIL ...so i f I rede f ine ORG MAIL, then DEFAULT should change as well,which doe sn't help me . Any help on this would be appre ciated

[david] DEFAULT is initially def ined as equal to ORGMAIL. Once procmail has started reading /etc/procmailrc (if it is the MDA ) or your .procmailrc, you can change the value of either withoutaff ecting the o ther.

In f act, you can even set DEFA ULT on the command line when you invoke procmail (I'm not sureabout doing that with ORGMAIL, though), and that value will override its normal initial value equal toORGMAIL.

What if it is possible that dropping to DEFA ULT f ails due to disk f ull? Then you would better have



another drop place in another file system. Peek at bdf (1) or df (1) to find out the diff erent mountedf ile systems.

# Place this to the end of your .procmailrc and define# DEFAULT_SECONDARY

:0 :$DEFAULT

:0 E$DEFAULT_SECONDARY

If you deliver explicitly to $DEFA ULT, procmail treats it like any other save-to-f older recipe, and if thewrite f ails, it continues reading recipes.

...If I had se t the "de live r" destination as ORGM AIL rather than DEFAULT, would it have made any di f f erence?

Nope. If you write a recipe for it, procmail just expands the variable and doesn't give a heck if ithappens to be the same destination as DEFAULT or ORGMAIL. DEFAULT is special to procmail only when

it uses it on its own after f alling of f the end of the rcf ile; ORGMAILis special only at startup (without -m) and when procmail f alls of f the end of the rcf ile and f inds that it cannot save the message toDEFAULT.

In general, i f procmail f alls o f f the end of the rc f ile, f ails to save to DEFAULT , and then f ails to save to ORGM AIL , doe s it reve rt to thecompiled-in value o f ORGM AIL ?

[philip] Procmail has no f allback beyond the current value of ORGMAIL. If delivery to both DEFAULT andORGMAIL f ail, then procmail gives up and exits with error code 73 (EX_CA NTCREAT) or 75(EX_TEMPFA IL), depending on whether the -t f lag was given. Setting EXITCODE would probablyoverride those. The message is logged as "*Bounced*".

19.28 When DEFAULT cannot be mailed toIf procmail gets to the end of the rcf ile without delivery (or without being directed to ano ther rcf ile byan INCLUDERC or HOST assignment), it assumes these:

:0:$DEFAULT

:0 e:$ORGMAIL

That is, it tries to deliver to $DEFA ULT and if it can't, it tries $ORGMA IL. If that f ails too ("deep, deeptrouble" as Stephen says in the man page), it exits without delivery and reports failure to the MTA ,which, depending on other f actors, will either requeue the letter and try delivering later or willbounce it to the sender.

19.29 Variable DROPPRIVS...I have procmail invoke d f rom a mailtable f or a virtual domain. Prese ntly that runs as root, inherited f rom se ndmail. I' d like tohave it run less privilege d. I tried chow n'ing the r c f ile to the us er I want used and se tting "DROPPRIVS = ye s". That didn't do it. S o I adde d "LOG NAME=user" and "US ER=$LOGNAME " bef ore the DROPPRIVS ass ignment and that didn't wo rk.

[philip] DROPPRIVS only has an ef f ect inside the /etc/procmailrc used when procmail is running indelivery mode (-d), not when it's running in mail f ilter mode (-m). USER and LOGNAME have no ef f ecton the working of DROPPRIVS, as procmail is just go ing to change to the uid/gid of the us er specif iedon the command line af ter the -d. Your mailtable entry should be specif ying the procmail mailer,which runs procmail in mail filter mode.

If the following are true:



procmail is running in mail filter modeno assignments we re given on the command linethe -p flag w as not specifiedthe rcfile specified is located under /etc/procmailrcs/ without backwards references ("/../"s)the rcfile is no t a directory (duh!)

then procmail will assume the uid and gid of the ow ner o f the rcfile. If the rcfile is actually a symlink,the procmail will assume the uid and gid of the link itself, not the underlying file. If your OS allowsanyone to give away ownership of files with chown, the procmail adds the following restriction to

those above:

/etc/procmailrcs must be owned by root and mode 700.

19.30 Variable HOME[david] Since procmail doesn't understand tilde, you have to use variable HOME instead.

CONTENT = `cat ~/file.txt` # Won't workCONTENT = `cat $HOME/file.txt` # ok

But accessing other user's home is another story. You could change the SHELL temporarily to getprocmail understand the reference, like this:

SHELL = /bin/cshCONTENT = `cat ~user/file.txt`SHELL = /bin/sh # restore original setting

Because the tilde is in $SHELLMETAS, when procmail sees a tilde, it will invoke a shell. It's better toskip the extra process o f a shell and use the $HOME variable: put a symlink somewhere under yourown home directory that points to the other user's file so that you can use the $HOME variable inyour .procmailrc and a void the shell invocation.

However, there are dangers on this too, because sysadm may move home directories and yoursymlinks may be out of date. If you expect such changes and broken links, then you could cache theneede d home directories at time you need them:

HOME_PHIL = `ksh -c "echo ~phil"`HOME_ED = `ksh -c "echo ~ed"`

19.31 Variable HOST[philip] If a assignment to the "HOST" variable occurs where the assigned value doesn't equal thehos tname of the machine on which procmail is running, procmail will stop reading the procmailrc, andif there are o ther procmailrcs specified o n the command line, it will start read ing them.

[david] It goes back to the early days of procmail, before Stephen thought of INCLUDERC or the "var

?? condition" syntax. When people had to use different code based on which local host machine wasprocessing a particular message, the method was to list a number of rcfiles on procmail's commandline. The first one wo uld start out with general code for all messages and all hosts and then have a

HOST = some.specific.machine



assignment, f ollowed by code f or mail delivered on that machine. If the first nine characters of "some.specific.machine" matched the real value of $HOST, procmail would stay in that rcfile; on amismatch, it would jump to the second rcf ile named o n the command line.

The second rcf ile would probably be for another particular machine, so (unless it f irst had someuniversal code f or all machines except the first one, or unless there w ere only two machines whe reprocmail might run) right a t the to p it would have

HOST = this.specific.machine

Again, a match for the f irst nine characters would keep procmail reading this rcf ile, but a mismatchwo uld make it jump to the next rcfile.

And so it went. A n incorrect HOST assignment (note that "HOST" alone attempts to unset thevariable, so it is always an incorrect assignment) in the last rcf ile on the command line made procmaildrop the message and exit. Since we almost never name more than one rcf ile on the command linenow , attempting to unset HOST in .procmailrc will have tha t eff ect.

I would guess that the only use of this original setup still around is in SmartList, where f list invokesprocmail with a number of rcf iles on the command line and uses things likeHOST=go.to.the.next.rcf ile.now to move from one to the next. A lso, procmail's -m f acility (which didn'texist back then) is incompatible with using HOST to jump among rcfiles, because it requires namingexactly one rcf ile on the command line.

Nowadays we can do something like this to use dif f erent rcf iles on dif f erent hosts:

:0* HOST ?? ^^\/[^.]+{

INCLUDERC = $HOME/.$MATCH.rc}

19.32 Variable L INEBUF...[manual] Length of the inter nal line bu f f ers , cannot be s et smaller than 128. All lines read f rom the rc f ile should not excee d $LINE BUF ch arac ters be f ore and af te r expansion. If not spe ci f ied, it def aults to 2048. This limit, o f course , doe s not apply to themail itsel f ...

Note: Beware of simply setting LINEBUF to a huge value: such an assignment causes procmail toimmediate ly allocate twice tha t much memory (procmail has two buf f er internally of size $LINEBUF).

[philip] Those 160 lines of condition are almost certainly overflowing LINEBUF. Y ou should either a)use one of the innumerable recipes sent to the list demonstrating the use of f grep; b) break it intomultiple recipes; or c) increase LINEBUF. If you modif y this list of domains regularly, then you shouldstrongly cons ider (a), as (b) and (c) just put of f it happening again.

LINEBUF only applies to lines f rom procmailrcs. Y ou generally only have to w orry about LINEBUF whenyou have a variable expansion or command expansion (back quotes) that doesn't have an obviousand reasonable bound on its size. procmail will avoid over running its LINEBUF length buf f er whendoing command expansions by ignoring the extra output, so you're safe there, as long as datatruncation is f ine. Variable expansion isn't checked like that, so you can cause procmail to core dumpby doing so mething like:

:0* ^Subject: \/.*|some-program $MATCH



then f eeding procmail a message with a huge Subject: header f ield: since no shell meta charactersappear in the action, the action line will be expanded and exec()ed by procmail directly instead of bythe shell. On the other hand, the following is f ine:

:0* ^Subject: \/.*

|some-program $MATCH ; ;

The semicolon f orces a shell invocation, and the shell should be safe. If your /bin/sh can buff eroverrun on variable expansion, then you're in more trouble than you know.

Action lines aren't the only place to watch your variable expansions. Variable assignments andcondition lines that have a leading dollar sign also undergo e xpansion. For example, this isn't saf e:

SUBJECT = `$FORMAIL -x Subject:`NEWSUBJ = "Subject: $SUBJECT"

procmail won't buf f er overrun in the f irst line, but a really long subject could cause the second to doso. The f ollowing should be saf e:

NEWSUBJ = "Subject: `$FORMAIL -x Subject:`"

but even then only if you're sure the shell is doing the expansion of NEWSUBJ.

Note that matching aga inst the value of a variable (using the "var ??" condition special) is saf e nomatter what the size of the contents of the variable. The problem is when you interpolate thevariable into something else .

I s the re any e asy w ay to know de f ault LIN EBUF value f or speci f ic procmail? I'm sure there 's a much easie r way, but this will work:

# Mitsuru Furukawa#$OUT = $HOME/tmp/linebuf.lst

:0 wc: $OUT$LOCKEXT

*$ ! ? $IS_EXIST $OUT| echo "$LINEBUF" > $OUT

[philip] If you examine the procmailrc manpage, you'll note that it lists f ourteen variables (amongthem DEFA ULT but not LINEBUF) whose values are reset in the environment by procmail, plus someadditiona l ones like IFS, ENV, PW D, and PA TH which come out of the top of config.h. Following this isa list of all of procmail's magic variables, including tho se f ourteen. The idea is that while procmail hasthirty magic variables, only fourteen of them are pu t into the environment by procmail.

The others may have default values, but they're 'input only': if wha t you're doing depends on one o f the o thers having a certain value, then you should just go ahead and set it to that value. I know of only two w ays to f ind out what value procmail is using by default: a) check the manpage (the manualpages should show the correct def ault f or the machine), or b) f ire up your f avorite debugger andhope that no one stripped the procmail binary.

There will be no error message when Procmail dumps core, even though the reason is apparently



precisely that LINEBUF is being exceeded too much.

I s the re a limit on the le ngth of a single line

[david] Yes, both before and af ter variable expansion and command substitution, it must be shorterthan LINEBUF characters. The exceptions are (1) comments and (2) commands that are run by a shellrather than directly by procmail. The entire condition must be under LINEBUF characters

Unf ortunately, LINEBUF seems to be a write-only variable; you can change its value but you can'tf ind out its current setting.

19.33 Variable L OG and LOGFILEIf you want to print something to the LOGFILE , you could do it like this

LOG = " This message goes to LOGFILE"LOG = " $NL$NL And this has linefeeds around $NL$NL"

Or like this, which proves to have some nice f eature in respect to VERBOSE setting:

dummy = " This message goes to LOGFILE"dummy = " $NL$NL And this has linefeeds around $NL$NL"

You see, if you set VERBOSE="off" Then the dummy lines are not printed and recorded to the LOGFILE .LOG messages are aways printed, and that's not very nice if you're trying to suppress messageswhile you call some subroutine:

saved = $VERBOSEVERBOSE = "off"

# Hope this subroutine does not use LOG# Eg. $PMSRC/pm-jaaddr.rc

INCLUDERC = $RC_ADDR

VERBOSE = $saved # restore original value

19.34 Variable TRAPHere is one example how to write to the log f ile, Be sure that you have preset all the variables, this

just demonstra ted the usage of TRAP. Pay attention to right use of single and double quotes if youpass the values to the shell. Like in this example where the /dev/ is removed f rom the FOLDERvariable's value.

TRAP = 'echo "FROM $FROMTO/CC $TO / $CCSUBJECT $SUBJECTFOLDER $LASTFOLDER" | sed -e "s#FOLDER /dev/#FOLDER #g"'

And if your MUA expects the file to be touched before it sees new incoming mail, here is recipe by



[david] :

TRAP = 'touch -m $HOME/Mail/$LASTFOLDER' # with strong quotes

Place it early in your rcf ile; then each recipe that saves to a directory can look simply like this, andthe trap will take care o f the touching:

:0 flags # no local lockfile needed for save to directory* conditionsdirectoryname/.

[david] Procmail terminates when it exits ... af ter f inal delivery of the message. It doesn't terminate(nor execute TRA P) af ter delivering a copy to a c recipe [however, a clone does execute TRA P w hen itterminates, unless you unset TRA P for it]. It doesn't execute the trap af ter a variable ass ignment, avariable capture recipe, a filtering recipe, nor any other non-delivering action.

On the other hand, it does execute the trap if you do a quick bail-out by unsetting or missetting$HOST.

[Recipe to record Subject lines on exit]

[Some Message had doubled Subject lines; david] ...this will list all subject lines in the log f ile uponexit if there are two or more. The earliest would appear twice: once in the trap output and once inthe logabstract.

:0* ^Subject:.*$(.+$)*Subject:

{# If there is already `TRAP' set, combine the# old trap recipe with this

TRAP = "${TRAP:+$TRAP ; }$FORMAIL -XSubject:"}

19.35 Variable UMASKThere is a be tter way to f ind out which f olders contain new mails if you are using procmail to f ilter the

mails. (This was a hack by one of my friends) procmail allows you to set UMASK on the folders. Sobef ore doing anything, set UMA SK to 076, which means the perms will be -rwx-----x to any folderwhich receives mails. now using find -perm -001, you can print the folders which have new mails.the shell script which does this w ill also have to chmod o-x on all these f olders.

...How doe s this work ? AFAIK umask only applies to new f iles cre ated and not to appe nding to existing f iles which is what procmail es se ntially doe s, rig ht?

[era] Procmail does interpret UMA SK this way, so this works, but I don't think it's a particularly goodsolution. It's actually hinted at in the documentation for UMA SK in procmailrc(5). find is a ratherheavy program to start up every time you want to look f or mail. (Haven't done any timings, though.)

I just grep -c '^From ' on my mail folders to see how many messages there are in them. (This

is only an approximation, in the case where one or more messages contain unescaped From_lines.)For a really pedestrian solution, keep all your spool files in their own directory (I think this is agood idea for other reaso ns as w ell) and do an ls -lrt on that directory, poss ibly piped into ased script to trim off files with time stamps older than, say, 24 hours.If your mail reader will rese t permissions on spoo l files when it gets mail from them, the UMASK



trick is a good base f or a mail checking script, but I would then only ls -l the spool f iles andlook f or files with an x01 permission.

19.36 UMASK and permissions My mail f older s ays -rw-r--r-x, Is there a bug in Procmail' s umask handling? (see last x bit )

[philip] That's a feature, not a bug! To quote the procmailrc(5) manpage:

UMASK : The name says it all (if it doesn't, then f orget about this one :-). A nything assigned toUMASK is taken as an octal number. If not specif ied, the umask defaults to 077. If the umask permitso+x, all the mailboxes procmail de livers to directly will receive an o+x mode change. This can be usedto check if new mail arrived.

Anyhow , normally , unde r Unix, the c re ate sys tem call wil l se t de f ault permiss ions o f 666 and the umask can only be us ed to masko f f the bits you don't want (and not to e.g. add x bits). S houldn't Procmail work this w ay, too, just to be consistent with the re st o f thesystem?

creat() will set the permissions to w hatever you w ant it to, modulus the umask. If the umask is zero,you can set the permissions to 7777, though that would be kind of stupid (and actually, mostversions of UNIX won't let you set both the sticky bit and an executable bit unless you're root, f orhistorical reasons). Most programs that call creat() or open(..,O_CREA T,...) give a mode argument of 0666, as they generally don't write out executables. Procmail just happens to call open() with amode argument o f 0667, to be modified by your umask.

19.37 Performance difference between back tick and "|"recipeProcmail sends the who le message to stdin whenever it sees back ticks used. A nd if you use recipe,you can add the h f lag to f eed only the header to the program, and not the whole message. Let'sask academic question: Which one of the choices below is efficient?

# Side effect: Do something with shelldummy = `echo hi there > some-file.txt`

:0 hwic| echo "hi there" > some-file.txt

Procmail sends whole message to first line and only headers to second recipe. A nswer: It doesn'tmatter. Either w ay procmail will make o ne w rite system call which will return 0 [bytes written] and of f it goes. Y ou should use the first one, because the latter affects the A and E f lags later, first one ismore clear overall.

While someone suggested f ollowing, it was rejected because it hurts perf ormance more [stephen] .The cat process is useless and directing to dev null does not buy anything.

:0 hwic| cat - /dev/null; echo "hi there" > some-file.txt

19.38 Procmail's temporary file names while writing file out...Any ideas w hat might make those .nf s* f iles? They contain message s which see m to have bee n successf ully processe d by

procmail in the late r parts of the .procmailrc . Howeve r, I doubt they'd eve r get cleaned up i f I didn't discove r them.

/disk3/home/foobar/Mail 119) ls -la backuptotal 22drwx------ 2 stanr 512 Nov 11 21:00 .drwx------ 3 stanr 2560 Nov 11 21:11 ..-rw------- 1 stanr 3063 Nov 4 03:31 .nfsA0c724.4

-rw------- 1 stanr 1780 Nov 3 23:00 .nfsA47da4.4-rw------- 1 stanr 849 Nov 3 23:22 .nfsA481f4.4-rw------- 1 stanr 2293 Nov 11 11:28 .nfsA737d4.4-rw------- 1 stanr 2598 Nov 11 20:39 msg.HCJB-rw------- 1 stanr 3127 Nov 11 21:00 msg.ICJB-rw------- 1 stanr 1884 Nov 11 20:45 msg.KCJB/disk3/home/stanr/Mail 120)

[david] procmail uses temporary name while it is trying to write a f ile out, which it renames if thingsgo well. I noticed that they all came f rom a 4h 31 span overnight; perhaps there w as so me systemswo rk being done o n your machine that screwed things up?

:0 ic| cd backup && rm -f dummy `ls -t msg.* .nfs* | sed -e 1,3d`

[aaron] When a f ile that is being used by a program on an NFS client gets unlinked the NFS serverrenames it to so mething like that. It should then actually get unlinked w hen the f ile is closed, but itlooks like the NFS server never got the close message f or those.

[Keith Pyle <keith A T ibmoto.com>] It is a result of using NFS, but the f ault lies with the operatingsystem on the NFS client. Keep in mind that NFS is stateless f rom the perspective of the NFS server.It keeps no inf ormation on how any file is being used. So, if a client tells the server to delete the f ile,the server deletes the file. This is not normally a problem, but many programs use a "trick" of Unixwhere the program opens a f ile, unlinks (deletes) it, and then continues to use the f ile. For all localf iles, the Unix kernel will not actually delete the file until all processes which have the f ile open exit.This works very well f or temporary files .

If a client tells an NFS server to delete a f ile, it will delete the f ile immediately because of thestateless nature of NFS. The server has no way of knowing if any client still has the f ile open. To

avoid this problem, if a client unlinks an open f ile on an NFS f ilesystem, the f ile is renamed to .nf s*where * is a unique value. The NFS client system is supposed to delete the .nfs* f ile when theprocess exits. However, there are some versions o f Unix which do not do this well (e.g., A IX). If oneof these OS's is used, it is common to find .nf s* files in various places. Theref ore, it is a good idea f orsystem administrators to periodically purge any .nfs* f iles over a certain age to eliminate theunsightly buildup in the f ilesystems.

19.39 Parameter $@[david] Of version 3.11pre7 procmail does not grok "$*", nor does it grok "$@" outside a pipe orf orward action. The only way to get the positional parameters all quoted together into "$*" issomething like this:

This doesn't work af ter all

ARGS = `echo "$@"`

Procmail subs titutes null f or "$@" there. T his wo rks, though:

:0 ir

ARGS=|echo "$@"

After that you use "$A RGS" instead of "$*".

If you try to set A RGS with A RGS="$@", procmail doesn't substitute for "$@" and makes $A RGS null.



If you try A RGS="$*" you ge t the literal text '$*'.

[philip] Of course, $A RGS dif f ers greatly f rom $@ in that $A RGS will either be split on whitespace (if unquoted) or one argument (if double-quoted). $@ has the cool property that if double quoted it'llstill be split into multiple arguments on the original argument boundaries. Since full-blown mailaddresses of ten have spaces, this distinction should not be casually dismissed. Note that w hile youmight not type in such an addresses, your MUA 's reply builder may.

19.40 Procmail variables are null terminated (detecting nullstring)You can't catch null in the message. Eg if you try like this

NUL=`/usr/bin/echo "\000"`

:0*$ HB ?? $\NUL{

LOG = "Caught NUL"}

[philip] It won't work as expected. The problem is that environment variables (and theref ore procmailvariables) are null-terminated, and theref ore cannot contain a nu ll. The above line creates an emptyvariable. The solution is to use an inverted character class :

NUL = `/usr/5bin/echo '[^\001-\377]'`

Note that procmail handles 8-bit characters except f or null in procmailrcs, so you can use a literalcontrol-A and o ctal-377 in your .procmailrc and save an echo and shell invocation right there.

19.41 FROM_DAEMON TO and TO_ and case-sensitiveness[david] ^TO is case-insensitive by def ault. Stephen once told me so mething to the ef f ect that tokenslike ̂ TO, ̂ TO_, ̂ FROM_DA EMON, and ^FROM_MAILER are always case-insensitive, even if the recipehas the D f lag, but I'm not positive that that was what he was saying, and we never pursued it.Certainly they are insensitive to case if there is no D.

[philip] If a regexp contains the ^FROM_DA EMON token, then that entire regexp is treated as case-insensitive. Other conditions in the recipe are not a f f ected by this. The other tokens have no ef f ecton the case-sensitivity. (This is w ith procmail 3.11pre4)

19.42 TO_ macro deciphered...What is the e sse ntial di f f ere nce be twe en TO and T O_ ?

[phil 1996-03-21] The difference is that ^TOalias1@site may match something like bobs-alias1@sitewhile ̂ TO_ won't.

[elijah 1997-09-16] Let's rewrite that in perl /x f ormat. See below. The def inition of the wordboundary in block (E). See below . The ^TO_ expans ion w as added in v3.11pre4. Y ou'll probably haveto just ^TO (no '_'), which should work almost as w ell.

/ # [begin regexp]( # [Block (A)]

^ # Anchor to start of line( # [Block (B)]

(Original-)? # Optionally proceed (C) with "Original-"

To: ([email protected]) {address}To: (fake {address}) [email protected]: Alice [email protected] , "W. Rabbit (late)"

 [email protected] , Gentle Reader <{address}>To: [email protected],

[email protected],[email protected], {address}, [email protected]

It will still f ail f or

To: (fake <{address}>) [email protected]

If someone is malicious enough to send you such mail.

19.44 FROM_DAEMON decipheredHere is the exploded FROM_DA EMON regexp as o f 3.11pre7

(^(Precedence:.*(junk|bulk|list)|To: Multiple recipients of|(

((Resent-)?(From|Sender)|X-Envelope-From):|>?From )([^>]*[^(.%@a-z0-9])?(

Post(ma?(st(e?r)?|n)|office)|(send)?Mail(er)?|daemon|m(mdf|ajordomo)|n?uucp|LIST(SERV|proc)|NETSERV|o(wner|ps)|r(e(quest|sponse)|oot)|b(ounce|bs\.smtp)|echo|mirror|s(erv(ices?|er)|mtp(error)?|ystem)|A(

dmin(istrator)?|MMGR|utoanswer

))( ([^).!:a-z0-9][-_a-z0-9]*)?

[%@> ][^<)]*($.*$.*)?)?$([^>]|$)

))

[era] explains the last regexps as follows:

(([^).!:a-z0-9] End of e-mail address token[-_a-z0-9] Another alpha token)? ... or maybe not;

[%@>\t ] Address separator -- either address@... or<address> or a bare address with whitespacearound it

[^<)]* Skip as long as we don't run into anotherbracketed address or end of comment(presumably to prevent this from matchinginside parenthesized comments in the firstplace)

($.*$.*)? Skip optional parenthesized comments andanything after them if found

)? ... or maybe not; maybe we just see an ...$ ... end of line instead([^>]|$) Uh, I should know what this is supposed to do,

but I can't quite remember what it's for. Ithink it had something to do with continuedheader lines ... Anyone?

Doe s ^FROM_MAILER match on the Return-Path: line ?

[david 1998-04-29] Apparently not, but it does match on the UNIX From_ line, which usually containsthe same address as the Return-Path: header.

Doe s anyo ne h ave an idea how I can use this macro but te ll it to ignore the Re turn-Path line in the h eade r?

There's probably some way within procmail without the extra f ork of f ormail, but this is easy to thinkof and easy to write:

:0h

HEAD_WITHOUT_FROM_=| formail -IReturn-Path: -I'From '

:0* HEAD_WITHOUT_FROM_ ?? ^FROM_MAILERaction

If you want to consider only the From: header, try this:

:0* ^\/From:.** MATCH ?? ^FROM_MAILERaction

20.0 Technical matters20.1 List of exit codesThe right place to look is /usr/include/sysexits.h, but the codes should be pretty much standard.These ones are f rom HP-UX 10 and the code that are mostly used are EX_NOUSER or EX_NOPERM. Ittells to the sender of UBE to "piss o f f and delete me from your list; I'm not here"

EX_OK 0 successful terminationEX__BASE 64 base value for error messages

EX_USAGE 64 command line usage errorEX_DATAERR 65 data format errorEX_NOINPUT 66 cannot open inputEX_NOUSER 67 addressee unknownEX_NOHOST 68 host name unknownEX_UNAVAILABLE 69 service unavailableEX_SOFTWARE 70 internal software errorEX_OSERR 71 system error (e.g., can't fork)EX_OSFILE 72 critical OS file missingEX_CANTCREAT 73 can't create (user) output fileEX_IOERR 74 input/output errorEX_TEMPFAIL 75 temp failure; user is invited to retryEX_PROTOCOL 76 remote error in protocolEX_NOPERM 77 permission denied

I thought that by us ing the E XITCODE, I w ould be ass ure d that the mail would be rejected but in f act Se ndmail 8.8.7 attempts todelive r the "use r unknown" to netcom.com, which is obviously w rong?

[sean] Sendmail accepts the message, then passes it on to Procmail, either as the local deliveryagent, or via a .forward file (depending on your system's configuration). Procmail says "gee, gotta lieabout not being here" and rejects the message, when is sent back into the spool, and deliveredaccording to who it appeared to come f rom.

Had SENDMA IL determined the user didn't exist (password file / aliases / virtusertable.txt), then itwould have rejected the message right when the remote was doing SMTP RCPT. But the user WA Svalid, and s o it accepted it.

Another scenario is when you have a mail secondary, and your primary (where the user account andprocmail are) is down. Some system goes to deliver mail to you, and resolves to your secondary –which s imply holds mail f or your primary – it hasn 't a clue which use r is valid and which isn't. Well, the(E)HELO (the system sending your primary the message) takes place during the SMTP session, themessage is coming from your secondary - not f rom the original sender. A t THAT point, if the userdidn't exist, I believe sendmail would be issuing an unknow n user error to the secondary, which inturn should mail that message back to who it thinks is the sender (I can't check my Bat book f romwhere I'm at - any sendmail pros are w elcome to e laborate).

is there any way at all to ge t around this ( f orce the reje ction at delive ry time)? Bette r yet, is ther e so me so rt of check to make surethat the Rece ived domain re asonably matche s the From: domain?

You'd need to have a ruleset in your SMTP Daemon (generally Sendmail) to check domains (whichWILL f ail on many valid messages , BTW) and reject it WHILE the SMTP delivery sess ion (actually, thenegotiation) is in progress. By the time Procmail has the message, you've completely accepted themessage, and any rejection you might hope to do is bouncing the mail - to the apparent sender.

Such is the problem with forged mail.

I wouldn't suggest this tactic for f ighting spam anyway - so much of it is f orged, and any bounce yousend out simply uses up system resources on your machine and those on the system that wasspoof ed. Spammers don't REMOVE addresses f rom their lists (they want the lists to look as big aspossible when they go to sell it to someone else) – some have even taken to GENERA TINGaddresses at domains and sending messages to them with the assumption that somebody willprobably have an account by that name ("bill@ joe@ dave@ ...").

Use procmail to trashbin (or otherwise file) all the junk and then manually take action on those whichget through.

20.2 List of precedence codes

The priorities most sendmails recognize are following. The lower the priority, the later the messagegets dealt with. A smart vacation program will ignore anything with a list, bulk, or junk priority. --Adam Shosta ck <adam A T bwh.harvard.edu>

0 first-class



30 list60 bulk100 junk100 special-delivery

[dan] You should use bulk when you distribute f iles via File Server. The value in the Precedence:header says absolutely NOTHING about the contents of the message itself , it merely suggests a

priority level to the mail system. From pp. 668 of the O'Reilly's sendmail book, bulk typically has avalue of -200 while junk -100; thus a message with junk will get higher priority than that of bulk(although this can be changed in the sendmail.cf f ile).

Other than on heavily loaded machines, this value won't matter anyway, since all mail will be quicklyprocessed.

[Stephen] ...Mail sent by a person is usually considered to be more important than autorepliesgenerated by some daemon. One way to express the lower priority of autoreplies is by adding a"Precedence: junk" f ield. This a llows mail transpo rt agents to make educated decisions about whichmail to f orward f irst (in case the mailqueue gets clogged).

Another point is: other autoreply services, like vacation . They try to make an effort not toaccidentally reply to a message generated by another daemon (e.g. yours). One way they detectthis is by looking at the Precedence f ield. If it contains junk , they know, this is not something weshould respond to.

20.3 Sendmail and -tse ndmail -t tag re ads T o, Cc, Bcc, etc, f or the rec ipient of the auto re sponse?

:0h* condition* !^X-Loop: foo@site\.com| ($FORMAIL -rA "X-Loop: [email protected]" ) | sendmail -oi -t

[david] That's not a problem, because f ormail -r will not generate any Cc: or Bcc: headers unless youtell formail to add them. The o nly line where sendmail -t will look f or recipients w ill be the To: line.

20.4 RFC822 Reply-To and formail problem with multiplerecipients[david] f ormail -r extracts only one return address, even when the Resent-Reply-To: or Reply-To:header contains more than one (and Stephen has told me he plans to leave it that way).

Looking for the best address to reply to is a completely different algorithm than looking for thebest group of addresses to reply to. Finding a group of addresses involves actuallydetermining that you even are searching for a group and not only for one address. Thenfinding out the best address for each. It's already a tricky business do ing this just for oneaddress.It makes thousands of autoreply recipes vulnerable to mail-storm attacks. Formail tries its bestto control the damage e ven if operated by someone who doe sn't know what he is do ing. If itwere to reply to multiple addresses at times, this damage control is severely undermined.

[dan]I understand these concerns; however RFC822 specifically allows for multiple recipients in aReply-To: header. Given that, it seems that there should be a straight-forward w ay to deal with thisin formail; even worse is that "formail" silently ignores multiple Reply-To: addresses .

For (a), wouldn't the Reply-To: (or Resent-Reply-To:) header supersede all other addresses and thusgreatly simplify the searching? For (b), how about o nly using multiple (Resent-)Reply-To: addresses if formail's "-t" option is also specified? Or if you are really worried about mail-storms and existingrecipes, a new formail option.

20.5 Procmail and IMAP server[ed] See also f tp://ftp.cac.washington.edu/mail/imap.vs.pop ...This paper is an e laboration on a shortnote entitled "Comparing Two A pproaches to Remote Mailbox A ccess: IMA P vs. POP", which waswritten in 1993 and recently updated. The purpose of this paper is to provide more extensivebackground on message access paradigms and protocols, and then to specif ically compare theInternet's Post Of f ice Protocol (POP) and the Internet Message A ccess Protocol (IMA P) in the contextof "online" operation.

...I lo g in to a se t o f NFS -e d se rvers (o r more pr ecis e ly AFS -e d), and my mail come s into anothe r se rve r (not a part of this se t )which is running IMAP. So se ndmail nev er delive rs mail into /var/mail/$LOGNAME on my login machines, and instead delive rsto the IMAP serve r. Since s endmail never reads my .f orward f ile in the ho me dire ctory, I f igure procmail never gets invoked.

You need a program which will fetch your e-mail f rom the IMA P server and then f eed it to procmail.One such program that can do this is f etchmail. Check out http://packages.debian.org/fetchmail Thebad news is that o nce you do this, you probably won't be able to use an IMA P client to read your e-mail anymore. But that might be good news if you pref er an MUA that reads mbox files but doesn'tgrok IMA P.

20.6 Machine which processes mail...The just-installed proc mail does not work and I am assuming that sendmail is trying to run procmail on another machine. I sthere anyway I could f ind out the appropriate ARCHITE CTURE f or that machine

[era] The f ollowing should tell you the name of the machine which processes mail f or the machineyou're asking about. You can then try to log in to that machine if you have shell access the re, whichis something you need to have in order to compile Procmail on it.

nslookup -q=mx machine # alternatively use host(1) command

If you don't have nslookup (doh) or don't understand wha t it says, try adding this to your .f orward

"|uname -a >/full/path/to/home/.uname.out"

i.e. this should be there in addition to what else you do. Otherwise this will lose your mailthoroughly, since it reads the mail but doesn't save it anywhere. Y ou might want to save a copy of allincoming mail to a saf ety mailbox, too, just in case. Like so:

/full/path/to/home/safetymailbox|"uname -a >/full/path/to/home/.uname.out"|"IFS=' '&& exec /usr/local/bin/procmail -Yf- || exit 75"

If you try this, it is very important that the f ile safetymailbox exists and is writable. ( man 5 forward if you have that – I don't seem to have this manual page on systems with newish versions of sendmail, is that correct?)

Try the uname command (and/or read the manual) to see what you should expect to f ind in the f ile.uname.out

20.7 Compiling procmail and MAILSPOOLHOME...I am compiling 3.11pre7 on a new sys tem and have a co uple of questions. I edited the makef ile to be the home directory" /ho me /a/abc" f or example. I def ined MAILS POOLHOME as " /mail". The inc oming mail is actually s tore d in " /us r/mail/ab c".W he n I pipe tes t message s through procmail (using "procmail< /us r/mail/abc" ), rathe r than the m ending up in my inbox, the y end up in a mailbox called "msg.gs.K B". What on earth did I go of up? As I sit here and think about this, sho uld MAILSPOOLH AS H be se t to 1 instead o f 0?

[philip] If incoming mail is supposed to be stored in /usr/mail/loginnamehere, then you should notdef ine MA ILSPOOLHOME at all, but rather def ine MAILSPOOLDIR to "/usr/mail/" and leaveMAILSPOOLHASH as 0. Def ining MA ILSPOOLHOME causes mail to be delivered to insides each user's

If you do n't use procmail and let Gnus to do a ll the splitting, you always see one huge inbox,$MAIL. It will not be split until you fire up Emacs and Gnus. If you're in a hurry, you may no thave time to start Emacs & Gnus, before reading the important messages. Your only option isto read all messages in $MAIL and try to find the o nes tha t consider e.g you work.

So, let procmail drop messages to their inboxes and Gnus to poss ibly "fine process" these inboxes.

21.4 Setting up Gnus for procmail - Basics

Procmail and Gnus communicate with each other very nicely when you use the mail backends like:nnml , nnmh and nnfolder . See Emacs info Gnus::Node: Select Methods for more.

Here are step by step instructions for reading the mail with nnml mail backend. We suppo se that youhave the following definition in your procmailrc so that the incoming mail is de livered to the rightdirectory.

The impor tant point here is that the name o f the g nus nnml group is ide ntical; exce pt the .spool suffix, to the s pool file whe re procmail write s. S o if you wr ite to list.procmail.spool , the group name in gnus is name d nnml:list.procmail

# .procmailrc excerpt

PMSRC = $HOME/pmMAILDIR = $HOME/MailSPOOL = $MAILDIR/spoolRC_LIST = $PMSRC/pm-jalist.rc

# The file name must be list.xxxxx.spool in order to# `nnml' to work in Gnus.Define procmail mailing list

PROCMAIL_SPOOL = $SPOOL/list.procmail.spool

# GNUS must have unique message headers, generate one# if it isn't there. By Joe Hildebrand [email protected] 

:0 fhw| $FORMAIL -a Message-Id: -a "Subject: (None)"

# detect mailing lists and store messages to spool directory

INCLUDERC = $RC_LIST

:0 :* ! LIST ?? ^^^^$SPOOL/list.$LIST.spool

Copy the Lisp code below to your ~/.gnusStart Gnus with M-x gnus-no-server (M-x means ESC followed by x). You will see Group bufferto appear.Make the new group with G m list.procmail RET nnml RET. You can read the group as usualand query new mail with g command.

(setqgnus-secondary-select-methods '((nnml ""));; See also nnmail-procmail-suffix which is .spool by;; default;;nnmail-use-procmail tnnmail-spool-file 'procmailnnmail-procmail-directory "~/Mail/spool/"



nnmail-delete-incoming t)

And then I have procmail always deliver to ~/Mail/spool/. If you add more inboxes, create theminside gnus Group buf f er with G m.

21.5 Gnus for procmail - More about itOkay, let's continue our journey in Emacs. What you read previously was the minimum you neededto get your Gnus to read procmail delivered f iles. However, if you're new to Gnus, here are somemore tips and basic instructions. The best advice I can give is that you go to each buf f er: In group,you press G C-h and in Summary C-h m and print the commands to printer that you see listed.

In Group buf f er

When you press g to get new mail to these groups, the group disappears if there is no mail. If you want the group to be permanently visible, then se t

(setq gnus-permanently-visible-groups "^nnml\\|^nnfolder")

In emergency, press `L' to list all groups.

If you made a mistake and wrote list.procmaill with an extra l accidentally in the groupname, use G r to rename group.

Raise o r lower the priority of your procmail mail groups with S l . Values 1 or 2 or 3 are goo d.Consider reserving 1 for your primary mail and 2 and 3 for mailing lists .

Whe n you exit a group and have read some articles, they won't show up next time you gothere. But by giving prefix argument before entering the group with SPC, Gnus will list a ll read

articles. You give the command like C-u SPC, where C-u is the prefix argument.

Settings

You want gnus to tell you everything it does

(setq gnus-verbose 10) ;; 0..10

You expire a rticles (get permanently rid of them) with the 'E' command in the Summary buffer.The default expiry time is 7 days. You can de fine the expiry time in days with

(setq nnmail-expiry-wait 7)

If you read mailing lists, you want automatic expiry when you have read the article. Use thefollowing to set up groups that use this automatic expiration.

(setq gnus-auto-expirable-newsgroups(concat

"procmail""\\|other-list""\\|and-some-other-list"))



B e in the Summary buffer expires current expirable articles .

If you want to kill an article; permanently remove it from disk, use B delete .

If you w ant to mark an article as persistent (never expires), use *

You don't want these mail groups cached because mail is already in "cache" format. The cacheis needed only when you read newsgroups and want to store message s locally.

(setq gnus-uncacheable-groups "^nn\$virtual\\|m[hlk]\\|db\$")

21.6 Emacs and Gnus – Fiddling with spool filesWell, to tell you the truth, managing Gnus is scary at first: You can make a lot of mistakes along theway or otherwise change your mind about group names and so on. It's a tricky task to move mailfrom one directory to another if you decide to rename the spool file name where procmail is puttingthe filtered mail.

Let's take an example: Say you decide to change the spool file name list.procmail.spool tomail.procmail.spo ol, because you come to think that all your mail groups should have the sa me prefix"mail." in your Gnus group buffer. You already changed procmail to output to that file, so now youhave two files sitting in your spool directory.

~/Mail/spool/list.procmail.spool~/Mail/spool/mail.procmail .spool # make sure this exists

Let Gnus read the old file as usual. Press g read new mail to list.procmail . list.procmail.spoolwill now be empty and merged to nnml backend file nnml:list.procmail.Make a new group with G m nnmail mail.procmail in Group buffer.Go to the old list.procmail group and select all articles with M P a . Move the messages w ith Bm to mail.procmail . You w ill see G marks appear to the be ginning o f moved a rticles.Exit the Summary buffer and hit g to see that the messages hat were transferred to your newmail.procmailKill the old group list.procmail with G DELOne more thing, remove that empty spool file. It is no longer used for anything.

% rm ~/Mail/spool/list.procmail.spool

21.7 Gnus article snippets[These articles have been collected from the GNUS hypertext archive]

I 'm also a bit confus ed with the propo se d solution of having procmail filter incoming mail in a nnmail-procmail-directory instead.

You have Procmail stuff mail in spool files, pre-sorted and filtered. Gnus then picks these up and s tuff the message s in the appropriate groups. Gnus uses movemail to actually move the mail out of thespoo l, and movemail uses locking that Procmail understands, so there is no danger o f mail loss .

Why are nnfolder-directory and nnmail-procmail-directory two differe nt dire ctories if nnmail-procmail-directory will contain the mail boxes that procmail appends to and nnfolder-directory is suppose d to be "All the nnfolder mail boxe s will be s tored under this directory"?

Because Procmail should stuff its mail in different folders, not in the ones that your regular mail isstored in.

Not all RFCs are standardsRFC 1036 specifically states that it is not an internet standard.The wording of RFC 1036 and 822 WRT to the RFC 1036 header is ambiguous. RFC 822specifically describes the format o f a mail message. It does not describe the complete formatof an electronic mail address.Nowhere in 1036 is there language requiring that the address be de liverable to. Further, 822provides language that w ould allow for a valid but not deliverable address to be acceptable.[822 doesn't describe addresses, it describes mailboxes , which are something s imilar but notidentical.]

The bottom line WRT RFCs that are informational is that when there is an ambiguity, or a differencebetween the RFC and the implementation, the implementation (which is wha t the RFC was trying todescribe in the first place) has precedent.

As much as y'all want it to be otherwise, the implementation of netne ws , (I. E. INND, NNTP) doesn'tcare about whether or not an address can be replied to. It is rumoured that some news postingsoftware checks the validity of an address . Such software is in a tiny minority.

[co unter argume nt 1998-03-25 gnu.emacs.gnus , Jan Vroo nhof <vro onho f A T fre ge .math.e thz.c h>] Now although IN ND and friends are impor tant parts of the Use ne t so ftware bundle the news READERS are e ve n more important. Now I 'll be t 99% re ade rs ,like f.i. Gnus, assume the addre ss in the he ader is the addres s to be re plyed to whe n the us er re ques ts to go into a privatediscussion w ith the author (i.e. reply instead of followup).

[marty] netnews is a public forum. mail is a private communication medium. Pos ting in a public forumdoes not require that I give you access to my private address, just a s speaking a t a public meetingdoes not require that I give you my unlisted phone number.

One thing is for certain: putting the burden on anyone wishing to send an mail to you, by requiringthem to decipher the address. Someone may never "reply by mail" to persons using those phonyaddresses. Anyone who wishes to send a personal mail cannot just hit 'reply'. People who do thisaccept this, which is they will watch the newsgroups for followups regularly. If someone eagerlywants to get personal, he can spend the extra minute to decipher the correct address for theperson. --Marty

[co unter argume nt, vroonho f] Howev er if you don't want to give me your phone number, w hy give me a false o ne? If peo ple withthis de sire at leas t put only their name and had no "<adress >" part then one co uld have the ne ws re ader s ay "Reply impossible , noaddress given".

[Counte r argume nt, unknow n] When I was using Pegasus Mail (Win95), it took me about 10 minutes to se t up filters that remo ve d ove r 75 % of the spam I re ce ived. 10 minutes is too gre at a burden to you? MY, what a busy pers on you are.

[timothy] What about the accounts from which I do not control (network at work) where I do nothave say over what software is installed? I can say to the sysadmin ``Hey I'd like Pegasus mailinstalled'' and he nods and mumbles something. He's got 2 years worth of backlog from there notbeing a real sysadmin around

[Counte r argume nt, unknow n] Furthe rmore , the re are a number of procmail rec ipes available on the net, that can be use d withminor adjustments to filter your mail. No he avy-duty Unix skills are req uired. Just the initiative to take res ponsibility for yo ur own

proble ms.

I know procmail very well, and spammers are still getting through. You know why? They refuse tofollow a ll the conventions w e depend o n. And they spam mailing lists, so I have to filter for that as

well. I have spent un told hours trying to develop better and be tter filters with lower numbers o f mis-hits. Nothing wo rks as w ell as no t giving more spammers my address.

...You s imply prefer to put the problem off on some body e lse , rather than take the time to deal with it yourse lf. Well, that kind of laziness doe s se e m to predominate in the "world of the I nte rnet" thes e days .

I have spent the time, learning from wha t others have done and seeking to improve them. You arecertain you are right and refuse to think about it anymore.... and that kind of laziness is all over theInternet.

The only one it wrongly inconveniences are those who need to mail me and have lost my mailaddress . If you w ant to followup a Usenet pos t, do it in Usenet. I'll be back here for followups. I getenough mail, and don't need mail for Usenet threads .

If you w ould like me to use a rea l address, please set me up an account w ith procmail where I canget all my Usenet related messages sent. --Timothy

22.2 Comments about addresses munging[1998-03- 24 gnu.emacs.gnus <drwho A T No-Spam-see- sig.xnet.com>]

...I am well aware that it is bad behavior, as I am well aware that it breaks s tandards. How ever, I'malso well aware o f the f act that I do no t need to have a mail-box f illed with spam every time I look atit. Things have quieted down considerably since I started altering my From: line. There's still theoccasional that gets me, though. It's not rea lly such a big deal right now, but af ter following the ne t-abuse newsgroups f or a while, it has become appa rent to me that spammers are trying new tacticsto grab mail addresses (msg id's, sender: lines , etc...).

Since I have to download most of my mail from a POP3 account, it takes time that I don't have towait f or all that spam to download. If breaking my headers means ge tting a few moments peace and

f reedom f rom spam, then s o be it.

[M. Maxwell <drwho A T No-Spam- see.sig> 1998-03- 26 gnus.emacs.gnus]

...Believe me, I d on't like having to do this header munging at all. But it saves me considerableaggravation. I also don't have to download my mail f rom a POP3 server (my ISP has a shell account),but I pref er to read mail off line s imply because I get so much of it with a ll those mail lists,

And since that's the case , I end up dow nloading plenty of junk along with the legit mail, af ter which,my local procmail puts it where it belongs. In other words, not in my inbox. A nd so I'll do what I haveto to f oil the spammers (until we get some so rt of legislation passed on junk mail). A nd those that doget past the f ouled headers are dealt with accordingly.

22.3 RFC and valid mail address charactersW hat characters are le gal in e-mail address es? S o f ar, I have uppercase , lowercase , digits, _ - + . @

[elijah] Most any 7bit character. For all practical purposes whitespace (space, tab, newline) are reallyinadvisable. This post is f rom a valid address. I also have ones with control characters – eg<@qz.to> (may not show up right in your newsreader). See RFC822 f or the f ull rules on generatingan address, but the quick and dirty thing is any o f the "specials" must be quoted to be used.

See definition of `specials' in RFCspecials = ()<>,;:\.[] and a double quote

If you don't believe me, there are mail toys to prove this. Best one I know of right now is TomPhoenix's "f red&barney"@example.com address . You can replace the "&" with just about any string Ibelieve. I've tried it with stuf f like "f red($)barney"@example.com and it seems pretty stable.

22.4 RFC and login-name@fdqn[1998-06-08 Message-ID: wkd8c jekay.f sf @m jf .vip.best.com Marty Fouts <Usenet-user A T usa.net> ingnu.emacs.help. Ref er also to summary of the whole thread in 1998-06-11 Message-ID:wk4sxs62ll.f sf @m jf .vip.best.com by Marty Fouts.]

>>>>> In article <[email protected]>,>>>>> Rich Pieri [email protected] enscribed:

> -----BEGIN PGP SIGNED MESSAGE-----

> Marty Fouts writes:

>> Sort of: system-name is not a hook into gethostbyname. The>> /variable/ system-name is set by a builtin defvar to>> gethostbyname. system-name returns the value of the /variable/

>> system-name, and the emacs lisp manual advises setting it if it>> is not correct.

> It still uses gethostbyname() to set the initial value.> gethostbyname() is supposed to return an fqdn on a networked



> host.

So? That the initial value is an FQDN is no indication that the value returned at any time thereaf terwill be. This is why emacs doesn't use system-name to create mail addresses, but has a separatef unction. If emacs itself doesn't rely on system-name to generate any mail addresses, why shouldgnus?

>>> user@fqdn is the agent responsible for submission of a>>> message to the network. user@fqdn is the RFC sender of the>>> message. user@fqdn therefore must be made to be a valid>>> mailbox.

>> This is just flat out wrong. There is no such requirement in>> any RFC or implied by any combination of RFCs.

> Premise: Gnus is used interactively. Premise: "user"> (user-login-name) is the login name of the person using Gnus.

And that's where you f ail first. There is no requirement anywhere in any RFC or combination of RFCsthat a login name even exist. Although your premise is true, it is irrelevant to your conclusion, asexplained below.

> Premise: "fqdn" (system-name, self-referential gethostbyname) is> the canonical network host name of the machine "user" is using at> the time.

And that's where you f ail second. There is no requirement anywhere in any RFC or combination of RFCs that the machine "user" is using be exposed as a part o f a mailbox. I am /allowed/ to do that,and if I do that I am required to support that mailbox as valid. I am not /required/ to do that.

I've already cited, and will repeat, that a TIP is a good example of such a machine. So is a POP3client. You are missing some more premises, most notably that user@f qdn is the sender of themessage in the sense of any RFC or combination of RFCs.

Most importantly, you are missing so me steps in your logic.

You have not es tablished tha t the /sender/ field's mailbox has to be the o ne you wouldconstruct from user-login-name@system-name, even on a system where such a combinationformed a valid mailbox.You have not e stablished that user-login-name@system-name be required to form a validmailbox, even if the system has the concept of a login-name and both user-login-name andsystem-name return what you expect them to.

Nor will you be able to, because there are no such requirements.

There is /no/ requirement /anywhere/ in any combination of RFCs that it be possible toconstruct a mailbox from the combination o f a "login-name" of any sort and an FQDN.There is /no/ requirement /anywhere/ in any combination of RFCs that a "login-name" evenexist.There is /no/ definition /anywhere/ in any combination of RFCs for the concept of a "login-name".

To put this as simply as possible:

You are incorrect to asse rt that there is any re quireme nt that a system s upport the mapping from (login-name,FQDN) to a mailbox of the form lo gin-name@FQDN.

Once you understand that this assertion is incorrect, it should be easy to see that all assertions

23.0 Introduction to E-mail Headers23.1 To find out more about mailReading Email Headers" - "A ll About Email Headershttp://www.stopspam.org/index.php?option=com_content&id=45 ...This document is intended toprovide a comprehensive introduction to the behavior of mail headers. It is primarily intended to helpvictims of unsolicited mail ("mail spam") attempting to determine the real source of the (generally

f orged) mail that plagues them; it should also help in attempts to understand any other forged mail.It may a lso be be neficial to readers interested in a general-purpose introduction to mail transf er onthe Internet.

[See also RFC po inters in the RFC section]

IMC – Interne t Mail Standardshttp://www .imc.org/mail-standards.html

FAQ archivehttp://www.f aqs.org/faqs/

UNIX EMail Sof tware

http://www.f aqs .org/faqs/mail/se tup/unix/part1/index.html ...This document is intended for systemadministrators who need to know how to set up their UNIX systems f or mail communication with theoutside world...UUCP, A ddresses, Domain A ddresses, FQDN, NIC, MX record, Bang-Paths, Gateways,Routers, Smarthost, MIME, X.400, "The maps", A liases

Plus addressinghttp://www.f aqs.org/faqs/mail/addressing/

The Unix MBOX, Berkeley, formathttp://www.qmail.org/qmail-manual-html/man5/mbox.html ...This f ormat comes to us f rom theancient UNIX mail program, V7 /bin/mail...Each message ends with tw o blank lines

[1998-09-06 PM-L Dallman Ross <dman A T netcom.com>] I would have thought the connection toBerkeley was /usr/ucb/mail (a.k.a. "Mail," with a capital "M"); not /usr/bin/mail (a.k.a. "/bin/mail").("UCB" stands f or "University of Calif ornia, Berkeley.") The two are close, though dif f erent enoughthat I get messed up if I try to use /bin/mail f or much. But "ancient UNIX mail program"? I use andpref er /usr/ucb/mail whene ver I'm in a UNIX shell. Many o thers do, too . <Yeesh.> (I don't like pine. Itf eels too GUI.)

Okay, sorry f or the digression, but you all were talking about the RFCs and From_ lines. If it's called"Berkeley Mail Format," then I'd inf er it comes f rom Berkeley Mail.

LiteratureDr. Bob's Pa inless Guide to the Interne t : & A mazing Things You Can Do With E-Mail by Bob Rankin NoStarch Press ISBN: 1886411093 List Price: $12.95

Netiquette by Virginia Shea Paperback 1 Ed edition (May 1994) A lbion Books ISBN: 0963702513Amazon.com Price: $19.95

The Elements of E-Mail Style : Communicate Ef f ectively Via Electronic Mail by David A ngell, Brent D.Heslop A ddison-Wesley Pub Co (C) ISBN: 0201627094 Paperback - 157 pages (A pril 1994) List Price:$12.95

All About Internet Mail (Internet Workshop Series, No. 7) by Lee David Jaff e Library Solutions Inst & Pr ISBN: 188220820X A mazon.com Price: $34.00

3 Rs of E-Mail : Risks, Rights and Responsibilities by Diane B. Hartman, Karen S. Nantz Crisp

Publications Inc. ISBN: 1560523786 Paperback - 153 pages (June 1996) List Price: $12.95E-mail Companion; Communicating Ef fectively Via the Internet and Other Global Networks by John S.Quarterman, Smoot Carl-Mitchell A ddison W esley Pub Co ISBN: 0201406586 Paperback - 318 pages(November 1994) List Price: $19.95



Date:Resent-Date: # if being redirectedFrom: # If not already presentSender: # if a From: is already presentX-Mailer: # what MUA composed this messageMime-Version:Content-Type: # what kind of stuff is in here

Content-Transfer-Encoding:Content-Length:

When the MT A receives the e-mail from the MUA at stage [3], it may insert additional headersshow ing the origination of the e-mail:

From # if local e-mail, automatic or by -f optionDate # If not already presentMessage-Id: # unique ID for the e-mail; the first MTA

# creates thisReceived: # shows inter-system e-mail tracking infoReturn-Path: # shows how to get back to the sender

As each MTA hands of f the e-mail, additional headers may get added, all as part of the MT A to MTAhandof f in stage [3]:

Received: # inserted by each MTA

As the f inal MT A hands the e-mail to a delivery agent (MDA ), in stage [4], there are still some moreheader insertions which may occur:

Apparently-To: # added if no To: header existsFrom # may get added if local e-mail

Some sites insert special rewrite rules and f iltering to occur to support virtual domains, and theseheader changes will occur at s tage [5], just bef ore the incoming mail is dropped. Generally, though,no new headers are added, except poss ibly one to avoid loops:

X-Loop: $USER@$HOST # inserted to avoid filtering loops

Finally, at stage [6] when the reader views his/her e-mail, most MUA s will apply a f ilter to the s toredmail causing selected headers to be omitted f rom the display. In a sense, then, this f iltering"removes" the headers from the user's view (although no headers are actually removed by theMUA).

The headers typically omitted are those inserted by the MT As, and those having to do with thetransport process and less with the contents.

23.3 Applied to received messages[alan] So, now that w e have a common understanding...

The first "From" is a Unix-mail From header (note the space). This is inserted automatically by MTA s,



unless o ne is already present and only then if it seems valid.

The second From: is generated by the MUA (your personal mailer), either by configuration, or by theuser. The rewrite rules in sendmail and most f iltering programs concern themselves with the From: ,To: , Cc: , Reply-To: headers.

I'll assume that if "From smmi" is not "correct", then you must be trying to hide the delivery process,and implementing something of a virtual domain.

In general, it is a bad idea to "correct" the automatic mail headers inserted by the MTAs. This is adif f erent matter than changing addresses to sho w virtual domains. The From_ header is part of thehistory of the message, showing how the mail was originated. Similarly, the "Received:" headersshould not be messed with. Changing the history of an e-mail message will make it very dif f icult todiagnose e-mail delivery errors.

That being said, and, since I also believe in the freedom of choice, I will now supply you with"enough rope to hang yourself " :^)

There are two places where you can have the From_ header corrected: just bef ore it gets droppedinto the mailbox (for incoming e-mail), or as it gets submitted to the MT A (for outgoing e-mail).

Changing the From_ before it gets dropped is easy. Just use a recipe like this:

FROM = `$FORMAIL -zxFrom:`DATE = ...construct the RFC date format

:0 fhw| $FORMAIL -I "From $FROM $DATE"

The From_ header is created automatically by the MTA (sendmail) when it receives a piece of mail. If the mail is sent through sendmail without using the '-f' option, then sendmail sets the def ault From_to that of the current user. If you are not root, or a "trusted user" (see the sendmail man page),then sendmail will ignore the From_ header and e ither remove it altogether o r replace it. Even if youare root, sendmail will replace the From_ , if the e-mail is being received locally (as opposed to f romthe netw ork).

If you wish to change the From_ , you must invoke sendmail, as root or a "trusted user", and use the"-f " option. EG: to set the From_ to match the From: header, use the f ollowing recipe, as root:

:0 hFROM=|$FORMAIL -zxFrom:

:0! -oi -t -f"$FROM"

Please read the man page on sendmail, noting the use of '-f '.

23.4 Bcc lecture by Alan StebbensProcmail mailing list 1996-11

Procmail most typically processes incoming mail at a destination site; the BCC f ormatting (or lack of

it) is done on outgoing mail, at the originating site.

For this discussion, let's make distinctions as to the kinds of mail there are: (a) incoming mail, and (b)outgoing mail. Bcc's are inserted into outgoing mail by the user, and the message is then handed toa MUA. T he MUA may then handle the BCC's or def er that to the Mail T ransport A gent (MTA ), such assendmail. Whichever agent performs the Bcc f unction, that f unction is perf ormed in at least three



dif f erent ways:

Many MUAs format outgoing mail without the Bcc: headers, so that the same message headercan be sent to all recipients. The Bcc: recipients receive an extra line in the message body,indicating the nature of the mail. The text of the message varies from MUA to MUA; The RandMailer, MH, for example inserts the lines around the o riginal text:

------- Blind-Carbon-Copy

...------- End of Blind-Carbon-Copy

Some MUAs will send the message , separately, to each Bcc: recipient, with the recipientaddress on the Bcc: header. Each Bcc recipient thus know s that they received the message byway of the Bcc, but do not know whom else was a Bcc recipient. All Bcc recipients are private,even to other Bcc recipients . (It would be nice if all MUAs behaved this way).

A few MUAs deliver the message without the Bcc:, but also without any special indication; youmust guess that it was a Bcc.

The original mail standard RFC822 says this about Bcc:

4.5 .3. BCC / RESENT-BCC

This field c ontains the ide ntity of additional recipients of the me ssage . The contents o f this field are not included in copie s o f themes sage s e nt to the primary and se condary rec ipients. S ome systems may choose to include the text of the "Bcc" field only in theauthor(s)'s copy, while others may also include it in the text se nt to all those indicated in the "Bcc" list.

So, procmail would handle Bcc's correctly if the sender's MUA included the Bcc in the header in thefirst place. But, since procmail is most typically used o n incoming mail, it will never have a chance todeal w ith Bcc: headers.

23.5 Bcc lecture by Philip Guenther

Procmail mailing list 1996-11The Bcc: header should in general not appear in an incoming message (if procmail is used forprocessing outgoing mail it may occur there). Most (?) Mail User Agents will send a bcc by justremoving the header entirely and putting the address in the envelope recipient list with the otherrecipients from the To: and Cc: headers. Done this way, the address to which the message wasbcc'ed *does not occur in the headers at all*, and you a re SOL.

By the time procmail is run (in the standard installation), the envelope is lost, which is the only wayyou would be able to process Bcc's with any possible regularity, and even that's suspect as if analias at ano ther site tha t contains your address is bcc'ed, then the envelope, by the time it reachesyou site, will only conta in your (local) address .

Furthermore, the whole point of the Bcc: header is that the people who receive the message do notknow the entire list of address to which the message was sent. If an alias is bcc'ed, it is not clearwhe ther the members of the alias should know that it was the alias that was bcc'ed and not just theindividual in question alone.

There MUST be some trace of the BCC des tination that trave ls with the e-mail. Otherw ise, how do es it know its de stination? If I'mright, then couldn't procmail use this to prope rly handle the me ssage ?

[alan]

Procmail mailing list 1996-11

Only the MTA knows the destination address because it is part of the "envelope", the information

which is passed on the "RCPT To: some-user" SMTP line. This information is how the MTA knows todeliver the mail, and not by the contents of the headers.

Of course, when invoked properly, many MTAs can read the headers to obtain the addresses neededon subsequent "RCPT" commands in the ensuing SMTP connections. In fact, the Bcc: header can beread along with the rest of the destination headers to obtain the recipient addresses, but the Bcc:

will also be removed f rom the headers.

The address by which an MTA receives a mail is known as the "envelope address", which may bedistinct f rom any headers in the message itself , or, the same as one of them, f or directly addressedmail.

With mailing lists, f or example, the addressee will never see his/her own address, but will see themailing list in the To: or Cc: header fields. Even here, when mail is addressed to more than onemailing list, there is a lack of standard f or determining the address by which a message is received.

There are lots o f conventions followed, and heuristics, bu t no clearly def ined standard to indicate thecause of delivery.

You may be able to conf igure your MTA to pass along the envelope in a new header, or pass it byargument to the local delivery program (which can be procmail). It is then up to the local deliveryprogram to use (or not) the envelope address inf ormation.

If you wish to understand the limits of your mail system, you should read RFC822 (mail f ormattingstandards) and RFC821, which describes the original language of SMTP. There are severalextensions in progress, but the basic commands of "MAIL", "RCPT", and "DA TA" should suf f ice.

24.0 Message headers24.1 What is correct From address syntax

C ase 1 is what is in RFC 822, as I re call. I re gard C ase 2 as "scre en syntax" f or inclusion in plain-text me ssage -body contexts. It could be use d f or the interactive pres entation of he aders , but I w ould be inclined to think any tool that doesn't accept the original

RFC-822 f orm is broken.

1. [email protected] (Personal Name)2. Personal Name [email protected] 

[1998-05-31 FA Q-L Simon Lyall <simon A T darkmere.gen.nz>] Both forms are legit but the way newsand standards documents are going is f or the f irst f orm to be discouraged. This efectively means thatsof tware should accept both forms but only generate the second (this is when the article is f irstcreated not by someone half way around the world).

The problem with the first f orm is that stuf f in brackets is actually a "comment" rather than the nameof the poste r. This means that there is no way using the f irst f orm to actually say what your name is,it is just that most people say their name in the comment field. They could just as easily saysomething else. This means that so f tware that displays the comment f ield as th name is just taking aguess.

The 2nd f ormat puts the name of the posted in a definite place that sof tware can work with andallows you to leave the use o f brackets f or comments . The current internet draft that on this that w illmost likely replace RFC 822 on this point is a t: http://tools.ietf .org/html/draf t-ietf -drums-msg-f mt-04

The bit is section 3.4 which says:

Note : Some le gacy imple me ntations used the s imple f orm whe re the addr-spe c appears w ithout the angle brack ets, but included the name o f the rec ipient in parenthes es as a comment f ollowing the addr-spe c. Since the meaning of the inf ormation in acomme nt is unspeci f ied, impleme ntations S HOULD use the f ull name-addr f orm of the mailbox i f a name o f the rec ipient is beinguse d instead of the legacy f orm. Also, be cause some legacy impleme ntations inter pret the comme nt, comme nts S HOULD NOT generally be used in addr ess f ields to avoid con f usion.

24.2 What's that X-UIDL header?[philip]

It's no t standardized, and will never be s tandardized by an RFC. (No X- header can be )Some servers use this to store information for the UIDL command.Some clients apparently store UIDL information in this header in the locally downloaded copy.(Note: the POP3 protocol doesn't let the client modify the message(s) stored on the server.)Some spamming software pa ckages include this header in messages they se nd to make somePOP3 clients that support client side filtering think that they've already filtered the message.



Filtering out incoming messages (pre-retrieval via POP3) seems 'fairly' safe, though s omelegitimate mail may include this header. Using it as a heavy weight (but not enough on itsown) in a procmail scoring recipe that detects spam appears to be reas onable.

[philip] If a message comes into your mailbox that has the X-UIDL: header, and do esn't haveyour address in the header, then I would have s trong doubts about its legitimacy.[ed] comments : E-mails with X-UIDL: headers a re almost definitely spam unless they've beenResent-To: me by someone . Also, valid X-UIDL: headers have 32 hexadecimal digits exactly.

[David] The advisability of trashing all mail with X-UIDL: headers has been discussed on procmail listrecently; apparently it's possible for one to appear in legitimate mail.

[Elijah] Yup. Very true. Mostly likely case would probably be for certain types of forwarded mail,including some moderated mailing lists. Fluffy's mod.* list had these until I pointed out the wide-spread file-to-/dev/null problem to Fluffy.

24.3 What is that first From_ header?[philip] the address on the From_ line is the envelope sender. If the message has a Return-Path:header, then it would probably be easier to use that instead, as then you don't have to deal withthe date as found at the end of the From_ header.

DON'T CONFUSE THE ENVELOPE WITH THE MESSAGE. The headers in the message are allowed tocontain a list of address in the To: and Cc: headers that are totally irrelevant to where the messageit going. For example, a message from a mailing list may simply sa y "To: p [email protected]", with no visible sign tha t "guenther A T gac.edu" is an address to which the message isbeing delivered. That information, where the message is currently in the process of being deliveredto, is found ONLY in the envelope.

Okay, where is this precious envelope? In SMTP the envelope consists of the MAIL FROM: and RCPTTO: SMTP commands. However, when a message is given to the local mailer, this information istypically lost. Well, the envelope sende r is usually saved now days in the Return-Path: header, butthe envelope recipient usually only appears in the form of the login name that the local mailer was

passed on the command line. This can be used, for example, by /etc/procmailrc scripts that check$LOGNAME to see where the message is set to go.

A problem arises however when people start creating virtual domains. When sendmail does thealiasing (usually by mailertable I believe?), it totally loses the original envelope recipient address inthe rewriting. All the addresse s get rew ritten to the sa me thing, and se ndmail thus has no reason todifferentiate them. Having lost their independent identities, the now-same multiple recipients aremerged to form one call to the local mailer.

The key point here is that once the envelope recipient is lost by the virtual domain alias, THERE ISNO WAY TO GET IT BACK! You can wave your hands and try faking it, but no one in the virtual domaincan ever get onto a mailing list or otherwise receive a piece of mail for which the header doesn't

explicitly contain his/her mail address. And furthermore, even doing that faking is extremely difficultto do right. What I show below does NOT correctly handle messages with Resent-* headers. Thiscan result in messages being received by people who shouldn't receive them, possibly violatingsomeone's privacy. Please keep all that in mind if you decide to use it. It handles a goodlypercentage o f the cases , but it'll bite you badly at some point in the future.

So you may ask, does this mean that virtual domains are hopeless? The answer is no, you just haveto be very careful in the sendmail.cf to keep the envelope recipient stashed somewhere long enoughthat it can be passed as an argument to the local mailer, usually by putting it in the 'host' part of themailer triple, though with sendmail 8.7.x, putting it into the local part with a '+' would probably beincredibly clean. In the end, it ends up being passed to procmail (standard /bin/mail has no way of handling this, but we already knew that) as another argument (i.e., -a orig-envelope-recip), though

with so me w ork it might be possible to do it via a new header, but that's uglier and no more efficient.I don't have the sendmail.cf (or m4 .mc) mods necessary to do this, but if you post tocomp.mail.sendmail (after checking the FAQ, I think it might be there) someone may be able to giveyou further pointers o n saving envelope recipients in virtual domain situations.

24.4 Message-Id header

...Are there known problems w ith "valid" mails with illegal Mess ageI Ds? For some s trange re ason, some pe ople are se nding out mail with bad me ssage id's. That wouldn't be much of a problem, e xcept that our MITS de partment won't ev en co nsider f ixing thebad-message-id unless it causes a problem somew here else .

Why would they not consider f ixing it? Their e-mail sof tware/gateway is broken, and needs f ixing.That's tha t. Direct them to RFC 822, sec 4.6.1. http://www.ietf .org/rf c/rf c0822.txt?number=822

[Gerald Oskoboiny <gerald A T impressive.net>] There are problems with Some of the problems withmail containing a bad message id

Some people (myself included) run f ilters to automatically delete incoming e-mail if its message-IDhas been seen recently, or if it looks bogus.

Some mailing list sof tware (including Smartlist) does not accept e-mail with a message-ID that hasbeen seen recently. Each message must have a unique message-ID. The best way to ensure thatmsgids are unique in a global context is to include a fully-qualif ied domain name af ter the '@'. Inparticular, a message-ID like <3.0.5.32.19971208192547.007db100@mailhub > is unacceptable f orthis reason (even if it didn't have a space at the end.)

Some mail archive so f tware (including some that I w rote) uses message-IDs as a unique identif ier f orthat message in the archive. It may reject messages that appear to be duplicates because theyhave a message-ID used by other messages . (as my software does.)

[generating message id]

[Stainless Steel Rat <ratinox A T peorth.gweep.net> 1998-03-13 in Emacs Gnus mailing list] ...it isstrongly recommended that Message-Id strings be generated by the MUA , rather than the MTA. Thereason being that a mail hub could be processing several messages at the same time (multipleCPUs), and so could accidentally generate duplicate Message-Id strings. The MTA should generateMessage-Id headers only when the MUA is stupid and f ails to do it.

[phil 1998-03-19 PM-L] ... let's do a quick work-up of a 'more complete' regexp to match Message-Id s.I'll take syntax lines f rom rf c822 with regexps that should match them. For ease of presenta tion, I'mgoing to work f rom the bottom up. Note: any brackets that only contain whitespace should reallycontain a space and a tab.

dq = '"' # (literal) double-quotebw = "\\" # (literal) backwhackws = "[ ]*" # whitespaceatom = "[-!#-'*+/-9=?A-Z^-~]+"word = "($atom|$dq([^$dq\]|$bw.)* $dq)'local_part = "$word($ws\.$ws$word)*"domain = "(\[$ws([^][\]|$bw.)*$ws\] |$atom($ws\.$ws$atom)*)"

:0*$ ! ^Message-Id:$ws $ws$local_part$ws@$ws$domain$ws {

...Catched illegal message id}

...I did start logging ids that match that condition. It matched two mes sage s s o f ar. One me ssage-id was cle arly bogus, but here'sthe other one (mailing list with 1 msg/we ek , no spam):

Message-Id: <[email protected].>

I s yo ur re ge xp incomple te wr t trailing dot in the domain part, or is the MUA/MT A broke n?

[philip] rf c822 doesn't allow a trailing dot. I just looked at the draf t of the new Internet MessageHeader Standard (the eventual replacement f or rf c822) and it doesn't either. Rather, it f urtherrestricts the syntax of generated Message-Id headers to disallow comments or f olding whitespacef rom occuring in the message-id itself.

however : bef ore you go tightening that regexp, note that the standard req uires that programs thatprocess messages must accept and parse messages tha t f it the obsolete syntax. This is because oldmail messages can hang around f or long periods of time in a way that most other internet dataf ormats don't see. The new requirements are on the generation of new messages, not on oldmessages.

[1998- 10-22 comp.emacs T oby S peight <T oby.Speight A T digitivity.com>]

It's more usual (and useful) to refer to news articles by Message-ID (that's what Message-IDs are

f or!). In this case

<news://[email protected] >

(though f or some reason this returns text/plain f or something which is clearly a message/rf c822).Either of which is an unambiguous URL, not subject to the same time-dependent changes. URLswere designed e xactly to remove the need for such descriptions.

24.5 Received header...Found another intere sting pattern, Rece ived he ader that are all on one line. Normally a Rece ived: he ader spans two lines, at leas t on all the mail I ge t. This f ilter lo cates the single line Rece ived: he aders and traps on that:

:0:*Received:\/( ?[^ ])*$mail/Spam

[Christopher Lindsey <lindsey A T ncsa.uiuc.edu>] No guarantees here. I just tried it out on sometest mailboxes (all known to have valid mail), and it matched like mad. A s f ar as I can tell, there's norequirement in RFC 822 f or multiple lines in a Received header.

[Reto Lichtensteiger <rali A T meitca.com>] The one line header vs. multi-line header is config'ed insendmail: A n older cf file (V8.7):

HReceived: $?sfrom $s $.$?_($?s$|from $.$_) \$.by $j ($v/$Z)$?r with $r$. id $i$?u for $u$.; $b

A later (V8.8) one:

HReceived: $?sfrom $s $.$?_($?s$|from $.$_)$.by $j ($v/$Z)$?r with $r$. id $i$?ufor $u; $|;$.$b

24.6 Return-Path...I've cre ate d a user (lo_mailer) with a .f orward and a procmailrc f ile to transpor t incoming mail to the right use r.T hat is working f ine, but the Re turn-Path: Line is se t to the local procmail use r (lo_mailer) and does not contain the original

Return-Path! W hat can I do to win back the original-line? Please help me :)

[david] Normally when you forward mail you should NOT keep the original return path. If thef orwarding des tination is invalid or unreachable, mail has to be returned to the f orwarder, who canf ix the f orwarding routine, not to the original sender, who can't do anything about it and probablynever even heard of the f inal destination address .

But, though you should change the return path, you do not have to lose the information that the

original return path contained. You can saf ely put that into the body or into ano ther header line. Trythis in lo_mailer's .procmailrc:

:0fwh # if there's a return path, save it as Old-Return-Path:* ^Return-Path:.*<.+>| formail -iReturn-Path: # lower-case i

:0Efwh # if there's no return path but there is a From_, use that* ^^From[ ]+\/[^ ]+| formail -A "Old-Return-Path: $MATCH"

:0Efwh # if there was neither a Return-Path: nor a From_| formail -A "Old-Return-Path: unknown"

The f irst set of brackets in the condition line of the second recipe enclose a space and a tab; thesecond set enclose caret, space, tab.

On the f orwarding leg f rom lo_mailer to the final recipient, the return path will be to lo_mailer, as it

should, but if the f inal recipient wants to know where it originated, he or she can look at the Old-Return-Path header.

There is one caution he re. If lo_mailer is taking mail to a general respo nse address and distributing itto specif ic people bas ed on sub ject or body content or just by rotation to balance the workload, f ine.But if you have a personal domain and your ISP is routing all mail f or any address in your domain toyour account on the ISP, and you're depending on procmail to deliver it to the right address in yourown domain by reading To: or Cc: headers, that is the wrong approach. The correct recipient will beon the envelope, which is removed from incoming mail bef ore procmail can see it. Your ISP has to dosomething that lets you know the true envelope recipient or recipients of a message, and othershere know a lot more about that than I do (and way, way more than I could tell you without makingmistakes).

[1998-11-11 Gnus-L Karl Kleinpaste] With regard to the standards f or Return-Path, RFC822 observesthat it should be a route back to the originator, i.e., it should show relay hops; RFC1123 in turn saysthat failure notifications should be sent back to the originator with the route information deleted,that is, "If the address is an explicit so urce route , it SHOULD be stripped down to its f inal hop." ???Then what's the point of providing the so urce route in the f irst place?

It seems to me that Return-Path's v alue has become very limited in an environment where source-routed mail is vastly deprecated, and just plain not supported by many. I know that, when I didserious sendmail work years ago , I shot all source routes on s ight.

You could very well substitute the use of user-login-name for the "-f" argument in sendmail with the

value user-mail-address; the result should give the eff ect you need, and not create anyinteroperability problems – mail will still show a proper way to return to you.

That said, this mailing list's requirement of matching Return-Path is indeed pretty peculiar.

24.7 Errors-To1) C an somebody con f irm that E rrors-To: is depre cated? 2) Is there an RFC f or this?

[1998-09-15 Liviu Daia <daia A T stoilow.imar.ro>] 1) It is an UUCP thing, and it's indeed deprecated.Here's the relevant quote f rom sendmail's manual. 2) Probably not, since UUCP-related RFCs haven'tbeen updated in a while.

I f errors occ ur anywhere during proces sing, this header will cause error me ssages to go to the listed addresse s. T his is intende d

f or mailing lists. T he Errors-To: heade r is o f f icially deprec ated and will go away in a f uture release .

24.8 X-Subscription-InfoThis is a header that is used by some mailing lists: it contains an mail address for un/subscribe, or aURL with said inf o. Imagine the reduction in bozo messages asking how to unsubscribe f rom mailing

The f law the IETF f ound is that Joe is equating his two mailboxes w ith his private and work accounts.There is no such correspondence as far as RFC 822 is concerned. If Joe is acting in a "private"f ashion, the system he is using is irrelevant; his private mailbox belongs in the From header and heshould put that mailbox there when he o riginates the message, regardless of where he physically iswhen he does so.

24.10 Mail-Copies-To header[Suggested by Lars Magne Ingebrigtsen, the A uthor of Emacs Gnus]

...Mail-Copies-To: is a header line used in messages on Usenet to direct copies by mail of f ollowupsto posts. http://www.newsreaders.com/misc/mail-copies-to.html

[SL Baur <steve A T xemacs.org>] The Mail-Copies-To: header should control how your mail (andUsenet) client prepares a f ollowup message. It gives control to the sender of a message whethercourtesy duplicate copies of messages should be sent. There are two forms:

Mail-Copies-To: never

Do not automatically include the sender of the message being respo nded to. There are two canonicalexamples.

Usenet:From: [email protected]: comp.emacs.xemacsMail-Copies-To: never

A f ollowup in a conf orming client sho uld generate in the response message headers:

Newsgroups: comp.emacs.xemacs

Email:From: [email protected]: [email protected]: [email protected]: never


To: [email protected]: [email protected]

The second form includes a properly formed RFC822 mail address as the parameter:

Mail-Copies-To: [email protected]

In this case, the sender of the message is specifically requesting that responses to the message notonly go to the main f orum (either mailing list or Usenet news group), but a duplicate copy should alsobe sent to [email protected] . There are (again) two canonical examples.

Usenet:From: [email protected]: comp.emacs.xemacsMail-Copies-To: [email protected]


Newsgroups: comp.emacs.xemacsCc: [email protected][1]

Email:From: [email protected]: [email protected]: [email protected]: [email protected]


To: [email protected]: [email protected], [email protected][2]

There is no requirement that the address in Mail-Copies-To match the From address. Footnotes: [1]Or `To: f [email protected]' [2] It is also acceptable to put [email protected] in the To: line.

24.11 Mail-Followup-To and Reply-To-Personal headers[21 Nov 1997, Mutt Development List <mutt-dev A T cs.hmc.edu]

Jacob Palme just today submitted an Internet-Draf t describing Mail-Followup-To. Jacob, the WorkingGroup chair Chris Newman and I all regard this as complementary to my own Reply-To-Personalproposa l, an early version of which I posted here and which was also submitted as an Internet-Draf t

just today. In f act had me w eek been a bit less ha rried Jacob and I wo uld have issued a joint draf t.Within a f ew days you should be able to view these draf ts in the IETF draf ts directory onds.internic.net under the names

draf t-ietf -drums-mail-followup-to-00.txt Jacob Palme's draf t on the proposed Mail-Followup-Toheader.

draf t-ietf -drums-replyto-personal-00.txt My draf t on Perso nal-Reply-To

24.12 Content-Length header and From_ specification[1996-05-17 From: Jamie Zawinski <jwz A T netscape.com> comp.mail.headers]

...I'm not saying that the BSD Mailbox f ormat is good. Just that the Content-Length variant of thatf ormat is wo rse.

Ok, so someone took the From_ f ormat, and extended it to no t require mangling by adding a lengthindicator to the f ormat. A t f irst glance, this may sound simple and e legant, but it breaks the world,and one shouldn't encourage its use to spread.

The thing that breaks is taking an existing, widely-implemented f ormat, and adding a requirementthat it have a length indicator. This means that any existing sof tware that already thinks it knowshow to manipulate that format is going to damage the file (any change to the data will cause thelength indicator to be wrong with respect to the new specif ication but not with respect to the oldspecification.)

If the content-length-based format was no t otherwise- indistinguishable f rom the ``From '' format,there wouldn't be a problem; the old sof tware would simply f ail to work with this new file format,instead of `corrupting ' the documents (in quotes, because it's really just a matter of which specyou're f ollowing.)

Also, mailboxes are by their nature a textual f ormat; but, the content-length header measures inbytes rather than lines. This means that if you move the file to a system which has a dif f erent end-of-line representation (Windows <=> Mac, or Windows <=> Unix) then the content-lengths willsuddenly be wrong, because the linebreaks no w take tw o bytes instead of one, or vice versa.

It's impossible for a mail client to look at a file, and tell which of the two formats (From_ or Content-Length) it is in; they are programmatically indistinguishable. The presence of a Content-Lengthheader is not enough, because suppose you were on a system which knew no thing at all about thatheader, and some incoming message just happened to have that header in it. Then that headerwo uld end up in your mailbox (because nobody w ould have know n to remove or recalculate it), and itwould possibly be incorrect. (Presume f urther that the header was not just incorrect, butintentionally malicious...)

Stricter parsing of the ``From '' separator line doesn't help either, because there are many, manyvariations on what goes in that line (since it was never standardized either); and also, some mailreaders include that line verbatim when forwarding messages (Sun's MailTool, for example) so a

stricter parser wouldn't help that case at all, because message bodies tend to contain valid matches.

Some mail readers attempt to cope with this by recognizing the case where the Content-Length isnot obviously spot-on-target, and then searching f orward and backward f or the nearest messagedelimiter; but this is obviously not f oolproof, and makes one's pa rser much more inef f icient (requiringarbitrary lookahead and backtracking.)

Conventional wisdom is, ``if you believe the Content-Length header, I've go t a b ridge to sell you.''

24.13 Moral about CC copies in UsenetSending CC

There has been very heated discussion in the gnu.emacs.gnus (e.g around 1999-03-20) newsgroupwhere many people argue f or sending CC replies to the person thet posted the question to thenewsgroup. The benef it of sending CC has been seen as:

The person gets fast answer.The person may not read the new sgroup regularly and appreciates the private answerThe newfeed for him may not be very reliable, so the answ er may not appear fast in the g roup(but we don't know this for sure)The newgroup expiry period may be too fast for him to catch the reply (but we do n't know thisfor sure).

In recent years the netnews has changed a lot and many people have started using non-existingmail address in order to prevent getting UBE mail. This has made the "CC" senders annoyed,because they get bounced mail from these non-existing addresses.

Not sending CC

Usenet is considered a public forum, which does not force anyone to reveal their "real" address if they don't want to. It's the same as lock in their doors. Some people don't want to see non-invitedpeople in their doors and that's why they don't like CC messages too :

The CC is superfluous: The answer has already posted to newsgroupa CC w on't help following a thread. Person has to visit the new sgroup to see the wholediscussion anyway.

A CC is subjected to mail delivery problems: Person has moved, mail delivery problem (keeptrying for N days), transient failure..He always wa nts to read the ne wsgroup and do esn't like CC copies to fill in his mailbox inexpensive ISP account.

A Clear munged address



An clear non-existing mail address that indicates that it is not the real destination is usuallyconsidered good manners:

[email protected]@[email protected]

Or partially modif ied, that a human mind can "decode" if a direct contact is wanted (but somewhathard to programs, because there are more creative choices that what program can ever expect tosee):

johnx.you-know-what-todo@not-here.skynet.example.comdoor.lock.mike@[email protected]

A valid looking address

But an address tha t looks like a "real", but is bogus , is not a po lite way to participate in Usene t. Thisaddress wold give an impression that persn is really there:

k f d l

procmail recipes

Documents