From: (Jari Aalto+mail.procmail) Subject: Procmail pointers Summary: This post contains excerpts from Procmail Tips page at which contains lot of infomation about Procmail, ube (aka spam), email headers, RFC headers etc. Athor's homepage is at http://poboxes.com/jari.aalto/ Newsgroups: comp.mail.misc, comp.answers, news.answers Followup-To: poster Date: 31 Dec 2003 02:02:26 GMT Organization: none Archive-name: mail/procmail/pointers Posting-Frequency: 2 times a month Maintainer: Jari Aalto Announcement: "Procmail resources" Last updated: 1999-04-21 v1.72 pm-tips.txt Faq archive is at http://www.faqs.org/faqs/ The following list below is an excerpt from a bigger document. http://www.procmail.org/jari/pm-tips.html My Procmail module library page is at http://www.procmail.org/jari/pm-tips.html Era's exellent procmail link page (including procmail FAQ) is at http://www.iki.fi/~era/procmail/links.html 4.0 Procmail pointers 4.1 Where to get procmail binary ftp://ftp.informatik.rwth-aachen.de/pub/packages/procmail/ 4.2 Where is procmail developed Philip Guenther is currently taking care of and coordinating procmail bug fixes. Please send any procmail bugs to the mailing list or to . The development mailing list is running SmarList at . Furher patch and bug info can be found at: http://www.gac.edu/~guenther/procmail/todo.html http://www.gac.edu/~guenther/procmail/warts.html Newest Procmail code: http://www.procmail.org/ ftp://ftp.procmail.org/ 4.3 About procmail's Y2K compliance Please consult Philip Guenther for more up to date details. Philip is the Procmail maintainer currently. [1998-09-23 Bennett Todd in Message-Id: <19980923164230.C30594@fcmc.com>] Well, from a simple ogle of the grep over the sources, it looks like there may be a Y2038 problem in the autoconf test code: unsigned otimet = time(). And another, possibly less likely to express itself, in formail.c: unsigned long h1 = time(). Those could express themselves when 32-bit signed time_t wraps; long before then the time_t define should have been changed to something that is bigger, even it's "long long". The above type-mixes may fail to profit from a suitably redefined time_t, and so may overflow on 2038. I don't see any Y2K problems, though. And email headers use four-digit years pretty consistently, so that should all be cool. This estimation doesn't constitute an in-depth Y2k audit of procmail, but the source code to procmail is ... kinda dense for in-depth auditing. [1998-09-25 Bennett Todd Message-Id: <19980925093902.B12428@fcmc.com>] As I see it there are at least three measures that a whole email system, taken in aggregate, could use for Y2K checking. First, capture a vast cross-section of traffic and make sure no email software is using 2-digit years. I don't recall having seen any, but it's still worth checking. Second, generate a load of traffic with 2000 and 2001 dates and shove it through all the channels. And third, run all the systems end-to-end with their system clocks rolling over the millenium. 4.4 Procmail mailing lists Traffic in this list is about 5-20 messages per day. Do not join if you can't handle that much traffic. The list is run by SmartList, which is a procmail based list software. ._MailingList_: questions/answers .subscription requests .digest request To get off the procmail mailing list To get off the list: send a message to *procmail-request* with: unsubscribe user@domain in the subject line unsubscribe first line in the body If that fails, try email to (purportedly that should go to a person). See also the original subscriptions message that you will received http://www.iki.fi/~era/procmail/welcome.txt 4.5 Procmail recipe modules and faqs Procmail is discussed in usenet newsgroup *comp.mail.misc*. "Procmail archive" ftp://ftp.informatik.rwth-aachen.de:/pub/packages/procmail/ Articles from procmail mailing list: covers from 1994-08 to 1995-05 (A .gz file: ~2Meg when uncompressed) And Latest articles can be found here, hosted by Achim Bohnet Covers from 1995-10 to the present day. . The www page has nice search capabilities. http://www.rosat.mpe-garching.mpg.de/mailing-lists/procmail/ http://www.rosat.mpe-garching.mpg.de/~ach/exmh/archive/procmail/ "Era's Procmail faq" http://www.iki.fi/~era/procmail/mini-faq.html http://www.dcs.ed.ac.uk/~procmail/faq/ [mirror] Also available by email, the ITEM can be: links.html, mini-faq.html, procmail-faq To: Subject: send ITEM "Era's Procmail Link collections" http://www.iki.fi/~era/procmail/links.html ...A page with full of good links to the world of procmail "Catherine's Getting Started With Procmail" http://shell3.ba.best.com/~ariel/nospam/proctut.shtml This is a quick tutorial intended to get a procmail neophyte started using procmail with as little trouble and fuss as possible. "Joe Gross's short Procmail tutorial" http://www.procmail.net/ ...Using procmail and a feature of ph you can set up your own mailing list without neededing root on your own machine. "Unix manpages" http://www.xs4all.nl/~pater/manpages/ ...If you don't have procmail manpages at hand, check this site. It contains wealth of Unix related manpages online. Jeroen Paternostre 4.6 Procmail Lint service (code check) _NOTE_: This service has closed 1999-04-20. It has been mentioned here for historical reasons only. If you have and can use Emacs, please download the Procmail programming mode, `tinypm.el', that [jari] has written. Lint is included in there and it can auto-correct mistakes on the fly. You can get it from the mentioned _uta_ ftp site (get tgz kit). Because not all people know how to use Emacs, how to use Emacs lisp packages or are otherwise clueless about Unix tools, I put up a procmail based Lint service where you can send your code. To: jari.aalto@poboxes.com Subject: send pm-lint.hlp This service is highly experimental and if traffic starts to get too high, I have to close it because every message to the lint starts a background Emacs process and it consumes server resources. The preferred way is that you get your own Emacs package and Lint your code locally. When you send a message to the Lint it will respond to you with a message similar to this one: *** 1997-11-24 22:13 (pm.lint) 3.11pre7 tinypm.el 1.80 cd /users/jaalto/junk/ pm.lint:010: Warning, no right hand variable found. ([$`'] pm.lint:055: Pedantic, flag orer style is not standard `hW:' pm.lint:060: Warning, message dropped to folder, you need lock. pm.lint:062: Warning, recipe with "|" may need `w' flag. pm.lint:073: Warning, Formail used but no `f' flag found. 4.7 Procmail module list Where to get the modules The UBE stop procmail modules are not listed here. See pointers in "procmail code" section later. o All pm-ja*.rc modules are in Jari's procmail kit. Also available ay Procmail code library page is at http://www.netforward.com/poboxes/?jari.aalto --> pm-code.html o Other modules are by Alan Stbbens http://reality.sgi.com/aks/ o 1998-12-08 Eli the Bearded <*@qz.to> announced in comp.mail.misc that he had put his own procmail modules to visible at http://www.qz.to/eli/src/procmail/. You may find interesting procmail code there but the modules itself are not general purpose *plug-in* modules that you could use right away. Some functionality included: Inline decoding of MIME text attachments (rc.mime-decode) Cleansing of obscure "Re:" formats in subject (rc.pre-list) Nifty autoresponder (rc.qz-2) Sophisticated dupicate email catching (rc.dupes) Example of using my mail bouncer (rc.lists-out) Detection of some classes of autoreplies (rc.daemon) Various junk mail filtering (rc.filter) Daily log files (rc.vars) Terminology *subroutine* = A piece of code that gets something in `INPUT' and responds with `OUTPUT'. Subroutine is not message specific. *recipe* = A piece of code, that is somewhat self containing: It reads something from the message or does something according to matches in message. Recipe may be message specific. Foreword to using modules In the module listing, some of the modules are recipes and some can be considered subroutines. Let's take the address explode module that was discussed a while ago. First, visualise following familiar programming language pseudo code: (ret-val1, ret-val2 ...) = Function( arg1, arg2, arg3 ...) *Function* may return multiple arguments and multiple arguments can be passed to it. Clear so far. Let's show how this applies to procmail modules: RC_FUNCTION = $PMSRC/pm-xxx.rc # name the subroutine/module RC_FUNCTION2 = ... INPUT = "value" # Set the arg1 for module INCLUDERC = $RC_FUNCTION # Call Function( $arg1 ) :0 # Examine function ret val * ERROR ?? yes ... This should be pretty clear too. You just have to look into the subroutine/module, which you intend to use, to find out what arguments it wants which you _need_ _to_ set (INPUT) before calling it. The documentation also tells you what values are returned, eg. one of them was ERROR. If it were recipe/module, the call would be almost the same, but instead of returning values, the recipe/module most likely does something to your message or writes something to the data files or etc. *Recipe/module* is much higher level hierarchy, because it may call multiple subroutine/modules. The distinction between subroutine and recipe module type is not crystal clear, but I hope the above will clarify a bit the Procmail module/subroutine/recipe concept. Header file modules These are like #include .h files in C, they define some common variables, but do not contain actual code. o pm-javar.rc -- Defines standard variables: SPC WSPC NSPC SPCL and perl styled \s \d \D \w \W and \a \A (alphabet only) o headers.rc -- From Alan's procmail-lib. Define standard regexp and macros: address, from, to, cc, list_precedence General modules o *pm-jafrom.rc* -- Derive FROM field without calling `formail' unnecessarily. If all fails, use formail. o *get-from.rc* -- From Alan's procmail-lib. get the "best" from address. Sets FROM and FRIENDLY, the latter being the "friendly" user name sans any address. o *pm-jaaddr.rc* -- Subroutine to extract various email components from INPUT. Like address=foo@some.com, net=com, account=foo... o *pm-jastore.rc* -- Subroutine for general mailbox delivery. Define MBOX as the folder where to drop message and this subroutine will store it appropriately. Supports single mboxes, ".gz" mbox files, directory files and MH folders with rcvstore. Date and time handling For these, you extract the date from somewhere first and then feed the string to some of these subroutines: o *pm-jatime.rc* -- a lowlevel subroutine. Parse time "hh:mm:ss" from variable INPUT o *pm-jadate1.rc* -- a lowlevel subroutine. Parse date "Tue, 31 Dec 1997 19:32:57" from variable INPUT o *pm-jadate2.rc* -- a lowlevel subroutine. Parse ISO standard date "1997-11-01 19:32:57" from variable INPUT o *pm-jadate3.rc* -- a lowlevel subroutine. Parse date Tue Nov 25 19:32:57 from variable INPUT o *pm-jadate4.rc* -- Call shell command "date" once to construct RFC "Tue, 31 Dec 1997 19:32:57" and parse the YY MM HH and other values. You usually use this subroutine if you can't get the date anywhere else. Date and time handling You use these recipes to get the date directly from the message: o *pm-jadate.rc* -- higher level recipe. Read date from message's headers: From_ Received, or call shell `date' if none succeeds. o *date.rc* -- higher level recipe. From Alan's procmail-lib: parse date or from headers Resent-Date:, Date, and From Forwarding and account modules o *pm-japop3.rc* -- Pop3 movemail implemented with procmail. You can send a "pop3" request to move your messages from account X to account Y. Each message is send separately. This recipe listens "pop3" requests. o *pm-jafwd.rc* -- controlling Forward remotedly. You can change the forward address with a "control message" or turn on/off the forwarding with a "control message" o *pm-japing.rc* -- Send short reply to subject containing word "ping" to show that the account is up and that email address is valid. o *correct-addr.rc* -- From alan's procmail lib. To help forward mail from an OLD address to a NEW address, and do some mailing list mail management. This recipe file is intended to make it easy for users to forward their mail from their old address to a new address, and, at the same time, educate their correspondents about it by CC'ing them with the mail. Vacation modules o *pm-javac.rc* -- A framework for your vacation replies. This recipe will handle the vacation cache and compose an initial reply; which you only need to fill in. (Like putting vacation message to the body) o *ackmail.rc* -- From Alan's procmail lib. procmail rc to acknowledge mail (with either a vacation message, or an acknowledgement) Message-id based modules o *pm-jadup.rc* -- Handle duplicate messages by Message-Id. Store duplicate message to separate folder. o *dupcheck.rc* -- From Alan's procmail-lib. If the current mail has a "Message-Id:" header, run the mail through "formail -D", causing duplicate messages to be dropped. Can use md5 cache. Cron modules o *pm-jacron.rc* -- A framework for your daily cron tasks. This recipe contains all the needed checks to ensure that your includerc is called whenever a day changes. (Day change is subject to messages you receive). Your own cron includerc is run once a day. Backup modules o *pm-jabup.rc* -- Save messages to backup directory and keep only N messages per day. Idea by John Gianni, packaged by Jari. Note: The implementation will always call shell for each message you receive; so using this module is not recommended if you get many messages per day. Instead, use the cron module to clean the messages' backup directory only once a day, and not everytime a message arrives. Confirmation modules o *pm-jacookie.rc* -- Handle cookie (unique id) confirmations. Also known as Procmail authentication service (PAS). This simple procmail module will accept messages only from user's who have returned a "cookie" key. You can use this to to protect your mailing list from false "subscribe" messages or from getting mail from unknown people, typically spammers who won't send the cookie back to you to "validate" themselves. Uses subroutine pm-jacookie1.rc, which generates the unique cookie; CRC 32 by default. o See also Michelle's confirmation module for SmartList File Servers o *pm-jasrv.rc* -- A Mime Procmail file server (MPFS) It contains all the instructions and supports several MIME encoding types: text/plain and gzip. The keyword SEND is configurable. You can set up as many files servers as you need to different directories by changing the SEND keyword. MPFS supports password for file access. o *commands.rc* -- From Alan's procmail-lib, check for commands in the subject line. Handles commands (send|get) [help|info|procmail info|procmail lib|procmailrc] and few others. o *send-file.rc* is a very simplistic piece of procmail code to send file (non-MIME support) requested in subject line. http://www.universe.digex.net/~mbr/unix/send-file.html Mime modules o *pm-jamime.rc* -- Subroutine to read MIME headers and put the mime version, boundary string, content-type information to variables. o *pm-jamime-decode.rc* -- recipe to decode quoted-printable or base64 encoding in the body. o *pm-jamime-kill.rc* -- Recipe for attachment killing: wipes out the extra mime cruft leaving only the plain text. Applications for killing: ms-tnef attachement (MS Explorer 7k), html attachements (netscape, MS Express) vcard (Netscape), PCX attachement (Lotus Notes). o *pm-jamime-save.rc* -- Recipe for saving simple file attachement. When you receive _ONE_ file attachement in a message, this recipe can save it to separate directory. The content is also decoded (base64,qp) while saving to file. Filtering message body or headers o *pm-jadaemon.rc* -- Handle DAEMON messages by changing subject to reflect the a) error reason b) to whom the message was originally sent c) original subject sent and what was the subject. Store the DAEMON messages to separate folder. o *pm-jasubject.rc* -- Standardize Subject "Re[32]: FW: Sv: message" or any other derivate to de facto "Re: message" o *pm-janetmind.rc* -- Reformat http://minder.netmind.com/ messages, The default 4k message is shortened to few important lines. Miscellaneus modules o *pm-jaempty.rc* -- check if message body is empty (nothing relevant) Define variable BODY_EMPTY to "yes" or "no" if message is empty. o *pm-janslookup.rc* -- Run nslookup on given address. If you compose return address with "formail -rt -x To:" you can verify if domain is registered before sending reply. Uses cache for already looked up domains. o *guess-mua.rc* -- Guess the Mail User Agent and set MUA: MH,PINE,MAIL Mailing list modules o *Microlist* a small mailing list by david hunt ...This version contains vars set for my environment and needs, and requires resetting of those vars before use. It's exact function and use will remain a mystery until I get a readme file written for it. If anyone wants to use it, I suggest you write to me first. If anyone has any suggestions or criticisms (no matter how harsh) please write http://www.west.net/~dh/homedir/microlist/microlist4.3 o *pm-jalist.rc* -- Subroutine to extract mailing list name from message. Do you need to add new recipe to your .procmailrc every time you subscribe to new mailing list? If you do, take a look at thsi module, which examines the message and defines variable `LIST' to hold the maling list name. You can use it directly to save the messages adaptively to correct folders. No more hand work and manual storing of mailing list messages. 4.8 Where to get Procmail code and modules "Alan's procmail modules" Send subject "send procmail library" to Alan Stebbens http://reality.sgi.com/aks/ "pm-code, Jari's Procmail modules" http://www.netforward.com/poboxes/?jari.aalto See pm-code.zip "Elijah's" http://www.qz.to/~eli/src/procmail/rc.master.html "Concordia scripts" http://alcor.concordia.ca/topics/email/auto/procmail/ ...We provide sample sets of recipes to get you started. The great thing about the concordia scripts is the fact that they are designed to run from a central location and be called from a .procmailrc installed in the user's ~/home directory. "Meng on procmail" http://icg.resnet.upenn.edu/procmail/ http://res2.resnet.upenn.edu/procmail/ ...goes into exhaustive detail about how I manage my mailing lists "David's" David Hunt http://www.west.net/~dh/homedir/pmdir/ ...My .procmailrc and .forward files can be viewed at 4.9 Procmail code to filter UBE _Sysadms_ _remember_ : Spam filtering is much more efficiently done in the MTA, especially if you just looking at From and To lines. For example, I you can setup in Exim a rule that blocks \d.*@aol\.com (that is any aol.com local part that begins with a digit). AOL guarantees that _none_ of their addresses being with a digit. Exim rejects such bogus addresses at the SMTP level before the message is received. "Daniel's smap filter" 1997-09-13 Daniel Smith sent excellent spam filter called `spamc.rc'. It used some nice heuristics and filters from various people, icluding [david] and [philip]. Later Dan made substantial changes to it and the new version is available from ftp://ftp.bristol.nl/pub/users/DanS/spamcheck "pm-jaube.rc Jari's ube filter (compiled from others)" After Daniel Smith posted his spam recipes to procmail mailing list, Jari investigated them and compiled other recipes to a general purpose UBE module that needs no special setup and can be installed via simple INCLUDERC. No additional ube-list files are used, all UBE all detected happens using procmail rules. The module is included in kit `pm-code.zip'. "Catherine A. Hampton's Spambouncer" http://www.best.com/~ariel/nospam/ ...The attached set of procmail recipes/filters, which I call The Spam Bouncer, are for users who are sick of spam (unsolicited junk mail email) and want to filter it out of their mail as easily as possible. These recipes can be used as shared recipes for a whole system, or by an individual for their own mailbox only. "Protect yourself from spam: A practical guide to procmail" http://www.sun.com/sunworldonline/swol-12-1997/swol-12-spam.html ...take you, step by step, through everything you need to know in order to enlist the aid of a Unix host in filtering unwanted e-mail traffic. This page is excellent to get you started with procmail and filtering with simple recipes and how to store messages to folders. Recommended for newcomers to Procmail. "Junkfilter" by Gregory Sutter http://www.pobox.com/~gsutter/junkfilter/ ...Junkfilter is a user-configurable procmail-based filter system for electronic mail. Recipes include checks for forged headers, key words, common spam domains, relay servers and many others. "Download procmail spam filters" http://www.telebyte.com/stopspamr This is excellent site and contains many other spam stop pointers. "SpamDunk" http://www.interlog.com/~waltdnes http://www.interlog.com/~waltdnes/beta/techie.htm ...This webpage shows a commented example of a working .procmailrc file that works for me. I have tried to make things as generic as possible, but there are no guarantees that it will work for anyone else.