On Fri, Feb 20, 2004 at 08:12:12AM +0000, Deepak Saxena wrote:
> I am on various linux mailing lists and everyone once in a while
> a thread will get cross posted so I will get it twice (sometimes
> three times if I happen to be CC:ed) and it will all end up in whatever
> procmail rule catches it first. Anyone know of a way to catch
> cross-posted messages via procmail and have it only store the first
> copy of it? Maybe do a md5 hash of something that is cross posted
> and everyime that same set of To:, From: and Cc: headers is seen, 
> discard the message if the hashes match and increment some counter.
> If the counter == number of lists I am on that would receive this
> message, delete it.
> 
> This sounds incredibly complicated and requires keeping state across
> procmail instances so I'm thinking there must be a _MUCH_ easier
> method. Anyone?

>From the procmailex man page:
> If  you are subscribed to several mailinglists and people cross-post to
> some of them, you usually receive several  duplicate  mails  (one from
> every  list).   The following simple recipe eliminates duplicate mails.
> It tells formail to keep an 8KB cache file in which it will store  the
> Message-IDs  of  the most recent mails you received.  Since Message-IDs
> are guaranteed to be unique for every new mail, they are ideally suited
> to  weed  out  duplicate mails.  Simply put the following recipe at the
> top of your rcfile, and no duplicate mail will get past it.
> 
>               :0 Wh: msgid.lock
>               | formail -D 8192 msgid.cache
> 
> Beware if you have delivery problems in  recipes  below  this one  and
> procmail  tries  to  requeue the mail, then on the next queue run, this
> mail will be considered a duplicate and will be thrown away.  For those
> not quite so confident in their own scripting capabilities, you can use
> the following recipe instead.  It puts duplicates in a separate folder
> instead  of  throwing them away.  It is up to you to periodically empty
> the folder of course.
> 
>               :0 Whc: msgid.lock
>               | formail -D 8192 msgid.cache
> 
>               :0 a:
>               duplicates

I gateway my mailinglist messages to a local newsgroup server so I
haven't tried the above recipe.  Should work though.

-Dale Farnsworth