lost headers

Thu Aug 4 10:25:41 MST 2005

The RFC for internet message format (822, later 2822) does allow, even require, MTA's to modify the header portion of messages for a variety of purposes (like adding to the backtrace).  That's what is happening here.  The body isn't modified (as far as the MTA is concerned), just the headers.  The problem is that the MTA is recognizing additional lines as headers because it's a lot looser about extra linefeeds between headers than the RFC requires.  This is done because older MUA's often generated extra blank lines.  Most modern MTA's handle this much more strictly, but some are still pretty loose.  SPAM filters often loosen up the rules to handle the fact that some SPAM senders are *slightly* breaking the RFC rules as a way of trying to fool the filters.  I think Kurt may be on the right 
track in that it's very likely a SPAM filter in the loop that's munging out the blank lines, since I think there may be 2 or 3 filters in my delivery chain  (and yes, the filters are SUPPOSED to modify the headers, how else do they add the X-JunkMail, additional backtrace headers, and other delivery modifications?)

==Joseph++

Erik Bixby wrote:
> RFCs aside, wouldn't something changing the message two users are
> sending two each be considered a bug?  Funny characters, and all...
> -Erik
> 
> On 8/4/05, Technomage <technomage-hawke at cox.net> wrote:
> 
>>: this is a test header
>>: this is not an actual header
>>: if this had been a real header, you would have been instructed
>>: on what to do with them.
>>
>>
>>ok. this is new to me.
>>
>>so, if I were to place a number of lines above that started with a ":"
>>followed by a comment, some clients would interpret that as header info?
>>
>>btw, this e-mail is also a test.....
>>
>>On Thursday 04 August 2005 02:21, Joseph Sinclair wrote:
>>
>>>I'm not missing anything, the information is just hidden from view.
>>>The first few lines (the ones with ":" characters) are being *properly*
>>>interpreted by Thunderbird as header lines, and those don't show unless I
>>>do a view source.
>>>
>>>The "tool" that's compressing out the blank lines between header fields is
>>>somewhere in the SMTP cloud between you and me.  I have no idea where, and
>>>no way of hunting that down.
>>>
>>>BTW Section 4.2 of RFC 2822 on the IETF web site states:
>>>    In the obsolete syntax, any amount of folding white space MAY be
>>>    inserted where the obs-FWS rule is allowed.  This creates the
>>>    possibility of having two consecutive "folds" in a line, and
>>>    therefore the possibility that a line which makes up a folded header
>>>    field could be composed entirely of white space.
>>>
>>>The EBNF in the spec states that a line with nothing but whitespace (i.e.
>>>tab or space) is allowed on input (compliant software should NEVER generate
>>>such a header), but a lot of older MTA's interpret that as "Blank Lines Are
>>>Allowed" on incoming messages. I know that's not perfect behavior, but
>>>speaking as someone who worked on MTA's about 15 years ago when the
>>>standing spec was RFC 822, a LOT of MTA's allowed completely blank lines,
>>>and a lot of other MTA's would remove those extra "fold" lines as a way of
>>>reducing message size without harming the content.  There are still a lot
>>>of MTA's out there that follow this, admittedly obsolete, reasoning, but
>>>there's also very little I can do to fix it, since I don't know which
>>>servers they are, and I couldn't do much about obsolete proprietary
>>>software even if I did.
>>>
>>>The point of all this is that it's a simple fix to just DON'T PUT LINES
>>>WITH ":" CHARACTERS IN THE FIRST LINE OF YOUR MESSAGE. Is it that hard to
>>>just accept that there are bad MTA's out there and take simple steps to
>>>avoid having them munge your message?
>>>
>>>I'm not asking you to do something terribly hard here, just to be a little
>>>more considerate when you put links in a message by prefacing them with
>>>something simple, like a line stating what the link is for.  It's not hard,
>>>and we're all used to doing simple things like this to work around flaws in
>>>widely used, but flawed, proprietary software (like IE).
>>>
>>>==Joseph++
>>>
>>>Jeremy C. Reed wrote:
>>>
>>>>On Wed, 3 Aug 2005, Joseph Sinclair wrote:
>>>>
>>>>>The User Agent isn't the only actor here.  The intervening mail
>>>>>servers are permitted to interpret the headers of the message as
>>>>>provided under section 4.2 of RFC 2822, and they're permitted to
>>>>>modify those headers to add trace information, reorder fields,,
>>>>>etc...  This is what's happening for me, and probably for some other
>>>>>people as well.
>>>>>One of more servers between you and me (at least 7 for the message
>>>>>below) is removing the blank line before your "fake" headers, and
>>>>>Thunderbird sees it, properly, as part of the header block.  According
>>>>>to the RFC 2822 spec, all conforming UA's SHOULD accept messages with
>>>>>blank lines between headers to accommodate sending UA's that are
>>>>>designed to the RFC 822 spec (which allowed this behavior).
>>>>
>>>>4.2 is the wrong section (unless different rfc2822 are sectioned
>>>>differently).
>>>>
>>>>The problem is that my two extra lines are not part of the header :)
>>>>
>>>>Please quote the specific section that allows putting "a line with
>>>>nothing preceding the CRLF" between the real headers and more headers.
>>>>
>>>>I'll admit that I don't know the mail specs perfectly. But I do have
>>>>experience with email message parsing: I have done POP3 server
>>>>development for around six years and have maintained vm-pop3d for over
>>>>four years; taught mail server administration for over four years; and I
>>>>have read a lot of exim, postfix and sendmail MTA code and mail.local
>>>>MDA code (and other SMTP mailers) and wrote and use "mailout" SMTP-out
>>>>sendmail replacement.
>>>>
>>>>
>>>>>Note, the replied messages below are NOT modified in any way, and your
>>>>>"fake" headers are not shown.  I don't just make this stuff up, and I
>>>>>rather resent your implication that I did.
>>>>
>>>>We should track down specifically what mail *tool* lost my important :)
>>>>email message. I agree with Kurt: it's a bug that needs to be reported.
>>>>
>>>>Please see
>>>>http://lists.plug.phoenix.az.us/lurker/message/20050804.005401.9719e855.e
>>>>n.html
>>>>
>>>>and
>>>>http://lists.plug.phoenix.az.us/lurker/message/20050804.013243.964e82c1.e
>>>>n.html
>>>>
>>>>
>>>>Does anyone know where the lines were lost?
>>>>
>>>>I am interested in solving this. Other readers: please let me know if my
>>>>two lines were missing for you.
>>>>
>>>> Jeremy C. Reed
>>>>
>>>>                 BSD News, BSD tutorials, BSD links
>>>>                http://www.bsdnewsletter.com/
>>>>---------------------------------------------------
>>>>PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>>>>To subscribe, unsubscribe, or to change  you mail settings:
>>>>http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>>
>>>---------------------------------------------------
>>>PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>>>To subscribe, unsubscribe, or to change  you mail settings:
>>>http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>>---------------------------------------------------
>>PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>>To subscribe, unsubscribe, or to change  you mail settings:
>>http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
> 
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change  you mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
> 