Openstreetmap-website: unable to reply to messages - "unparsable header"

[richlv]

recently the feature that allowed to reply to osm messages in email got broken.
now messages bounce with :

WARNING: Ignoring unparsable header "From sender@email Mon Dec 18 00:23:54 2017\nReceived: from shenron.openstreetmap.org ...": invalid header name syntax: "From sender@email Mon Dec 18 00"
/srv/www.openstreetmap.org/rails/app/models/message.rb:52:in `from_mail': undefined method `new_offset' for nil:NilClass (NoMethodError)
from /srv/www.openstreetmap.org/rails/script/deliver-message:30:in `<main>'

the "header" it complains includes all headers and full message

[gravitystorm]

I've had a look into this, and three things come to mind in particular. It looks like you've edited the error message to remove personal details, but I need to check:

• In the original error message, is there a colon : after the From?
• In the original error message, was there anything between the email address and the date?
• In the original error message, were there any \r\n or were all the headers separated by just \n?

For anyone else looking, I mainly suspect this is the handling of \n as a header separator by the mail gem in 2.7.0, which appears to be "restored" recently in an unreleased commit (https://github.com/mikel/mail/pull/1168) - although it could be something else entirely too.

[richlv]

thank you for your help. the only edit was changing email address to "sender@email" in two locations, and trimming the full error/message.

• there is no colon after "From"
• nothing between email and date (the part until "shenron.openstreetmap.org ..." is exact copypaste)
• no rn anywhere

[tomhughes]

Oddly I just tried replying to a message by email and it worked just fine...

[tomhughes]

@richlv can you look at the headers of the bounce message you got and see which machine it originated from? it should be spike-01/2/3 and I'd like to know which...

[richlv]

Received: from Debian-exim by spike-01.openstreetmap.org with local (Exim 4.86_2)
id 1eQjDv-00045A-Je

[richlv]

looking at my sent message history, last time i was able to reply by email was on 12 November 2017. first time it failed was on 14 December 2017. has failed for 2 out of 2 attempts since then.

[martini]

@richlv I have a similar issue "invalid header name syntax: "From sender@email Mon Dec 18 00"" with mail 2.7.0 and 2.7.1.edge (at 2-7-stable@34aef0d) gem.

Can you verify if it only happens with utf8 chars? Only ascii mails are working fine?

[richlv]

hmm, can you please send me two messages - one with, one without non-ascii chars ?

https://www.openstreetmap.org/user/richlv

[tomhughes]

I tried to reproduce this some weeks ago with an exact copy of a failing message and was unable to so I'm afraid I don't really have any ideas here. It's unlikely to be something as simple as character set or I would have been able to reproduce it.

[martini]

hmm, can you please send me two messages - one with, one without non-ascii chars ?

Attached two simple mails. You can verify it via rails console (call mail gem directly
examples.zip)

> Mail.new(IO.binread('path/to/mail-with-umlaut.eml'))
WARNING: Ignoring unparsable header "From [email protected]  Mon May  7 07:45:48 2012\nFrom: [email protected] (Marc Smith)\nTo: <[email protected]>\nSubject: Re: XXXX Betatest Ticket Templates [Ticket#11162]\nContent-Type: text/plain; charset=UTF-8; format=flowed\nContent-Transfer-Encoding: 8bit\n\nAm 07.05.2012 08:10, schrieb:\n> Hallo Marc,\n>\n> Here an Umlaut is comming \"\xC3\xBC\". :-D": invalid header name syntax: "From [email protected]  Mon May  7 07"
 => #<Mail::Message:70344875104420, Multipart: false, Headers: > 

Without it's working fine:

> Mail.new(IO.binread('path/to/mail-without-umlaut.eml'))
 => #<Mail::Message:70344865634620, Multipart: false, Headers: <From: [email protected] (Marc Smith)>, <To: <[email protected]>>, <Subject: Re: XXXX Betatest Ticket Templates [Ticket#11162]>, <Content-Type: text/plain; charset=UTF-8; format=flowed>, <Content-Transfer-Encoding: 8bit>> 

If you downgrade mail gem 2.6.6 it's working fine.

[tomhughes]

It's weirder than that - you also need the mbox delimiter line (ie the first line that isn't a proper header) as without that it works even with the umlaut which is why I failed to reproduce it before as I didn't have that line.

[tomhughes]

I think the problem is that https://github.com/mikel/mail/blob/master/lib/mail/message.rb#L2004 fails to match and remove the envelope from line meaning that it then tries to parse it as a header.

[tomhughes]

So although those example files have simple LF line endings something has converted it to CRLF in the non-unicode one but not in the unicode one and that stops the end of line being matched.

[martini]

For me in mail gem 2.7.0 it works if I disable the condition of https://github.com/mikel/mail/blob/master/lib/mail/field.rb#L133 this check was introduced with 2.7.0.rc2 by https://github.com/mikel/mail/commit/c0662469a2be815ace17c18962bf02bef6a3474c#diff-480c9cbc16ce0b8fb4b396b9f46187f6

[tomhughes]

Sure, but it shouldn't be getting there at all because the envelope from line should already have been removed (you can tell by whether the envelope_from attribute of the parsed message is set).

[richlv]

yay, thank you for debugging & fixing.