Google-api-ruby-client: HTTPClient::KeepAliveDisconnected: Broken pipe on send_user_message call

[ian]

Workes on multiple attachments (tested with 1-3) when their sizes are under 5MB. Docs on attachments makes note of 5MB but I can't find a definitive response to this.

I'm wondering if this is due to an issue with httpclient, as google-cloud-ruby makes a reference to this issue and patches Faraday: https://github.com/GoogleCloudPlatform/google-cloud-ruby/pull/268/files#diff-1f8b3b22f2178a557864952dcc16fbdbR675

To reproduce, create a Mail object and add an attachment > 5MB (I have verified that this causes an issue: https://pivit.com.au/images/files/pdf/10mb-test-file.pdf

mail = Mail.new( to: email, from: "#{user.name} <#{user.email}>", subject: subject )
mail.part content_type: 'multipart/alternative' do |part|
  part.html_part = Mail::Part.new(body: email.body), content_type: 'text/html; charset=UTF-8')
  part.text_part = Mail::Part.new(body: strip_tags(email.body))
end
mail.attachments[a.name.to_s] = {
  filename: File.basename(url),
  content_transfer_encoding: 'base64',
  content: Base64.encode64(open(url) { |f| f.read })
}

If you remove the mail.attachments =, it works fine. When you set the 10MB attachment, you get:

Sending HTTP post https://www.googleapis.com/gmail/v1/users/me/messages/send?
Caught error HTTPClient::KeepAliveDisconnected: Broken pipe
Error - #

HTTPClient::KeepAliveDisconnected: HTTPClient::KeepAliveDisconnected: Broken pipe
from /Users/ian/.rvm/gems/ruby-2.4.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:524:in > `rescue in query'

Not sure what the solution is but other google projects seem to be switching to Faraday

[TheRoyalTnetennba]

Thanks for reporting this! We’ll look into this and see how best to handle large files.

[jwg2s]

@TheRoyalTnetennba we're still suffering from this problem - and unfortunately there's not a clear example that shows the best practice for delivering a Mail::Message object that contains a large attachment. I think many people would benefit from that.

Specifically, how to reliably send a message using this gem, the ruby Mail::Message object, with an attachment greater than 10mb.

[jwg2s]

@TheRoyalTnetennba any update on this? Unfortunately a pretty critical part of our workflow and we haven't been able to get this to work with resumable uploads. We can upload the large file, but then trying to finish sending the email, we get Recipient address required even though it's part of the request. I think a quick markdown guide of how to send a message using this gem with a 20mb file for example would be extremely useful. I'm happy to write it up once we get this figured out.

[TheRoyalTnetennba]

@jwg2s have you tried reaching out at the gmail apis site? Are you using the gmail client google/apis/gmail_v1?

[jwg2s]

@TheRoyalTnetennba I've asked for help here - https://stackoverflow.com/questions/51714674/using-gmail-api-to-send-attachments-larger-than-10mb, we'll see if anyone comes back with anything. I find it surprising there's not a single example that ties this all together, but I guess I can make one once I get to the bottom of this and help others out!

[jeremy]

https://github.com/nahi/httpclient/issues/185#issuecomment-58159196

[jwg2s]

So the issue was that when sending attachments greater than 5mb and less than 35mb (but also works on messages without attachments), you do NOT base64 encode the body of the request, and use multipart as the uploadType. Unfortunately the docs don't mention this at all, and the error messages don't indicate that either.

Here's a working example that was able to send a 20mb attachment. Hopefully this will help anyone else who has wasted countless hours trying to figure this one out!

  result = api_client.execute!(
    :api_method => gmail_api.users.messages.to_h['gmail.users.messages.send'],
    :body => rfc822_message_string,
    :parameters => {
      :uploadType => 'multipart',
      :userId => 'me'
    },
    :headers => {
      'Content-Type' => 'message/rfc822',
    }
  )

[hashwin]

I found that using the upload_source over sending a message object works best as this plugs into the resumable upload API automatically and avoid this error:
Change this:

service.send_user_message('me', { raw: mail_message.to_s }, content_type: 'message/rfc822')

to this:

service.send_user_message('me', nil, upload_source: StringIO.new(mail_message.to_s), content_type: 'message/rfc822')

[hengwoon]

Is there a way to use the upload_source and set the threadId when sending the message?

[ryansch]

@hengwoon

service.send_user_message('me', { thread_id: 'asdf' }, upload_source: StringIO.new(mail_message.to_s), content_type: 'message/rfc822')

The second positional param doesn't have to be nil, it can contain everything except for raw that the API allows.

[hengwoon]

Thanks @ryansch . I wasn't sure if we could pass in a non-Google::Apis::GmailV1::Message object without any content.