Google-api-ruby-client: Google API client is very memory hungry

Created on 15 Jun 2016  路  8Comments  路  Source: googleapis/google-api-ruby-client

I'm using google-api-client v0.9.4 in my Rails app.
I'm using only Google Plus API and Analytics. Here's my Gemfile:

gem 'google-api-client', '>= 0.9.4', require: [
  'google/apis/plus_v1', 'google/apis/analytics_v3', 'googleauth', 'google/api_client/client_secrets'
]

I tried monitoring the app RAM usage using derailed gem. It seems like this gem is using 20+ mb of RAM:

Report using bundle exec derailed bundle:mem

google/apis/plus_v1: 26.707 MiB
    google/apis/plus_v1/service.rb: 24.1211 MiB
      google/apis/core/base_service: 22.293 MiB (Also required by: google/apis/plus_v1/classes.rb, google/apis/plus_v1/representations.rb, and 3 others)
        google/apis/core/batch: 10.8594 MiB
          google/apis/core/upload: 10.5703 MiB (Also required by: google/apis/core/base_service)
            mime-types: 10.4219 MiB
              mime/types: 10.3672 MiB
                mime/types/registry: 9.707 MiB
        google/apis/core/http_client_adapter: 7.7148 MiB
          httpclient: 7.6406 MiB
            httpclient/cookie: 5.4961 MiB
              http-cookie: 5.25 MiB
                http/cookie: 5.25 MiB (Also required by: http/cookie_jar)
                  domain_name: 4.9102 MiB
                    domain_name/etld_data: 4.418 MiB
            httpclient/session: 0.8945 MiB (Also required by: httpclient/auth)
              httpclient/http: 0.3867 MiB (Also required by: httpclient)
            httpclient/auth: 0.457 MiB
        googleauth: 1.9375 MiB (Also required by: TOP)
          googleauth/compute_engine: 0.9063 MiB
            googleauth/signet: 0.8047 MiB (Also required by: googleauth/service_account, googleauth/user_refresh, and 3 others)
              signet/oauth_2/client: 0.7891 MiB
        google/apis/core/api_command: 1.4766 MiB (Also required by: google/apis/core/upload, google/apis/core/download)
          google/apis/core/http_command: 1.4258 MiB (Also required by: google/apis/core/batch, google/apis/core/upload)
            hurley: 0.7383 MiB (Also required by: hurley_patches, google/apis/core/batch, and 3 others)
            pp: 0.3008 MiB
      google/apis/core/json_representation: 1.6914 MiB (Also required by: google/apis/plus_v1/classes.rb, google/apis/plus_v1/representations.rb, and 3 others)
        representable/json: 1.4648 MiB (Also required by: representable/json/hash)
          representable/hash: 1.3438 MiB
            representable: 1.0898 MiB (Also required by: representable/decorator)
    google/apis/plus_v1/representations.rb: 1.6055 MiB
    google/apis/plus_v1/classes.rb: 0.9648 MiB
... other gems here...

Report using bundle exec derailed exec perf:mem:

  application: 66.4375 MiB
    google/apis/plus_v1: 24.5273 MiB
      google/apis/plus_v1/service.rb: 23.2109 MiB
        google/apis/core/base_service: 22.2578 MiB (Also required by: google/apis/plus_v1/classes.rb, google/apis/plus_v1/representations.rb, and 3 others)
          google/apis/core/batch: 9.9141 MiB
            google/apis/core/upload: 9.5859 MiB (Also required by: google/apis/core/base_service)
              mime-types: 9.3633 MiB
                mime/types: 9.3516 MiB
                  mime/types/registry: 8.6445 MiB
          google/apis/core/http_client_adapter: 8.6719 MiB
            httpclient: 8.5625 MiB
              httpclient/cookie: 5.957 MiB
                http-cookie: 5.6016 MiB
                  http/cookie: 5.6016 MiB (Also required by: http/cookie_jar)
                    domain_name: 5.1836 MiB
                      domain_name/etld_data: 4.5469 MiB
              httpclient/session: 1.1211 MiB (Also required by: httpclient/auth)
                httpclient/http: 0.4883 MiB (Also required by: httpclient)
              httpclient/auth: 0.543 MiB
          google/apis/core/api_command: 2.2148 MiB (Also required by: google/apis/core/upload, google/apis/core/download)
            google/apis/core/http_command: 2.1211 MiB (Also required by: google/apis/core/batch, google/apis/core/upload)
              hurley: 1.082 MiB (Also required by: hurley_patches, google/apis/core/batch, and 3 others)
                /Users/progm/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/hurley-0.2/lib/hurley/client: 0.3125 MiB
              pp: 0.4023 MiB
          googleauth: 0.8945 MiB (Also required by: application)
            googleauth/compute_engine: 0.5156 MiB
              googleauth/signet: 0.4258 MiB (Also required by: googleauth/service_account, googleauth/user_refresh, and 3 others)
                signet/oauth_2/client: 0.3828 MiB
          addressable/template: 0.3359 MiB (Also required by: google/apis/core/api_command, google/apis/core/http_command)
        google/apis/core/json_representation: 0.7539 MiB (Also required by: google/apis/plus_v1/classes.rb, google/apis/plus_v1/representations.rb, and 3 others)
          representable/json: 0.6406 MiB (Also required by: representable/json/hash)
            representable/hash: 0.6055 MiB
              representable: 0.4766 MiB (Also required by: representable/decorator)
      google/apis/plus_v1/representations.rb: 0.9141 MiB
      google/apis/plus_v1/classes.rb: 0.3906 MiB
... other gems here...
feature request
Source
picture ProGM

Most helpful comment

One quick fix is I could do is lazy require mime-types. It's only used during media uploads and even then, only if the content type isn't specified. For apps that don't hit that condition we could avoid having to load all that.

picture sqrrrl on 22 Jun 2016
👍2

All 8 comments

Thanks. Looks like the biggest offender is the mimetype registry. It's not super critical to have it, but makes media uploads easier since we can infer the type if omitted.

Unfortunately can't remove that without breaking compatibility. Would have to be done on the next major release. Might look at making it optional (use it if it exists...)

sqrrrl picture sqrrrl on 22 Jun 2016
👍2

One quick fix is I could do is lazy require mime-types. It's only used during media uploads and even then, only if the content type isn't specified. For apps that don't hit that condition we could avoid having to load all that.

sqrrrl picture sqrrrl on 22 Jun 2016
👍2

This would be pretty great 馃憤

ProGM picture ProGM on 22 Jun 2016

Looks like just switching to the columnar store knocks off about 5mb and also appears to lazy load the data as needed.

Before:

## Impact of `require <file>` on RAM

Showing all `require <file>` calls that consume 0.3 MiB or more of RSS
Configure with `CUT_OFF=0` for all entries or `CUT_OFF=5` for few entries
Note: Files only count against RAM on their first load.
      If multiple libraries require the same file, then
       the 'cost' only shows up under the first library

TOP: 19.9922 MiB
  google/apis/drive_v3: 16.5938 MiB
    google/apis/drive_v3/service.rb: 15.7383 MiB
      google/apis/core/base_service: 14.5547 MiB (Also required by: google/apis/drive_v3/classes.rb, google/apis/drive_v3/representations.rb)
        google/apis/core/batch: 7.9492 MiB
          google/apis/core/upload: 7.8672 MiB (Also required by: google/apis/core/base_service)
            mime-types: 7.793 MiB
              mime/types: 7.793 MiB
        addressable/uri: 3.6758 MiB (Also required by: addressable/template, google/apis/core/api_command, and 8 others)
          addressable/idna: 2.5625 MiB
            addressable/idna/pure: 2.5039 MiB
        google/apis/core/http_client_adapter: 0.8984 MiB
          httpclient: 0.8594 MiB
            httpclient/cookie: 0.582 MiB
              http-cookie: 0.5742 MiB
                http/cookie: 0.5703 MiB (Also required by: httpclient/cookie)
                  domain_name: 0.4609 MiB
                    domain_name/etld_data: 0.3633 MiB
        googleauth: 0.8789 MiB
          googleauth/compute_engine: 0.4961 MiB
        google/apis/core/api_command: 0.8203 MiB (Also required by: google/apis/core/upload, google/apis/core/download)
          google/apis/core/http_command: 0.6797 MiB (Also required by: google/apis/core/batch, google/apis/core/upload)
            hurley: 0.4336 MiB (Also required by: hurley_patches, google/apis/core/batch, and 3 others)
      google/apis/core/json_representation: 0.5664 MiB (Also required by: google/apis/drive_v3/classes.rb, google/apis/drive_v3/representations.rb)
        representable/json: 0.4766 MiB (Also required by: representable/json/hash)
          representable/hash: 0.4688 MiB
            representable: 0.3633 MiB (Also required by: google/apis/core/json_representation)
    google/apis/drive_v3/representations.rb: 0.7148 MiB
  rack/test: 1.332 MiB
    rack: 0.5703 MiB (Also required by: TOP)
  rack/file: 0.7891 MiB
    rack/mime: 0.6719 MiB
  rack/lint: 0.4922 MiB
  rack/response: 0.4375 MiB

after

## Impact of `require <file>` on RAM

Showing all `require <file>` calls that consume 0.3 MiB or more of RSS
Configure with `CUT_OFF=0` for all entries or `CUT_OFF=5` for few entries
Note: Files only count against RAM on their first load.
      If multiple libraries require the same file, then
       the 'cost' only shows up under the first library

TOP: 14.8633 MiB
  google/apis/drive_v3: 11.457 MiB
    google/apis/drive_v3/service.rb: 10.6523 MiB
      google/apis/core/base_service: 9.875 MiB (Also required by: google/apis/drive_v3/classes.rb, google/apis/drive_v3/representations.rb)
        addressable/uri: 3.6523 MiB (Also required by: addressable/template, google/apis/core/api_command, and 8 others)
          addressable/idna: 2.4766 MiB
            addressable/idna/pure: 2.4375 MiB
        google/apis/core/batch: 3.0391 MiB
          google/apis/core/upload: 2.9727 MiB (Also required by: google/apis/core/base_service)
            mime/types/columnar: 2.9531 MiB
              mime/types: 2.8398 MiB
        google/apis/core/http_client_adapter: 1.5039 MiB
          httpclient: 1.5 MiB
            httpclient/cookie: 0.668 MiB
              http-cookie: 0.6172 MiB
                http/cookie: 0.6172 MiB (Also required by: httpclient/cookie)
                  domain_name: 0.6094 MiB
                    domain_name/etld_data: 0.5117 MiB
            httpclient/session: 0.3906 MiB (Also required by: httpclient/auth)
        googleauth: 0.7656 MiB
          googleauth/credentials_loader: 0.3672 MiB (Also required by: googleauth/service_account, googleauth/user_refresh)
            os: 0.3359 MiB
              yaml: 0.3086 MiB
                psych: 0.3086 MiB
        google/apis/core/api_command: 0.6094 MiB (Also required by: google/apis/core/upload, google/apis/core/download)
          google/apis/core/http_command: 0.5 MiB (Also required by: google/apis/core/batch, google/apis/core/upload)
    google/apis/drive_v3/representations.rb: 0.5625 MiB
  rack/test: 1.3242 MiB
    rack: 0.5703 MiB (Also required by: TOP)
  rack/file: 0.7969 MiB
    rack/mime: 0.6641 MiB
  rack/lint: 0.4922 MiB
  rack/response: 0.4414 MiB

Not sure why your domain_name require is so much bigger than mine... what version is showing in your Gemfile.lock?

sqrrrl picture sqrrrl on 23 Jun 2016

Here's my Gemfile.lock:

    google-api-client (0.9.4)
      addressable (~> 2.3)
      googleauth (~> 0.5)
      httpclient (~> 2.7)
      hurley (~> 0.1)
      memoist (~> 0.11)
      mime-types (>= 1.6)
      representable (~> 2.3.0)
      retriable (~> 2.0)
      thor (~> 0.19)
ProGM picture ProGM on 23 Jun 2016

However, now I notice, you are loading google/apis/drive_v3, instead I was loading google/apis/plus_v1. Maybe this could cause is the difference?

ProGM picture ProGM on 23 Jun 2016

Should be slightly better with 0.9.10

sqrrrl picture sqrrrl on 24 Jun 2016

This is an older issue and I'm going to close it. Since the last activity this gem has switched to the mini_mime gem, which a) reduces memory overhead further than mime-types and b) is also used by rails.

Using the Gemfile from the issue description with the most up to date version of google-api-client we see the following:

TOP: 16.9922 MiB
  google/apis/plus_v1: 11.9688 MiB
    google/apis/plus_v1/service.rb: 11.1211 MiB
      google/apis/core/base_service: 10.3711 MiB (Also required by: google/apis/plus_v1/classes.rb, google/apis/plus_v1/representations.rb, and 3 others)
        googleauth: 5.6328 MiB (Also required by: TOP, google/api_client/client_secrets)
          googleauth/application_default: 5.5703 MiB
            googleauth/compute_engine: 4.9531 MiB
              googleauth/signet: 3.1992 MiB (Also required by: googleauth/service_account, googleauth/user_refresh, and 3 others)
                signet/oauth_2/client: 3.1328 MiB (Also required by: googleauth/credentials)
                  jwt: 2.6523 MiB (Also required by: googleauth/service_account)
                    jwt/decode: 2.5234 MiB
                      jwt/signature: 2.4531 MiB
                        openssl: 2.4453 MiB (Also required by: httpclient/ssl_config)
                          openssl/ssl: 1.4141 MiB
                            ipaddr: 0.3828 MiB
                          openssl.so: 0.918 MiB
              faraday: 1.6367 MiB (Also required by: signet/oauth_2/client)
                /Users/blowmage/.gem/repos/apiary-mem-test/gems/faraday-0.17.0/lib/faraday/options: 0.3398 MiB
                /Users/blowmage/.gem/repos/apiary-mem-test/gems/faraday-0.17.0/lib/faraday/connection: 0.3164 MiB
            googleauth/default_credentials: 0.5781 MiB (Also required by: googleauth)
              googleauth/credentials_loader: 0.5352 MiB (Also required by: googleauth/service_account, googleauth/user_refresh, and 3 others)
                os: 0.4766 MiB
                  yaml: 0.4102 MiB
                    psych: 0.4102 MiB
        addressable/uri: 1.668 MiB (Also required by: addressable/template, google/apis/core/api_command, and 8 others)
          addressable/idna: 0.7266 MiB
            addressable/idna/pure: 0.7031 MiB
        google/apis/core/api_command: 1.0898 MiB (Also required by: google/apis/core/upload, google/apis/core/download)
          google/apis/core/http_command: 0.9648 MiB (Also required by: google/apis/core/batch, google/apis/core/upload)
            pp: 0.4844 MiB
            retriable: 0.3789 MiB (Also required by: google/apis/core/api_command)
              timeout: 0.3594 MiB (Also required by: faraday, httpclient/session)
        httpclient: 0.9141 MiB
          httpclient/session: 0.5977 MiB (Also required by: httpclient/auth)
            httpclient/http: 0.3203 MiB (Also required by: httpclient)
        google/apis/core/batch: 0.8164 MiB
          google/apis/core/upload: 0.625 MiB (Also required by: google/apis/core/composite_io, google/apis/core/base_service)
            tempfile: 0.4492 MiB
              tmpdir: 0.3828 MiB
                fileutils: 0.3711 MiB
      google/apis/core/json_representation: 0.5977 MiB (Also required by: google/apis/plus_v1/classes.rb, google/apis/plus_v1/representations.rb, and 3 others)
        representable/json: 0.4844 MiB
          representable/hash: 0.4766 MiB
            representable: 0.4609 MiB (Also required by: representable/decorator)
    google/apis/plus_v1/representations.rb: 0.5859 MiB
  google/apis/analytics_v3: 4.918 MiB
    google/apis/analytics_v3/representations.rb: 2.2031 MiB
    google/apis/analytics_v3/classes.rb: 1.7383 MiB
    google/apis/analytics_v3/service.rb: 0.9609 MiB

I am unable to get derailed to run against a simple Rails 6 app, but I assume it should show a similar lowered memory overhead.

If someone is reading this issue and have a similar concern, please open a new issue that references this issue instead of commenting on this issue.

blowmage picture blowmage on 12 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jponc picture jponc  路  5Comments
bekkou68 picture bekkou68  路  3Comments
MattMencel picture MattMencel  路  3Comments
davidraj picture davidraj  路  6Comments
jiggneshhgohel picture jiggneshhgohel  路  6Comments