Sidekiq: Can I put environment config variables inside sidekiq.yml file?

[zulhfreelancer]

Hi,

I'm new Sidekiq user and I want to ask about Advanced Options sidekiq.yml file. Can I set values in my environment config variables?

For example:

:pidfile: tmp/pids/sidekiq.pid
staging:
  :concurrency: ENV['SIDEKIQ_CONCURRENCY']
production:
  :concurrency: ENV['SIDEKIQ_CONCURRENCY']
:queues:
  - default

Since I have different set of web dynos, workers and RedisCloud plan on staging and production, I think it will much more easier for me to just change the values from my Heroku config variables.

Ruby version: 2.3.1
Sidekiq / Pro / Enterprise version(s): 4.1.4
Rails version: 4.2.6

[ryansch]

Sidekiq will run the config file through ERB.

Your example:

:pidfile: tmp/pids/sidekiq.pid
staging:
  :concurrency: <%= ENV['SIDEKIQ_CONCURRENCY'] %>
production:
  :concurrency: <%= ENV['SIDEKIQ_CONCURRENCY'] %>
:queues:
  - default

Cheers!

[mperham]

Also, master (4.1.5) supports the new RAILS_MAX_THREADS variable which Sidekiq and Puma will pick up automatically, no extra config needed at all.

https://github.com/mperham/sidekiq/blob/master/Changes.md#head

[zulhfreelancer]

Thanks @ryansch and @mperham!

[zulhfreelancer]

I get this error when I try to run bundle exec sidekiq:

concurrency: 0 is not a valid value

I think that's because Heroku converts all config vars to string.

ᐅ heroku run console
Loading staging environment (Rails 4.2.6)
irb(main):001:0> ENV['SIDEKIQ_CONCURRENCY']
=> "249"

I tried to put <%= ENV["SIDEKIQ_CONCURRENCY"].to_i %> inside the sidekiq.yml file, but still get the same error.

Any idea how to fix this issue?

Thanks.

@ryansch @mperham

[concept47]

For posterity: its because of the way sidekiq loads, dotenv doesn't get to run and load env variables before it boots so its will default to nil which I guess the yml file reads as 0.

[mperham]

you can use the dotenv binary to bootstrap the env first.

[concept47]

@mperham I was able to do this with puma but I couldn't figure out how to do it with sidekiq

in puma.rb I just wrote this at the very top of the file. since RAILS_ENV is set in every other environment, it works.

if ENV.fetch("RAILS_ENV") { "development" } == "development"
  require "dotenv/load"
  Dotenv.load
end

but because sidekiq.yml is a yaml file I was stumped on how to accomplish the same thing

[mperham]

dotenv bundle exec sidekiq

[jhirn]

Be careful if you're using dotenv-rails and environment specific override files such as .env.production when using the dotenv executable. You need to provide the -f argument in order for it to respect anything other than .env file.

e.g. dotenv -f ".env.production,.env" bundle exec sidekiq

Even then it takes the order of the -f argument completely ignoring Rails.env.

[aesyondu]

Be careful if you're using dotenv-rails and environment specific override files such as .env.production when using the dotenv executable. You need to provide the -f argument in order for it to respect anything other than .env file.

e.g. dotenv -f ".env.production,.env" bundle exec sidekiq

Even then it takes the order of the -f argument completely ignoring Rails.env.

Thanks for this. For some reason -f ".env.production,.env" wasn't working with capistrano+systemd. So I removed the .env:

ExecStart=/bin/bash -lc 'dotenv -f .env.production bundle exec sidekiq # blah blah'