Sidekiq: Sidekiq does not support keyword arguments

[richardjortega]

First off, thank you for Sidekiq! Hopefully this issue is a valid one.

Problem

Sidekiq does not seem to support keyword arguments for its worker perform method. Ruby introduced in 2.0 the ability to do _keyword arguments_ and in 2.1 provide a way to do _required keyword arguments_. It does not seem to accept required keyword arguments which are expressed via keyword_argument: with the trailing colon.

Note: Possibly changing to double spat operator ** may solve this issue.

Analysis

Below is code and backtrace showing that when you run perform_async you'll get an ArgumentError: wrong number of arguments (1 for 0). This only occurs when running with perform_async, separately I ran it with Worker.new.perform arguments and it was fine but that's synchronously so not sure if it's related to how it's sent/retrieved from Redis after.

For now, I just use old syntax for method arguments as workaround.

Version info

• Rails: 4.2.1
• Ruby: 2.2.2p95
• Sidekiq: 3.3.4
• Redis: 3.0.1
• Docker 1.6
• CentOS 7.1
  

  Code (which won't work)

class TestWorker
  include Sidekiq::Worker
  sidekiq_options queue: :default, retry: false, backtrace: true

  def perform required_keyword_argument:, another_keyword_argument: 'test'
    begin
      puts required_keyword_argument
      puts another_keyword_argument
    rescue => ex
      logger.info ex.message
      logger.info ex.backtrace.join("\n")
    end
  end
end

TestWorker.perform_async(required_keyword_argument: 'test_string', another_keyword_argument: 'test_2_string')

Backtrace

worker_1      | 2015-06-01T22:46:38.601Z 1 TID-ori7rdvmg TestWorker JID-fb882113cd4c899f048d1855 INFO: start
worker_1      | 2015-06-01T22:46:38.602Z 1 TID-ori7rdvmg TestWorker JID-fb882113cd4c899f048d1855 INFO: fail: 0.001 sec
worker_1      | 2015-06-01T22:46:38.603Z 1 TID-ori7rdvmg WARN: {"class"=>"TestWorker", "args"=>[{"required_keyword_argument"=>"test_string", "another_keyword_argument"=>"test_2_string"}], "retry"=>false, "queue"=>"default", "backtrace"=>true, "jid"=>"fb882113cd4c899f048d1855", "enqueued_at"=>1433198798.593975, "error_message"=>"wrong number of arguments (1 for 0)", "error_class"=>"ArgumentError", "processor"=>"732b5246d3e5:1", "failed_at"=>1433198798.6015728, "error_backtrace"=>["/usr/src/app/app/workers/test_worker.rb:5:in `perform'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:75:in `execute_job'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:52:in `block (2 levels) in process'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/active_record.rb:6:in `call'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'", "/usr/local/bundle/gems/sidekiq-failures-0.4.3/lib/sidekiq/failures/middleware.rb:9:in `call'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/logging.rb:15:in `block in call'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/logging.rb:24:in `with_context'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/logging.rb:11:in `call'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:132:in `call'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:132:in `invoke'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:51:in `block in process'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:98:in `stats'", "/usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:50:in `process'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'", "/usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'"]}
worker_1      | 2015-06-01T22:46:38.603Z 1 TID-ori7rdvmg WARN: wrong number of arguments (1 for 0)
worker_1      | 2015-06-01T22:46:38.603Z 1 TID-ori7rdvmg WARN: /usr/src/app/app/workers/test_worker.rb:5:in `perform'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:75:in `execute_job'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:52:in `block (2 levels) in process'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      | /usr/local/bundle/gems/sidekiq-failures-0.4.3/lib/sidekiq/failures/middleware.rb:9:in `call'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/logging.rb:15:in `block in call'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/logging.rb:24:in `with_context'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/logging.rb:11:in `call'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:132:in `call'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:132:in `invoke'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:51:in `block in process'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:98:in `stats'
worker_1      | /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:50:in `process'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
worker_1      | /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'
worker_1      | 2015-06-01 22:46:38.603 [ERROR] Actor crashed!
worker_1      | ArgumentError: wrong number of arguments (1 for 0)
worker_1      |     /usr/src/app/app/workers/test_worker.rb:5:in `perform'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:75:in `execute_job'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:52:in `block (2 levels) in process'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      |     /usr/local/bundle/gems/sidekiq-failures-0.4.3/lib/sidekiq/failures/middleware.rb:9:in `call'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/logging.rb:15:in `block in call'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/logging.rb:24:in `with_context'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/server/logging.rb:11:in `call'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:132:in `call'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/middleware/chain.rb:132:in `invoke'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:51:in `block in process'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:98:in `stats'
worker_1      |     /usr/local/bundle/gems/sidekiq-3.3.4/lib/sidekiq/processor.rb:50:in `process'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'
worker_1      |     /usr/local/bundle/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create' (pid:1)

Code (will work)

class TestWorker
  include Sidekiq::Worker
  sidekiq_options queue: :default, retry: false, backtrace: true

  def perform required_keyword_argument, another_keyword_argument = 'test'
    begin
      puts required_keyword_argument
      puts another_keyword_argument
    rescue => ex
      logger.info ex.message
      logger.info ex.backtrace.join("\n")
    end
  end
end

TestWorker.perform_async('test_string', 'test_2_string')

Backtrace

worker_1      | 2015-06-01T22:58:05.635Z 1 TID-os0yrozo4 IncomingBinaryMessageProcessorWorker JID-d27c87af04e9fe9bc5fada98 INFO: done: 0.064 sec
worker_1      | 2015-06-01T22:58:07.462Z 1 TID-os0yrozo4 TestWorker JID-a64deb68be64a0613e52d2e3 INFO: start
worker_1      | test_string
worker_1      | test_2_string
worker_1      | 2015-06-01T22:58:07.463Z 1 TID-os0yrozo4 TestWorker JID-a64deb68be64a0613e52d2e3 INFO: done: 0.0 sec

[mperham]

Sidekiq serializes the perform method arguments as an array of JSON entities. It doesn't support symbols or more complex serialization. AJ does support both at the cost of more complex serialization.

[richardjortega]

Yes, familiar with Sidekiq not accepting complex arguments parameter values. However, I wasn't familiar it also related to the method signature itself.

Closing this out and if anyone runs into this same gotcha, searching can point them here.

[tobico]

Can this be reopened? Seems like a big limitation for Sidekiq to fail on a standard and commonly used Ruby feature.

[mperham]

@tobico It's a fundamental design decision I made: arguments are serialized as plain JSON, for interoperability with all programming languages. https://github.com/mperham/sidekiq/wiki/Best-Practices#1-make-your-job-parameters-small-and-simple

[tobico]

@mperham good point, I see the value in serializing as plain JSON. So the reason named parameters can't be supported is that they would have to be passed into the method call as Ruby symbols, but there's no way to represent a Ruby symbol in JSON?

[mikegee]

@tobico symbols are converted to strings in the JSON, so we can't know whether it should be a string or symbol when parsing that JSON.

irb> require 'json'
irb> puts({foo: :bar}.to_json)
{"foo":"bar"}

[b-nik]

In the rare cases I need to call my worker with keyword arguments I do something like this:

# worker
def perform(id, options={})
  arg1 = options["arg1"] || []
  arg2 = options["arg2"] || []
  # ....
end

# usage
MyWorker.perform_async(id, arg1: [1, 2, 3], arg2: [a, b, c])

[DannyBen]

Since I really want to have strong keyword args in my jobs, I have been toying with this approach:

# jobs/job.rb
class Job
  include Sidekiq::Worker

  # ... more default options here

  def perform(params={})
    params = params.to_h.transform_keys(&:to_sym)
    execute params
  end
end

# jobs/greet_job.rb
class Greet < Job
  def execute(name: "there")
    puts "Hi #{name}!"
  end
end

So now all jobs simply define execute instead of defining perform.

And it even works nicely in specs:

# spec/greet_job_spec.rb
expect(Greet).to have_enqueued_sidekiq_job(name: "Lloyd Christmas")

Anyone seeing any problem with this pattern?

[axos88]

@mperham, why exactly would keyword arguments introduce extra complexity?
Without delving too deep into how Sidekiq::Worker works (ha!), I think it could be as simple as

  def perform_async(*args, **kwargs)
    client_push('class' => self, 'args' => args, 'kwargs' => kwargs)
  end

And then deserialize and call the perform method accordingly where required. If the issue is that the hash sent to client_push needs to be flat, one could argue that it could be converted to json and back, without a serious (or probably even measurable for most cases) performance hit.

[axos88]

Although I don't see why client_push would require a flat hash, since args is an array itself, which would suggests it (probably) doesn't need to be.

[axos88]

As far as I see JSON.generate gets called in the end, so client_push should be able to take a non-flat hash.