Sidekiq: Passing a block to .perform_async method

[nicolasiensen]

Hey,

I'm implementing a worker that uses Sidekiq in order to run it asynchronously.

Inside the perform method I'm doing an API request, followed by a database update that depends on the API response.

class RequestWorker
  include Sidekiq::Worker

  def perform param1, resource_id
    response = API.request(param1)
    Resource.find(resource_id).update_attribute :attr1, response["attr1"]
  end
end

My Rails model looks like this:

class Resource < ActiveRecord::Base
  after_create do
    RequestWorker.perform_async("request param", self.id)
  end
end

It's working perfectly fine, but I would like to extend this worker to make it possible to process the same API request for any Rails model, so the code I wish I could implement on the models is:

class Resource1 < ActiveRecord::Base
  after_create do
    RequestWorker.perform_async("request param") do |api_response|
      self.update_attribute :attr1, api_response["attr1"] 
    end
  end
end

class Resource2 < ActiveRecord::Base
  after_create do
    RequestWorker.perform_async("request param") do |api_response|
      # Do whatever you want to do with the api_response
    end
  end
end

And the worker perform method would be like this:

class RequestWorker
  include Sidekiq::Worker

  def perform param1
    response = API.request(param1)
    yield(response)
  end
end

Unfortunately it did't work, I get an error no block given (yield) in the perform method. This should be happening because I'm calling perform_async and passing the block to it, but it's not passing the block ahead to the perform method.

My question is if there is a way to pass this block, so I can run it inside the perform method. If the answer is no, what do you think I should do to extend this worker behavior?

Thanks!

[jonhyman]

Sidekiq pro batch callbacks is what you want.

Sent from my mobile device
On Oct 17, 2014 9:01 AM, "Nícolas Iensen" [email protected] wrote:

Hey,

I'm implementing a worker that uses Sidekiq in order to run it
asynchronously.

Inside the perform method I'm doing an API request, followed by a
database update that depends on the API response.

class RequestWorker
include Sidekiq::Worker

def perform param1, resource_id
response = API.request(param1)
Resource.find(resource_id).update_attribute :attr1, response["attr1"]
endend

My Rails model looks like this:

class Resource < ActiveRecord::Base
after_create do
RequestWorker.perform_async("request param", self.id)
endend

It's working perfectly fine, but I would like to extend this worker to
make it possible to process the same API request for any Rails model, so
the code I wish I could implement on the models is:

class Resource1 < ActiveRecord::Base
after_create do
RequestWorker.perform_async("request param") do |api_response|
self.update_attribute :attr1, api_response["attr1"]
end
endend
class Resource2 < ActiveRecord::Base
after_create do
RequestWorker.perform_async("request param") do |api_response|
# Do whatever you want to do with the api_response
end
endend

And the worker perform method would be like this:

class RequestWorker
include Sidekiq::Worker

def perform param1
response = API.request(param1)
yield(response)
endend

Unfortunately it did't work, I get an error no block given (yield) in the
perform method. This should be happening because I'm calling perform_async
and passing the block to it, but it's not passing the block ahead to the
perform method.

My question is if there is a way to pass this block, so I can run it
inside the perform method. If the answer is no, what do you think I
should do to extend this worker behavior?

Thanks!

Reply to this email directly or view it on GitHub
https://github.com/mperham/sidekiq/issues/2010.

[nicolasiensen]

@jonhyman could you exemplify how can I use this batch feature to solve the problem? Many thanks!

[jonhyman]

Actually re-reading I realize that you want to do something with the API _response_ in the callback. So batch callbacks won't work for you. Have you considered using inheritance to subclass your jobs and have Resource1 call AttrFooUpdater.perform_async and Resource2 call AttrBarUpdater.perform_async with a common AttrUpdaterBase job class?

[mperham]

You can't pass complex arguments like blocks to a job and, as Jon pointed out, Batch callbacks allow you to do something after one or more jobs are complete but I think you really should build this logic into the worker itself, like so:

SyncModel.perform_async(self.class.name, self.id)

class SyncModel
  include Sidekiq::Worker

  def perform(klass, id)
    model = klass.constantize.find(id)
    make_request("request param") do |api_response|
      model.update_attribute :attr1, api_response["attr1"] 
    end
  end
end

[seuros]

@nicolasiensen , since you are using AR, you can use GlobalId to serialize your model.

[nicolasiensen]

@mperham, @seuros thanks for your reply