Sidekiq: Circular Dependency

[PeteMichaud]

I'm upgrading an app from 3.2 to 4.0, and sidekiq won't run.

I have a model that uses concerns, and everything works fine in the application and on the console.

When I try to run "bundle exec sidekiq -q reminder_queue -q scheduler_queue -q default"

It gives me this:

  Circular dependency detected while autoloading constant Contact::Avatarable
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:460:in `load_missing_constant'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:183:in `const_missing'
  /Users/pete/Projects/_Apps/contactio/app/models/contact.rb:3:in `<class:Contact>'
  /Users/pete/Projects/_Apps/contactio/app/models/contact.rb:1:in `<top (required)>'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:423:in `load'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:423:in `block in load_file'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:615:in `new_constants_in'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:422:in `load_file'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:323:in `require_or_load'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:462:in `load_missing_constant'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:183:in `const_missing'
  /Users/pete/Projects/_Apps/contactio/app/models/concerns/contact/avatarable.rb:1:in `<top (required)>'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:423:in `load'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:423:in `block in load_file'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:615:in `new_constants_in'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:422:in `load_file'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:323:in `require_or_load'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:288:in `depend_on'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:206:in `require_dependency'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/railties-4.0.0/lib/rails/engine.rb:465:in `block (2 levels) in eager_load!'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/railties-4.0.0/lib/rails/engine.rb:464:in `each'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/railties-4.0.0/lib/rails/engine.rb:464:in `block in eager_load!'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/railties-4.0.0/lib/rails/engine.rb:462:in `each'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/railties-4.0.0/lib/rails/engine.rb:462:in `eager_load!'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/sidekiq-2.15.2/lib/sidekiq/cli.rb:199:in `boot_system'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/sidekiq-2.15.2/lib/sidekiq/cli.rb:42:in `parse'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/gems/sidekiq-2.15.2/bin/sidekiq:7:in `<top (required)>'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/bin/sidekiq:23:in `load'
  /Users/pete/.rvm/gems/ruby-2.0.0-p247/bin/sidekiq:23:in `<main>'

There's definitely no circular dependency, contact and avatarable are pretty basic. Not sure how to proceed.

[mperham]

This is a Rails issue with your models, not a Sidekiq issue. I'm just calling eager_load!

[mperham]

Your Avatarable class should have no reference to Contact at all.

[PeteMichaud]

Here is the concern, in its entirety:

module Contact::Avatarable
  extend ActiveSupport::Concern

  included do
    mount_uploader :avatar, AvatarUploader
  end

end

It pretty much can't be more basic, and there's no circular dependency. It works fine in the app, and in the console. The only time I get an issue is when I try to run sidekiq.

Just for kicks, I moved the mount_uploader into the model itself and stopped including the concern. If I do that, sidekiq runs fine.

I've read in other places that concerns have been causing circular dependency issues in some gems, but my impression was that it was all controller concerns, it had something to do with unloadable. That doesn't seem relevant, but maybe there's a clue?

[mperham]

The circular dep is in the name itself. Does Contact explicitly require Avatarable?

[PeteMichaud]

class Contact < ActiveRecord::Base
  include Meetable
  include Avatarable

  paginates_per 10
  default_scope order(:name)

  belongs_to :user
  belongs_to :relationship_type
  has_many :contact_bits

  validates :name, presence: true
  validates :user, presence: true
  validates :relationship_type, presence: true

end

There's Contact. As you can see, Contact does have another concern that runs fine including with sidekiq (Meetable is named similarly, ie Contact::Meetable). At least one other model includes namespaced concerns as well. It's apparently just this Avatarable thing. Maybe sidekiq actually doesn't get along with CarrierWave+concerns?

[PeteMichaud]

Wow, something confusing just happened.

Like I said before, when I tried this:

class Contact < ActiveRecord::Base
  include Meetable
  #include Avatarable

  paginates_per 10
  default_scope order(:name)

  belongs_to :user
  #has_many :important_dates
  belongs_to :relationship_type
  has_many :contact_bits

  mount_uploader :avatar

  validates :name, presence: true
  validates :user, presence: true
  validates :relationship_type, presence: true

  def relationship_type=(value)
    if value.nil?
      write_attribute(:relationship_type_id, value)
    else
      write_attribute(:relationship_type_id, value.id)
    end
    super
  end

  def relationship
    return relationship_type.label unless relationship_type.nil?
    nil
  end

end

Sidekiq ran fine. I just got rid of the Avatarable concern, and it ran with the Contact::Meetable without issue.

So just to see what would happen, I put Avatarable back in, but this time I changed it to a general model concern, instead of one specific to Contact. When I ran sidekiq, it didn't complain about Avatarable, but it said there was a circular dependency in Meetable now!

[PeteMichaud]

When I comment out the Avatarable concern and don't mount the uploader at all, it also runs fine. It only complains about Meetable if I try to include Avatarable.

[mperham]

It looks to me like you just can't nest those concerns within the top-level Contact class. I wish I knew more but this is a new error to me.

[PeteMichaud]

I created a dummy concern called Testable, that does absolutely nothing but extend ActiveSupport::Concern.

First, I namespaced it to Contact, ie Contact::Testable, and I included it (still without Avatarable). That ran fine in sidekiq, with Meetable.

Then I moved it so it was a general concern, ie ::Testable. That also ran fine with Meetable.

The rest of the behavior is identical -- if I include Contact::Avatarable, it says circular dependency on Contact::Avatarable. If I include ::Avatarable, it says circular dependency on Contact::Meetable. Including or not including Testable has no effect, and namespacing it or not namespacing it has no effect.

So then I decided to try to make Testable an iota less trivial, and changed this:

module Testable
  extend ActiveSupport::Concern

end

To this:

module Testable
  extend ActiveSupport::Concern

  included do

  end

end

Then I included Testable and Meetable in Contact, and got the circular dependency error on Contact::Meetable. So I changed Testable to Contact::Testable, thinking that it would cause a circular dependency error on Contact::Testable, but it didn't--it caused a circular dependency error still on Contact::Meetable! Including Testable before Meetable makes no difference.

[jbgo]

Going back to the original example, I believe the circular dependency is actually on Contact and not Contact::Avatarable. This may be a superclass mismatch in disguise. If Rails is eager loading Contact::Avatarable first, it would create a Contact module that conflicts with your Contact model.

In an empty rails 4 app, if I change module Contact::Avatarable to

class Contact < ActiveRecord::Base
  module Avatarable

...the circular dependency goes way for me.

[pixeltrix]

@jbgo :+1: to your solution, but it is a circular dependency not a superclass mismatch. What's happening is that the class definition in contact.rb hasn't been completed so Contact still doesn't exist when it loads the concern. By defining the module in the concern as module Contact::Avatarable it triggers const_missing so ActiveSupport::Dependencies will try to load contact.rb. It sees that this has already been loaded so raises the circular dependency error. By changing the concern to class Contact < ActiveRecord::Base you eliminate the call to const_missing allowing the class definition to complete successfully.

[macool]

:+1:

[chaizhenhua]

hi, I also got circular dependency error.

in lib folder i have two worker class WorkerA, WorkerB include the same module Handler::Poller.
when I run WorkerA.async_perform and WorkB.async_perform in rails console and then start sidekiq. i found crash msg in the log. and one of the message is lost. after the first crash, I run async_perform again, and it works fine.

/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/dependencies.rb:463:in `load_missing_constant'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/dependencies.rb:184:in `const_missing'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/inflector/methods.rb:228:in `const_get'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/inflector/methods.rb:228:in `block in constantize'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/inflector/methods.rb:224:in `each'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/inflector/methods.rb:224:in `inject'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/inflector/methods.rb:224:in `constantize'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/sidekiq-2.16.1/lib/sidekiq/processor.rb:42:in `block in process'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/sidekiq-2.16.1/lib/sidekiq/processor.rb:83:in `do_defer'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/sidekiq-2.16.1/lib/sidekiq/processor.rb:37:in `process'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize'
/home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'
2014-01-08T18:53:58Z 5727 TID-cc93o ERROR: Sidekiq::Processor crashed!
RuntimeError: Circular dependency detected while autoloading constant Handler::Poller
    /home/chaizhenhua/.rvm/gems/ruby-2.0.0-p247/gems/activesupport-4.0.1/lib/active_support/dependencies.rb:461:in `load_missing_constant' 

[mperham]

@chaizhenhua As the issue states, this is a problem in your app code, not Sidekiq. You'll have to fix the bug, we can't.

[chaizhenhua]

@mperham OK. I will check my codes. but why it works after first crash?

[pixeltrix]

@chaizhenhua because the constant has now been loaded and load_missing_constant is not called

[chaizhenhua]

@pixeltrix thanks for your reply, but i still don't know how to fix it. do you have any suggestion?

[pixeltrix]

@chaizhenhua not without seeing your code, no - other than defining things like this:

module Foo
  class Bar
    # stuff
  end
end

rather than

class Foo::Bar
  # stuff
end

but you'll have to post some code if you want anything more specific.

[chaizhenhua]

@pixeltrix I tried again. when i set concurrency to 1 in sidekiq.yml and restart sidekiq it wont crash. but when set concurrency back to 25 sidekiq crash again.

My codes looks like this, Handler::Poller will be included by multi classes.

#lib/handler/poller.rb
module Handler
  module Poller
    base.class_eval {
      include Sidekiq::Worker
      sidekiq_options queue: :default
      # sidekiq_options queue: self.to_s
    }
    base.extend(ClassMethods)
    module ClassMethods ... end;

    def perform(*args)
      // processing
    end
  end
end

#lib/module_a/a.rb
module ModuleA
  class A < Base
    include Handler::Poller
  end
end

#lib/module_a/b.rb
module ModuleA
  class B < Base
    include Handler::Poller
  end
end

[pixeltrix]

Sorry, but this isn't your actual code is it? If you enter pseudo code I can't tell whether you've changed something in creating it. For example the lib/handler/poller.rb doesn't look right the base.class_eval should be inside a def self.included(base) method I'm assuming. Also what is the Base class? Please use your actual code - if you think it's too much to embed inline then use a gist.

[chaizhenhua]

hi @pixeltrix @mperham, please take a look at this gist https://gist.github.com/chaizhenhua/8335236. I think the crash is due to thread race condition. in my module code there are many dynamic methods. the module need some time to load, and other thread try to load the same module again, and then sidekiq crash.

Is it a bug? or how to avoid the crash?

[pixeltrix]

You've got a sleep(1) before the include Sidekiq::Worker so the jid= method hasn't been defined yet when Sidekiq starts processing. The error isn't the same as the issue subject and your original backtrace so I'm assuming that this is still pseudo-code. It's also it's completely contrived so I don't what you're trying achieve by posting it.

I'm sorry but I can't help any further.

[chamnap]

@pixeltrix, I faced the circular dependency problem as well. In my app, there are around 100 models. I want to make them to be organized in a proper namespace and good directory structure. Therefore, I layout my models in this way:

app/models/yoolk/listing.rb
app/models/yoolk/listing/communication.rb
app/models/yoolk/listing/extra_communication.rb

The question is that: I have to define those models this way?

require_dependency "yoolk/core/base_model"

module Yoolk
  class Listing < Core::BaseModel
  end
end

require_dependency "yoolk/core/base_model"
module Yoolk
  class Listing < Core::BaseModel
    class Communication < Core::BaseModel
    end
  end
end

[pixeltrix]

@chamnap yes, that should work because the Yoolk constant will be defined in the require_dependency.

[chamnap]

@pixeltrix, I tried it, and it still has problem. I have no idea to fix this.

Circular dependency detected while autoloading constant Yoolk::Listing::Communication

[pixeltrix]

@chamnap I just replicated your structure in a new Rails app and it worked fine - I suggest you create a new issue on the Rails bug tracker with simple steps to reproduce and a backtrace. Once you've done that I will take a further look. Either way this is isn't a problem with Sidekiq so we should move the discussion.

[chamnap]

@pixeltrix, I fixed the problem now. The thing is that this problem only exists in development environment. In production or staging environment, it works quite well. In development, it happens when concurrency is greater than 1. It logs a few errors (circular dependency) at the beginning, and then it works as normal.

In my app, those models are inside rails engine, which is used among other rails apps. I check the sidekiq codebase, and I see in this line, https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb#L201. It eager loads the rails app. However, I don't understand why it doesn't eager load my rails engine. The solution for now is:

Yoolk::Core::Engine.eager_load! if File.basename($0) == "sidekiq"

Thanks for your help.

[pixeltrix]

@chamnap the general rule of thumb I follow is to properly nest class/module definitions and require_dependency is your friend. There's often a discrepancy between development and production because a class contains Object in its ancestors whereas a module doesn't, e.g:

>> Module.new.ancestors
=> [#<Module:0x007fbdf38953d8>]
>> Class.new.ancestors
=> [#<Class:0x007fbdf38920e8>, Object, Kernel, BasicObject]

This causes problems because when const_missing is triggered inside a class, ruby will then search for the missing constant in Object which if a top level constant matches (e.g Core) it will then look inside that for Core::Object rather than Yoolk::Core::Object - this all happens before ActiveSupport::Dependencies can intervene so there's nothing we can do to fix it. If the missing constant is inside a module then our load_missing_constant is triggered and Rails can do it's magic. As you've discovered everything is fine in production because everything is eager loaded.

As for why your engine is not being eager loaded, perhaps Sidekiq is calling eager_load! too early? i.e. before Rails has added the engines to the eager load paths - I'd have to investigate further.

[chamnap]

@pixeltrix, Thanks for the advise.

I think the problem in development is because ActiveSupport Depedencies is not threadsafe, http://blog.plataformatec.com.br/tag/eager-load/. That's why sidekiq tries to eager load everything in https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb#L201.

In the codebase, I didn't change anything:

require_dependency "yoolk/core/base_model"
module Yoolk
  class Listing::Communication < Core::BaseModel
  end
end

[pixeltrix]

I think the problem in development is because ActiveSupport Depedencies is not thread safe

Any kind of code loading is not thread-safe - even just standard require or autoload.

[chamnap]

From jose valim blog post, he said:

module Foo
autoload :Bar, "path/to/bar"
end
Now, the first time Foo::Bar is accessed, it will be automatically loaded. The issue with this approach is that it is not thread-safe, except for latest JRuby versions (since 1.7) and Ruby master (2.0).

[afunction]

I notice if your named concerns module prefix same as model class name will get error

For example:

Your have a "Order" model, then include Conerns "Order::Validate", and you will get error when you run sidekiq process.

Do you have any solutions to fix it?

Ruby 2.1.0
Rails 4.0.2
Sidekiq: 2.17.5

[pixeltrix]

This is the scenario described earlier in this discussion - have you tried writing it this way:

# app/models/concerns/order/validate.rb
class Order < ActiveRecord::Base
  module Validate
    extend ActiveSupport::Concern

    included do
      # do stuff
    end

    module ClassMethods
      # class method definitions
    end

    # instance method definitions
  end
end

# app/models/order.rb
require_dependency 'order/validate'

class Order < ActiveRecord::Base
  include Order::Validate
end

I haven't tested this but this is the layout you need to adopt - my only concern is that Validate may clash with an existing Active Record constant.

To any future person looking at this ticket please leave @mperham alone - this isn't anything to do with Sidekiq - use one of the regular Ruby/Rails support channels and if you think you've found a bug with autoloading in Rails please report it on our issues page.

[afunction]

Thanks, I fixed it, it runs great.

[afunction]

Oh, sidekiq process is ok, but the web is faild
"Circular dependency detected while autoloading constant Order::Validate"

[pabloivan57]

Hey guys, I know this is probably our code's fault, but haven't been able to figure out why so far... This is the code where the problem occurs.

module MyApp
  module APIIntegration
    class Client          
       def driver
         @driver ||= MyApp::APIIntegration::Driver.new
       end  
    end
  end
end 

We keep getting "Circular dependency detected while autoloading constant MyApp::APIIntegration::Driver" on production environment. What are we doing wrong? any suggestions?

[pratik60]

I initially changed this setting

config/environments/development.rb

config.eager_load = true

This didn't help though!

And I needed to add

config/initializers/eager_load.rb:

Rails.application.eager_load! unless Rails.env.test?

You also need to eager load the entire lib folder, if you are using one (autoload isn't enough).

config/application.rb

config.eager_load_paths += Dir["#{Rails.root}/lib/**/"]

Also, make sure that if devise is used in initializer it is renamed to 01_devise.rb or something, as initializers are loaded alphabetically, and your User or Admin, will reference it.

Probably suggest that in tests eager_load is skipped as it is suboptimal, and if you are not using threads internally, skip it altogether!

[bf4]

@pratik60 that advice might be appropriate for Rails 3.x, but for modern Rails, it is sufficient to add to your config/application.rb something like

    # http://blog.arkency.com/2014/11/dont-forget-about-eager-load-when-extending-autoload/
    # https://github.com/rails/rails/blob/4-2-stable/railties/lib/rails/engine/configuration.rb#L42-L51
    # Also:
    #    https://github.com/mperham/sidekiq/wiki/FAQ#why-doesnt-sidekiq-autoload-my-rails-application-code
    #    https://github.com/rails/rails/blob/4-2-stable/railties/lib/rails/engine.rb#L466-L475
    #    https://github.com/rails/rails/blob/4-2-stable/railties/lib/rails/paths.rb#L58-L61
    config.paths.add 'lib', load_path: true, eager_load: true
    config.paths.add 'app/workers', eager_load: true
    config.paths.add 'app/workers/concerns', eager_load: true

[mperham]

I would strongly urge people not to adjust their paths at all. Stick with the Rails defaults and design your code to work with them.