Sidekiq: Bundler::HTTPError Could not fetch specs from https://gems.contribsys.com/

[geoffreylitt]

Hi @mperham ,

We're not able to build our app because bundle install fails trying to fetch the sidekiq-pro gem.

The status website (https://status.sidekiq.org/) shows that the gem server is down, so I'm presuming the effects we're seeing are caused by an infrastructure problem there.

Do you have a sense of when the gem server will be back up? If it might be a while, do you have a best practice recommendation for working around the issue?

Thanks,
Geoffrey

p.s. I wasn't sure if a Github issue was the best way to report an infrastructure issue like this; please let me know if there is a better way for next time.

Version Details:

Ruby version: 2.1.5
Sidekiq / Pro / Enterprise version(s): Sidekiq v4.2.10 / Sidekiq Pro v3.5.0

Please include your initializer and any error message with the full backtrace:
Bundler::HTTPError Could not fetch specs from https://gems.contribsys.com/

[mperham]

A customer is DOSing the server with an CI infinite loop. They are working on stopping it now but I don't have a timeline when it will be fixed. It's subsided a bit as I've manually blocked the following IP ranges:

34.*
35.*
54.*
52.*

But if any customers are in those ranges, they'll be caught too. Unfortunately this CI service is distributed, using dozens of machines and IP addresses so I can't easily block 5-10 individual IPs or even smaller subnets.

[geoffreylitt]

Thanks for the update. We're starting to see some builds go through, at least after several retries.

Assuming the issue continues to persist, is there any recommended way to bypass the gem server in our builds?

[mperham]

Vendor the gem. A gem is just ruby code unpacked into a directory on your disk.

[mbasset]

@mperham The status page https://status.sidekiq.org/ seems to indicate everything is good. Perhaps this could be updated so we know when to try restarting our builds again?

[mperham]

My monitoring shows the server is back to normal now, as of 10:45 Pacific.

[jimiray]

@mperham Are the IP ranges above still blocked?

[richessler]

Everything seems to be back up now.

[comjf]

Can you please let us know when you'll remove the Blocks. We fall within that 54.* (and I think a lot of AWS does too, at least us-east-1)

[mperham]

@jimiray No, I've switched the server back to one without IP blocks.

[mperham]

Ok, customer reports their CI is disabled now until the service is fixed. I will look into adding an emergency Cloudflare cached mode to handle DDOSes in the future.

[mperham]

3597

[geoffreylitt]

Thanks for resolving this!

Just for future reference -- is it within the terms of our license to vendor the gem on an ongoing basis to reduce the need for this dependency in our build process?

[mperham]

It's absolutely within the license. As long as your subscription is active, you can cache/vendor the code for resiliency.

[geoffreylitt]

Awesome, thanks so much.

[aHaydenLSS]

Server is down again. How exactly do you vendor a gem? @mperham