Sidekiq: Bundler::HTTPError Could not fetch specs from https://gems.contribsys.com/

[ACPK]

Ruby version:
Sidekiq / Pro / Enterprise version(s): Pro

I started receiving the following error about an hour ago when trying to deploy to production.

Bundler::HTTPError Could not fetch specs from https://gems.contribsys.com/

[kelishrestha]

Same here 👎
Ruby version: 2.3.0

Could not fetch specs from https://gems.contribsys.com/

[TheWudu]

Same here ..
JRuby-Version 9.0.4.0.

Bundler::HTTPError Could not fetch specs from https://enterprise.contribsys.com/

[TheTeaNerd]

+1, I'm getting Bundler::HTTPError Could not fetch specs from https://enterprise.contribsys.com/

[smeyfroi]

Same: this is blocking our production deployments using sidekiq-ent.

@mperham Could we get a status update soon so we can plan around the issue please?

[matthewbennink]

Same. Heroku build hangs with Fetching source index from https://gems.contribsys.com/.

[iMacTia]

Same here, https://gems.contribsys.com/ is timing out.
Production deploy blocked.
Anyone aware of a quick fix? We don't really need to update Sidekiq, we just need to deploy with Capistrano

[dawidpawliszko]

+1

[austinwang]

+1

[saveyou]

+1

[a-setyadi]

+1. Same here.

[oricodes89]

+1

[rohandaxini]

Same issue for us as well, unable to deploy due to these failures for bundle install
URL https://gems.contribsys.com/ is down

[ffiller]

+1

[scottrobertson]

+1

[TruptiPotdar-Edmodo]

Let's click thumbs up to indicate if you are facing the same issue on this comment or the top most description.

[pachkovsky]

Our CD is broken because of this!

P.S. in the end I just temporarily cached it in the app

[thapakazi]

Damn, it could break our autoscaling 😠 minions...

[ryantownsend]

According to various people on Twitter, this outage is caused by Digital Ocean – https://status.digitalocean.com

"Our engineering team is aware of connectivity issues in the SFO2 region, and we are actively working towards a resolution. During this time you may experience slow, or no connection to your droplets. We will provide an update soon." – 2017-04-11 07:00:01 UTC.

[asanghi]

We vendored the gem from our development environment and checked it in into our repo to proceed.

[prasathsarath]

@asanghi : Im getting error Could not find gem 'sidekiq-pro' in source at 'location_of_gem_folder in vendor'
Source does not contain any versions of 'sidekiq-pro'
while doing bundle install
Any suggestions?

[Bivek]

@iMacTia if it working fine in your dev env you can bundle package as a workaround for now

[iMacTia]

@Bivek I was able to bundle package from another project and got the .gem file to use in production so we're now fine.
Anyway, it's not an absolute working solution as in another project bundle package is trying to contact the https://gems.contribsys.com server

[douglascamata]

"enterprise" gem's host doesn't have a mirror outside of the Digital Ocean SF datacenter that is affected? A bit lame.

[zoras]

Guys, try mirroring your own local gem server with gemirro. I've partial success with it
http://guides.rubygems.org/run-your-own-gem-server/

[svenfuchs]

@prasathsarath you'll need to copy the gemspec over, too

mkdir -p vendor
gem=$(bundle show sidekiq-pro)
cp -r $gem vendor/sidekiq-pro
cp "$gem/../../specifications/sidekiq-pro-3.4.0.gemspec" vendor/sidekiq-pro/

# Gemfile:
# gem 'sidekiq-pro', path: 'vendor/sidekiq-pro'

bundle install

[mperham]

Server outage has been resolved. I do have a warm backup server running at all times but use DNS failover which needs to be triggered manually. I have server uptime monitoring which worked perfectly but unfortunately I was asleep during this outage. I apologize for the delayed response; I will look into how I might get my phone to wake me up in this instance.

[mperham]

As someone pointed out, the root cause was an extended datacenter outage at DigitalOcean SFO2.

[svenfuchs]

@mperham #ophugs ❤️

[smeyfroi]

@mperham have you considered (something like) https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover-configuring.html to avoid manual intervention?

[douglascamata]

@mperham we don't need you to wake up, it needs to be done automatically.

[thapakazi]

Thanks @mperham and all

Next time maybe, we would have some dedicated workers ready to SIG9 you up bad :neckbeard:

..jk

[ryansch]

@mperham I've had good experiences with using PagerDuty. I can set it to call me twice inside of 5 minutes to punch through iOS's do not disturb.

[corytheboyd]

On the subject of PagerDuty, their mobile application has a setting to override your phone's system volume. Useful for when you don't want normal notifications to wake you up, but you want pages to be loud and clear. Furthermore, the alert sounds feature such hits as "Cry", "Network's Down", and "Server's On Fire" from a Barbershop quartet ;)

[mperham]

PagerDuty is sounding pretty good, I'll evaluate it. My current monitor, UptimeRobot, doesn't have any notification support beyond a simple email.

[mperham]

For future reference, this is the best place to ask about server status. Everyone sending me individual emails and DMs only jams my inbox.

Some steps I've taken this morning:

1. New status site: https://status.sidekiq.org/
2. PagerDuty purchased, installed on phone and integrated with UptimeRobot.
3. New gem server replica built (the two current servers are in Toronto and Frankfurt)

Again, sorry for the extended outage and I hope these steps will prevent any further late night outages.

[mperham]

(If anyone wants to send me a 400x200 image that doesn't look terrible for that status page logo, I would be forever grateful)

[ACPK]

@mperham - I noticed that this was closed. Will there be an automated DNS change-over in the future?

[mperham]

It's certainly possible and something I'd like to get to but there's no quick solution that I know of today.

On Apr 13, 2017, at 15:34, Andrew Charles Potter Kelley [e-mail] wrote:

@mperham - I noticed that this was closed. Will there be an automated fix in the future?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[rhazegh]

Happening now:

       Fetching source index from https://gems.contribsys.com/

       Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://gems.contribsys.com/
       Retrying fetcher due to error (3/4): Bundler::HTTPError Could not fetch specs from https://gems.contribsys.com/
       Retrying fetcher due to error (4/4): Bundler::HTTPError Could not fetch specs from https://gems.contribsys.com/
       Could not fetch specs from https://gems.contribsys.com/
 !
 !     Failed to install gems via Bundler.
 !
 !     Push rejected, failed to compile Ruby app.
 !     Push failed

[minghz]

+1

[r1williams]

+1

[aHaydenLSS]

+1

[aleccarper]

+1

[liyucun]

+1

[ideaoforder]

+1

[g24fitzgerald]

+1

[jsilvia721]

+1

[aHaydenLSS]

Looks to be up and running again. Worked for me, anyways.

[remotezygote]

Status page says up, but it's still completely unreachable for me.

[mperham]

DNS caching.

On Jul 10, 2018, at 13:08, John Bragg [e-mail] wrote:

Status page says up, but it's still completely unreachable for me.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[chenxie1]

+1
Problem has come back...

[scottrobertson]

Fine here

[thiagoao]

+1

[tinbka]

+1

Erm... seems it's the issue with Russian internet policy which blocks a good amount of DO. VPN helps.

[mperham]

The gem server moved from NYC to Amsterdam this weekend (due to scheduled DO maint in NYC). I had one other customer report trouble and it was because they were geo-fencing their allowed network connections.

If you suddenly have problems, please check the https://status.sidekiq.org page. If that is green, please assume the trouble is on your end. Either due to Bundler misconfig, network policies or account cancellation.