Sharp: Status 403 from https://github.com/lovell/sharp-libvips/releases/download/...

[zaarheed]

I'm pretty new to Heroku, but I have managed to get the npm install command to work on my Linux server, Windows machine and my Mac. On Heroku, I get this error when deploying my app:

-----> VIPS app detected
-----> Installing libvips
       Downloading libvips and unpacking
       libvips installed
-----> Node.js app detected

-----> Creating runtime environment

       NPM_CONFIG_LOGLEVEL=error
       NODE_VERBOSE=false
       NODE_ENV=production
       NODE_MODULES_CACHE=false

-----> Installing binaries
       engines.node (package.json):  unspecified
       engines.npm (package.json):   unspecified (use default)

       Resolving node version 10.x...
       Downloading and installing node 10.15.3...
       Using default npm version: 6.4.1

-----> Restoring cache
       Caching has been disabled because NODE_MODULES_CACHE=false

-----> Installing dependencies
       Installing node modules (package.json + package-lock)

       > [email protected] install /tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/sharp
       > (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

       info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz
       /tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/sharp/install/libvips.js:80
                 throw new Error(`Status ${response.statusCode}`);
                 ^

       Error: Status 403
           at /tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/sharp/install/libvips.js:80:17
           at f (/tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/once/once.js:25:25)
           at ClientRequest.protocol.request.res (/tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/simple-get/index.js:63:5)
           at Object.onceWrapper (events.js:277:13)
           at ClientRequest.emit (events.js:189:13)
           at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:556:21)
           at HTTPParser.parserOnHeadersComplete (_http_common.js:109:17)
           at TLSSocket.socketOnData (_http_client.js:442:20)
           at TLSSocket.emit (events.js:189:13)
           at addChunk (_stream_readable.js:284:12)
       make: Entering directory '/tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/sharp/build'
         TOUCH Release/obj.target/libvips-cpp.stamp
         CXX(target) Release/obj.target/sharp/src/common.o
       ../src/common.cc:25:22: fatal error: vips/vips8: No such file or directory
       compilation terminated.
       sharp.target.mk:126: recipe for target 'Release/obj.target/sharp/src/common.o' failed
       make: *** [Release/obj.target/sharp/src/common.o] Error 1
       make: Leaving directory '/tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/sharp/build'
       gyp ERR! build error 
       gyp ERR! stack Error: `make` failed with exit code: 2
       gyp ERR! stack     at ChildProcess.onExit (/tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
       gyp ERR! stack     at ChildProcess.emit (events.js:189:13)
       gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
       gyp ERR! System Linux 4.4.0-1038-aws
       gyp ERR! command "/tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/.heroku/node/bin/node" "/tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/.heroku/node/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
       gyp ERR! cwd /tmp/build_de25ad290908a11f95fbe6f4c89bd6e4/node_modules/sharp
       gyp ERR! node -v v10.15.3
       gyp ERR! node-gyp -v v3.8.0
       gyp ERR! not ok 
       npm ERR! code ELIFECYCLE
       npm ERR! errno 1
       npm ERR! [email protected] install: `(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)`
       npm ERR! Exit status 1
       npm ERR! 
       npm ERR! Failed at the [email protected] install script.
       npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

       npm ERR! A complete log of this run can be found in:
       npm ERR!     /tmp/npmcache.Fqauu/_logs/2019-03-28T16_16_10_448Z-debug.log
-----> Change to Node.js build process 
       Heroku has begun executing the "build" script defined in package.json
       during Node.js builds.
       Read more: https://devcenter.heroku.com/changelog-items/1573
-----> Build failed

       We're sorry this build is failing! You can troubleshoot common issues here:
       https://devcenter.heroku.com/articles/troubleshooting-node-deploys

       Some possible problems:

       - Node version not specified in package.json
         https://devcenter.heroku.com/articles/nodejs-support#specifying-a-node-js-version

       Love,
       Heroku

 !     Push rejected, failed to compile Node.js app.
 !     Push failed

And as suggested in other issues, I have this buildpack added: https://github.com/alex88/heroku-buildpack-vips.git. Any suggestions on what the issue might be?

[mfrobben]

This is happening on circleci as well.

```> [email protected] install /home/circleci/repo/node_modules/sharp

(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz
/home/circleci/repo/node_modules/sharp/install/libvips.js:81
throw new Error(Status ${response.statusCode});
^

Error: Status 403
at /home/circleci/repo/node_modules/sharp/install/libvips.js:81:17
at f (/home/circleci/repo/node_modules/once/once.js:25:25)
at ClientRequest.protocol.request.res (/home/circleci/repo/node_modules/simple-get/index.js:63:5)
at Object.onceWrapper (events.js:273:13)
at ClientRequest.emit (events.js:182:13)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:555:21)
at HTTPParser.parserOnHeadersComplete (_http_common.js:109:17)
at TLSSocket.socketOnData (_http_client.js:441:20)
at TLSSocket.emit (events.js:182:13)
at addChunk (_stream_readable.js:283:12)
make: Entering directory '/home/circleci/repo/node_modules/sharp/build'
TOUCH Release/obj.target/libvips-cpp.stamp
CXX(target) Release/obj.target/sharp/src/common.o
../src/common.cc:25:22: fatal error: vips/vips8: No such file or directory
#include
^
compilation terminated.
sharp.target.mk:128: recipe for target 'Release/obj.target/sharp/src/common.o' failed
make: * [Release/obj.target/sharp/src/common.o] Error 1
make: Leaving directory '/home/circleci/repo/node_modules/sharp/build'
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack at ChildProcess.emit (events.js:182:13)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
gyp ERR! System Linux 4.4.0-141-generic
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/circleci/repo/node_modules/sharp
gyp ERR! node -v v10.11.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN [email protected] requires a peer of ajv@^6.9.1 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/nodemon/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/circleci/.npm/_logs/2019-03-28T16_41_50_493Z-debug.log```

[bradak1]

Happening in Circle for me as well:

info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz
/home/circleci/project/node_modules/sharp/install/libvips.js:80
          throw new Error(`Status ${response.statusCode}`);
          ^
Error: Status 403
    at /home/circleci/project/node_modules/sharp/install/libvips.js:80:17

[mfrobben]

@lovell I tried vendoring but the underlying binaries generated by npm install aren't compatible cross platform. Any suggestions or workarounds?

[mfrobben]

Aha - I saw in the docs you have SHARP_DIST_BASE_URL as an overrideable env var. If you're here wondering how to unbreak your build, I think putting a prebuilt binary in a public S3 bucket and overriding that should do the trick...working on it.

[mfrobben]

That works.

1. Pull down one of the tarballs that matches your sharp version and destination platform, e.g.:
   https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz

2. Throw them on s3 / dropbox / whatever.
   Change your build script to either set the SHARP_DIST_BASE_URL environment variable for the container, or directly prepend the npm install command with the override:
   SHARP_DIST_BASE_URL= https://s3.amazonaws.com/<YOUR BUCKET> npm install

3. Build is working again.

[lovell]

Given three people have just reported this problem within a few minutes of each other, I'd guess there are currently (or were recently) temporary networking/permission problems somewhere between GitHub and AWS, which hosts GitHub releases.

I'm aware of some intermittent problems earlier today in the AWS us-west-2 region that might be related.

[cbelsole]

I am able to recreate the issue with wget on circle

wget --header="accept-encoding: gzip, deflate" --header="Host: github.com" --header="Connection: close" https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz

--2019-03-28 17:53:55--  https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found

Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/104502835/71a4e080-bc4b-11e8-82df-da868f4615d4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190328%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190328T175355Z&X-Amz-Expires=300&X-Amz-Signature=d68b252599122a33e61d0fc93c9095205c68d7d96cd01298199d91ac676c1a39&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlibvips-8.7.0-linux-x64.tar.gz&response-content-type=application%2Foctet-stream [following]

--2019-03-28 17:53:55--  https://github-production-release-asset-2e65be.s3.amazonaws.com/104502835/71a4e080-bc4b-11e8-82df-da868f4615d4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190328%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190328T175355Z&X-Amz-Expires=300&X-Amz-Signature=d68b252599122a33e61d0fc93c9095205c68d7d96cd01298199d91ac676c1a39&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlibvips-8.7.0-linux-x64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.1.24
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.1.24|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2019-03-28 17:53:55 ERROR 403: Forbidden.

When you remove the host header you get a 200:

wget --header="accept-encoding: gzip, deflate" --header="Connection: close" https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz

--2019-03-28 17:46:52--  https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found

Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/104502835/71a4e080-bc4b-11e8-82df-da868f4615d4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190328%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190328T174652Z&X-Amz-Expires=300&X-Amz-Signature=e0a25d0a384a055f9bbce83d991c4f928d82babdad8fa083268987c8b4abb2ab&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlibvips-8.7.0-linux-x64.tar.gz&response-content-type=application%2Foctet-stream [following]

--2019-03-28 17:46:52--  https://github-production-release-asset-2e65be.s3.amazonaws.com/104502835/71a4e080-bc4b-11e8-82df-da868f4615d4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190328%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190328T174652Z&X-Amz-Expires=300&X-Amz-Signature=e0a25d0a384a055f9bbce83d991c4f928d82babdad8fa083268987c8b4abb2ab&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dlibvips-8.7.0-linux-x64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.176.19
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.176.19|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8352468 (8.0M) [application/octet-stream]
Saving to: ‘libvips-8.7.0-linux-x64.tar.gz.1’

libvips-8.7.0-linux-x64.tar.g 100%[================================================>]   7.96M  --.-KB/s    in 0.08s

2019-03-28 17:46:52 (98.3 MB/s) - ‘libvips-8.7.0-linux-x64.tar.gz.1’ saved [8352468/8352468]

It looks like the common denominator here is s3 which returns a 403 if you don't have permissions to read the file. This looks like a problem in how simpleGet handels the redirect.

[DiegoTc]

@mfrobben so basically if my S3 is test it will be
SHARP_DIST_BASE_URL= https://s3.amazonaws.com/test npm install
and it should be working?

[mfrobben]

Yep - might need a trailing slash.

On Thu, Mar 28, 2019 at 3:48 PM Diego [e-mail] wrote:

@mfrobben https://github.com/mfrobben so basically if my S3 is test it
will be
SHARP_DIST_BASE_URL= https://s3.amazonaws.com/test
https://s3.amazonaws.com/test npm install
and it should be working?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lovell/sharp/issues/1632#issuecomment-477745396, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACBOL9TZ0zhcwYRiwDz0927u2x5uFryRks5vbRyrgaJpZM4cQmRq
.

[DiegoTc]

Adding information @LA1CH3
https://github.com/lovell/sharp/issues/1634#issuecomment-477734169

We too are facing this issue and have our own internal repo setup for binaries using the SHARP_DIST_BASE_URL environment variable but when installing in Docker today it seemed to ignore the environment var and install from public Github, giving us the exact same error as logged above.

[DiegoTc]

I can confirm your solution work for me @mfrobben

RUN SHARP_DIST_BASE_URL=https://s3.amazonaws.com/MYS3NAME/ npm install

[lovell]

I've contacted GitHub support about this.

[LA1CH3]

@DiegoTc Ah, our SHARP_DIST_BASE_URL is just another Github repo where we host them so thats probably why ours is still failing. I bet if I move to S3 then then it would work. Will wait to see what @lovell hears but otherwise will move our self-hosted binaries to S3 or another location if necessary.

[fyodorvi]

Same thing for us, CircleCI and node-alpine.

[TimDurward]

Same thing for us too, we're in the process of temporarily hosting our own libvips on S3.

[TimDurward]

We've made our fix public (all versions in Github Releases are available)
Add this as an environment variable in CircleCI:
SHARP_DIST_BASE_URL=https://s3-us-west-2.amazonaws.com/sharp-distro/

Temporary until real fix comes in from Github Release

[farfromrest]

@TimDurward I am getting a 403 with that URL.

[JakeMakesStuff]

@farfromrest Same, I just made my own mirror: https://s3.magiccap.me/sharp/

[joshbench]

@TimDurward I am getting a 403 with that URL.

Oops, we only uploaded one version there. All versions on the releases page should be there now.

[DiegoTc]

@lovell has GitHub reply back?

[aarondack]

Not trying to pile it on or anything, but seeing this as well in circle.

[unixzii]

And I'm wondering if the install script is correct, it seems that some header files are missing to finish the fallback compiling.

[zivagolee]

I did some investigation and I am not quite sure why simple-get keeps returning 403 from a circleci node. However, when I switched to use node-wget, it worked fine. Not sure if anyone would like my changes to install/libvips.js?

[JakeMakesStuff]

Here's a thing that I think might be related: Electron 4.0.5 does the same for me in CircleCI

[lovell]

There has been no response from GitHub support so I have sent another message both providing and requesting more details.

Additional evidence of the same problem affecting Electron can be seen at https://twitter.com/mipearson/status/1111533780706062336

My best guess is that (new?) rate limiting by source IP address is being applied to the shared AWS S3 bucket used by GitHub for releases. Perhaps Microsoft want to rate limit what they pay Amazon.

If anyone affected by this is currently paying GitHub or CircleCI for support then I suggest you contact them.

It looks like you've all found the SHARP_DIST_BASE_URL environment variable, which was added to workaround exactly this kind of scenario.

[JakeMakesStuff]

@lovell Thanks for doing all you can considering the situation, it is appreciated. I am in the process of making a mirror for Electron 4.0.5 at the moment.

UPDATE: My Electron mirror is now up: ELECTRON_MIRROR="https://s3.magiccap.me/electron/". It currently has Electron v4.0.5, but if anyone wants anything else, I'll happily add.

[lovell]

A reply from GitHub:

"I have escalated this to our data center team. I do not have an ETA currently but we'll follow up as soon as we have some news. I'm sorry for the trouble this is causing in the meantime!"

[LA1CH3]

Just want to echo @zivagolee , albeit our project is on a corporate network requiring proxy, we run our CI jobs in a Docker container in Jenkins and even trying the public mirrors are all still returning 403 for me, which makes me think its an issue with something in the install script- though, that wouldn't explain why others are able to use the public mirrors fine. We never had any issues pulling in Sharp binaries in our environment until the issues began with everyone else yesterday.

[JakeMakesStuff]

@LA1CH3 I don't think it's anything script related, Electron is having issues too.

[zivagolee]

@LA1CH3 @JakeMakesStuff @lovell I'm not exactly sure why when using simple-get it returns a 403 "Your access to this site has been restricted." Maybe a user-agent or some other type of header github expects? As mentioned, I tried using the node-wget module and there seems to be no issues. Commit I made here my fork:

https://github.com/zivagolee/sharp/commit/1d36a9426adfb5a2f46b15ac44117065c5fd43f4

[JakeMakesStuff]

@zaarheed I didn't think of it like that, it is possible, could also explain Electron's issues.

[cbelsole]

@zivagolee Check my post above. I ran simpleget in node to get the headers it was passing. On a redirect it looks like it deletes the host header, but that was not the behavior I was seeing. I removed the host header via wget and got a 200.

[zivagolee]

@cbelsole good findings. I agree on that simple-get does not properly figure out redirects (without some work). With my commit, I find that node-wget seems to do this on properly on its own.

[jclangst]

I don't believe that it actually has to do with simple-get. Here is a test script that demonstrates the issue:

const https = require('https');

const opts = {
  agent: null,
  hostname: 'github.com',
  port: null,
  protocol: 'https:',
  auth: null,
  path:
   '/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linux-x64.tar.gz',
  headers: { 'accept-encoding': 'gzip, deflate' }
}

const req =  https.request(opts, (res) => {
 console.log(res.statusCode);
});

req.end();

This script seems to work EXCEPT for running on AWS (regardless of the base image) with Node 10 (other versions of node work fine). We are able to run that script on other environments (local, Digital Ocean, etc.) with any version of Node 8+.

I don't know if anyone else can confirm or if others are experiencing different results.

Edit:

By work I mean returns 302 rather than 403.

[LA1CH3]

FWIW we are no longer receiving 403s on our end and Sharp install is back to normal.

[adrw]

Moving to S3 artifacts instead of relying on Github probably is a good idea long term, will that continue to be pursued?

[lovell]

Thanks for the reports of this situation returning to normal. I'm going to leave this open until I hear back from GitHub as I suspect the lack of errors may be because the shared S3 bucket used for releases is no longer experiencing peak traffic and is therefore no longer being rate-limited (if that is the reason).

The prebuilt libvips binaries provided by sharp were previously hosted on Bintray but they imposed a limit, originally 1TB/month then increased to 4TB/month in 2017. See https://github.com/lovell/sharp/issues/964 for the kind of problem that led to the use of GitHub Releases instead.

It's a good example of the tragedy of the commons, albeit Microsoft-provided, so if you work at a for-profit organisation then please consider self-hosting these files in your CI environments.

[JakeMakesStuff]

@lovell Thank you for being so responsive about this, I took this opportunity to move all required binaries for my application to an S3 bucket. Should save any of this in the future!

[lovell]

The closing response from GitHub:

"This wasn't the result of any rate limits. This was a configuration change that was rolled back - we're sorry for the trouble this caused!"

Whilst I have everyone's attention, it's always great to learn more about what people are using sharp for in the survey at #35 if you're able to share details publicly.

[lcnvdl]

I'm having this problem. I really don't understand what I have to do :(