Ghost: 502 on image upload, but works fine with `ghost run`

[ekerstein]

I'm having a similar issue as #3827 and the post on the forum here.

I just updated to 2.31.1 and now whenever I upload an image I get a 502 error. I'm using a DO droplet.

I tried debugging with ghost run but then it works. However, when I go back to ghost start it gives me 502s again.

Technical details:

• Ghost Version: 2.31.1
• Node Version: 10.16.3

[ekerstein]

Update:
I rolled back my ghost version to 2.30.2 and it started working again.

[andreborud]

As you got it working again by downgrading I'm not sure if nginx is the problem. On my droplet running CentOS I had to change the max size of any file being uploaded in the base config of nginx located in /etc/nginx/nginx.conf. I don't remember the default size 2 or 5mb.

http {
    ...
    client_max_body_size 50M; <- change this line
    ...
}

[label-actions[bot]]

Hey @ekerstein :wave: We ask that you please do not use GitHub for help or support 😄. We use GitHub solely for bug-tracking and on-going feature development.

Many questions can be answered by reviewing our documentation. If you can't find an answer then our forum is a great place to get community support, plus it helps create a central location for searching problems/solutions.

FYI: Many projects have their own support guidelines and GitHub will highlight them for you as it did here, or the project owners will use issue templates to point you in the right direction. Reading the guidelines or issue templates before opening issues can save you and project maintainers valuable time.

[ekerstein]

@andreborud What do you recommend changing the client_max_body_size to?

[andreborud]

Up to you, depends on how large files you plan to upload. I set it to 50 so I don't need to think about it too much.

[ivRodriguezCA]

Dunno if it was the same problem. But in my case, the issue was that some images were stored in /content/images/{year}/{month}/ but referenced at content/images/size/w{number}/{year}/{month}/ when a GET request to size/w{number}/... came in Ghost would crash and restart. I had to manually copy and move some of them around.

For example:

I had to copy content/images/2019/06/hero-image.jpg into content/images/size/w2000/2019/06/hero-image.jpg for it to be loaded on the post and avoid Ghost crashing.

Edit: Adding this comment because editing /etc/nginx/nginx.conf didn't help with my problem.

[andreborud]

@ekerstein Did you manage to get it working? I have run in to the same issue now. I haven't tried to upload an image for a while, tried last night and I get a 502 error.

[andreborud]

In ghost I get "Something went wrong :(" and when inspecting I see this error: POST https://andreborud.com/ghost/api/canary/admin/images/upload/ 502

But going in to the error logs of ghost there is no 502 errors logged, the only persistent error is the following:

{
    "name": "Log",
    "hostname": "host",
    "pid": 4586,
    "level": 50,
    "req": {
        "meta": {
            "requestId": "1e7a7421-62f8-4ad2-9a32-a24020798489",
            "userId": null
        },
        "url": "/users/me/?include=roles",
        "method": "GET",
        "originalUrl": "/ghost/api/canary/admin/users/me/?include=roles",
        "params": {},
        "headers": {
            "host": "andreborud.com",
            "x-real-ip": "192.145.124.124",
            "x-forwarded-for": "192.145.124.124",
            "x-forwarded-proto": "https",
            "connection": "close",
            "sec-fetch-mode": "cors",
            "dnt": "1",
            "app-pragma": "no-cache",
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
            "content-type": "application/json; charset=UTF-8",
            "accept": "application/json, text/javascript, */*; q=0.01",
            "x-requested-with": "XMLHttpRequest",
            "x-ghost-version": "2.31",
            "sec-fetch-site": "same-origin",
            "referer": "https://andreborud.com/ghost/",
            "accept-encoding": "gzip, deflate, br",
            "accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
            "cookie": "**REDACTED**"
        },
        "query": {
            "include": "roles"
        }
    },
    "res": {
        "_headers": {
            "x-powered-by": "Express",
            "cache-control": "no-cache, private, no-store, must-revalidate, max-stale=0, post-check=0, pre-check=0",
            "content-type": "application/json; charset=utf-8",
            "content-length": "321",
            "etag": "W/\"141-MpHclrM9KZiRaDmHfy2VdplNIzo\"",
            "vary": "Accept-Encoding"
        },
        "statusCode": 403,
        "responseTime": "9ms"
    },
    "err": {
        "id": "a7a38f30-e8cb-11e9-91b9-f398e47b2304",
        "domain": "https://andreborud.com",
        "code": null,
        "name": "NoPermissionError",
        "statusCode": 403,
        "level": "normal",
        "message": "Authorization failed",
        "context": "\"Unable to determine the authenticated user or integration. Check that cookies are being passed through if using session authentication.\"",
        "stack": "NoPermissionError: Authorization failed\n    at new NoPermissionError (/home/user/andreborud.com/versions/2.31.1/node_modules/ghost-ignition/lib/errors/index.js:114:23)\n    at authorizeAdminApi (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/authorize.js:76:25)\n    at Layer.handle [as handle_request] (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/layer.js:95:5)\n    at next (/home/user/andreborud.com/versions/2.31.1/node_modules/express/lib/router/route.js:137:13)\n    at /home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/middleware.js:119:20\n    at /home/user/andreborud.com/versions/2.31.1/node_modules/express-session/index.js:495:7\n    at Child.SessionModel.findOne.then (/home/user/andreborud.com/versions/2.31.1/core/server/services/auth/session/store.js:24:28)\n    at Child.tryCatcher (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:517:31)\n    at Promise._settlePromise (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:574:18)\n    at Promise._settlePromise0 (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:619:10)\n    at Promise._settlePromises (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/promise.js:699:18)\n    at _drainQueueStep (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:138:12)\n    at _drainQueue (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:131:9)\n    at Async._drainQueues (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:147:5)\n    at Immediate.Async.drainQueues [as _onImmediate] (/home/user/andreborud.com/versions/2.31.1/node_modules/bluebird/js/release/async.js:17:14)"
    },
    "msg": "Authorization failed",
    "time": "2019-10-07T06:28:20.007Z",
    "v": 0
}

The NGINX error logs reports this:

upstream prematurely closed connection while reading response header from upstream, client: 192.145.124.124, server: andreborud.com, request: "POST /ghost/api/canary/admin/images/upload/ HTTP/2.0", upstream: "http://127.0.0.1:2371/ghost/api/canary/admin/images/upload/", host: "andreborud.com", referrer: "https://andreborud.com/ghost/"

I have also tried to temporarily disable SELinux, no difference.

[ivRodriguezCA]

Actually no. Yesterday I was publishing an article and had to ghost stop and ghost run, upload the image, then CTL + C and ghost start. Then manually copy the image from content/images/2019/10 to content/images/size/w2000 for it to work.

[andreborud]

Stopping ghost and then start it with run and then start works for me as well.

[ekerstein]

@andreborud Honestly, I haven't tried against since rolling back to 2.30.2. So I just tested it again now:

• Checked my nginx configuration and it was already at 50M. Also, the file I uploaded when I got a 502 error was only 1-2mb. So unless the config file is in another folder, I'm not sure that's the issue.

• The nginx config also doesn't explain why rolling back to a previous version would make it work again.

  I came across this issue related to the config server.port being 2369 while nginx was on 2368. I realized I had the same issue! So I did ghost config set server.port 2368, restarted, and updated to the latest version. My site loaded fine but when I went to try and access my posts it gave me a 500 error. I did ghost run and it gave me this:

InternalServerError: Cannot read property '0' of undefined
    at new GhostError (/var/www/ghost/versions/2.32.0/core/server/lib/common/errors.js:10:26)
    at _private.prepareError (/var/www/ghost/versions/2.32.0/core/server/web/shared/middlewares/error-handler.js:51:19)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:71:5)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at Layer.handle_error (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/layer.js:67:12)
    at trim_prefix (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:315:13)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:335:12)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:275:10)
    at /var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:635:15
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/index.js:260:14)
    at next (/var/www/ghost/versions/2.32.0/node_modules/express/lib/router/route.js:127:14)
    at apiImpl.then.then.catch (/var/www/ghost/versions/2.32.0/core/server/api/shared/http.js:97:17)
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:725:18)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:93:12)

TypeError: Cannot read property '0' of undefined
    at forEach (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relative-to-absolute.js:22:43)
    at Array.forEach (<anonymous>)
    at Object.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/utils/mobiledoc-relativ                        e-to-absolute.js:21:39)
    at UrlUtils.mobiledocRelativeToAbsolute (/var/www/ghost/versions/2.32.0/node_modules/@tryghost/url-utils/lib/index.js:387:22)
    at Object.forPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/url.js:34:36)
    at Object.mapPost (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/utils/mapper.js:35:9)
    at frame.response.posts.models.data.map.model (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/post                        s.js:15:56)
    at Array.map (<anonymous>)
    at Object.all (/var/www/ghost/versions/2.32.0/core/server/api/canary/utils/serializers/output/posts.js:15:36)
    at serializeOptionsShared (/var/www/ghost/versions/2.32.0/core/server/api/shared/serializers/handle.js:107:58)
    at /var/www/ghost/versions/2.32.0/core/server/lib/promise/sequence.js:10:31
    at tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Object.gotValue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:168:18)
    at Object.gotAccum (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/reduce.js:155:25)
    at Object.tryCatcher (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromiseCtx (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/promise.js:641:10)
    at _drainQueueStep (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:97:12)
    at _drainQueue (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/var/www/ghost/versions/2.32.0/node_modules/bluebird/js/release/async.js:15:14)

I rolled back to 2.30.2 again and everything is working. So now I'm not sure if the issue is related to the port 2368 that I changed or something else.

[ekerstein]

FYI - v.2.34.0 fixed my 500 error issue.

However, now my 502 error on image upload (not occuring during ghost run) is back :(

[ErisDS]

I have a feeling this is to do with sharp. I think maybe it doesn't always install properly, and when it doesn't, we're not falling back gracefully like we should.

In the case of a "recommended stack" setup, it should always install. @ekerstein can you look through your versions/xx/node_modules folders and see if sharp is in some or all of them?

We are looking into this, but it's slow work & very hard to track down because we haven't reproduced it yet.

Also can you run ghost doctor?

[andreborud]

Updated to 2.34.0 this morning and the problem persists.

And sharp is installed on this and previous versions.

[ErisDS]

What does ghost doctor say?

[andreborud]

Well I'm running ghost on CentOS so the doctor isn't to excited about that. You can see the output below.

$ ghost doctor
✔ Checking system Node.js version
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
System checks failed with message: 'Linux version is not Ubuntu 16 or 18'
Some features of Ghost-CLI may not work without additional configuration.
For local installs we recommend using `ghost install local` instead.
? Continue anyway? Yes
System stack check skipped
ℹ Checking operating system compatibility [skipped]
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_andreborud
? Sudo Password [hidden]
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability

However, I have been running ghost for a long time on CentOS without this problem, it popped up recently, not entirely certain what version introduced the bug. And stopping ghost and running it with ghost run instead make the image upload work again.

Is there any other information I can provide for you guys to make things easier?

[ekerstein]

@ErisDS Checked some of my versions folders. The version I kept rolling back to that worked (2.30.2) has sharp version 0.23.0. The latest version of Ghost has 0.23.1. Not sure if that means anything.. but I'm happy to provide more details if I can help.

[timskap]

I've got the same problem and image uploading works only with "ghost run". When I try use "ghost start" i've got error 502.

[andreborud]

This problem still happens in Ghost 3.0.0

[shanteshpatil]

This problem still happens in Ghost 3.0.0

I was really hoping this would be fixed in such a major version update. I am facing the same problem

Ghost CLI Version is 1.12.0

Hosted on a Digital Ocean droplet

[ekerstein]

I tried going into the current folder and downgrading sharp to 0.23.0 just out of curiosity (since that was the version on an earlier working ghost install). It was messy because some of the dependencies got messed up, but I was able to get the server started again. No luck though, same issue persisted.

[ekerstein]

Maybe discovered something? I decided to just add logging extensively throughout the image processing files.

In lib/image/manipulator.js I added logging to everything. Then I uploaded a file to see the normal process vs broken process.

Based on the logging, this is my understanding of what happens when you upload a file. This is using ghost run and works fine.

1. makeSafe
2. makeSafe require('sharp')
3. unsafeProcess
4. unsafeResizeImage
5. unsafeProcess return fs.writeFile

So then I tried it in production with ghost start. The server restarted in the makeSafe function at the same point every time. Right here:

const makeSafe = fn => (...args) => {
    common.logging.info('log 1');
    try {
        common.logging.info('log 2');
        require('sharp');
    } 
    ...

Every time it fails, "log 1" is logged, and then it never makes it to "log 2". It restarts immediately after "log 1" like this:

[2019-10-26 17:56:14] INFO log 1
[2019-10-26 17:56:20] INFO Ghost is running in production...
[2019-10-26 17:56:20] INFO Your site is now available on https://example.com/
[2019-10-26 17:56:20] INFO Ctrl+C to shut down
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 0
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 1
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] WARN Tries: 2
[2019-10-26 17:56:20] WARN Retrying...
[2019-10-26 17:56:20] WARN Can't connect to the bootstrap socket (localhost 8000) ECONNREFUSED
[2019-10-26 17:56:20] INFO Ghost boot 4.203s

I assume it was failing on require('sharp'). So I removed that line and it continued on to unsafeProcess and then unsafeResizeImage where it failed again on this line where it's loading sharp again.

const unsafeResizeImage = (originalBuffer, { width, height } = {}) => {
    const sharp = require('sharp');

Seeing as the issue is sharp, I tried reinstalling Ghost, downgrading sharp to 0.23.0, and 0.22.1, but no luck.

[1c7]

Ghost 3.0

Unable to upload image. version 2.3 work fine. I just update Ghost to 3.0 yesterday and this happen.

I didn't post log because I am not sure which log I should look at.
let me know which log I should copy&paste here~

[ErisDS]

@ekerstein Your debug info is very helpful but also baffling!

Is there anything in your journal or syslog at all?

Given that the second log doesn't even get output, that means the require is dying and causing the process to crash without an exception and before the log even gets sent to stdout - there's a try-catch block there on purpose, it's meant to catch any problems with requiring sharp, but clearly it isn't able to.

This points at a problem with the install of sharp's c-libraries.

Could you possibly change directory into /versions/[latest] and try 1) deleting the sharp directory, and then running npm install sharp just to see if there's any output?

I'm sorry to make you jump, but I've not been able to reproduce this issue. It's happening to many people so it's definitely an issue, but without reproduction we don't have a hope in hell at fixing it.

It's also almost certainly a problem with the sharp install process, but we don't even yet understand it enough to report it.

---

To anyone else - please don't just comment here saying "me too". We know it's a problem.

If you want to help, share your full system information (os, node version, npm version, node install method, ghost-cli version or ghost install method) and if possible ghost & system logs.

[ErisDS]

Oh P.S. @ekerstein can you also share what OS and version are you running on & what version of node and npm you are on? Is it still node 10.16?

[ErisDS]

FWIW I keep trying to reproduce this and I can't! I do however have a potentially useful debug process - it's tricky, but here we go:

Step 1. Modify ghost-cli to be super verbose:

which ghost will tell you roughly where ghost-cli is installed

E.g. on a DO droplet it should be /usr/local/bin/ghost.

We want to edit the file lib/tasks/yarn-install.js. The path to the actual files will be up 2 levels from which ghost, in lib/node_modules/ghost-cli.

So, if your which ghost gives you /usr/local/bin/ghost then you need to edit the file /usr/local/lib/node_modules/ghost-cli/lib/tasks/yarn-install.js.

Change line 65 from:

const observable = yarn(['install', '--no-emoji', '--no-progress'], {

to:

const observable = yarn(['install', '--no-emoji', '--no-progress', '--verbose'], {

Step 2. Rerun update

First figure out what version of Ghost you are currently on with ghost version. Then update to that exact version again, with the force and verbose flags:

e.g. ghost version -> 2.37.0

Then run ghost update v2.37.0 --force --verbose.

This is going to give TONNES of output from yarn, but thankfully the interesting bit is at the end.

You should see some output like:

node-pre-gyp info install unpacking node-v64-linux-x64/node_sqlite3.node
node-pre-gyp info tarball done parsing tarball
[sqlite3] Success: "/var/www/ghost/versions/2.37.0/node_modules/sqlite3/lib/binding/node-v64-linux-x64/node_sqlite3.node" is installed via remote
node-pre-gyp info ok
[16:06:22] → verbose 27.014 info sharp Using cached /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz
[16:06:23] → Done in 26.92s.
[16:06:23] Installing dependencies [completed]

See that line in the middle with verbose 27.014 info sharp? That's the bit we're looking for - some output from sharp.

Please follow these steps if you can and share any sharp-related output at the end of the yarn output.

---

Additional info: The output from sharp comes from this file, in my test case, this line which says it's reusing cached sharp.

I did these steps, then I deleted the path that it mentioned:

rm /home/ghost-mgr/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz

Re-ran the force upgrade, and this time I got a different line:

verbose 28.056 info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz

Image uploads continue to work fine.

[andreborud]

@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.

Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4 manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json to use library v8.7.4 instead of v8.8.1 and now it works again.

I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.

What OS is @everyoneElse using?

[timskap]

@andreborud I'm using Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-66-generic x86_64)
I Installed this version from Digital Ocean
Ghost-CLI version: 1.12.0
Node: v10.16.3

[andreborud]

So its probably not the OS then. I was using Node v10.16.3 until today. But just updating to Node v12.13.0 didn't solve it. But I am on DigitalOcean as well.

[timskap]

@andreborud Maybe DigitalOcean make some trouble?

[andreborud]

I don't see how. But could you try to downgrade to libvips and see if it works for you as well?

[timskap]

@andreborud I don't tried to downgrade but found intresting issue with VipsJpeg 8.8.1: https://github.com/libvips/libvips/issues/1372

[Pathel]

Don't mean to just add another +1 here, but I've got two blogs running affected by this. Both are on a (somewhat dated, but kept up to date via regular apt-get upgrades) ghost DO droplet.

Ghost-CLI version: 1.12.0
Ghost version: 3.0.0 (at /var/www/ghost)
Linux personal-blog 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux`

I've run through these instructions from above and oddly enough, the 502s persist despite getting nearly identical output. Initially, it had been using the cached version from ghost-mgr's npm cache, then after deleting it, it downloaded a new one from https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz.

Happy to try and share the results of any other ideas anyone has. :)

*Edit: * Downgrading libvips to 8.7.4 in sharp's package.json seemed to do the trick, as here. Guess I'll just keep doing that with every release for a while. 🤷‍♂

[johnpoelman]

@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.

Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4 manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json to use library v8.7.4 instead of v8.8.1 and now it works again.

I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.

What OS is @everyoneElse using?

I fixed this problem as described by @andreborud and it works now with ghost 3.0.2 on DIgital Ocean. Droplet runs on the ubuntu Ghost package from their market.

[timskap]

@ErisDS I've tried what you suggested. But I started with the last part, I had 4 versions of the libvips library. Deleted all and made a forced re-install. Got exactly the same output as you.
Then I thought maybe its a problem with the libvips library. So I re-downloaded v8.7.4 manually with wget. Then I modified ghostFolder/current/node_modules/sharp/package.json to use library v8.7.4 instead of v8.8.1 and now it works again.
I'm guessing version 8.8.1 has a problem with CentOS 7.x since its working for you.
What OS is @everyoneElse using?

I fixed this problem as described by @andreborud and it works now with ghost 3.0.2 on DIgital Ocean. Droplet runs on the ubuntu Ghost package from their market.

Can you explain step by step how you downgrade package?

[johnpoelman]

1. Download the library https://github.com/lovell/sharp-libvips/releases/download/v8.7.4/libvips-8.7.4-linux-x64.tar.gz

2. Place the tarball in the /home/ghost-mgr/.npm/_libvips/ folder (you can download it in here through wget)

3. Edit ghostFolder/current/node_modules/sharp/package.json file, change the current version to 8.7.4
4. Restart blog, I don't know if it's necessary but I did it

[lovell]

Hi, I'm the sharp maintainer, is this problem only occurring with Digital Ocean VMs / "droplets"? Please can someone provide full details of which type/size/location this occurs with, including the output of cat /proc/cpuinfo.

If anyone is able to use gdb node to get a backtrace of the crash that would be amazing. Here's an example of what to type (run this from the directory that contains node_modules):

$ gdb node
GNU gdb (Ubuntu 8.2.91.20190405-0ubuntu3) 8.2.91.20190405-git
Copyright (C) 2019 Free Software Foundation, Inc.
...

(gdb) run -e "require('sharp')"

Starting program: /usr/bin/node -e "require('sharp')"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
[Detaching after fork from child process 15193]
[Inferior 1 (process 15183) exited normally]

In this example there is no crash, but if there was you can type backtrace to get full details.

[Pathel]

I'll answer what I can.

Most of us seem to be running off the Ghost DO droplet (it's sort of a premade Ubuntu 18.04 image with mySQL, Nginx, et cetera), but seems like at least one CentOS box has the problem.

Here's my /proc/cpuinfo. That instance has 2GB of memory.

Will have a go at bumping libvips back to 8.8.1 and attaching gdb to node.

[colroyce]

@lovell Hi! I’m also having this issue on DO Droplet (Ubuntu 18.04.3 LTS / 1 GB memory). And here’s my /proc/cpuinfo.
Downgrading libvips library worked for me and solved the problem.

[lovell]

I created a droplet using https://marketplace.digitalocean.com/apps/ghost and all the latest versions seemed to be installed without error. I did notice a mixed use of root and ghost-mgr users, which I think is based on the way the Digital Ocean bootstrap script works (are these open source?).

In addition to a backtrace of the crash, please can someone provide a complete listing of the sharp directory from a machine on which the crash is occurring.

$ ls -alR /var/www/ghost/versions/3.0.2/node_modules/sharp

[andreborud]

@lovell thanks for trying to solve this problem. The output from cpuinfo, gdb node and ls -alR ended up being quite a lot. Instead of adding a long list here I made a .txt file you can view here: https://www.dropbox.com/s/g45li4af40i4jsf/sharp.txt?dl=0.

I'm on a 2 CPUs, 4GB RAM droplet running Centos 7.6.

Let me know if I can provide anything else.

[lovell]

Thank you, the output from gdb suggests that requiring sharp didn't crash. Does running the following commands crash or produce a list of versions?

$ cd /var/www/ghost/versions/3.0.2
$ node -e "console.log(require('sharp').versions)"

[andreborud]

Nope it does not:

$ node -e "console.log(require('sharp').versions)"
{
  cairo: '1.17.2',
  croco: '0.6.13',
  exif: '0.6.21',
  expat: '2.2.7',
  ffi: '3.2.1',
  fontconfig: '2.13.91',
  freetype: '2.10.1',
  fribidi: '1.0.5',
  gdkpixbuf: '2.36.12',
  gettext: '0.20.1',
  gif: '5.1.4',
  glib: '2.56.4',
  gsf: '1.14.46',
  harfbuzz: '2.5.3',
  jpeg: '2.0.2',
  lcms: '2.9',
  orc: '0.4.28',
  pango: '1.42.4',
  pixman: '0.38.4',
  png: '1.6.37',
  svg: '2.45.5',
  tiff: '4.0.10',
  vips: '8.8.1',
  webp: '1.0.3',
  xml: '2.9.9',
  zlib: '1.2.11'
}

[lovell]

Thanks, this suggests it's not sharp alone that's causing the problem but perhaps a conflict with another native module or shared library.

To help narrow down native module conflicts, what do the following return when run in the /var/www/ghost/current directory (when it is in a state that crashes)?

$ npm ls nan
$ npm ls -g nan

To help narrow down shared library conflicts, please can someone add the LD_DEBUG=files environment variable and provide the full log of ghost start until the crash.

[vweevers]

@lovell If it helps, here's a list of native addons in the dependencies of ghost master: [email protected], [email protected], [email protected], [email protected]. I used about-native-modules as a source of known native addons (includes more than nan dependents) (but in this case all 4 depend on [email protected]). Testing these addons against each other (injecting one addon into the process of another, running unit tests) didn't reveal any obvious conflicts (on Ubuntu 18.04, excluding dtrace-provider).

As another lead, for CentOS specifically, your prebuild might be built against a glibc that's too new? In leveldown we solved this by compiling the prebuilds on CentOS (instead of Travis' default Ubuntu).

[vweevers]

@lovell Another lead is that some versions of sharp can't be loaded side by side into the same process. Probably doesn't apply to this ghost issue (because there should only be one version of sharp) but who knows.

| | 0.23.2 | 0.23.1 | 0.23.0 | 0.22.1 |
| ---------| -------- | -------- | -------- | -------- |
| 0.23.2 | - | OK | OK | ERR |
| 0.23.1 | OK | - | OK | ERR |
| 0.23.0 | OK | OK | - | ERR |
| 0.22.1 | ERR | ERR | ERR | - |

[andreborud]

@lovell do you need any more info from us?

[lovell]

@andreborud Are you able to provide any of the details requested in https://github.com/TryGhost/Ghost/issues/11129#issuecomment-549082570 ?

[andreborud]

No problem

$ npm ls nan
[email protected] /home/architect/andreborud.com/versions/3.0.2
├─┬ [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └── [email protected]  deduped
├─┬ [email protected]
│ └── [email protected] 
└─┬ [email protected]
  └── [email protected]  deduped

and

$ npm ls -g nan
/usr/local/lib
└── (empty)

[andreborud]

Not sure how to do this part:

To help narrow down shared library conflicts, please can someone add the LD_DEBUG=files environment variable and provide the full log of ghost start until the crash.

[rushilsrivastava]

It seems that most people here are running on a DO droplet. I am running on my own Ubuntu server and after downgrading libvips to 8.7.4, I was able to upload images again.

[lovell]

Please can someone get a backtrace of the crash.

From your ghost installation directory, where you might normally run npm start, use the following steps instead:

$ sudo apt-get install gdb
$ gdb node

At the (gdb) prompt enter run index:

(gdb) run index
Starting program: /usr/bin/node index

Perform the actions you know will cause the process to crash, then enter backtrace at the (gdb) prompt:

(gdb) backtrace

[MaskyS]

I just hit this problem myself, everything was working, but trying to upload any images would give the server error/maintenance message. I followed https://ghost.org/faq/node-versions/ suggested by another github issue with the same error, then went through https://github.com/TryGhost/Ghost/issues/11129#issuecomment-548303143.

Upon restart, I got a 502 error while trying to access my site. I ran ghost status which revealed that my Ghost port had suddenly been changed to 2369 (probably from way before I followed the above, I just hadn't restarted Ghost since the update/port change). I set it back to 2368 using ghost config set server.port 2368, restarted everything, and image upload is working once again. For reference, I was on Ghost 3.0, updated to 3.0.3 while debugging. It's an Ubuntu DO droplet.

[andreborud]

@lovell so I've tried what you suggest with gdb. I'm running ghost on another port, 2371, if I do what you suggest and run gdb in my_ghost_installation/current where index.js is and change the default.json in core/server/config to use the same port (quicker than messing with nginx and setting up a new subdomain). This starts a new blog from scratch, so I have to make the initial setup again.

Did all that, and uploading an image kind of worked. Upload worked, but it didn't manage to display the preview.

Here is the output:

(gdb) run index
Starting program: /usr/local/bin/node index
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff6dcd700 (LWP 4532)]
[New Thread 0x7ffff65cc700 (LWP 4533)]
[New Thread 0x7ffff5dcb700 (LWP 4534)]
[New Thread 0x7ffff55ca700 (LWP 4535)]
[New Thread 0x7ffff4dc9700 (LWP 4536)]
[New Thread 0x7ffff7fea700 (LWP 4537)]
Detaching after fork from child process 4538.
[New Thread 0x7fffdffff700 (LWP 4539)]
[New Thread 0x7fffdf7fe700 (LWP 4540)]
[New Thread 0x7fffdeffd700 (LWP 4541)]
[New Thread 0x7fffde7fc700 (LWP 4542)]
[2019-11-11 14:20:25] INFO Ghost is running in development...
[2019-11-11 14:20:25] INFO Listening on: 127.0.0.1:2371
[2019-11-11 14:20:25] INFO Url configured as: http://localhost:2368/
[2019-11-11 14:20:25] INFO Ctrl+C to shut down
[2019-11-11 14:20:25] INFO Ghost boot 2.935s
[2019-11-11 14:20:41] ERROR "GET /ghost/api/v3/admin/posts/?limit=30&page=1&filter=status%3A%5Bdraft%2Cscheduled%2Cpublished%5D" 403 23ms

Authorization failed

"Unable to determine the authenticated user or integration. Check that cookies are being passed through if using session authentication."

Error ID:
    70d536f0-048e-11ea-8211-17b6ab584cd5

----------------------------------------

NoPermissionError: Authorization failed
    at new NoPermissionError (/home/architect/andreborud.com/versions/3.0.2/node_modules/ghost-ignition/lib/errors/index.js:114:23)
    at authorizeAdminApi (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/authorize.js:27:25)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:137:13)
    at /home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/session/middleware.js:119:20
    at session (/home/architect/andreborud.com/versions/3.0.2/node_modules/express-session/index.js:468:7)
    at getSession (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/session/middleware.js:48:12)
    at authenticate (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/session/middleware.js:106:5)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:137:13)
    at authenticate (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/api-key/admin.js:40:16)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:281:22
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/cors/lib/index.js:228:13
    at handleCORS (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/api/cors.js:71:16)
    at corsMiddleware (/home/architect/andreborud.com/versions/3.0.2/node_modules/cors/lib/index.js:204:7)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at Function.handle (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:174:3)
    at router (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:47:12)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at cacheControlHeaders (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/cache-control.js:32:9)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at checkVersionMatch (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/api/version-match.js:23:5)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at maintenance (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/maintenance.js:18:5)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express-query-boolean/index.js:6:5

(node:4528) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
[2019-11-11 14:20:48] INFO "GET /ghost/" 200 35ms
[2019-11-11 14:20:49] INFO "GET /ghost/assets/vendor.min-e80c50b3effa149734ab96b786349748.css" 200 10ms
[2019-11-11 14:20:49] INFO "GET /ghost/assets/ghost.min-c4e0057f46b262e4edcf840522559848.css" 200 37ms
[2019-11-11 14:20:49] INFO "GET /ghost/assets/ghost.min-b75322d062d69b67325425a7d04d4c28.js" 200 75ms
[2019-11-11 14:20:49] INFO "GET /ghost/assets/vendor.min-4fe157c12fff6582cb230b46e7c16b15.js" 200 219ms
[2019-11-11 14:20:50] INFO "GET /ghost/assets/img/medium-7359075af28d69523987ff4c0e2067c5.png" 200 2ms
[2019-11-11 14:20:50] INFO "GET /ghost/assets/img/favicon.ico" 200 3ms
[2019-11-11 14:20:50] ERROR "GET /ghost/api/v3/admin/users/me/?include=roles" 403 4ms

Authorization failed

"Unable to determine the authenticated user or integration. Check that cookies are being passed through if using session authentication."

Error ID:
    7675cc00-048e-11ea-8211-17b6ab584cd5

----------------------------------------

NoPermissionError: Authorization failed
    at new NoPermissionError (/home/architect/andreborud.com/versions/3.0.2/node_modules/ghost-ignition/lib/errors/index.js:114:23)
    at authorizeAdminApi (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/authorize.js:27:25)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:137:13)
    at /home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/session/middleware.js:119:20
    at session (/home/architect/andreborud.com/versions/3.0.2/node_modules/express-session/index.js:468:7)
    at getSession (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/session/middleware.js:48:12)
    at authenticate (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/session/middleware.js:106:5)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:137:13)
    at authenticate (/home/architect/andreborud.com/versions/3.0.2/core/server/services/auth/api-key/admin.js:40:16)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:281:22
    at param (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:354:14)
    at param (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:365:14)
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:410:3)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/cors/lib/index.js:228:13
    at handleCORS (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/api/cors.js:71:16)
    at corsMiddleware (/home/architect/andreborud.com/versions/3.0.2/node_modules/cors/lib/index.js:204:7)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at Function.handle (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:174:3)
    at router (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:47:12)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at cacheControlHeaders (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/cache-control.js:32:9)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at checkVersionMatch (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/api/version-match.js:23:5)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)
    at next (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:275:10)
    at maintenance (/home/architect/andreborud.com/versions/3.0.2/core/server/web/shared/middlewares/maintenance.js:18:5)
    at Layer.handle [as handle_request] (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:317:13)
    at /home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/home/architect/andreborud.com/versions/3.0.2/node_modules/express/lib/router/index.js:335:12)

[2019-11-11 14:20:50] INFO "GET /ghost/api/v3/admin/site/" 200 8ms
[2019-11-11 14:20:51] INFO "GET /ghost/api/v3/admin/authentication/setup/" 200 38ms
[2019-11-11 14:20:53] INFO "POST /ghost/api/v3/admin/session" 201 263ms
[2019-11-11 14:20:54] INFO "GET /ghost/api/v3/admin/config/" 200 15ms
[2019-11-11 14:20:54] INFO "GET /ghost/api/v3/admin/settings/?type=blog%2Ctheme%2Cprivate%2Cmembers" 200 73ms
[2019-11-11 14:20:54] INFO "GET /ghost/api/v3/admin/users/me/?include=roles" 200 82ms
[2019-11-11 14:20:54] INFO "GET /ghost/assets/ghost-dark-44a25cd4ab8c3c56721231196b853737.css" 200 22ms
[2019-11-11 14:20:54] INFO "GET /ghost/api/v3/admin/notifications/" 200 28ms
[2019-11-11 14:20:58] INFO "GET /ghost/api/v3/admin/posts/?limit=30&page=1&filter=status%3A%5Bdraft%2Cscheduled%2Cpublished%5D" 200 206ms
[2019-11-11 14:20:58] INFO "GET /ghost/api/v3/admin/tags/?limit=all" 200 44ms
[2019-11-11 14:20:59] INFO "GET /ghost/api/v3/admin/users/?limit=all&include=roles" 200 60ms
[2019-11-11 14:21:04] INFO "GET /ghost/api/v3/admin/themes/" 200 61ms
[2019-11-11 14:21:04] INFO "GET /ghost/api/v3/admin/tags/?limit=all" 200 77ms
[2019-11-11 14:21:04] INFO "GET /ghost/api/v3/admin/users/?limit=all&include=roles" 200 90ms
[New Thread 0x7fffc78aa700 (LWP 4543)]
[New Thread 0x7fffc70a9700 (LWP 4544)]
[New Thread 0x7fffc68a8700 (LWP 4545)]
[New Thread 0x7fffc60a7700 (LWP 4546)]
[New Thread 0x7fffc58a6700 (LWP 4547)]
[New Thread 0x7fffc50a5700 (LWP 4548)]
[Thread 0x7fffc50a5700 (LWP 4548) exited]
[Thread 0x7fffc58a6700 (LWP 4547) exited]
[Thread 0x7fffc68a8700 (LWP 4545) exited]
[Thread 0x7fffc60a7700 (LWP 4546) exited]
[Thread 0x7fffc70a9700 (LWP 4544) exited]
[Thread 0x7fffc78aa700 (LWP 4543) exited]
[2019-11-11 14:21:10] INFO "POST /ghost/api/v3/admin/images/upload/" 201 249ms
^C
Program received signal SIGINT, Interrupt.
0x00007ffff6eccac7 in epoll_pwait () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64
(gdb) backtrace
#0  0x00007ffff6eccac7 in epoll_pwait () from /lib64/libc.so.6
#1  0x00000000012e652f in uv__io_poll (loop=loop@entry=0x29f4500 <default_loop_struct>, timeout=1000) at ../deps/uv/src/unix/linux-core.c:284
#2  0x00000000012d4c5b in uv_run (loop=0x29f4500 <default_loop_struct>, mode=UV_RUN_DEFAULT) at ../deps/uv/src/unix/core.c:373
#3  0x0000000000a1afa7 in node::NodeMainInstance::Run() ()
#4  0x00000000009ab298 in node::Start(int, char**) ()
#5  0x00007ffff6df0505 in __libc_start_main () from /lib64/libc.so.6
#6  0x0000000000949035 in _start ()
(gdb) 

[lovell]

@andreborud Thank you, it look like you pressed Ctrl-C to end the process rather than experiencing a crashed process. Is this correct? Are you, or anyone else, able to get a backtrace of the process that crashes?

[andreborud]

@lovell as I wrote, I didn't experience the usual crash by using gdb. Eventually I had to press ctrl+c to come back to the command line to run backtrace.

[Maultasche]

I have a Digital Ocean droplet that I recently upgraded to 3.0.3 with this same issue, and I can confirm that the workaround posted by @johnpoelman worked for me. I can now upload images again.

[LanceMaverick]

The workaround fixed image uploading for me but does not fix the issue of Sharp not generating the image sizes, so my ghost site does not display most of the images. This is also on a Digital Ocean droplet and with upgrading to 3.0.3

[BlueHatbRit]

I've been having what seems to be the same issue, however I'm also getting 502's when loading up the front page or articles in the blog from most of the images it attempts to load. Using ghost run works totally fine but using ghost start causes the same issue.

I managed to fix this as you'll see below but thought my environment and outputs might be useful to help trace the issue.

I'm on a DO droplet as well so I'm going to guess it's likely the same issue. I've updated to 3.0.3 as the comment above suggests. I am not however using the pre-built DO ghost image. Here is my environment:

OS: Ubuntu 16.04.3 LT
Node version: 12.13.0
Ghost CLI version: 1.13.1
Ghost version: 3.0.3 (just updated from 3.0.0 to try and fix this, issue persists)

Ghost doctor output:

$ ghost doctor
✔ Checking system Node.js version
✔ Checking logged in user
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking operating system compatibility
✔ Checking for a MySQL installation
+ sudo systemctl is-active ghost_www-elliotblackburn-com
Instance is currently running
ℹ Validating config [skipped]
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies

The results of the debug steps by @ErisDS:

verbose 26.077586082 info sharp Using cached /home/elliot/.npm/_libvips/libvips-8.8.1-linux-x64.tar.gz

Deleting the cached version and forcing an update once again returned:

verbose 37.684216704 info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.8.1/libvips-8.8.1-linux-x64.tar.gz

This seemed to correct the issue on my blog and I'm no longer getting 502's on my home page / other articles.

Let me know if there's any other info I can provide to help debug this issue.

[shanteshpatil]

@BlueHatbRit When you say the issue was corrected on your blog were you only referring to the 502 errors or the image uploading as well

[Pathel]

I'm also curious if that fixed image uploading for you; I had tried that as well, but the only thing that did anything for me was downgrading sharp to 8.7.4.

[andreborud]

Please can someone get a backtrace of the crash.

From your ghost installation directory, where you might normally run npm start, use the following steps instead:

$ sudo apt-get install gdb
$ gdb node

At the (gdb) prompt enter run index:

(gdb) run index
Starting program: /usr/bin/node index

Perform the actions you know will cause the process to crash, then enter backtrace at the (gdb) prompt:

(gdb) backtrace

@shanteshpatil @Pathel @BlueHatbRit @LanceMaverick @Maultasche @rushilsrivastava @ekerstein guys, the workaround I suggested by downgrading the sharp library to 8.7.4 is not a fix of the problem and as @LanceMaverick writes it doesn't seem to generate all the correct sizes of the images. Plus you'll have to make this workaround every time ghost is updated.

As @lovell (who made the sharp library) writes in the quote above he needs more info to fix the problem. As all of you have shell access can you do the steps described above and send the crash to lovell so this can get a permanent fix, please?!?! I have tried but with my configuration it doesn't seem to work.

[rushilsrivastava]

Thanks for the update @andreborud. Ghost seems to launch in development mode which breaks all the urls on my site as they redirect over to localhost so I am not able to test this upload feature correctly. How were you able to avoid this?

[Pathel]

I tried a while ago but wasn't able to get anywhere with attaching gdb:

ghost-mgr@ghost-multiblog:/var/www/ghost/current$ gdb node
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from node...done.
(gdb) run index
Starting program: /usr/local/bin/node index
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff6c81700 (LWP 29769)]
[New Thread 0x7ffff6480700 (LWP 29770)]
[New Thread 0x7ffff5c7f700 (LWP 29771)]
[New Thread 0x7ffff547e700 (LWP 29772)]
[New Thread 0x7ffff4c7d700 (LWP 29773)]
[New Thread 0x7ffff7fe8700 (LWP 29774)]
[New Thread 0x7fffdffff700 (LWP 29781)]
[New Thread 0x7fffdf7fe700 (LWP 29782)]
[New Thread 0x7fffdeffd700 (LWP 29783)]
[New Thread 0x7fffde7fc700 (LWP 29784)]
[2019-11-20 08:01:09] INFO Creating table: posts
[2019-11-20 08:01:09] INFO Creating table: posts_meta
[2019-11-20 08:01:09] INFO Creating table: users
[2019-11-20 08:01:09] INFO Creating table: posts_authors
[2019-11-20 08:01:09] INFO Creating table: roles
[2019-11-20 08:01:09] INFO Creating table: roles_users
[2019-11-20 08:01:09] INFO Creating table: permissions
[2019-11-20 08:01:09] INFO Creating table: permissions_users
[2019-11-20 08:01:09] INFO Creating table: permissions_roles
[2019-11-20 08:01:09] INFO Creating table: permissions_apps
[2019-11-20 08:01:09] INFO Creating table: settings
[2019-11-20 08:01:09] INFO Creating table: tags
[2019-11-20 08:01:09] INFO Creating table: posts_tags
[2019-11-20 08:01:09] INFO Creating table: apps
[2019-11-20 08:01:09] INFO Creating table: app_settings
[2019-11-20 08:01:09] INFO Creating table: app_fields
[2019-11-20 08:01:09] INFO Creating table: invites
[2019-11-20 08:01:09] INFO Creating table: brute
[2019-11-20 08:01:09] INFO Creating table: webhooks
[2019-11-20 08:01:09] INFO Creating table: sessions
[2019-11-20 08:01:09] INFO Creating table: integrations
[2019-11-20 08:01:09] INFO Creating table: api_keys
[2019-11-20 08:01:09] INFO Creating table: mobiledoc_revisions
[2019-11-20 08:01:09] INFO Creating table: members
[2019-11-20 08:01:09] INFO Creating table: members_stripe_customers
[2019-11-20 08:01:09] INFO Creating table: members_stripe_customers_subscriptions
[2019-11-20 08:01:09] INFO Creating table: actions
[2019-11-20 08:01:09] INFO Model: Tag
[2019-11-20 08:01:09] INFO Model: Role
[2019-11-20 08:01:09] INFO Model: Permission
[2019-11-20 08:01:09] INFO Model: User
[2019-11-20 08:01:10] INFO Model: Post
[2019-11-20 08:01:10] INFO Model: Integration
[2019-11-20 08:01:10] INFO Relation: Role to Permission
[2019-11-20 08:01:10] INFO Relation: Post to Tag
[2019-11-20 08:01:10] INFO Relation: User to Role
[2019-11-20 08:01:13] ERROR (EADDRINUSE) Cannot start Ghost.

(EADDRINUSE) Cannot start Ghost.

"Port 2368 is already in use by another program."
"Is another Ghost instance already running?"

Error ID:
    ebc31ba0-0b6b-11ea-8aba-d9c3955286e2

----------------------------------------

InternalServerError: (EADDRINUSE) Cannot start Ghost.
    at new GhostError (/var/www/ghost/versions/3.0.3/core/server/lib/common/errors.js:10:26)
    at Server.<anonymous> (/var/www/ghost/versions/3.0.3/core/server/ghost-server.js:79:30)
    at Server.emit (events.js:210:5)
    at Server.EventEmitter.emit (domain.js:476:20)
    at emitErrorNT (net.js:1327:8)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)

[Thread 0x7ffff5c7f700 (LWP 29771) exited]
[Thread 0x7ffff4c7d700 (LWP 29773) exited]
[Thread 0x7ffff547e700 (LWP 29772) exited]
[Thread 0x7ffff6480700 (LWP 29770) exited]
[Thread 0x7fffdffff700 (LWP 29781) exited]
[Thread 0x7fffde7fc700 (LWP 29784) exited]
[Thread 0x7fffdeffd700 (LWP 29783) exited]
[Thread 0x7fffdf7fe700 (LWP 29782) exited]
[Thread 0x7ffff7fe8700 (LWP 29774) exited]
[Thread 0x7ffff7fe9780 (LWP 29765) exited]
[Inferior 1 (process 29765) exited with code 0377]

Taking down the site whose directory we're in doesn't seem to make it go away, and unfortunately this is my prod server so I can't bring everyone down to try with no ghost running at all. :

[andreborud]

@rushilsrivastava good point. thats what went wrong for me too.

@lovell its a bit problematic running gdb nodeand then run index as this defaults to the development environment and not our main instances. Then at least for me if I set the production environment variable, gdb complains about not getting root access to the db.

(gdb) set env NODE_ENV=production
(gdb) run index
Starting program: /bin/node index
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff6bc5700 (LWP 25728)]
[New Thread 0x7ffff63c4700 (LWP 25729)]
[New Thread 0x7ffff5bc3700 (LWP 25730)]
[New Thread 0x7ffff53c2700 (LWP 25731)]
[New Thread 0x7ffff7ff8700 (LWP 25732)]
[New Thread 0x7ffff4bc1700 (LWP 25733)]
[New Thread 0x7fffe7fff700 (LWP 25734)]
[New Thread 0x7fffe77fe700 (LWP 25735)]
[New Thread 0x7fffe6ffd700 (LWP 25736)]
[2019-11-20 08:19:18] ERROR ER_ACCESS_DENIED_ERROR: Access denied for user 'root'@'localhost' (using password: NO)

ER_ACCESS_DENIED_ERROR: Access denied for user 'root'@'localhost' (using password: NO)

"Unknown database error"

Error ID:
    500

Error Code: 
    ER_ACCESS_DENIED_ERROR

And the config file for the db is located in the parent directory of where index.js is. Is there a way of running gdb with ghost-cli instead as the problem happens when running ghost start and not with ghost run?

@rushilsrivastava maybe you have better luck adding the env?

(gdb) set env NODE_ENV=production
(gdb) run index

[rushilsrivastava]

@Pathel You need to shut down your running Ghost instance as it is currently running. ghost stop. Run through gdb from there.

[rushilsrivastava]

@andreborud I am having the same problem when moving to production version.

[rushilsrivastava]

@andreborud I have found a fix, move the configuration file to /current and then run:

(gdb) set env NODE_ENV=production
(gdb) run index

Should solve the problem, compiling the logs right now.

[rushilsrivastava]

I don't see the problem when running in gdb and on the latest Ghost version - unclear of what has changed. Maybe others have different results.

[andreborud]

@rushilsrivastava thanks moving the config worked for me as well, but like you I can not reproduce the problem.

This conversation is growing very long so let me try to summarize it.

Uploading images for some people on mainly digitalocean droplets can not upload images which results in a 502 and ghost say 'oops something went wrong :('.

Stopping ghost and instead starting it with ghost run (instead of the usual ghost start) makes photo upload work again.

Downgrading libvips 8.8.1 to 8.7.4 makes it work again, but as @LanceMaverick commented:

The workaround fixed image uploading for me but does not fix the issue of Sharp not generating the image sizes, so my ghost site does not display most of the images. This is also on a Digital Ocean droplet and with upgrading to 3.0.3

The only difference I can see between running ghost start and ghost run is the user. Running ghost start uses the ghost user while ghost run is the currently logged in user. Looking in my ghost/system/files/ghost_xxx.service file the user is set to ghost.
If I use the ghost user and run everything using gdb I can't produce the error either.

@ErisDS is there a way to start ghost using gdb to mimic as much as possible of ghost start?

[BlueHatbRit]

@shanteshpatil:
@BlueHatbRit When you say the issue was corrected on your blog were you only referring to the 502 errors or the image uploading as well

@Pathel
I'm also curious if that fixed image uploading for you; I had tried that as well, but the only thing that did anything for me was downgrading sharp to 8.7.4.

Sorry I should have specified, it has not fixed image uploading, just viewing the blog. Image uploading is still giving a 502, my main worry was to get the site back online again.

[1c7]

@johnpoelman Solution works for me!
2019-11-27

Ghost 3.0.3

Run on a $5 Digital Ocean droplet & using Ubuntu

[ErisDS]

@ErisDS is there a way to start ghost using gdb to mimic as much as possible of ghost start?

@acburdine or maybe @vikaspotluri123 can I ask one of you to answer this question? I am not clear on the differences between ghost start and ghost run that could be causing this, but it's clearly a problem for a lot of people 😞

[vikaspotluri123]

I am not clear on the differences between ghost start and ghost run that could be causing this

I'm kind of in the same boat ~

ghost run spawns the Ghost instance (node current/index.js) as the proper user and sets up IPC

ghost start runs doctor and tells the process manager to start the service

• The service's command to run is ghost run
• env, user, cwd, etc. are all configured at install time

As I was writing this, I can think of 2 things that might be the issue:

1. The environment - If I remember correctly, ghost run runs ghost in _development mode_. This issue can be tested by running NODE_ENV=production ghost run to ensure ghost runs in production mode
2. The user - ghost run runs ghost as (e.g.) ghost-mgr rather than ghost. This issue can be tested by running sudo -u ghost NODE_ENV=production ghost run to run Ghost in production mode as the ghost user

[jagracey]

Can confirm:
- migrating v2 to v3 fails, NODE_ENV=production ghost run* is a temp fix (blogs=3)
- fresh v3 install works fine (blogs=2)

We're running on small Digital Ocean droplets, using the marketplace Ghost app.

sudo -i -u ghost-mgr
cd /var/www/ghost
NODE_ENV=production ghost run

[vikaspotluri123]

@jagracey can you confirm if sudo -u ghost NODE_ENV=production ghost run works?

[BrnoPCmaniak]

So I had the same issue running ghost from alpine docker image https://hub.docker.com/_/ghost/, when I tried to upload a picture my nginx showed 502 and the container just died with no log or anything.

So I tried to do the gdb debug as @lovell proposed just with the change to attach to already running process. When I tried to do that it showed ptrace: Operation not permitted quick google search reveals this https://stackoverflow.com/a/46676868 that the docker restrictions prevent GDB to work properly and that I have to supply --cap-add=SYS_PTRACE --security-opt seccomp=unconfined in order to do debug. And weirdly the issue disappeard. After some more testing it was the seccomp=undefined option which fixed the issue.

Not sure if it helps the debug on non-dockerized environment.

I tried to do gdb from the host machine but that didn't yield much info because of docker. But strace did produce some usable output https://gist.github.com/BrnoPCmaniak/66068fbbcacff1a748b98fb59dcad826 the last thing it did was that it loaded sharp's libharfbuzz.so.0 and then it crashed.

TL;DR

I run ghost in docker:alpine after 3.0 version this issue arises and --security-opt seccomp=unconfined fixed it.

[lyf-coder]

So I had the same issue running ghost from alpine docker image https://hub.docker.com/_/ghost/, when I tried to upload a picture my nginx showed 502 and the container just died with no log or anything.

So I tried to do the gdb debug as @lovell proposed just with the change to attach to already running process. When I tried to do that it showed ptrace: Operation not permitted quick google search reveals this https://stackoverflow.com/a/46676868 that the docker restrictions prevent GDB to work properly and that I have to supply --cap-add=SYS_PTRACE --security-opt seccomp=unconfined in order to do debug. And weirdly the issue disappeard. After some more testing it was the seccomp=undefined option which fixed the issue.

Not sure if it helps the debug on non-dockerized environment.

I tried to do gdb from the host machine but that didn't yield much info because of docker. But strace did produce some usable output https://gist.github.com/BrnoPCmaniak/66068fbbcacff1a748b98fb59dcad826 the last thing it did was that it loaded sharp's libharfbuzz.so.0 and then it crashed.

TL;DR

I run ghost in docker:alpine after 3.0 version this issue arises and --security-opt seccomp=unconfined fixed it.

I run ghost in docker:ghost:3.1.0-alpine with --security-opt seccomp=unconfined,but not works, so I tried to use docker:ghost:3.1.0,it works...(don't need --security-opt seccomp=unconfined)

[jagracey]

@vikaspotluri123

@jagracey can you confirm if sudo -u ghost NODE_ENV=production ghost run works?

No, the above runs for v3.0.3, but image uploading does not work due to a bug with the sharp module. Module did not self-register.

I've been using NODE_ENV=production ghost run as ghost-mgr user.

[brinehart]

The problem with running NODE_ENV=production ghost run as ghost-mgr is that it'll only work so long as you have a terminal open. The moment you close your terminal ghost shuts down.

Surely there must be a better image uploading package that doesn't have all of these issues that can be used especially because the main way suggested for implementing Ghost on an outside server (via the pre-built Digital Ocean droplet) is the one that is broken.

This is a huge issue for a lot of us and desperately needs to be fixed.

[BlueHatbRit]

This certinaly isn't an ideal solution, but if you desperately need it to work with images right now, you could run NODE_ENV=production ghost run as ghost-mgr using tmux or gnu screen. You can then disconnect from those sessions and the ssh session and it'll stay running as-is until you connect to the session and stop it. That will work for you in the mean time.

If you are going to do this and haven't used a terminal multiplexer before, I'd recommend tmux, here are some resources to get going. It should a simple case of installing tmux, running tmux new-session -t ghost running your commands and then hitting Ctrl+D to disconnect from the session. You can then connect again with tmux a when you SSH back in later.

• Tmux
• Tmux crash course

Far from a great solution, but if ghost run is working then it's a fairly painless solution for the time being.

[cereallarceny]

I can confirm this happens on 3.1.1 as well.

[chrishawn]

someone mentioned it only happened on an upgrade not on a new install, can anyone else confirm that?

[praveenweb]

@chrishawn - I can confirm this happens on 3.1.1 too. I upgraded from v2 to v3 and it seems to happen since then. Also upgraded node version to be v10.16.0 and rebuilt dependencies.

Edit: @johnpoelman's solution works :)

[chrishawn]

ok to summarize, it seems the problem is being cause by the newer libvips in the sharp package not working correctly on some OS mainly shown in the thread to be CentOS and DigitalOcean's Ubuntu image

[brinehart]

I can confirm that is my situation. It is an upgraded instance.

[dilettanteCoder]

We’re running into the image upload problem as well after upgrading to 3.2.0. All was going well until the upgrade.

Running Ghost on Ubuntu 18.04.3 LTS on an AWS instance.

[nyneava]

Upgraded from 2.31.0 to 3.2.0 using "ghost upgrade" and having the same issue as everyone else, npm version 6.12.1 and [email protected] "ghost doctor" shows no sign of any issue.

I am able to upload SVG files but not PNGs or JPGs.

Everything works if using "ghost run" in lieu of "ghost start"

I'm using a tmux instance as a workaround for the time being. To add on to what @BlueHatbRit said, you can also write tmux new -s name then to detach press ctrl + b and then d, and to re-attach, tmux a -t name

I'll try to post a backtrace later.

[ekerstein]

FYI - in case anyone needs a quick way to implement @johnpoelman's solution after an update you can use this command. This assumes you already have libvips 8.7.4 installed.

sed -i 's/"libvips": "8.8.1"/"libvips": "8.7.4"/' /var/www/ghost/current/node_modules/sharp/package.json

[theverything]

| Key | Value |
|-----|------|
| OS | Ubuntu 18.04.2 |
| Node | 12.14.0 |
| NPM | 6.13.4 |
| Node install | https://github.com/tj/n |
| Ghost-CLI | 1.13.1 |
| Ghost | 3.2.0 |
| Ghost install | Digital Ocean marketplace |

This happened to me as well after upgrading to ghost v3. I can confirm that downgrading libvips to 8.7.4 fixed the problem.

[elyseholladay]

Confirming that @johnpoelman's downgrade of libvips to 8.7.4 worked for me also, on a DO Ghost droplet.

OS Ubuntu 18.04.2
Node 12.14.1
Ghost-CLI 1.13.1
Ghost 3.2.0

Thank you @johnpoelman and all for the thorough issue description and fix. 💖

[johnpoelman]

@elyseholladay thank you! I'm glad I could help. I must admit that for the time being aka "as long upgrading is buggy" I upgrade to every major version with reinstalling ghost and reimporting everything. I created a small roadbook for it and it took me less then 20 minutes to do it without the need for hacking of any kind :)

[darrenmhill]

Is there an actual fix coming for this in terms of a ghost or ghost-cli release? Please can someone summarise the final solution, if indeed there is one. Happy to make system level changes, but downgrading libraries doesn't seem ideal.

[andreborud]

@darrenmhill No idea if a fix is coming as this bug seems hard to reproduce. However, if you want a permanent fix you can install a new version of ghost. Not sure how you can do that with a Ghost droplet if thats what you have, but here is what I did on my CentOS droplet to keep the downtime to a minimum.

Warning, this worked for me and I won't take any responsibility if it doesn't work for you!!!

Create a new folder in whatever parent you have for your current ghost installation. If you ghost installation is in home/user/ghost then you could create folder home/user/ghost2.

user $ mkdir ghost2 && cd ghost2

Inside the new folder run ghost setupand install a new version of ghost.

When you come to url, set the same one as for your already running ghost blog. And for the database use the same host, username and password as your already installed ghost instance. Setup the systemd but don't start the blog yet. Next we will change the port ghost is running on. Open production.config.json with your favorit editor.

user $ vi production.config.json

Change the port and save. Your current ghost installation is probably running on port 2368, so you could use 2369 as an example.

Next, copy everything from ghost/content to ghost2/content. This command should do the trick.

ghost2 $ cp -r ../ghost/content ./

Next you want to make sure your ghost user owns the content.

ghost2 $ sudo chown -R ghost:ghost ./content

Just to make sure everything is fine, run ghost doctor and take any medicine the doctor prescribes. If all is good you should be able to start ghost now. ghost start.

If ghost is running its time to change your nginx or apache setup to point to the new port instead of the old one. I use nginx on CentOS which places my config files in /etc/nginx/conf.d/. If you are using apache or nginx on a different OS you'll have to find the the right config files.

ghost2 $ sudo vi /etc/nginx/conf.d/ghost.conf

Find localhost:2368 and change it to localhost:2369 and save. If you have content serving on you will also have to update the path of Ghost from ghost/ to ghost2/. Then let nginx test that the config file is fine.

ghost2 $ sudo nginx -t

If all is good restart nginx.

ghost2 $ sudo systemctl restart nginx

And DONE! If all went well you will have a down time of whatever time it took for nginx to restart. ~1-5 sec.

If you are running SELinux you will have to allow traffic to port 2369 which can be done like this:

ghost2 $ sudo semanage port --add --type http_port_t --proto tcp 2369

A fresh ghost install will fix the image upload problem, BUT DO NOT install ghost or later move ghost to the same folder as your broken ghost is installed in, then the problem comes back. I have tried and had to reinstall ghost 3 times before I got it working. @vikaspotluri123 this could be a clue to the problem, it seems to be linked to the installation folder and comes back even with a new install if its installed in the same location as the previously broken installation.

[johnpoelman]

@andreborud isn't it easier to download your images folder and exporting your content through the adminpanel? Then you can rebuild your DO droplet and reinstall ghost as you did the first time and after that import your content and reupload your images folder.

Also sometimes it is necessary to reset the folder's ownership back to the ghost user. I have written this procedure down in a dutch blogpost but it isn't published yet so I can't link it here just yet.

[andreborud]

@johnpoelman sure, you can do it that way. You'll have to remember your theme as well. The main point is that you need a new ghost installation in a different folder to fix the upload issue. So yes, you can absolutely do it that way as well, and import the exported content in your new installation, re install the theme and re-upload the images. If you are creating a new droplet its absolutely the way to go.

But doing everything through the terminal with no downloading could be quicker. ;) up to you!

[johnpoelman]

@andreborud i forgot the theme as I use casper :) . I did it as I described in under 5 minutes as with the rebuilding link on the DO administration page it just resets your droplet without creating a new one. I have misread your earlier post and missed that you were running centos and not the Ghost droplet from the market, my bad.

For everyone who is using the droplet from te market, you can try it my way and see if it is convenient. For file copying from the droplet to your computer and vice versa just use the scp command. For backing up and restoring your ghost content use Ghost itself.

Have a nice day!

[andreborud]

@johnpoelman sweet! And image uploading is working again?

[johnpoelman]

@andreborud fully functional :)

[kalinof]

Running ghost 3.4.0 on AWS EC2. I had the problem since ghost ver 2. Today I checked my nodeJs version. It was version 8. After updating to the latest version 10 everything seems to work again.

[brettdewoody]

Getting the same error on Ghost v3.0.1 and Node v10.13.0. 502 response from /ghost/api/v3/admin/images/upload/ and the logs show ghost is restarting when I try to upload an image.

From the above conversation it looks like installing [email protected] and updating sharp to use that version is one solution. There's an open PR for updating sharp from 0.23 > 0.24, but no mention in the PR about this issue. The changelog for the new sharp version (0.24) has this commit:

Upgrade to libvips v8.9.0-rc4, drop support for Node.js 8

but I'm unsure if that addresses the issue we're seeing here.

Happy to help debug in any way.

[tonilampela]

I came across this issue related to the config server.port being 2369 while nginx was on 2368. I realized I had the same issue! So I did ghost config set server.port 2368, restarted, and updated to the latest version.

Oh my god... I've been struggling with the 502 error with many recent 3.x updates and had to always roll back. This was my exact problem too. Something is messed up with the 3.x upgrades regarding nginx port setup.

[elyseholladay]

Update:

I created a new DO droplet (for unrelated reasons, moving my site to a subdomain) with:
Ubuntu 18.04 Ghost Droplet
Ghost-CLI version: 1.13.1
Ghost version: 3.5.1
NodeJS: 10.16.3

I ran into an issue where my permissions on my content folder was incorrect, but after I fixed that, I can upload files just fine, without the libvips/sharp package version fix. 🤷‍♀ If creating a new instance is doable for you, it seems to be fixed now.

[LanceMaverick]

I've tried every fix here, including downgrading libvips version used by sharp and installing a brand new ghost instance and copying over content and although it solves the 502 error on upload, responsive images are no longer generated. Respective folders and images are not created inside content/images/size/. Has anyone managed to fix this part of the issue?

[darrenmhill]

This is a major issue! I'm shocked that it has still not been fixed given the number of Ghost releases. Is this bug not considered high priority?

[vikaspotluri123]

@darrenmhill regardless of the importance of this issue, it's quite difficult to fix as we're not able to create a RCA with the updated version of libvps. In the meantime, many people have mentioned that migrating to a new droplet or downgrading libvps and upgrading sharp in your installation is a valid workaround.

[nickad]

I got the same image upload problem described above when I installed a fresh copy of Ghost today on CentOS 7 from the zip file: https://ghost.org/zip/ghost-latest.zip

The relevant versions were:

Ghost-CLI version: 1.13.1
Ghost version: 3.7.0
Node 12.14.1

I had been running Ghost via systemctl so, after reading the above posts, I tried running it via "ghost run". It started ok but when I tried to upload an image I got this error:

Something went wrong installing the "sharp" module

/lib64/libz.so.1: version ZLIB_1.2.9 not found (required by /my/path/versions/3.7.0/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)

The advice was to:

• Remove the node_modules/sharp directory, run _npm install_ and look for errors
• Consult the installation documentation at https://sharp.pixelplumbing.com/en/stable/install/
• Search for this error at https://github.com/lovell/sharp/issues

I noticed my CentOS 7 distribution is running zlib 1.2.7.
I removed the /my/path/versions/3.7.0/node_modules/sharp directory then reinstalled sharp via:

npm install sharp

Then I started Ghost via systemctl and the image upload worked fine.

Just thought I would share that in case it is of any help in resolving this.

[BeanBagKing]

Edit: It looks like it did work. However, I had to specify sharp v0.24, e.g. npm install [email protected]. The default npm install sharp installed v0.23.4

Having the same issue. Image uploads work fine under "ghost run", but fail when ghost is started normally.

I tried removing sharp from my current version (3.7.0), and then reinstalling via npm install sharp per the above instructions, but no luck. I actually had a few hours of screwing things up, force re-installing the current version, having dependency issues, fixing things, breaking them again. I can't begin to say exactly what I did, but none of it worked anyway.

Running Ubuntu 18.04.4 LTS in AWS. I'll be happy to provide any other information that might be useful, or try anything else that someone wants me to for troubleshooting. Just let me know what commands to run.

[tud0r]

This thread has been incredibly helpful thanks all who have contributed to the investigation of this! My set up too is a pre-installed Ghost set up on a DigitalOcean box. After reading through all this I tried the installation of a new ghost instance which ended up working.

I ran ghost install in a new ghost directory and backed up the old one.
The new installation required inputting details for a new mysql database, user, ssl etc etc. In order to get the new installation to work I had to go back and reconfigure the .conf file and map everything to my old database and also make sure all the permission were correct as mentioned in previous comments in this thread, before it finally worked.

[daniellockyer]

Hi all! Just to let you know that Ghost 3.9.0 has shipped with sharp 0.24.1. You might need to update with --force if you run into trouble with the binary dependencies.

[colroyce]

@daniellockyer Thank you! I had this 502 issue and avoided upgrading Ghost in case it breaks temporary solution that’s described here. I upgraded Ghost (it didn’t require --force) a few minutes ago, and it works now. Seems like image upload problem is no longer with us.

[darrenmhill]

Excellent news! Confirmed working fine for me now. I'd missed this being closed and didn't see anything in the main release notes, but very happy my problems are now solved. Great job all.

[ErisDS]

We weren't 100% sure the issue was fixed - it's been impossible to reproduce, but have updated the release notes now.

[darrenmhill]

It's fixed for me and I and my team massively appreciate it! Thank you :)

[karai17]

I'm getting this issue using the ghost:latest docker image as of a couple days ago. Is there no plans to actually push a real fix for this, or are users expected to manually upgrade and downgrade various libraries themselves?

[pascalandy]

I confirm this was fixed as well. Thanks!