Rocket.chat: Uploading picture gives 403 sporadically

[lunitic]

Your Rocket.Chat version: (make sure you are running the latest) 0,36

Very often when someone uploads a clipboard files it gets failed a 403 is showed in the console log.

If the person do a refresh crtl+r of page then the image is showed!
Using GridFS.

This is very very annoying. Running the RC as 1 instance at the moment without any nginx loadbalancing.
However the nginx sits infront of the RC but only has 1 server upstream. Previously it had 3 upstream instances. But changed it to 1 to rule out session issues.
Anyone else have the same experience.

[AlecTroemel]

The issue could be coming from this line of code here https://github.com/AlecTroemel/Rocket.Chat/blob/develop/lib/fileUpload.coffee#L56

Which could be because rocketchat is unable to get the rawCookies (or its missing the uid or token). Maybe something with nginx

[lunitic]

@AlecTroemel this is my config on the nginx i got the config from @Sing-Li
Any NGINX pro that could look at it??

`# Upstreams

upstream backend {
server 127.0.0.1:3000;
server 127.0.0.1:3001;
server 127.0.0.1:3002;
}

server {
listen 443;
server_name chat.example.com;
error_log /var/log/nginx/example.access.log;

    ssl on;
    ssl_certificate /etc/nginx/certificate.crt;
    ssl_certificate_key /etc/nginx/certificate.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 ;

    error_page 497 https://chat.example.com;

location / {
    proxy_pass http://backend/;
    #proxy_pass http://192.168.102.2:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forward-Proto http;
    proxy_set_header X-Nginx-Proxy true;
    proxy_redirect off;
}

}

server {
listen 80;
server_name www.example.com;
error_log /var/log/nginx/www.example.access.log;
#return 301 https://chat.example.com:81;

location / {
    proxy_pass http://192.168.102.7/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forward-Proto http;
    proxy_set_header X-Nginx-Proxy true;
    proxy_redirect off;

}

}

server {
listen 80;
server_name chat.example.com;
error_log /var/log/nginx/example.access.log;
return 301 https://chat.example.com;
}
`

[lunitic]

And my config is identical with

https://rocket.chat/docs/installation/manual-installation/configuring-ssl-reverse-proxy-with-nginx

[lunitic]

This problem occur on 2 RC installations haven't anyone else seen it?

[lunitic]

Just to add this is windows 10 64bit clipboard

On @Sing-Li suggestion i tested to use S3 instead. Behaves the same

[lunitic]

Not only clipboard files gives this result.. Also happend from regular (a png) pictures.

[lunitic]

This driving me nuts..

Just tested to disable firewall IDS system infront of the chat and the ngingx..
The RC instances now sits directly on the internet nothing in between
Still get this 403 now and then when i scroll back in messages.

This system was a 0.20 from the beginning and been updated.

[pierreozoux]

One of my user is hitting the same issue on his instance, let me know if I can help debug.

[engelgabriel]

We are updating the GridFS packages, can you guys please retry with the develop branch when https://github.com/RocketChat/Rocket.Chat/issues/4235 gets closed?

[pierreozoux]

@engelgabriel Thanks, but I can't test easily until it hits a release. Just let me know when it is released, and then I let know my user.

(Or do you have a develop docker image?)

[engelgabriel]

@pierreozoux and @lunitic can you please try with version 0.40.1 ?

[lunitic]

@engelgabriel just gonna update our live server to 0.39 then we will migrate to 0.40.1 on our lab server.

[lunitic]

@engelgabriel finally had time to upgrade lab server and it still exists in 0.43

[lunitic]

This weekend I will replace the Sophos firewall in front of the instance. It's the last thing I can come up with

[lunitic]

Just notice that his happend on demo.rocket.chat too

[lunitic]

[rodrigok]

Any way to reproduce? I can't :(

[engelgabriel]

@lunitic @AlecTroemel @pierreozoux Is this still happening?

[pierreozoux]

@engelgabriel I confirm that it is fixed, thanks (and sorry for delay)

[lunitic]

Sorry i saw the comment now.. I am at 0.43 and there it still exists.
Have not been able to upgrade to any later revision yet.. (waiting for a stable release)
@engelgabriel @rodrigok

[garychapman]

Definitely not fixed. I can easily replicate it just by logging out and in again - refreshing the browser (while in the channel where the image is posted) fixes it for the duration of that particular login session.

I did some digging in packages/rocketchat-file-upload/server/lib/requests.js and it looks like it's picking up the rc_token value from the previous login. As soon as you refresh, you get a new rc_token (i.e. one that hashes correctly to the current hashedToken in the user record) and then the findOneByIdAndLoginToken() call succeeds.

[bbrauns]

I can confirm this bug:
docker image: rocket.chat:0.52
Ubuntu 14.04.5 LTS
haproxy:1.7 as reverse proxy

[lunitic]

Started to dig in this issue again. 0.55.1 is still having it.. (our current server) It seems that the Rocket.Chat client makes it worse.

[Khuman2]

Have the same issue with 0.74.3