Rocket.chat: rate limiter using wrong IP

[idarlund]

Description:

RateLimiter is using clientaddress instead of x-forward-for when behind loadbalancer/reverse proxy.

Steps to reproduce:

1. Set up rocketchat behind a reverse proxy (with x-forward-for configured on the reverse proxy)
2. Bruteforce login page
3. Look at journald log

Expected behavior:

Rate limiter should have used "x-forward-for" instead of "clientAddress".

Actual behavior:

Rate limiteris using "clientAddress" instead of "x-forward-for".

Server Setup Information:

• Version of Rocket.Chat Server: 1364
• Operating System: Fedora 30
• Deployment Method: snap
• Number of Running Instances: 1

Additional context

I'm not a js developer, but the bug might be here:
https://github.com/RocketChat/Rocket.Chat/blob/develop/app/lib/server/startup/rateLimiter.js#L154

It seems like it will always use clientAddress and not “x-forwarded-for” such as this example here: https://github.com/RocketChat/Rocket.Chat/blob/develop/app/api/server/api.js#L235

Relevant logs:

Logs from systemd (journalctl -f -u snap.rocketchat-server.rocketchat-server):

DDP RATE LIMIT: limit by connectionId per method
{
  "allowed": false,
  "timeToReset": 2359,
  "numInvocationsLeft": 0,
  "userId": null,
  "clientAddress": "10.5.15.1",
  "type": "method",
  "name": "login",
  "connectionId": "xx",
  "broadcastAuth": false
}

tcpdump from rocketchat server (which show the forwarding of header variables are being done). I did the connection trough tor so I didn't have to mask the IPs (tcpdump -s 0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'):

Host: chat.domain.tld
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
Accept: image/webp,image/apng,image/*,*/*;q=0.8
Referer: https://chat.domain.tld/home
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,en-US;q=0.8,nb;q=0.7,sv;q=0.6,da;q=0.5
Cookie: connect.sid=s%ss
X-Forwarded-Proto: https
X-Client-IP: 87.118.116.12
X-Real-IP: 87.118.116.12
x-forwarded-for: 87.118.116.12
X-Forwarded-For: 87.118.116.12

Please let me know if you want some more information or want me to test patches or anything else.

[wreiske]

Please keep in mind that it is possible (in some cases) to spoof the x-forwarded-for address.

https://remotalks.blogspot.com/2013/12/spoofing-http-header-field-x-forwarded.html

Just wanted to throw that in there for whoever may fix this or who may be relying on x-forwarded-for to always be accurate.

[idarlund]

You can write (spoof) whatever you like in the header, not just x-forwarded-for.

The fix for this bug should probably be configurable; if the server is behind a reverse proxy then accept x-forwarded-for, if not then don't.

If you spoof x-forwarded-for in front of the reverse proxy, the reverse proxy will handle that according to your reverse proxy config.

[geekgonecrazy]

@rodrigok this one I think I’ve seen effecting several people. Every case has involved a reverse proxy in front

[Sing-Li]

@gabrielengel confirmed - this one currently affecting our partner out in the field. Using a reverse proxy.

[geekgonecrazy]

Lots of false positives here I see them in our own logs too. Rate limiter is getting hit way more often than it should. I can only imagine this leading to a worse experience with actions taking multiple tries in some cases when load gets heavy

[idarlund]

I suspect that this issue: https://github.com/RocketChat/Rocket.Chat/issues/15017 might be linked to the issue we're talking about here

[rodrigok]

Hey.

Unfortunately we do not control the clientAddress variable since it is passed by meteor here

As you can see here the clientAddress is computed here and it takes an environment variable in count.

So, to get the correct IP the sys admin needs to define the HTTP_FORWARDED_COUNT env var to a value of the number of reverse proxies in front of it.

We may add a new setting to be able to configure this value via the admin console.

[rodrigok]

Please take in count the comment in the code describing a security reason to not try to get the ip directly from x-forwarded-for:

    // Typically the first value in the `x-forwarded-for` header is
    // the original IP address of the client connecting to the first
    // proxy.  However, the end user can easily spoof the header, in
    // which case the first value(s) will be the fake IP address from
    // the user pretending to be a proxy reporting the original IP
    // address value.  By counting HTTP_FORWARDED_COUNT back from the
    // end of the list, we ensure that we get the IP address being
    // reported by *our* first proxy.

[idarlund]

Thanks! I was unaware of the custom environment variables.

For the issue's sake and google foo, I'll write down here how I got it working:
Found a documentation here: https://rocket.chat/docs/installation/manual-installation/ubuntu/snaps/#how-do-i-change-other-environmental-variables-in-my-snap and added HTTP_FORWARDED_COUNT to my environment:

cat /var/snap/rocketchat-server/common/my.env
HTTP_FORWARDED_COUNT=1

That solved the issue with clientAddress. It's now reporting the correct address.

[reetp]

Does this affect #15017 ?

[idarlund]

Does this affect #15017 ?

Just try to add HTTP_FORWARDED_COUNT=1 to /var/snap/rocketchat-server/common/my.env (if you're using snap - if not put it in the correct path) to see if it works.

[idarlund]

I also proposed an updated documentation so that people don't forget to use the correct env variable when setting up reverse proxy: https://github.com/RocketChat/docs/pull/1370

[reetp]

Mine are all docker so I guess it will need adding to compose (there are probably a lot of docs to update)