Rocket.chat: RateLimiter and CPU consumption

Created on 12 Mar 2020 · 16Comments · Source: RocketChat/Rocket.Chat

Description:

We have a lot of RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method and RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method with different names such as stream-notify-room - most of all, stream-room-messages, UserPresence:away, stream-notify-logged, stream-notify-user.
It begin since upgrade to 3.0.2.

In that moments we see server CPU utilization is increases.
I think because there is a thousands of such messages in logs in very moment.

It's looks like some clients issue.
Like something in stuck on client side and that utilized server side.

It is often a mobile client (IP address of cellular network in logs)

Expected behavior:

Actually I don't know.
We need RateLimiter to be save of DDOS or something like that.
But RateLimiter itself now acting like flood in log messages.

Actual behavior:

Server Setup Information:

Version of Rocket.Chat Server: 3.0.2
Operating System: CentOS7
Deployment Method: docker
Number of Running Instances: 12
DB Replicaset Oplog: Enabled
NodeJS Version: v12.14.0
MongoDB Version: 4.0.13

Client Setup Information

Desktop App Version: 2.17.7
Operating System: Windows 10

Additional context

IP address in logs (213.87.150.76) is defiantly mobile client.

Relevant logs:

March 12th 2020, 07:01:16.032 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",
-- | -- | -- | --

  | March 12th 2020, 07:01:16.032 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:16.032 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:16.027 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:16.027 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:16.027 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:16.026 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:16.026 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:16.026 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:16.026 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 34998,

  | March 12th 2020, 07:01:16.026 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:16.025 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:16.025 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:16.025 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:16.020 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:16.020 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:16.020 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:16.019 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:16.019 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:16.019 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 34998,

  | March 12th 2020, 07:01:16.019 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:16.018 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:16.018 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:16.018 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:16.013 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:16.008 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:16.008 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:16.008 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:16.007 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35000,

  | March 12th 2020, 07:01:16.007 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:16.007 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:16.007 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:16.007 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:16.006 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:16.006 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:16.005 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:16.000 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:16.000 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:16.000 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:16.000 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.999 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35000,

  | March 12th 2020, 07:01:15.999 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.999 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.999 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.999 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.998 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.998 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.993 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.992 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.992 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.992 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.992 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.992 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.991 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.991 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.991 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.991 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35002,

  | March 12th 2020, 07:01:15.991 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.987 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.987 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.987 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.987 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.986 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.986 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.986 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.986 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.986 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.985 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.981 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.981 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.981 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.981 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35002,

  | March 12th 2020, 07:01:15.980 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.980 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.980 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.980 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.980 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.979 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.975 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.975 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35004,

  | March 12th 2020, 07:01:15.975 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.974 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.974 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.974 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.974 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.974 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.972 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.956 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.956 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.956 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.956 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.955 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.955 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.955 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.955 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35004,

  | March 12th 2020, 07:01:15.954 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.953 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.950 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.950 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.950 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.949 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.949 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.949 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35006,

  | March 12th 2020, 07:01:15.949 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.949 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.948 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.948 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.944 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.944 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.943 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.943 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.943 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.943 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.943 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.943 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.942 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35006,

  | March 12th 2020, 07:01:15.942 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.941 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.938 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.937 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.937 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.937 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.937 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.937 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.936 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.936 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.936 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.936 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.930 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35008,

  | March 12th 2020, 07:01:15.923 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.923 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.923 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.922 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.922 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.922 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.922 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.922 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.922 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.920 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.918 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.918 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.918 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.918 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35008,

  | March 12th 2020, 07:01:15.917 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.917 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.917 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-room",

  | March 12th 2020, 07:01:15.917 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.917 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.915 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.912 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.912 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.912 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.911 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.911 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35011,

  | March 12th 2020, 07:01:15.911 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.911 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.911 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.911 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.909 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.905 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.905 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.905 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.905 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.905 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-room",

  | March 12th 2020, 07:01:15.904 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.904 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.904 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.904 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35011,

  | March 12th 2020, 07:01:15.901 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.899 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.899 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-room",

  | March 12th 2020, 07:01:15.899 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.898 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.898 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.898 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35013,

  | March 12th 2020, 07:01:15.898 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.898 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.897 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.896 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.893 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.893 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.893 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.893 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.892 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.892 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.892 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.892 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.892 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-room",

  | March 12th 2020, 07:01:15.891 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35013,

  | March 12th 2020, 07:01:15.887 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.887 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.887 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.887 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.887 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.886 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.886 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.886 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.886 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.885 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.885 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.879 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.879 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.879 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.879 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.879 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35015,

  | March 12th 2020, 07:01:15.878 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.878 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.878 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.878 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.878 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.872 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.872 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.872 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.872 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35015,

  | March 12th 2020, 07:01:15.872 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.871 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.871 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.871 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.871 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.870 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.867 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.867 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.867 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.867 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.866 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.866 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35017,

  | March 12th 2020, 07:01:15.866 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.866 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.866 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.863 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.861 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.861 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.861 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.861 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.860 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.860 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.860 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.860 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.860 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35017,

  | March 12th 2020, 07:01:15.858 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.855 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.855 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.855 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.855 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.854 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35020,

  | March 12th 2020, 07:01:15.854 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.854 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.854 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.854 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.853 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.847 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.847 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.847 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.847 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.847 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.847 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.846 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.846 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.846 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.846 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.843 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35029,

  | March 12th 2020, 07:01:15.840 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.839 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.839 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.839 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.839 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.839 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.838 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.838 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.838 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.838 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.837 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.833 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.833 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35033,

  | March 12th 2020, 07:01:15.833 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.833 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.832 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.832 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.832 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.832 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.832 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.828 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.827 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.827 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.827 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35032,

  | March 12th 2020, 07:01:15.827 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.826 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.826 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.826 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.826 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.826 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.825 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.821 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.821 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.821 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.821 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35035,

  | March 12th 2020, 07:01:15.821 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.820 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.820 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.820 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.820 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.820 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.819 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.814 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.814 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.813 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35035,

  | March 12th 2020, 07:01:15.813 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.813 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.813 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.813 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.812 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.812 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.812 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.812 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.811 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.806 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.806 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.806 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.806 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.805 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.805 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.805 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.805 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35037,

  | March 12th 2020, 07:01:15.805 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.804 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.800 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.800 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.800 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.800 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.799 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.799 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.799 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35037,

  | March 12th 2020, 07:01:15.799 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.799 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.795 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.793 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.792 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.792 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.792 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.792 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.791 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.791 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.791 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.791 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.788 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35039,

  | March 12th 2020, 07:01:15.786 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.786 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.786 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.786 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.786 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.785 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.785 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.785 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-notify-user",

  | March 12th 2020, 07:01:15.784 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.781 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.778 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35039,

  | March 12th 2020, 07:01:15.778 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.777 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.777 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.777 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.776 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.776 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.776 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.776 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.775 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.769 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.769 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.768 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35041,

  | March 12th 2020, 07:01:15.768 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.768 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.768 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.767 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.767 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.767 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.767 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.766 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.760 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.760 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.760 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.760 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.759 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.759 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.759 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.759 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35041,

  | March 12th 2020, 07:01:15.759 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.758 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.758 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.758 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.752 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.752 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.752 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.752 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.751 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.751 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.751 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.751 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.751 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35043,

  | March 12th 2020, 07:01:15.750 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.748 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.744 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.744 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.744 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.744 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.743 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.743 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35043,

  | March 12th 2020, 07:01:15.743 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.743 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.743 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.742 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.742 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.737 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.737 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.737 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.736 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.736 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.736 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.736 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35045,

  | March 12th 2020, 07:01:15.736 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.736 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.735 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.733 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.731 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.731 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.730 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.730 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.730 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.730 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.730 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.729 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.729 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.729 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35045,

  | March 12th 2020, 07:01:15.725 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.725 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.725 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.724 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.724 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.724 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.724 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.724 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.724 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.723 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35047,

  | March 12th 2020, 07:01:15.720 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.719 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.719 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.719 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.719 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.719 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.718 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.718 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.718 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.718 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.716 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.714 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35047,

  | March 12th 2020, 07:01:15.714 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.714 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.714 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.714 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.714 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.713 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.713 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.713 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.712 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.711 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.709 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.709 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.709 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35049,

  | March 12th 2020, 07:01:15.709 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.708 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.708 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.708 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.708 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.707 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.707 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.702 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.701 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.701 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35049,

  | March 12th 2020, 07:01:15.701 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.701 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by userId per method

  | March 12th 2020, 07:01:15.701 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.701 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.700 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

  | March 12th 2020, 07:01:15.700 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.700 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "connectionId": "vadQFgBP6ExP622y5",

  | March 12th 2020, 07:01:15.699 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "name": "stream-room-messages",

  | March 12th 2020, 07:01:15.696 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "timeToReset": 35051,

  | March 12th 2020, 07:01:15.696 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "userId": "9xJjz3ZJZDTzqjjAf",

  | March 12th 2020, 07:01:15.696 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "numInvocationsLeft": 0,

  | March 12th 2020, 07:01:15.696 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "allowed": false,

  | March 12th 2020, 07:01:15.696 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "clientAddress": "213.87.150.76",

  | March 12th 2020, 07:01:15.696 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "type": "subscription",

  | March 12th 2020, 07:01:15.695 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | "broadcastAuth": false

  | March 12th 2020, 07:01:15.695 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info DDP RATE LIMIT: limit by connectionId per method

  | March 12th 2020, 07:01:15.695 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | RateLimiter ➔ info {

  | March 12th 2020, 07:01:15.695 | s-rc-srv-02 | rocketchat_rocketchat_2_1 | }

Source

ankar84

All 16 comments

Why it is discussion? It is a bug some where in RC logic or in client behavior.
Here is a example from chat - https://open.rocket.chat/channel/support?msg=jxt46hpi4uc8nQMjR
And here I temporary enabled all DDP rate limiters and set it to default values:
CPU

Network

@rsjr please remove discussion tag

ankar84 on 1 Apr 2020

👍1

@ankar84 I removed it, but discussion in this specific case was used for us to filter out issues that we need to discuss internally (as this is not a trivial correction neither trivial analysis).

Thanks for your understanding.

rsjr on 1 Apr 2020

Hi, @rsjr
Any update about that problem or about your internal discussion about that problem?
I'm afraid, that with DDP Rate limit disabled my servers are not protected from internal malicious users, who can somehow DDOS servers and I never now about that.
I'm afraid that #17310 could be result of such internal DDOS attack.
But when I try to enable DDP Rate limit I always get CPU high load for Rocket Chat servers.
I need a solution for that.

ankar84 on 17 Apr 2020

I enabled DDP Ratelimit after one reboot of one of our 3 servers and see, that method: loadMissedMessages did a 1.4 mil requests

Then I started to increase limits, and DDP calmed down.

Limits, that in current moment work for us

I will check on users peak today

ankar84 on 27 Apr 2020

👍1

Yesterday with DDP Rateimits like here was totally fine.
IMHO, average CPU load was even less then usual. Which is great!
But we still have a lot of log entries of DDP limits was executed.
@rsjr
I think it would be great to log only one event of many and other to ignore.
For example, user get DDP Ratelimit with 100 events overhead - now we see 100 events in logs.
I suggest in that case - only one (first) event writes to logs (so Admin can see that) and other are ignored. In that case log will not be overflooded by DDP Ratelimiter.
And that setting need to be configurable in Admin UI in DDP Ratelimits section.

ankar84 on 28 Apr 2020

👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] on 1 Jul 2020

Issue still in place.
We will try to upgrade next month and I will check DDP rate limiter and will write a result.

ankar84 on 1 Jul 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] on 7 Oct 2020

I'm on 3.6.3 now.
I'll try to test DDP Rate Limiter early morning...

ankar84 on 7 Oct 2020

3.6.3 looks much better in that problem perspective - one day with DDP Rate Limiter and without high CPU peaks - good.
Still watching.

ankar84 on 9 Oct 2020

The problem is that some RC code issues leads to DDP Rate Limiter fire up.
Here is an good example https://github.com/RocketChat/Rocket.Chat/issues/19007
Just open Federation Dashboard in Admin UI and get a lot of DDP Rate limiter events.

ankar84 on 22 Oct 2020

3.7.1 DDP work in general, but anyway can stop all RC stuff

ankar84 on 17 Nov 2020

@sampaiodiego can team implement some log message grouping to decrease amount of that type of messages, like host last message repeated <counter> times. I see this behavior in Linux system log for example. And it could decrease CPU usage a lot, because we have thousands of that same log messages sometimes.
I still can't enable DDP Rate limiter for a long time use, because it consumes CPU (major load by it loggings system, I think)

ankar84 on 20 Nov 2020

Hi @ankar84 .. I see what you're suggesting and we might implement that in the future, but for now I'd if you want less logs you should just lower the log level on admin settings.. Rate limiter logs are "info level", so you can definitely enable the rate limiter and lower the log level to errors only and you should be fine.

sampaiodiego on 20 Nov 2020

👍1

Hi @ankar84 .. I see what you're suggesting and we might implement that in the future, but for now I'd if you want less logs you should just lower the log level on admin settings.. Rate limiter logs are "info level", so you can definitely enable the rate limiter and lower the log level to errors only and you should be fine.

Thanks, Diego!
I set DDP Rate Limiter to default values and set logging level to 0 - Errors only
Will check that configuration for a while.
Thanks again for that advice!

ankar84 on 20 Nov 2020

@rsjr @sampaiodiego I think it's a pretty good solution of that issue for us.
Here is a 2 days graphs for DDP Rate Limiter, you can see that there was some executions, but there was no CPU peaks, all was nice and fluent.

So, here is solution for that problem: