Rocket.chat: [NEW] Server Session Monitor (SAU)

[renatobecker-zz]

Server Session Monitor for SAU(Simultaneously Active Users), DAU(Daily Active Users) and MAU(Monthly Active Users)

The main idea on this implementation is to provide a server session monitor(Meteor.server.sessions), which will be responsible for monitoring the life cycle of all connections on the server.
Through this monitor we will be able to store important informations about the sessions, such as sessionID, clientAddress , connection UserID, connection time, OS, platform, etc.

By default, the object returned when a new session is created contains the following fields:

{ 
  id: 'SzczLqirqXc3jFT6W',
  close: [Function: close], 
  onClose: [Function: onClose],
  clientAddress: '127.0.0.1',
  httpHeaders:
    { 
      'x-forwarded-for': '127.0.0.1',
      'x-forwarded-port': '3000',
      'x-forwarded-proto': 'ws',
      host: 'localhost:3000',
      'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
              'accept-language': 'pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6' 
    } 
}

Once the user-agent is provided, we can split the string on this field and populated specific fields on the model, like below:

{
    id: 'SzczLqirqXc3jFT6W',
    ...
    browser: { name: 'Chrome', version: '67.0.3396.99', major: '67' },
    os: { name: 'Mac OS', version: '10.13.6' },
    engine: { name: 'WebKit', version: '537.36' }
    ...
}

Implementation

To track the creation of the sessions, we can rely on Meteor.onConnection(callback), and to handle the session closure, we can rely on connection.onClose callback.
Aiming to identify the User of the session, we can update the session model after login using the Accounts.onLogin(callback) method(e.g.):

Accounts.onLogin(login => {
    const sessionID = login.connection.id;
    const userId = login._id;

    Session.update(sessionID, {
        $set: {
            userId,
            loginAt: new Date()
        }
    });
});

Obviously the connection.onClose callback will not be called if the server dies, so we need to define the best strategy to keep monitoring the session life cycle.
The initial idea is to map the Meteor.server.sessions and update the records in the database related to the current sessionID's. We can think on creating a specific array to store/remove the SessionID's according to the session life cycle, but I'm not sure how difficult it could be to update the records in the database using that array of sessionID's.
If we start by this approach, a time interval needs to be set to trigger the method in which the array of sessionID's will be read. If the array is large, maybe would be possible to separate it into batches of sessionID's.
Any suggestion on this implementation is very welcome.

Results

At the end of this implementation, we will be able to analyze the information regarding the activities of the sessions on the server.

Steps:

• [x] Design session model

• [x] Implementation

• [x] Tests(In Progress)

[geekgonecrazy]

@renatobecker when splitting the user agent might want to take in mind samples from iOS app and Android app. Because they send user agents with version information

@rafaelks do you have any samples of those?

[rafaelks]

@geekgonecrazy @renatobecker Yes!!! They look like this:

Information

1. RC Mobile is the default one to both platforms;
2. System & System Version;
3. App version and App Bundle;

iOS

RC Mobile; iOS 12.0; v3.0.0 (196)

Android

RC Mobile; Android 6.0; v2.5.0 (2750)

[renatobecker-zz]

@rafaelks and @geekgonecrazy,

I have already tested on mobile versions(iOS), I will handle these cases to store this information correctly.

Thanks.

[geekgonecrazy]

Awesome! I'm looking forward to this! This I think will be fantastic information

[sampaiodiego]

I would say to start off defining what exactly should be the end result. like defining the result's data structure. that way will be easier to define how to accomplish (or what to do) to get that result.

I guess we'll need to store all sessions/connections on DB and then at some point run a map reduce on them to extract the desired data on the defined time range.

[renatobecker-zz]

@sampaiodiego, the session documents look like these:

{
    "_id" : "XSM2orqtSPWGYoQT2",
    "year" : 2018,
    "month" : 7,
    "day" : 20,
    "sessionId" : "ySTQpjSifhFfXWDis",
    "userId" : "uk294ojDzhYFQK53P",
    "ip" : "127.0.0.1",
    "host" : "localhost:3000",
    "browser" : {
        "name" : "Chrome",
        "version" : "67.0.3396.99",
        "major" : "67"
    },
    "os" : {
        "name" : "Mac OS",
        "version" : "10.13.6"
    },  
    "createdAt" : ISODate("2018-07-20T17:06:32.480Z"),  //when the session is created
    "lastActivityAt" : ISODate("2018-07-20T17:10:01.166Z"), //when the session monitor updates current sessions in => Meteor.server.sessions
    "loginAt" : ISODate("2018-07-20T17:06:32.491Z"), //user login
    "logoutAt" : ISODate("2018-07-20T17:10:07.311Z") //user logout
}

{
    "_id" : "mX3QphXJfJhr7EdAw",
    "year" : 2018,
    "month" : 7,
    "day" : 20,
    "sessionId" : "vzm6vSMiCBwoeNfMs",
    "userId" : "uk294ojDzhYFQK53P",
    "ip" : "200.34.239.117,127.0.0.1",
    "host" : "e06f367a.ngrok.io",
    "os" : {
        "name" : "Android",
        "version" : "6.0"
    },
    "device" : {
        "type" : "mobile"
    },
    "createdAt" : ISODate("2018-07-20T17:06:32.480Z"),
    "loginAt" : ISODate("2018-07-20T17:06:32.491Z")
}

{
    "_id" : "C3Jzv8J8otgy43Jiw",
    "year" : 2018,
    "day" : 20,
    "month" : 7,
    "sessionId" : "rzw8zp6GNo3eMkNv7",
    "ip" : "127.0.0.1",
    "host" : "localhost:3000",
    "browser" : {
        "name" : "Chrome",
        "version" : "37.0.2062.120",
        "major" : "37"
    },
    "os" : {
        "name" : "Windows",
        "version" : "7"
    },
    "createdAt" : ISODate("2018-07-20T15:21:09.275Z"),
    "closeAt" : ISODate("2018-07-20T15:21:09.282Z"),
    "lastActivityAt" : ISODate("2018-07-20T15:57:29.182Z")
}

@rafaelks, I was thinking about the user-agent you're using for mobile apps, do you know if we have a list that identifies all of Rocket.Chat's applications?
I mean, we could create a field in the session document to store the appName, which would be passed by user-agent like you are doing.(e.g):

• Mobile Apps = { app: 'RC Mobile' }
• Livechat = { app: 'RC Livechat'}

What do you guys think about it?

[rafaelks]

@renatobecker I think that could work, yes! 👍

Will there also be a way to differentiate between iOS and Android?

[renatobecker-zz]

@rafaelks yes!
If you see the second document, it has a field os:

...
"os" : {
    "name" : "Android",
    "version" : "6.0"
}
...

But we will need to improve the user-agent string to get all informations correctly, I guess..

[rafaelks]

@renatobecker Nice! The User-Agent already have all the OS information and the app version.

[geekgonecrazy]

Under device could we maybe include the version like done in the browser? I think will be important to know.

Or maybe just drop the raw useragent in just in case it ever contains a tiny bit more info?

[renatobecker-zz]

Hi @rafaelks and @geekgonecrazy!

The mobile session documents will look like the doc below:

{
    "_id" : "YLoJ68zjA6WfNFt3b",
    "day" : 26,
    "instanceId" : "LPvDxJh8fyDy6QsYz",
    "month" : 7,
    "sessionId" : "wTrw5SNHa7wfudxaz",
    "year" : 2018,
    "ip" : "200.34.239.117,127.0.0.1",
    "host" : "d16877f6.ngrok.io",
    "os" : {
        "name" : "iOS",
        "version" : "11.4.1"
    },
    "device" : {
        "type" : "mobile"
    },
    "app" : {
        "name" : "RC Mobile",
        "version" : "v3.0.2",
        "bundle" : "(202)"
    },
    "createdAt" : ISODate("2018-07-26T18:57:49.475Z"),
    "userId" : "uk294ojDzhYFQK53P",
    "loginAt" : ISODate("2018-07-26T18:57:49.812Z"),
        "lastActivityAt" : ISODate("2018-07-26T18:58:03.834Z"),
    "closedAt" : ISODate("2018-07-26T18:58:03.834Z")
}

What do you guys think?

[rafaelks]

@renatobecker Looking great to me! Just one thing: can you remove the v. and the () from the bundle and version? Thanks, excited for this!

[geekgonecrazy]

Looks awesome to me!

[renatobecker]

The Server Session Monitor(SAU) has been delivered a long time ago, so I'm closing this issue.