Rocket.chat: Admin ability to view history for all channels, direct messages and private channels

[gt50]

Your Rocket.Chat version: 0.26.0

I would like to be able to view all conversations that have occurred on the RocketChat server. I would like to use it in an work environment. There is no expectation of privacy and the information would be accessed when needed for auditing purposes.

I imagine this either being added to the history area, or allowing an admin to view a room, or add themselves to a room from the rooms area in adminstration.

[edman545]

I agree, there is some expectation from upper management that all electronic communication is being logged somewhere. Even if if it just dumped every message sent to a log file somewhere, that would be enough for our purposes. It is very rare that I have to dig into such information, but sometimes there is good reason.

[tholu]

Could be implemented in the admin rooms view (where all direct messages are already listed as well), when clicking on the room to show the logs of it. Or in a separate audit view.

[kdahmani]

I also think that from a compliance point of view it would be very important to be able to access all chats on the server

[heytimc]

Definitely need logging for compliance here. Anything would be great; text, syslog, d/b, don't care how. We won't be able to go live until this is available.

[dereckhall]

same requirement for our office as well. currently utilizing openfire jabber for interoffice communication w/ their archive plugin. something similar or anything really would be great.

[thelilmatt]

+1 for this. It's the only thing holding us back from implementing RocketChat

[dupree972]

I would love this functionality too...

On Jun 10, 2016, at 3:36 PM, thelilmatt [email protected] wrote:

+1 for this. It's the only thing holding us back from implementing RocketChat

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/RocketChat/Rocket.Chat/issues/2869#issuecomment-225288931, or mute the thread https://github.com/notifications/unsubscribe/AOVtMEqR3suEpU_gECJHTbf_otfHbdZdks5qKcrGgaJpZM4IGrw8.

[widget7h15]

Rocketchat is a fantastic platform but I would really like to see this feature implemented so we can use this in our organization. Pretty much every other chat style program has at least one way to audit conversations for HR purposes.

[dupree972]

Is there any progress or has anyone even found a work around for this. We are a SEC and CFTC regulated entity and our compliance officer has the need to randomly review a certain small percentage of messages per month. Also we have the need to archive messages. It does not need to be fancy, even if they all dropped into a text file or something that could be ingested somewhere like a smarsh or other archive that would be terrific.

[slowbiz]

This is also functionality we will need before moving to production in our environment.

[heytimc]

Yay! Merged to develop :)

[slowbiz]

I see the Smarsh Connector in the latest build, which is great. Is there any plan to offer a built-in log/archive/history page to audit conversations?

[patit72]

Here too, we need that function before moving to production

[megabit-computing]

This functionality is vital before we can consider using rocket Chat hosted version!

[rajsardhara]

+1

[Nepoxx]

You can already do this by connecting to your Mongo database.

List the rooms with db.rocketchat_room.find({}), grab the ID of the room you want the history for.

Then you can see that room's history with db.rocketchat_message.find({"rid": "your_chatroom_id"}, {"u.username": 1, "msg": 1, "_id": 0}).sort({ts: 1}).map(function (d) {return d.u.username + ": " + d.msg})

It's not as intuitive as a web interface, but auditing your users' chat logs should seldom be done therefore this is a viable workaround in the meantime.

This topic attracts a great deal of controversy, as it should. That being said, this method is a workaround until there is a corresponding UI. This also means that you are already being recorded when chatting using RocketChat since your messages are saved in the database unencrypted (unless you use OTR). If you allow your users to edit/delete their messages, they are indeed modified/deleted from the database, so take that into account.

[dereckhall]

@Nepoxx For the majority of users here this is a business requirement.

Also, your method doesn't take into account that users can still edit their original message, which modifies the original within mongo.

[philippevk]

@dereckhall He just gave us a workaround. You can disagree about the spying part (I don't), but where I'm from when someone help you, you say "thank you"

@Nepoxx thanks!

[xandershelley]

If this feature is enabled it should be made clear the chat is being recorded. RocketChat is already being used for private chat servers as well.

[patit72]

It might be required for forensic audit, and its use should be logged too. For sure it is a feature needed in business environments

[christopherturner]

Are there still plans to integrate this into the UI?

[sebastientaggart]

+1

[devact]

To have history for audit is one of the major mandatory business requirements for any company. So +1 for an easy UI interface. 10x!

[Nemra1]

we need that for admins only..we dont know what bad planes happend in direct rooms

[evaldoprestes]

+1

[mauriciovirtual]

+1

[anhnt]

-1

[Tallestthomas]

Has there been anyone working on this? We love rocketchat at my company, but can't move forward with using it until theres some sort of logging in place. I've started to build an external logger using the api, but it would be great if there was an in app solution to this.

[valentincollado]

+1

[timiil]

+1!!!

[Darkestlyrics]

All Messages are logged in the logs as debug events.

You will have to change you logging level from Error only to Error, Information and Debug.

This will then allow you to get the following from the logs

I20180227-08:59:05.806(0) rocketchat_logger.js:278 Integrations ➔ Outgoing WebHook.debug Got the event arguments for the event: sendMessage { event: 'sendMessage', message: { _id: 'eXePYf2wfXgs46k2J', rid: 'bWQH3mKraFHwcyMNvrJY49bWrYTtd67m5E', msg: 'test', ts: 2018-02-27T08:59:05.780Z, u: { _id: 'bWQH3mKraFHwcyMNv', username: 'USER1', name: 'USER1' }, mentions: [], channels: [], _updatedAt: 2018-02-27T08:59:05.782Z, sandstormSessionId: null }, room: { _id: 'bWQH3mKraFHwcyMNvrJY49bWrYTtd67m5E', _updatedAt: 2018-02-27T08:57:51.741Z, msgs: 7, t: 'd', ts: 2018-02-26T10:13:17.126Z, meta: { revision: 7, created: 1519639997131, version: 0, updated: 1519721871744 }, '$loki': 15, username: 'USER1', lm: 2018-02-27T08:57:51.730Z, usernames: [ 'USER1', 'USER2' ] } }

[leonardoap]

+1 !!!

[paulomatheusgrr]

Do we have any news about this?

I can´t install rocket chat in my business while this don't works!!

[sadkins76]

Tallestthomas,
Did you make any progress with the external logger? If so, would you mind sharing? I need this badly....

[vynmera]

It's probably fair to (if this does get implemented in a way where this is possible) add a warning to private channels and DMs that admins can read these chats (especially for public instances).

As it is right now, DMs could have a warning if the Read other IMs option is enabled in server settings.

I believe transparency is key with these things!

[gt50]

Last year I created a python script that I could run to get the data that I needed. I just uploaded it to a gist. Enjoy!

https://gist.github.com/gt50/82005184d9a030cbe6e2ea91a4ae39b7

[victor-freire]

After trying to get some room's histories I was thinking: is it possible to use the REST API to read the messages? I was trying to get the users message as an admin and I couldn't because I didn't have the user's credentials to login into the API.

[vynmera]

@dotsomething You _can_ get DM messages, yes. You can use https://rocket.chat/docs/developer-guides/rest-api/im/messages-others/ together with https://rocket.chat/docs/developer-guides/rest-api/im/list-everyone/ for this.
Simply use im.list.everyone and read the DMs with im.messages.others.
To use this, you'll need to turn on the Enable Direct Message History Endpoint setting!

[Nepoxx]

@dotsomething Do you have access to the mongo database? If so you can query it to see all the messages you need. See my comment above which contains a sample query.

https://github.com/RocketChat/Rocket.Chat/issues/2869#issuecomment-280046547

[victor-freire]

@vynmera I have tried that method.

My scenario is:

I'm logging in into my admin account and retrieving my authToken and using the im.list.everyone. But am I supposed to do the following?

            method: 'GET',
            url: URL + "/api/v1/im.list.everyone",
            headers: {
                "X-Auth-Token": 'my token',
                "X-User-Id": 'the user's ID I wanna get the conversations from',
                "Content-Type": "application/json"
            }

Because it returns this:

{ status: "error", message: "You must be logged in to do this." }

@Nepoxx Yes, I do have access to the database. But I was going to build a static page as a web interface to audit in case something wrong happens at the company. Now I'm going to take a look at the solution @vynmera gave me and perhaps use Nodejs with mongodb.

Thank you, guys!

[vynmera]

@dotsomething You run im.list.everyone as yourself, then find the IM you want to look into. (Hint: the _id field for a DM consists of the 2 user IDs that are talking)
Once you have a _id, use im.messages.others with that id, again as yourself.

[victor-freire]

@vynmera Thank you, dude! Going to work on that and if it turns out to be a good piece of software I'll provide it here as a repo. 😉

[engelgabriel]

Here are the screenshots of what we are developing.

And each time the admin looks into conversations, it will leave a log that can be searched to autiding the auditor :)

Since this a sensitive feature, we will be releasing it as one of our first Rocket.Chat Apps available via our marketplace next month.

[rajsardhara]

Sensitive Feature 🤔 🤔

[engelgabriel]

I forgot to say, if you want to be updated when it is available and are interested on being beta testers, please send an email to [email protected]

[engelgabriel]

@rajsardhara sorry for the short "Sensitive Feature", let me elaborate.

We always thought that since all the msgs were available on the DB, that this would be already enough for most use cases. But, as this thread here shows, it was clear not the case and so we proposed that we were going to build that feature in the admin panel.

Then we received a lot of feedback from the community explaining that they specifically don't want a feature like that to be available on the admin panel by default, making far to easy for admins to "spy" on users' conversations.

Thats why we considered it a "Sensitive Feature", because the more we looked into it, the more complicated it got.

We realised that to make it simpler, we had to take it out of the main project, and deal with it as an extra plugin (now called Apps) so only the admins of deployments that wanted that feature would install it.

We also realised that this is a very enterprise focused functionality and we believe that this can be one of the ways to monetise the project and we are studding a few licensing models for this app, and we would love to hear your thoughts about it.

[sadkins76]

I LOVE the way this is looking. Not sure how I feel about the monetization of an app to add functionality that should be part of the program to begin with. I really have no issue paying for the functionality, but please, for the love of all that is holy, make it a flat rate fee. Not another subscription? PLEASE?

I don't understand the issue people have with an ADMIN seeing conversations, especially in a corporate environment. There are far more valuable things that an admin has access to than a chat conversation log to worry about. Looks amazing though guys! Count me in as a tester when you make it available!

[vynmera]

Looking very good, count me in!
On the topic of monetization: it's not a bad idea at all. However, we should always be wary of greed, and make sure we don't restrict access to basic features (unlike Mattermost, in which a basic permission system is already paid). Furthermore, we should inform users beforehand on the website what features are paid and what aren't!

@sadkins76 Well, I suppose the idea is that some people use Rocket.Chat for public services - having your conversations recorded and delivered to the admin on a silver platter may feel dirty to users. I'd be one of them, surely. Of course, an admin can get all this data already if they desire, but that takes extra effort and would break the trust between the user and the admin. A simple warning would allow the admin to look at the DMs, yet keep the user's trust (and also get a nicer UI to read the messages with).

[engelgabriel]

And also for clarity, ALL the messages are stored on the DB, so they can be searched and viewed by the sysadmins and DBAs.

Regarding the edited msgs, there is a setting in the admin panel that when turned on, will save all version of the messages on the DB also.

@sadkins76 in a corporate environment most people agree with you, but on other environments, people seems to disagree :)

[engelgabriel]

@vynmera indeed the topic of monetisation is a delicate one, but very important for the continuity of the project, so we will get the community input on this to make sure we are aligned.

[sadkins76]

@engelgabriel I understand the concerns in other environments, but Rocket.Chat has Slack square in its crosshairs...and is billed as a replacement for it.
I cannot wait to be able to have this functionality...I do understand the implications of using it for Public chat spaces and the problems that come along with it.

[gt50]

When I created this issue, I never realized that it would be such a divisive subject among some users. This sort of functionality has been in place for many messaging applications that could be self-hosted for longer than I have been involved with technology (20+ years).

Regarding public servers, someone owns the server and is paying for it. If it is not the admin, the admin has been appointed by the owner. Why should users assume that the admin will not act in the best interest of the organization? Any self-hosted database backend can be accessed by an admin if given enough time. Users are naive if they think otherwise. The access will come without friendly messages warning them of what happened.

Releasing it as a paid option is a reasonable compromise.

Before I created this issue, I had spent time evaluation mattermost and rocket.chat. The only advantage that mattermost had to rocket.chat was that it used an RDBMS. I have used most of the RDBMSs in existence. MongoDB was a scary thing for me, but I am glad that I chose rocket.chat over mattermost.

[engelgabriel]

Thanks you for your support guys, we are looking forward to have you to test the Audit App and we will give you Early Adopters licences :)

[victor-freire]

As I promised I made a poor workaround for my company. But keep in mind that I've never worked with web development before.

Repo

It just lists users Direct Messages and you need to be an Admin to Log-in. Sorry if it's in Portuguese, I didn't have time to translate it to you guys.

Ps: You need to change this line with your Rocket.Chat URL. And if you're a newbie as I am, you need to serve it as a HTTP server.

[leonardorocc0]

I think that an opersource project should guarantee privacy among their users.
Do you want this feature implemented for your company? Well, pay for development :)

[gt50]

If your goal is an FOSS project with guaranteed privacy then you may want to try Signal by Open Whisper. For anyone paranoid enough to be concerned with privacy on messaging apps, anything less than end to end encryption is not acceptable.

[leonardorocc0]

Don't get me wrong, I'm not saying rocket chat should be the safest app in the world, but I think that company features should be a low priority, we as an open source community should prioritize features useful for NGOs, open source communities, etc.
As for example, I'm part of a demoscene group which releases open source productions and can't afford paying slack for the staff.

[sadkins76]

You cannot offer a product as a replacement for Slack and the like, and not offer BASIC features that an enterprise REQUIRES. It’s that simple. There is no expectation of privacy in a corporate environment.

[paulomatheusgrr]

when will we have a beta to test the app?

[piotrkochan]

Since this a sensitive feature, we will be releasing it as one of our first Rocket.Chat Apps available via our marketplace next month.

Ok, but where is Rocket Chat Apps Marketplace?

[ghost]

When is this coming out?

[keithhenning]

Any update? Not yet in marketplace.

[FaizanZahid]

Our thing would love this thing to be in some form of UI

[guilhermecfviana]

+1 I need this

[guilhermecfviana]

When it's released, can I participate in the tests? I already have the rocket chat in the company and as you are going to launch this I have abhorred the development of a tool for it to pick up from the REST API.

[Cetrana]

Do we have this app yet?

[almacedo68]

+1

[suyashdixit]

+1

[suyashdixit]

+1

[FaizanZahid]

i need this too, it will be a game changer +1

[rmundel]

+1

[necil2201]

Have there been any recent updates on this functionality?

This is the one hurdle preventing my company from fully migrating to Rocket Chat and kicking Slack to the curb.

[sasha2002]

TWO years of +1 and no attention.... yeah serious company....
Disappointed!

[engelgabriel]

@sasha2002 @necil2201 @rmetzler @FaizanZahid and other, please send your details to [email protected] and our team will get in contact to talk to you regarding the History Audit app.

[piotrkochan]

Why The heck not on The public github issue page?
W dniu śr., 26.12.2018 o 21:24 Gabriel Engel [e-mail]
napisał(a):

@sasha2002 https://github.com/sasha2002 @necil2201
https://github.com/necil2201 @rmetzler https://github.com/rmetzler
@FaizanZahid https://github.com/FaizanZahid and other, please send your
details to [email protected] and our team will get in contact to talk
to you regarding the History Audit app.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/RocketChat/Rocket.Chat/issues/2869#issuecomment-450020728,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFXy-dJVsCGdMaiXnR-2Zvuw_sxEAbGOks5u89sGgaJpZM4IGrw8
.

[ellisgeek]

Any news on when this will be available?

[sasha2002]

@engelgabriel WE choose to switch to another software because of multiple bugz and feature that is not present, and near future of this features is also unclear!
To all users, think twice before install!

[vynmera]

@sasha2002 Sorry to hear that. Where official support is lacking, I still very much enjoy using the APIs and JS to integrate my own code. I feel that's the power of RC.

[faz1993]

@vynmera can i test it ??

[spivey76]

Any news on this front? I'm currently testing several solutions and this is by far the most difficult to pin down.

[necil2201]

Everyone,

A new feature appeared on the Pricing page for the Enterprise version called "Message Auditing Panel". I think this puts the nail in the coffin of ever getting a dedicated Marketplace App for compliance auditing purposes.

This is particularly sad because I would have gladly paid for a dedicated app but I'm not going to pay thousands of dollars for a bunch of other features I don't want or need.

If your company works in an industry that requires compliance auditing you would do well to look elsewhere.

[rmundel]

What about the community join forces and develop an alternative plugin anyway?

[bkraul]

Yup...I don't think Rocket.Chat is actively saying their auditing panel is the end all be all. I also believe that the Pricing page is not the final word, you can talk to them and I am sure they can tailor services as well. I also do not think they would be far from rethinking their position should enough demand for the feature in self-hosted environments arise.

[paulomatheusgrr]

I was more than 2 years waiting to receive this bad news.
That makes me sad. I will have to remove the RC from my company for lack of this possibility, does anyone know another open source Software for it?

[graywolf336]

We will be eventually moving the Enterprise feature of Auditing over to an App in our interface. We are just lacking the support inside the App system to actually do it. It is on the long road map. But if you need it now, we have the Enterprise edition.

[Sing-Li]

Definitely don't wait. Even if we do actually eventually move it to apps, it will always be an Enterprise feature. Contact sales if you need this urgently.

[bkraul]

@Nemra1, I'm sorry, what is the point of your comment? You are directing us to a platform whose 2 main touted features are: "end-to-end encryption" and "de-centralized". Can you help us understand how this solves the issue of auditing, especially in terms of business auditing requirements?

[ghost]

Hello! I am an employee. I read this topic and it explains a lot to me. I left my companie's rocketchat never to return and I want te let you know that reading private messages is really really wrong. No matter how you justify your compliance, if you, as HR department read your employee's private messages, you are a cunt. If an employee is living a situation that need intervention, he can give you a copy of that conversation.

(ok, I took that part out... no need for this, you are right)

[bkraul]

@Emaelqc what exactly is your point? A company's rocketchat server is the property of the company. Contractually they have every right to audit for very real legal reasons. An employee is always free to either use OTR or not engage on personal conversations outside of what pertains to business matters. You really don't seem to understand business auditing.

Also, rather unhelpful attitude.

[ghost]

You are right. But we are in 2020, and I win, not you. Treat your employees like your clients. Thats what my point is. The company cannot afford losing me at all, and I am not using the chat anymore. So what is your point? The purpose of the chat is not reached if employees do not use it. You have no respect.

Sit down and take notes.

[Nepoxx]

You have no respect.

I mean, you just called people names...

Contractually they have every right to audit for very real legal reasons.

@bkraul The laws are not the same everywhere. Some countries allow employees to have a reasonable expectation of privacy, so these are really muddy waters...

@Emaelqc Arrête d'agir comme un enfant. C'est vrai que c'est cave de vouloir lire les messages privés de ses employés, mais tes commentaires n'aident pas.

[ghost]

Je suis d'accord que c'est plutôt une discussion pour LinkedIn que pour GitHub, et à cet effet, j'admets ne pas être au bon endroit.

(I agree that this is more of a discussion for LinkedIn than for GitHub, and as such, I admit I'm in the wrong place.)

[bkraul]

@Nepoxx I understand the laws are not the same. I speak for US law. And again, it is all on how the chat server is deployed. Obviously if I deploy a chat in my company, contractually granting employees an assurance of expectation of privacy and I breach such agreement, my company could be in deep waters. It's all about the disclosure.

[ghost]

So let's make something else clear : I will never use this product again. You can count on your right hand people who do what I do in Canada, so unfortunatly for employers... Anyway, I will drop this. I just had the need to express my feelings about such a functionality.

Have a good life,

[Darkestlyrics]

let's say around 75% of the messaging platforms you use will have a way for the messages to be retrieved unless you self host them. Are you going to stop using each and every single one of those?

Also, did you create a github profile just to complain about this?

[rmundel]

No one is irreplaceable. You're not the client. Just don't use it.

[ghost]

Naa I had a git hub profile for a while, email was not activated. Anyway, you are also right, everybody can be replaced, but that is not entirely true in the current state of events... lol. Anyway. like I said... im out ;P both from here and rocketchat.