Node-slack-sdk: API/Bot Client High Memory Usage

[kurisubrooks]

This bug's been around from quite a long while and i'm honestly surprised as to why no one else has picked up on it yet. I originally launched the issue through Slack's support, but it's apparent that it hasn't gotten through to the dev team.

Using the given example in the readme, produces a continual increase in memory usage with no other modules and minimal code. Let me know if you require any more details, but both of my 2 slack bots have had this issue, as well as 2 of my friends.

Edit: We've been able to reproduce this on 3 different systems, with 5 seperate programs, by 3 different people.

[evansolomon]

Can you explain this a bit more? In particular I'd be interested in how you measured memory usage and how quickly it increased. But really any information on what you did/what happened/what you expected would probably help make sure the problem you're describing is correctly reproducible.

Fwiw I tried a quick test with the code in the readme plus this:

startedAt = Date.now()
dumpMemoryUsage = () ->
  console.log Math.round((Date.now() - startedAt) / 1000), process.memoryUsage()

setInterval dumpMemoryUsage, 1000

My total heap size actually decreased after a few minutes ¯_(ツ)_/¯

[kurisubrooks]

To measure memory usage i was using PM2.
Via their status command, here are the results i've collected over time.

13m    │ 45.734 MB
102m   │ 55.215 MB
2h     │ 60.563 MB 
2h     │ 64.199 MB

Last time I had this running it seemed to cap out around 80-100mb after a couple hours.

Over a period of about 2 months (the time i've had my bots), it's generally been the same. some variation in memory use, but, this is for an absolute minimum program. Which, is kind of silly.

[kurisubrooks]

In the 17 minutes that have passed since I sent the last message, the memory continues to go up:

2h     │ 67.805 MB
2h     │ 71.906 MB

[evansolomon]

If it caps out, I don't think think it's quite a "leak" in the usual sense. Whether 80-100mb is too much is hard to say. On one hand, sure it's just a web socket and some event listeners so what the hell is it doing with all those bytes. On the other hand, running a node program that is _just_ the piece of code I pasted above consumes something like 20mb of memory, which seems equally silly.

In my experience, node/v8 does not try particularly hard to be memory efficient. There are often good reasons for this, too. For example, in many use cases, more gc pauses are a worse problem than "wasting" memory. I am not familiar enough with the internals, but in my observations node often seems happy to burn a few tens of megabytes of memory usage in small programs running on systems with with plentiful memory available.

So I guess it's not totally clear to me that there's a memory leak here. Does that make sense?

[kurisubrooks]

I understand. I'm going to leave my program open over night and check the memory usage in the morning. I still don't quite understand why the memory usage goes up over time for something that I wouldn't expect to.

[evansolomon]

Cool.

Not to give you more work or anything, but an interesting comparison might be to leave open a program that's just connected to a websocket or something like that. There's a nice demo server provided by websocket.org (https://www.websocket.org/echo.html) that will respond with whatever you send it. A program that maintained a connection and periodically sent/received some data could be a useful baseline since that's sort of equivalent to what this library does. Just a thought.

[kurisubrooks]

Hello. Just ran the command to check memory usage and it seems to have stuck around the 80mb mark, as I predicted before.

17h    │ 79.211 MB

The issue might not be a memory leak, but rather just a very large memory memory footprint. Let me know what you think.

[evansolomon]

I don't really have any specific thoughts myself. Whether 80 mb is very/too large seems like a matter of taste. If you have gigs of free memory I'd be inclined to say no big deal. If you're on a Raspberry Pi or something then maybe it's an important issue.

Do you have some ideas to reduce memory usage? Going out and trying to find optimizations just to reduce this number doesn't seem very valuable to me unless there's some problem it's causing. Just my 2 cents.

[kurisubrooks]

I host on a very cheap vps, which only has about the same amount of ram as a raspberry pi. Memory usage is quite important for something with as low memory as this.

I've read through the code but I don't see anything obvious that would cause such high memory, this might be a problem with websocket or one of the dependencies that the client runs on, who knows.

I think i'll change the title of the issue and leave it open to see if anyone else can provide some input.

[dudarino]

80MB? that's adorable. My guess is this client just caches everything and why invalidate cache because memory is an infinite resource? What's going on here, i have only one channel and it's not terribly busy. A dozen users, tops. what's going on here?

http://i.imgur.com/0Ns8tnt.png

[kurisubrooks]

Your memory usage would be from the Chromium Wrapper that Slack runs on, and not the API itself. I think your issue is seperate. This repo is for the Node.js implementation of the Slack API.

[ghost]

Ok, first off, I apologize for how long it's taken us to resolve this issue.

I just pushed the 2.0.0 release for this library, which is a ground up refactor with the aim of building a lib that both Slack and our external contributors can easily work with and improve.

For this particular issue, you should be able to create a client without attaching a memory data store. That way your bot will simply receive messages, but not attempt to cache the data from the team attached to it, which should reduce the overall memory footprint.

So, I'd suggest that the best way to resolve these issues is to upgrade to the latest version. If you have any problems in doing that, please create a new issue and I'll try and help you in there.

[Brain2000]

Version 2 is still running over 400MB with 6 processes. I don't know how to "create a client without attaching a memory data store."

The only preference that looked like it might be similar is "Disable Local Storage" under advanced. However, Local Storage indicates the local hard drive, not RAM. So this can't be it.

[Brain2000]

Actually, version 2 is taking up over 500MB.

[ghost]

The default client doesn't create a memory data store, so likely something else is going on.

Can you give me a sense of:

• what size team are you attached to?
• how long has the client been attached to that team?
• what are you doing with the client, e.g. just listening to RTM events, sending messages etc.?

[Brain2000]

Our team is made up of 10 people. We just started using slack about two months ago. I just listen to events. I have sent about three messages using it. We have the Visual Studio TFS plugin.

[ghost]

I'm not familiar with the Visual Studio TFS plugin, could you give me a sense of what it's doing in this context?

I'm running memwatch-next on an RTM connection against a roughly 500 person team, so should have a sense of if there are any obvious memory leaks from that.

[Brain2000]

We have the TFS plugin set up as a channel which sends a message every time a build takes place, which ends up being around 20-30 messages a day in our environment. I notice the memory when first opening the Slack app. It doesn't seem to go up, so I don't think there is a memory leak.

[clavin]

So, I feel like there's some confusion happening here. I'm going to just requote something @Brain2000 said:

The only preference that looked like it might be similar is "Disable Local Storage" under advanced.

This shows he's talking about the actual slack client. App. The thing you get from https://slack.com/downloads. I've unpacked it, it doesn't use the module this repository hosts, so his issue shouldn't really go here.

Furthermore, @l12s is talking about the module that this repository is dedicated to, hence the whole "testing a RTM connection".

Basically, this ordeal should've been mentioned to one of the awesome people at the Slack Help Center-- if anywhere. I will agree that 500 MB is quite a lot of memory being used as my client always hovers right under 60 MB and I'm connected to 4 different teams.

I can also see where the confusion comes from. The only thing in the name of this repository that indicates it's a node module is the "node-" prefix. That leaves "Slack Client", with which most would imagine the slack app.

[Brain2000]

Oh. That would be my fault, I didn't realize you had a Node.js client. All clients must follow the same versioning too, since they are both on 2.0.

[kurisubrooks]

This repo's for the backend API of Slack through Node.js, serves an API for Bots and Integrations. The client is closed source embedded webkit. The repo itself is poorly named, but that's not our problem.

The reason the desktop client, the program (not the API) has such a high memory footprint, is due to the fact that it runs on Chrome's WebKit Engine. High Memory use is to be expected.

[ghost]

Thanks again @Everesty for that explanation.

Closing this issue for now.

[bewitchingme]

Running the Slack client on Kubuntu 16.04 and after 3.5 hours of running it consumed vast amounts of system memory, causing my desktop to nearly grind to a halt. I have a 64bit Core i7 processor and 8GB of dual-channel RAM. I wanted to recommend this for the team I'm working with, but given that we're all running the same OS and hardware, I can't support use of the desktop application.

[ghost]

Hey there. The best place to get help on that is https://get.slack.help/hc/en-us

This repo is for the node client for the Slack APIs

[NikolayS]

Meanwhile (Slack app 2.0.3 for MacOS):

The link https://get.slack.help/hc/en-us is useless, it says just this:

App uses a lot of memory

Lots of unread channels? Reading them (or marking them as read) should help. 
Use the /collapse command in channels with lots of images and videos.
Sign out of any inessential teams.

-- and that's it, nothing more.
I have 5 teams and 22+19+1=42 unread messages. It's not worthing 2.1GB of RAM.
Do you really think that I'll be sitting checking RAM consumption each hour and being ready to type /collapse in each channel?

Here is the public discussion and hey you, Slack team, pay attention to this issue, it's so important that you're losing clients.
It's very important, just escalate this question on your next team meeting.

[NikolayS]

UPDATE:

Marked everything as "read", and restarted the app, it quickly (less than in 1 minute) grown to 462MB:

462MB for what?!

Notice the Gitter app -- I have dozens of teams and channels there, it's running for couple of weeks and has hundreds of unread messages. Still 180MB of memory.

[mvaragnat]

It's true, but this is a thread related to the Node API client, not the Mac app...

[NikolayS]

@mvaragnat thanks for pointing this out, I see, but I wanted to discuss it publicly and haven't found better place than this one.

So two cases already, Linux and Mac.

[clavin]

There's more than just public help topics; the right choice here is to contact one of the lovely supporters. It's not public, but it does allow you to get your message to Slack and a response from them, which is a lot less likely here.

Even then, I know of only one slacker in this repository, and I believe he's more of a back-end dude.

[kurisubrooks]

The reason Slack's desktop app uses so much memory is because of the underlaying Electron wrapper. Electron is based on Chromium, which is Chrome, which is where the ridiculous memory usage would be coming from.

[DEGoodmanWilson]

This forum is not an appropriate place to discuss the Slack client itself. Please, let us limit ourselves to the topic of this discussion: Memory usage of the Node SDK.

[vedmant]

Yeah, I'm forced to close slack client when I'm using virtual machines and need more memory...

[kurisubrooks]

Changing the name in hopes people will bother reading up what this issue is _actually_ for before commenting...

[NikolayS]

Very good, @kurisubrooks but the problem of RAM consumption in Slack client is so annoying.. and definitely there is a lack of places where people would express their feeling publicly. So I'm afraid that you need to do more changes (like delete all our comments and avoid phrases such "memory usage" next to "slack") to completely get rid of commenters like me.

Not long ago Slack client devs decided to use multiple processes -- like in Chrome. So now we can easily observe several hundreds of MBs times # of teams you work with -- sometimes it takes gigabyte or even more. So things became even worse.

If Slack developers don't bother to listen to their customers, and if the continue doing so, there will probably no need in developing any API, bots or smth else for Slack because some other project will win.

[clavin]

That still doesn't take away from the fact that this isn't the appropriate place to discuss the issue for various crucial reasons:

• People who work on and look at this repository don't work on the actual Slack client. As such, we have no control over what happens with the official Slack client.
  
  
  
  • That also means that people who do work on the Slack client don't check here for their todo list.
  
  
• Similarly, we don't have control over electron, which Slack runs on.
• Again, we don't have control over chromium, which electron (and therefore Slack) runs on.

We can't fix your Slack client issues and nothing posted here about the client is likely to be seen by anyone who works on the client.

If you want to see changes to the issue at hand, then choose one of the following:

• Publish your own changes to Chromium.
• Publish your own changes to electron.
• Talk to Slack support (if it's a Slack-specific issue).

If there is an issue with the module this repository provides (which, mind you, is not used in the official Slack client) then this is the place to discuss that issue.

[aoberoi]

We value building this Slack SDK in open source so that we are held accountable, because in the end of the day, our aim is to give developers the best slack client they can get.

I see some confusion in the terminology, so let me help define a few things:

• Slack Developer Kit for Node / Slack SDK for Node / Node Slack client / Slack client: this is a package developers use to work with the Slack API and Platofrm from Node.js. its the thing you get when you npm install @slack/cleint.

• Slack Desktop App / Slack App / Slack for macOS / Slack for Windows: this is the thing a user of Slack downloads and runs to communicate with his or her Team. Its built (for Mac) on top of Electron.

This is the place to talk about memory usage for the first thing, but it is NOT for the second thing. I think this issue is quite a bit off track. If there are specific memory concerns about the first thing, we should make more specific issues (such as #252). If there are memory concerns with the second thing, @Everesty's comment has a good list of places to look. Seeing as that covers all cases, and that this issue is closed, i'm going to lock it for conversation. _This is not meant to silence our community, its merely a way of routing everyone to the places where they can find the best help_.