Matterbridge: [Slack] users list not populated under high API usage

[patcon]

Describe the bug
When deploying the amazing new refactor on Heroku, no messages were sending. I've debugged it to find that the b.users slice is empty, so I get this in the logs with my debug output (see full log):

&errors.errorString{s:"could not find information for user with id U4MCXJKNC"}

Having said that, about half the time running locally (where my connection is slower), the b.users populated with 4000+ users (I'm also on a high-usercount slack)

To Reproduce
Add about 11 bridges over 2 gateways on a no too slow connection, with a large slack team of 4200 users.

Expected behavior
Would expect b.users to be populated and messages to send.

Screenshots/debug logs
Logs: https://gist.github.com/patcon/b45c1e80cfd249eb54ce6f877a62e9ea

Environment (please complete the following information):

• OS: osx
  
  
  
  • working on the head of master
  
  

Additional context
Added some debug code to make use of the RateLimitEvent from slack's rtm api, and not seeing it raise a flag when the b.users var fails to populate, so I'm thinking it must be something else

I'm wondering whether it might be related to the giant team and not being able to use the exclude_members flag which slack's api docs warn are heavy on big teams: https://github.com/nlopes/slack/pull/169

Might be worth reviving that PR in the scope of this, if we run out of options

cc @Helcaraxan if you have any thoughts :)

[patcon]

Not a solution, but adding a one-second sleep into populateUsers and populateChannels, and this consistently resolves things. It allows a new heroku build to work for the first time in a few weeks -- my deploys were failing even after working locally, and i just now had time to figure out that it was my janky translation branch causing it 😜

I'll keep working at it!

[Helcaraxan]

@patcon please tell if you are able to reproduce this after moving to the latest master. I do not unfortunately have access to a large enough Slack instance for me to try and reproduce this.

[patcon]

Thanks so much, @Helcaraxan :) I'll give it a shot when I get a sec, but fwiw, i threw in some debug code to catch ratelimit events, and they didn't seem to be arriving when there were troubles, so I worry it might be more subtle

But regardless, your PR makes this tool better!

[patcon]

Ah, ok, I got my answer -- just found that 1000 is the max returned users, so you're right that pagination is the crux of it. Still unclear why this did not affect things before the refactor, but *shruggie man*

[Helcaraxan]

So can you confirm that the problem has disappeared? :) Just checking if we can close the issue.

[patcon]

Def will, just not working on this project today

[Helcaraxan]

Closing this for now. Please re-open if the issue is observed again.

[patcon]

Sorry, forgot to follow-up, but this is still active. I'm still using a one-second sleep in populateUsers and populateChannels to mitigate :)

EDIT: As mentioned, it doesn't seem related to ratelimiting, as that event never throws