Bull: Do keys generated by bull expire?

[kentnek]

Hi guys,

I'm using Bull in a personal project. After running for a while, I notice bull generated a lot of keys and Redis starts hogging lots of RAM. Will these keys expire?

Thanks!

[nibblesnbits]

At the moment there are no keys that require any sort of expiration. It is true that Bull will quickly accumulate keys in redis if you don't regularly clean jobs. Are you using Queue##clean at all?

[manast]

It may be worth to implement an expire option for completed (and failed jobs), so the user is free to configure if he wants automatic key expiration after a certain optional time, or no expiration at all.

[nibblesnbits]

That shouldn't be too difficult. I'll get back into it and see if I can do this cleanly.

[kentnek]

Nice!

[CCharlieLi]

Any update here? I saw no expire option in the latest reference, and staled keys in Redis are still eating my RAM even with Queue##clean.

[manast]

@CCharlieLi why don't you use the option removeOnComplete ? https://github.com/OptimalBits/bull/blob/master/REFERENCE.md#queueadd

[CCharlieLi]

Thanks for reminding @manast , I guess it's the same as queue.clean(0) (please correct me if I m wrong). I also wonder if there's a way to remove keys left by previous queue instances which were already shutdown. Or do I have to use script to delete them from Redis?

[manast]

@CCharlieLi you should be able to instantiate the old queue and then call clean on it.

[carlhopf]

It would be very useful to be able to set TTL, for example in an environment where Nodes go up and down all the time, with unique queue names for inter-server communication.

F.ex.: two AWS spot instances communicate with each other, each instance uses a unique named queue based local instance id. both instances get shut down at the same time by aws. remaining data on redis never expires, unless manually keeping track of instances (or queue) names for cleaning up.

(edit for better example)

[joe-at-startupmedia]

In my use case removeIOnComplete isn't an ideal option because I would like to keep track within a given time interval which records have completed. For example: I remove all completed queues after 2 weeks because by then I have seen and acknowledged they were completed. It brings peace of mind to know queues are being completed and not just purging them immediately after they complete.

[nidhhoggr]

See my implementation here:
https://github.com/nidhhoggr/bull/commit/486eabb35f85621c2eff6fd4eaffc6edfa6d78c9

Several Issues

1. The injection of queue settings into the constructor job. Currently Job doesn't inherit settings from Queue in this manner. Job sets this.expiresIn from queue.settings.
2. The order of arguments for lib/commands/moveToFinished-6.lua was shifted up one before the arguments from the 9 position used to avoid the next roundtrip. I though it was necessary to keep the argument next to the relative arguments for the particular job so offset 7 (before the even data) made more sense to me.
3. The keys are expired but not removed from the respective sorted sets in redis (e.g. bull:[job_name]:completed) . One way to do this is to add a redis event listener for expired keys and remove the keys in the event handler. Another way is to write a maintenance script to flush the expired keys. The cons of the first approach is that the key is not technically expired and purged from redis until it is attempted to be accessed per documentation below:

https://redis.io/topics/notifications#timing-of-expired-events

[nidhhoggr]

Implementation to expire expired keys from a job redis key:
https://github.com/nidhhoggr/bull/commit/1f34aa025c21066a86e3ae60facb25d2082c4f8d

[ritter]

@nidhhoggr glancing at your implementation I have immediate concerns about jobId being expected and cast to be number/integer, my understanding is that jobId can also be a string.

[ritter]

Also, this code is subscribing to every key matching ":expired", and probably should only be matching those set under bull's namespace, like: "bull:*:expired", and even then probably only for a specific queue?

Disregard that, I misread it.

[nidhhoggr]

@ritter Disregard what? These are both valid concerns, I was unaware there was a possibility of jobId as string with non-numeric keys. The issue with having a string as a key means that the jobId must be confirmed as an actual job entry of type hash, because bull has other keys with the convention bull:${queueName}:[string] that it uses to store completed, failed, active, stalled-check etc. So something needs to be devised to assert the jobId is an actual job before calling the removeJob.lua script with attempts to remove the entries by the jobId. regarding only watching for bull:*:expired that is a pretty simple fix:

          client.psubscribe(_this.toKey('expired'), function(err, result) {

The event handler also needs error checking as well. In conclusion, my implementation isn't PR ready but feel free to provide suggestions and/or take from it what you will to submit your own PR.

[nidhhoggr]

Actually, regarding the expired keys events listener my example above would not work because the name of the keyspace event looks like this:

"pmessage","__key*__:*","__keyevent@0__:expired","bull:updateCauseTotalSavings:stalled-check"

When I attempted to listen for

client.subscribe("__keyevent@0__:expired") 

I wasn't getting anything so I changed it to the following with success.

client.psubscribe("*:expired") 

[sarneeh]

This feature is really missing. For now, the only option to automate cleaning up the queue is to:

• have a ioredis instance which updates the entries on completed and/or failed adding an EXPIRE
• have a repeatable job which runs every X and cleans up the jobs.

This is a tedious task. I'd suggest adding an option to .add or extending removeOnComplete/removeOnFail to accept an object like: { expireIn: 30000 } instead of a boolean.

[manast]

@sarneeh I will try to improve this in the following days.

[sarneeh]

@manast But don't feel like you have to. It's open source, if I'd need it very much I'd do it myself. It's just a suggestion if you'd like to improve it somehow :smiley: Your work already saves a dozen of hours creating such a solution (bull) manually :smiley:

[manast]

I need it myself in some project :).

[manast]

It is not really expiring all the keys, this would be a complete different story, we may implement it in the future for ephemeral queues.