Stackexchange.redis: About deleting many keys in Redis

[d-saravanan]

Hi

We have a scenario that there is a cache for every user and that there are times like when there is a change in the role permission, we delete the keys for all the users so that the changes are evenly distributed. There are few applications and we are also using the redis pub-sub to sync up the changes across the connected applications. However, when checked for approx 800 user ids using a redis batch like the one given below, there is a delay in the subscriber getting notified on the key deletion.

Please suggest any other efficient approach for this problem. I have an idea to try the batch to use many keys instead of one so that the batch size comes down also to explore on lua scripting [completely new and not sure if it will really help]

Current code snippet:

if (redisDB != null)
{
    if (redisDBKeys.Length == 1)
    {
        await redisDB.KeyDeleteAsync(redisDBPrefix + redisDBKeys[0]);
    }
    else if (redisDBKeys.Length > 1)
    {
        var redisBatch = redisDB.CreateBatch();
        var tasks = redisDBKeys.Select(key => redisBatch.KeyDeleteAsync(redisDBPrefix + key));
        redisBatch.Execute();
        await Task.WhenAll(tasks);
// when the above statement has await, the lines below never get executed.
// If the above line has .Forget(), below lines are getting executed, but there is a delay in the pub-sub callback to the other applications
    }
    await redisDB.PublishAsync(CallbackChannel + key.ToString(), JsonConvert.SerializeObject(redisDBKeys.ToArray()), CommandFlags.FireAndForget);
}

[mgravell]

How big is the batch? How many subscriptions are there? Presumably you're
using a wildcard subscription, since you're concatenating the key into the
channel name.

If the batch is huge, then yes - it will do all the deletes before it
starts the publish. I wonder if you could interleave a delete and publish
per key so that it is always del/publish/del/publish/etc.

On 3 Feb 2018 11:23 am, "Saravanan" [e-mail] wrote:

Hi

We have a scenario that there is a cache for every user and that there are
times like when there is a change in the role permission, we delete the
keys for all the users so that the changes are evenly distributed. There
are few applications and we are also using the redis pub-sub to sync up the
changes across the connected applications. However, when checked for approx
800 user ids using a redis batch like the one given below, there is a delay
in the subscriber getting notified on the key deletion.

Please suggest any other efficient approach for this problem. I have an
idea to try the batch to use many keys instead of one so that the batch
size comes down also to explore on lua scripting [completely new and not
sure if it will really help]

Current code snippet:

if (redisDB != null)
{
if (redisDBKeys.Length == 1)
{
await redisDB.KeyDeleteAsync(redisDBPrefix + redisDBKeys[0]);
}
else if (redisDBKeys.Length > 1)
{
var redisBatch = redisDB.CreateredisBatch();
var tasks = redisDBKeys.Select(key => redisBatch.KeyDeleteAsync(redisDBPrefix + key));
redisBatch.Execute();
await Task.WhenAll(tasks);
// when the above statement has await, the lines below never get executed.
// If the above line has .Forget(), below lines are getting executed, but there is a delay in the pub-sub callback to the other applications
}
await redisDB.PublishAsync(CallbackChannel + key.ToString(), JsonConvert.SerializeObject(redisDBKeys.ToArray()), CommandFlags.FireAndForget);
}

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/777, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABDsJoJtHawu38zv1MBlHTx-S1uwZp1ks5tREGVgaJpZM4R4Iru
.

[d-saravanan]

Hi Marc, Thanks for the response.
There are approx 800 keys in the batch on the Test Environment. There are 3 subscriptions now.

The following code snippet is used in the subscription

private static async Task InitializeSubscriptionAsync()
{
    var redisSubscription = redisCache.GetSubscriber();
    try
    {
        await redisSubscription.SubscribeAsync((CallbackChannel + "*"), (channel, payload) =>
        {
            try
            {
                var cacheKeys = JsonConvert.DeserializeObject<string[]>(payload);
                foreach (var key in cacheKeys)
                {
                    memCache.Remove(key);
                }
            }
            catch { }
        });
    }
    catch (TimeoutException e)
    {
        Logger.LogException(e);
        // try again
        InitializeSubscriptionAsync().Forget();
    }
}

1. I am also finding that the KeyDeleteAsync takes a collection of keys so, I can partition the key set into collections of 50 and then send it.

2. I am exploring on the Redis Set. I hope that if I can have a set in the Redis cache, then I can just delete the set which will be a very simple call and more efficient.

Please let me know if I am correct

[mgravell]

The only way to answer a performance question is to race the horses :) but
yes, deleting an entire set is faster than deleting keys individually, and
batching things can improve throughput as long as you don't get too extreme

On 4 Feb 2018 6:15 a.m., "Saravanan" [e-mail] wrote:

Hi Marc, Thanks for the response.
There are approx 800 keys in the batch on the Test Environment. There are
3 subscriptions now.

The following code snippet is used in the subscription

private static async Task InitializeSubscriptionAsync()
{
var redisSubscription = redisCache.GetSubscriber();
try
{
await redisSubscription.SubscribeAsync((CallbackChannel + "*"), (channel, payload) =>
{
try
{
var cacheKeys = JsonConvert.DeserializeObject foreach (var key in cacheKeys)
{
memCache.Remove(key);
}
}
catch { }
});
}
catch (TimeoutException e)
{
Logger.LogException(e);
// try again
InitializeSubscriptionAsync().Forget();
}
}

1.

I am also finding that the KeyDeleteAsync takes a collection of keys
so, I can partition the key set into collections of 50 and then send it.
2.

I am exploring on the Redis Set. I hope that if I can have a set in
the Redis cache, then I can just delete the set which will be a very simple
call and more efficient.

Please let me know if I am correct

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/777#issuecomment-362883889,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABDsBe5TL9gIMTl_MgvTGuyg62bUmFmks5tRUr-gaJpZM4R4Iru
.

[d-saravanan]

@mgravell
I have tried to use the Redis Hashes to implement this scenario and the following is the logic followed

1. There is a cache hash created
2. The individual user's data is added as a key with the userid and the value for the hash
3. When I wanted to remove the individual item within the hash, I use the hashkey and the userid key and remove
4. When i need to flush the cache for all the user's, the hashkey will be removed by a call to the KeyDeleteAsync passing the hashkey as the argument which wipes off the entire key-value-pairs in the hash.
5. Once the hash is cleared, we publish a message so that the remaining subscriber apps can clear or sync up their local caches.

I have put the code sample here, can you please check the same and let me know so that I can reuse the similar logic for my application too.

I would like to prefer a simple approach like this so that there is less load on REDIS and less load on the network and my pub-sub calls are also expected to be fast. Please share your thoughts on this model.

Note: Since this is a sample, I have used Wait() and other methods which will not be in the real code.

[d-saravanan]

@mgravell
I have tested using the hash and found that deleting the hash is faster than the previous ones. Further to this, I observed in the local environment that the PublishAsync method with the command flag as FireAndForget does not result in the subscription call back for the subscriber. I am checking on the client list redis-cli and trying to analyze. If you have any inputs for this point can you please post here.

[d-saravanan]

I solved the problem with the Redis Hash and found to be working fine.

[Jerome2606]

@d-saravanan Can you share the code you finally used. I come to a similar issue.

Here's mine (source: AbpBoilerplate Framework):

database.ScriptEvaluate(@"
                local keys = redis.call('keys', ARGV[1]) 
                for i=1,#keys,5000 do 
                redis.call('del', unpack(keys, i, math.min(i+4999, #keys)))
                end", values: new RedisValue[] { prefix });

[mgravell]

Just for the record and to be clear: I do not personally advocate or endorse any such solution - anything that involves KEYS (or SCAN in a loop in a server-side script) : can cause significant performance degradation. If you need to run it periodically for some reason: fine, have fun - just... don't do it casually :)

[Jerome2606]

@mgravell Can you propose an alternative ?
Actually I'm testing with:

public static void KeyDeleteWithPrefix(this IDatabase database, IServer server, string prefix)
        {

            if (server == null)
            {
                throw new ArgumentException("Server cannot be null", nameof(server));
            }

            if (string.IsNullOrWhiteSpace(prefix))
            {
                throw new ArgumentException("Prefix cannot be empty", nameof(server));
            }

            foreach (var key in server.Keys(pattern: prefix))
            {
                database.KeyDelete(key);
            }
        }

[mgravell]

@Jerome2606 that's definitely much less harmful in terms of single large operations at the server; personally I'd probably be tempted to increase the page size slightly (optional param to Keys), but: yeah, that works if you need to do that. I'd rather avoid the need to do it, personally :)

[Jerome2606]

@mgravell mmh just tested and it is very slow.
The first option is 110 ms on my cache example, the code pasted below took 1.5 sec.
I need it for the implementation of reset cache for a specific tenant.
I make a service that reset cache during night after database import.
I don't know why it should be avoid to want to clear a part of a cache.
For me a CacheClient should implement: Get, Set and Clear.

[mgravell]

@Jerome2606 great; if the issue is flushing an entire tenant: FLUSHDB (and use a db per tenant). If the data is more granular and you only want to delete keys of a particular type, then consider instead of using string-per-value (foo-abc = ..., foo-def = ..., foo-ghi = ... and "delete everything like foo-*"): use a hash per notional area, (foo : abc = ..., foo : def = ..., foo : ghi = ..., and then simply "delete foo")

[Jerome2606]

@mgravell I already use hash per notional area.
But for deletion I don't find a command to delete a "branch" / namespace.
How to simply delete foo ?

[mgravell]

if foo is a hash: KeyDelete("foo")