Stackexchange.redis: SeachKeys: implementation KEYS vs. SCAN

[gzannerini]

Hi, posted here:
https://stackoverflow.com/questions/48079957/redis-stackexchange-keys-vs-implementation-via-scan-usage-of-redisresult
Question on performance of KEYS command. SCAN is non-blocking, but will allow better performance for large data sets.
Tried to implement on my application, using Execute() method and RedisResult class, but inner result is non accessible.
Will submit separate issue.
Thanks,
Gabriele

[mgravell]

SE.Redis always prefers SCAN over KEYS; this detail is abstracted away behind the IServer.Keys(...) method, and makes some decisions based on the detected server version. Basically: you shouldn't need to do anything here except use IServer.Keys(...). See also: https://stackexchange.github.io/StackExchange.Redis/KeysScan

I'm intrigued about what you say about Execute, though. It should "just work". If there are inner-results, then there is a cast:

public static explicit operator RedisResult[] (RedisResult result) {...}

[gzannerini]

Wrote test method to compare performance.

    public static List<string> GetSchemas(string filter)
        {
            List<string> schemas = new List<string>();
            System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();

            NewtonsoftSerializer serializer = new NewtonsoftSerializer();
            StackExchangeRedisCacheClient cacheClient = new StackExchangeRedisCacheClient(serializer);

            #region With SCAN and MATCH

            sw.Start();
            int nextCursor = 0;
            do
            {
                RedisResult redisResult = cacheClient.Database.Execute("SCAN", new object[] { nextCursor.ToString(), "MATCH", "pdr:*", "COUNT", "1000" });
                var innerResult = (RedisResult[])redisResult;

                nextCursor = int.Parse((string)innerResult[0]);

                List<string> resultLines = ((string[])innerResult[1]).ToList();
                schemas.AddRange(resultLines);
            }
            while (nextCursor != 0);
            LoggerManager.Debug("SCAN and MATCH: Got {0} elements of \"pdr:\" in {1}msec.", schemas.Count, sw.ElapsedMilliseconds);

            #endregion With SCAN and MATCH

            #region With method SearchKeys

            sw.Restart();
            IEnumerable<string> schemas1 = cacheClient.SearchKeys("pdr:*");
            LoggerManager.Debug("SearchKeys: Got {0} elements of \"pdr:\" in {1}msec.", schemas1.Count(), sw.ElapsedMilliseconds);

            #endregion With method SearchKeys

            return schemas;
        }

Results are:

04/01/2018 15.19.07;;;;;;9;GetSchemas;;;SCAN and MATCH: Got 977370 elements of "pdr:" in 3018msec.
04/01/2018 15.19.52;;;;;;;9;GetSchemas;;;SearchKeys: Got 977370 elements of "pdr:" in 45144msec.

Run 4-5 times, at every run there's a factor 10 (3 sec. vs. 40 sec. circa).

Do you have an explanation for this behavior?

Thanks in advance.
Regards, Gabriele

[mgravell]

Well, what is SearchKeys? That doesn't sound like my code, so can I see it?
In particular: what page size is it specifying? Is it 1000 like the first
test?

On 4 Jan 2018 2:46 pm, "gzannerini" [e-mail] wrote:

Wrote test method to compare performance.

public static List GetSchemas(string filter)
{
List schemas = new List();
System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();

        NewtonsoftSerializer serializer = new NewtonsoftSerializer();
        StackExchangeRedisCacheClient cacheClient = new StackExchangeRedisCacheClient(serializer);

        #region With SCAN and MATCH

        sw.Start();
        int nextCursor = 0;
        do
        {
            RedisResult redisResult = cacheClient.Database.Execute("SCAN", new object[] { nextCursor.ToString(), "MATCH", "pdr:*", "COUNT", "1000" });
            var innerResult = (RedisResult[])redisResult;

            nextCursor = int.Parse((string)innerResult[0]);

            List<string> resultLines = ((string[])innerResult[1]).ToList();
            schemas.AddRange(resultLines);
        }
        while (nextCursor != 0);
        LoggerManager.Debug("SCAN and MATCH: Got {0} elements of \"pdr:\" in {1}msec.", schemas.Count, sw.ElapsedMilliseconds);

        #endregion With SCAN and MATCH

        #region With method SearchKeys

        sw.Restart();
        IEnumerable<string> schemas1 = cacheClient.SearchKeys("pdr:*");
        LoggerManager.Debug("SearchKeys: Got {0} elements of \"pdr:\" in {1}msec.", schemas1.Count(), sw.ElapsedMilliseconds);

        #endregion With method SearchKeys

        return schemas;
    }

Results are:

04/01/2018 15.19.07;;;;;;9;GetSchemas;;;SCAN and MATCH: Got 977370
elements of "pdr:" in 3018msec.
04/01/2018 15.19.52;;;;;;;9;GetSchemas;;;SearchKeys: Got 977370 elements
of "pdr:" in 45144msec.

Run 4-5 times, at every run there's a factor 10 (3 sec. vs. 40 sec. circa).

Do you have an explanation for this behavior?

Thanks in advance.
Regards, Gabriele

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/763#issuecomment-355300226,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABDsHTOEoHcn__YyCVvvQlnMolc0b25ks5tHORQgaJpZM4RTByG
.

[gzannerini]

I Mark, it's a method of StackExchange.Redis.Extensions, got in my C# project via NuGet.
Below a snippet of class definition.

Can't specify on SearchKeys method the page size.

Is it maybe my mistake, i.e. not a package you're maintaining? In that case, can you please post how to get same result via method provided by StackExchange.Redis library?

Regards, Gabriele

#region Assembly StackExchange.Redis.Extensions.Core, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null
// C:\DevAMM\RetiSAC\packages\StackExchange.Redis.Extensions.Core.2.4.0\lib\net45\StackExchange.Redis.Extensions.Core.dll
#endregion

using System;
using System.Collections.Generic;
using System.Runtime.CompilerServices;
using System.Threading.Tasks;
using StackExchange.Redis.Extensions.Core.Configuration;

namespace StackExchange.Redis.Extensions.Core
{
    //
    // Summary:
    //     The implementation of StackExchange.Redis.Extensions.Core.ICacheClient
    public class StackExchangeRedisCacheClient : ICacheClient, IDisposable
    {
        public StackExchangeRedisCacheClient(ISerializer serializer, IRedisCachingConfiguration configuration = null);
        //
    // ..............
    // 
        //
        // Summary:
        //     Searches the keys from Redis database
        //
        // Parameters:
        //   pattern:
        //     The pattern.
        //
        // Returns:
        //     A list of cache keys retrieved from Redis database
        //
        // Remarks:
        //     Consider this as a command that should only be used in production environments
        //     with extreme care. It may ruin performance when it is executed against large
        //     databases
        public IEnumerable<string> SearchKeys(string pattern);
        //
        // Summary:
        //     Searches the keys from Redis database
        //
        // Parameters:
        //   pattern:
        //     The pattern.
        //
        // Returns:
        //     A list of cache keys retrieved from Redis database
        //
        // Remarks:
        //     Consider this as a command that should only be used in production environments
        //     with extreme care. It may ruin performance when it is executed against large
        //     databases
        public Task<IEnumerable<string>> SearchKeysAsync(string pattern);

        //
    // ..............
    // 
    }

[mgravell]

StackExchange.Redis.Extensions is nothing too do with me and is not
officially affiliated with StackExchange.Redis in any way. I cannot comment
on what parameters of offers. However, as per the link I sent you
yesterday: StackExchange.Redis offers full support for KEYS/SCAN via the
IServer API, including the ability to specify the page size.

On 4 Jan 2018 8:59 p.m., "gzannerini" [e-mail] wrote:

I Mark, it's a method of StackExchange.Redis.Extensions, got in my C#
project via NuGet.
Below a snippet of class definition.

Can't specify on SearchKeys method the page size.

Is it maybe my mistake, i.e. not a package you're maintaining? In that
case, can you please post how to get same result via method provided by
StackExchange.Redis library?

Regards, Gabriele

region Assembly StackExchange.Redis.Extensions.Core, Version=2.4.0.0, Culture=neutral, PublicKeyToken=null// C:\DevAMM\RetiSAC\packages\StackExchange.Redis.Extensions.Core.2.4.0\lib\net45\StackExchange.Redis.Extensions.Core.dll

endregion

using System;using System.Collections.Generic;using System.Runtime.CompilerServices;using System.Threading.Tasks;using StackExchange.Redis.Extensions.Core.Configuration;
namespace StackExchange.Redis.Extensions.Core
{
//
// Summary:
// The implementation of StackExchange.Redis.Extensions.Core.ICacheClient
public class StackExchangeRedisCacheClient : ICacheClient, IDisposable
{
public StackExchangeRedisCacheClient(ISerializer serializer, IRedisCachingConfiguration configuration = null);
//
// ....
//
//
// Summary:
// Searches the keys from Redis database
//
// Parameters:
// pattern:
// The pattern.
//
// Returns:
// A list of cache keys retrieved from Redis database
//
// Remarks:
// Consider this as a command that should only be used in production environments
// with extreme care. It may ruin performance when it is executed against large
// databases
public IEnumerable SearchKeys(string pattern);
//
// Summary:
// Searches the keys from Redis database
//
// Parameters:
// pattern:
// The pattern.
//
// Returns:
// A list of cache keys retrieved from Redis database
//
// Remarks:
// Consider this as a command that should only be used in production environments
// with extreme care. It may ruin performance when it is executed against large
// databases
public Task> SearchKeysAsync(string pattern);
}

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/763#issuecomment-355396933,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABDsB6uHkoyUxd8i6OHehR0TFzlvA9Wks5tHTuggaJpZM4RTByG
.

[gzannerini]

Ok, sorry about that. Closing the issue.
Regards, Gabriele.

[mgravell]

@gzannerini I'm still very curious: if you use the IDatabase API to search the keys specifying your chosen page size: does the performance problem disappear?

edit: I meant IServer API

[gzannerini]

When bypassing StackExchange.Redis.Extensions.Core using direct call to the IDatabase API, performance problem disappear (3 seconds to load 997.000 circa elements).

[mgravell]

that's great, thanks

[gzannerini]

Can you please explain?

[mgravell]

you said that the performance issue disappears; I replied saying "that's great", meaning: I'm glad that the problem is solved when using the IServer API directly.

Because : if it wasn't solved, I'd need to fix it :)

[gzannerini]

Thanks Marc! Need to investigate with developers of other package. Closing (definetly) the issue. Regards, Gabriele