Ioredis: Best method for iterating large scan results

[zorro-fr24]

I couldn't find an obvious way of iterating a full set scan results from a large dataset ( x million rows ).

Alternatively perhaps there a clean method of feeding the cursor back into the scan function after it completes each pass?

or is it better to use a redis stream library for this kind of task.

thanks

note: I'm also using pattern matching: eg. var promise = redis.scan(0, 'match', 'keyspace:*', 'count', '1000');

[luin]

There's no special way of handling the scan command in ioredis. I just found a library https://github.com/brycebaril/redis-scanstreams doing the trick, which however only works with node_redis. I'm thinking of adding some convenient methods to ioredis.

[zorro-fr24]

thanks for the quick response.

Yup, I had investigated redis-scanstreams and node-redis-streamify and they both seem to rely on extending node_redis's client prototype which won't work with ioredis without modification.

I didn't need anything fancy, so I tested node-redis-streamify rather than scanstreams ( which looks like a better implementation, using proper piped streams )

In the case of modifying streamify, it's literally 1 line of code, or you can work around it in userland by:

var Redis = require('ioredis'),
    RedisStreamify = require('node-redis-streamify');

/* Add reference to ioredis client with node_redis property */
Redis.RedisClient = Redis;

RedisStreamify(Redis);

/* ... continue as normal */

It works, but it's ugly, so I'd love to see a native implementation.

Cheers

[luin]

I'm going to work on it this weekend.

[zorro-fr24]

Fantastic!
I look forward to it.

[luin]

Added 1.6.0. You can check out the unit tests for the usage. (It takes longer time to write documentation for a non-native speaker 😢)

[zorro-fr24]

awesome work, looks very neat - I'll give it a spin on monday - thanks!

[luin]

Just updated documentation: https://github.com/luin/ioredis/blob/a170ac7e95e01ab20e3ccdba0abcd437fb17c90e/README.md#streamify-scaning

[damianobarbati]

@luin is is possible to ad a promise'd version of it?

const result = await redis.scanAsync({ match: 'user:*', count: 100 });

Faster when developer is expecting small set of items from a scan without using the "evil" keys nonetheless.

[luin]

@damianobarbati It's not the recommended way to use scan since there may be millions of keys in a redis server, thus this will blow up the memory.

[vweevers]

In node v10 you can use readable[Symbol.asyncIterator]() to for await (x of stream).