Ioredis: Handle "LOADING Redis is loading the dataset in memory"

[shaharmor]

Hi,

When a slave is first being connected to a master it needs to load the entire DB, which takes time.
Any command that is send to that slave during this time will receive a LOADING Redis is loading the dataset in memory response.

I think we should handle this and retry the command (Maybe even to a different node within the same slot).

@luin thoughts?

[shaharmor]

Its possible that during a failover to a slave, the old master will sync from the new master and cause this error to be returned, which makes the whole failover mechanism not so failsafe.

[luin]

ioredis already supports detecting loading in standalong version: https://github.com/luin/ioredis/blob/master/lib/redis.js#L420-L428. Seems we just need to wait for the "ready" event of the new redis node here: status.https://github.com/luin/ioredis/blob/master/lib/cluster/connection_pool.js#L58-L63

[shaharmor]

@luin something like this?

redis = new Redis(_.defaults({
      retryStrategy: null,
      readOnly: readOnly
    }, node, this.redisOptions, { lazyConnect: true }));

    var _this = this;
    redis._readyCheck(function (err) {
      // TODO: handle error
      _this.nodes.all[node.key] = redis;
      _this.nodes[readOnly ? 'slave' : 'master'][node.key] = redis;

      redis.once('end', function () {
        delete _this.nodes.all[node.key];
        delete _this.nodes.master[node.key];
        delete _this.nodes.slave[node.key];
        _this.emit('-node', redis);
        if (!Object.keys(_this.nodes.all).length) {
          _this.emit('drain');
        }
      });

      _this.emit('+node', redis);

      redis.on('error', function (error) {
        _this.emit('nodeError', error);
      });
    });

Also, how should we handle an error in the _readyCheck function?

[luin]

Hmm...I just checked the code, and it seems that when a node has not finished loading data from the disk, the commands sent to it will be added to its offline queue instead of sending to the redis immediately.

[shaharmor]

So that means that this should already be fixed? I've seen this happen in production, so its definitely an issue.

Could it be that it happens only to slaves or something? or when using scaleReads?

[shaharmor]

Its also possible that it happens if the slave was once connected, but then got restarted for some reason

[luin]

That's strange. Either the node is a slave or a master doesn't affect the support of offline queue. Are you able to reproduce the issue? Or enable the debug log maybe?

[kishorpawar]

I found this issue when I did following.

1. Accidentally I ran FLUSHALL on redis-cli, I tried to do ctrl-d.
2. Without stopping redis-server I copied backed up rdb to dump.rdb and restarted redis-server. I found that the copy did not happen actually.
3. I stopped redis-server and then copied backed up rdb to dump.rdb and started redis-server. Copy worked.
4. Started redis-cli
5. Ran command KEYS * and got error (error) LOADING Redis is loading the dataset in memory

[kaidiren]

@shaharmor So how did you deal with it finally ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[shaharmor]

Hey @luin , I just encountered this issue again, and I think we should see how we can fix it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[Eywek]

Hello,

Any news on this? I got the same error on ioredis v4.0.10

[alavers]

@Eywek @shaharmor Do you have any more details on how you reproduce this issue?

Is it possible you're connected to a slave that has begun a resync? E.g if the master it was pointing to performed a failover? A redis slave would return -LOADING errors during a resync which might explain how you encounter them without a connection reset.

What happens if you implement a reconnectOnError that returns 2 when a LOADING error is encountered?

[xiandong79]

any update

[alavers]

^I have a hypothesis that an error handler like this:

    reconnectOnError: function(err) {
      if (err.message.includes("LOADING")) {
        return 2;
      }
    }

_might_ solve this problem and if so should perhaps be made a default ioredis behavior. But I haven't built a repeatable way to reproduce this issue.