Azure-sdk-for-js: [Service Bus] scheduleMessage is returning the same seq number for multiple messages

[diego1686]

Hi! When I'm trying to schedule messages in parallel sharing the same sender (for example, scheduling from a message handler with maxConcurrentCalls > 1), the scheduleMessage method is always returning the same sequence number. Here is a similar scenario using Bluebird's Promise.join function:

'use strict';

const { ServiceBusClient } = require('@azure/service-bus');
const moment = require('moment');
const Promise = require('bluebird');

(async function main() {
  const serviceBusClient = new ServiceBusClient(process.env.CONNECTION_STRING);
  const sender = await serviceBusClient.createSender(process.env.QUEUE_NAME);

  try {
    let array = [];
    for (let index = 0; index < 5; index++) {
      array.push({body: { id: index.toString() } });
    }

    const date = moment().add(30, 'seconds').toDate();

    // Schedule multiple messages at once
    const seqNumbers = await sender.scheduleMessages(date, array);
    console.log(seqNumbers);

    // Schedule messages one by one
    const seqNumber1 = await sender.scheduleMessage(date, { test: true });
    const seqNumber2 = await sender.scheduleMessage(date, { test: true });
    const seqNumber3 = await sender.scheduleMessage(date, { test: true });
    console.log(seqNumber1);
    console.log(seqNumber2);
    console.log(seqNumber3);

    // Schedule messages in parallel
    const inParallel = await Promise.join(
      sender.scheduleMessage(date, { test: true }),
      sender.scheduleMessage(date, { test: true }),
      sender.scheduleMessage(date, { test: true })
    );
    console.log(inParallel);

  } catch (err) {
    console.error('Error %O', err);
  } finally {
    await sender.close();
    await serviceBusClient.close();
  }
})();

This is what I got:

// "Schedule multiple messages at once" result
[ Long { low: 8559, high: 0, unsigned: false },
  Long { low: 8560, high: 0, unsigned: false },
  Long { low: 8561, high: 0, unsigned: false },
  Long { low: 8562, high: 0, unsigned: false },
  Long { low: 8563, high: 0, unsigned: false } ]

// "Schedule messages one by one" results
Long { low: 8564, high: 0, unsigned: false }
Long { low: 8565, high: 0, unsigned: false }
Long { low: 8566, high: 0, unsigned: false }

// "Schedule messages in parallel" results (all the messages with seq = 8567)
[ Long { low: 8567, high: 0, unsigned: false },
  Long { low: 8567, high: 0, unsigned: false },
  Long { low: 8567, high: 0, unsigned: false } ]

I'm using node v10.17.0 and "@azure/service-bus": "^7.0.0-preview.2" (works much better than the v1, thanks!)

[guillegette]

I was able to replicate the same issue.

[ramya-rao-a]

Thanks for reporting @diego1686 and @guillegette!

I'm using node v10.17.0 and "@azure/service-bus": "^7.0.0-preview.2" (works much better than the v1, thanks!)

While @HarshaNalluru takes a look into this, can you share what parts are working better than v1? We would love to get feedback on the preview versions we are working on

[diego1686]

@ramya-rao-a thanks!

Honestly I'm just starting to use it, but the sintax is more clear, which is obviously better and the most important thing is on this kind of scenarios we got race conditions using the v1, I mean, scheduling operations taking a lot to get solved (between 15 or 25 seconds), duplicated messages, lock and timeout errors, etc. All those things are not happening with the new version :)

@HarshaNalluru FYI this not happens if before the scheduling operation I create a new ServiceBusClient and a new sender, which are closed after the scheduling... but I think that's not a good practice. If I have 1000 concurrent messages, I'd need to create 1000 clients at the same time... It doesn't sound good

[HarshaNalluru]

You're right, @diego1686. We definitely don't want you to create 1000 clients at the same time just to send a message with each of them. It is recommended to reuse the same client and the same sender wherever possible.

I was able to repro the issue, the sequence numbers are the same when the scheduled messages in parallel. Most likely, the issue is with the service since the sequence numbers are generated by the service. We'll see if we can avoid the issue from the SDK side as well as followup with the service to resolve the issue.

[guillegette]

Most likely, the issue is with the service since the sequence numbers are generated by the service.

@HarshaNalluru if you don't mind me asking, is that really a possibility? because if you run multiple Node.js processes in parallel scheduling messages, you don't encounter this issue. You can peak the messages that are scheduled and each of them have a different sequence number. Could it be a bug in the SDK when multiples requests are starting at the same time ? has something like this been reported in another Bus Service SDK ?

Thanks!

[HarshaNalluru]

@guillegette, thanks for the comment. It is the responsibility of the service to assign unique sequence numbers to the messages, SDK has no role in assigning the sequence number for a message. It is possible that the sequence numbers are generated based on the attributes provided while making the scheduling request at the SDK level and those relevant attributes might have been the same when we schedule messages in parallel.

I agree the cause to be firing multiple scheduling requests starting at the same time using the same sender and the same ServiceBusClient. This hasn't been reported before to my knowledge.
Like I've mentioned in the previous comment, we'll investigate from the SDK side to see if we can avoid the issue by tweaking the scheduling code.

[guillegette]

Thanks @HarshaNalluru for the response. What I was trying to say is that, I (personally) don't think is a problem with the service. Giving a unique and incremental sequence number is core to the queue capabilities, failing at this would be a huge issue for everyone using the service.

I think when multiple requests are executed in parallel, the SKD library is not managing the response well for the different values and end up returning the same for all requests. (the first one)

I know I could be wrong, but I think implying an issue on the actual queue seems unrealistic considering that it would be a huge issue for all customers.

I will look into more details the codebase to see if I can find the potential issue.

Thanks!

[HarshaNalluru]

You're right, @guillegette, the issue is similar to what you've mentioned.
Here is the PR with the bug fix and relevant details - https://github.com/Azure/azure-sdk-for-js/pull/9503

[HarshaNalluru]

The version 7.0.0-preview.4 for the @azure/service-bus package has just been released which contains the bug fix. 🎁
Here is the changelog for version 7.0.0-preview.4.

In an effort to standardize the API, method names and signatures have been updated.
scheduleMessage API is updated as follows,
In 7.0.0-preview.3:

scheduleMessage(scheduledEnqueueTimeUtc: Date, message: ServiceBusMessage, options?: OperationOptions): Promise<Long>;
scheduleMessages(scheduledEnqueueTimeUtc: Date, messages: ServiceBusMessage[], options?: OperationOptions): Promise<Long[]>;

In 7.0.0-preview.4:

scheduleMessages(scheduledEnqueueTimeUtc: Date, messages: ServiceBusMessage | ServiceBusMessage[], options?: OperationOptionsBase): Promise<Long[]>;

Please check out the library and let us know your feedback :)