Rasa: Duplicate messages in Slack channel

[selphe]

Rasa version: 1.1.4

Python version: 3.6.8

Operating system (windows, osx, ...): Windows + ngrok

Issue:
When I send message to Slack channel, bot answering to me multiple times:
Here was an issue with same problem: https://github.com/RasaHQ/rasa/issues/2120

At ngrok logs I see, what User-Agent: Slackbot 1.0 keeps POST me messages with headers:

X-Slack-Retry-Num: 1
X-Slack-Retry-Reason: http_timeout

X-Slack-Retry-Num: 2
X-Slack-Retry-Reason: http_timeout



md5-7ae6cecde33929cf38559591b7a8dcfe



slack:
  slack_token: "xoxb-****"
  errors_ignore_retry: "http_timeout"

But it doesn't work.
(same problem on production server with docker)

Summarizing:

1. Slack duplicate messages to bot with reason of http_timeout even when no custom_actions executed (just utter from domain.yml)
2. Bot replies to every retry attempt
3. I can't bypass http_timeout with errors_ignore_retry

Could you advice please?

[cyrilthank]

I am encountering this similar issue.
This works fine on rasa shell

Also

1. i see the message being re-sent from rasa itself
2. i am unable to reproduce this on the moodbot

[akelad]

Thanks for raising this issue, @erohmensing will get back to you about it soon.

[erohmensing]

@selphe @cyrilthank Which responses are getting duplicated? Are they dispatched from within a custom action?

[cyrilthank]

Thank yo for your attention to this

The bot’s responses are repeate

For example:

Expected:

User : hi
Bot : are you human

===============

Observed

User: hi
Bot : are you human
Bot : are you human
Bot : are you human

Cyril

---

On Fri, 6/28/19, Ella Rohm-Ensing [e-mail] wrote:

Subject: Re: [RasaHQ/rasa] Duplicate messages in Slack channel (#3882)
To: "RasaHQ/rasa" [e-mail]
Cc: "cyrilthank" [e-mail], "Manual" [e-mail]
Date: Friday, June 28, 2019, 12:10 PM

Which responses are getting
duplicated? Are they dispatched from within a custom
action?

—
You are receiving this because you are subscribed to this
thread.
Reply to this email directly, view
it on GitHub, or mute
the thread.

[erohmensing]

@cyrilthank Yeah, I understand what it looks like, but am trying to figure out where the lag causing the timeout is coming from. Can you show me a story for when this happens?

[cyrilthank]

Thank you for your reply

This happens only when the action server is up even while doing utter but within actions.py

Doesn’t happen in shell or when the action types are utter*

Can you please share if there is anything particular I need to check in the stories

Requesting since I will be back only after a couple of days so appreciate if you could share some sort of steps to execute to debug/diagnose

Thank you

Cyril

---

On Fri, 6/28/19, Ella Rohm-Ensing [e-mail] wrote:

Subject: Re: [RasaHQ/rasa] Duplicate messages in Slack channel (#3882)
To: "RasaHQ/rasa" [e-mail]
Cc: "cyrilthank" [e-mail], "Mention" [e-mail]
Date: Friday, June 28, 2019, 3:43 PM

@cyrilthank yes i
realize this. can you show me a story for when this
happens?

—
You are receiving this because you were mentioned.
Reply to this email directly, view
it on GitHub, or mute
the thread.

[erohmensing]

@cyrilthank Okay thats very helpful actually! If it only happens when using the utter within actions.py, then it is a duplicate of https://github.com/RasaHQ/rasa/issues/3569. Slack expects some sort of response within 3 seconds and if it doesn't get it will re-send the request. As for

I can't bypass http_timeout with errors_ignore_retry

@selphe Looks like it should be a tuple as of here: https://github.com/RasaHQ/rasa/blob/master/rasa/core/channels/slack.py#L156
But this is strange because it looks like we actually try in our code to ignore the timeout error already. So this errors_ignore_retry not working looks like a bug in itself.

[cyrilthank]

ThNk you for your reply

Can you please advice if there is a way to increase the 3 secs and/or ignore/reset this limit

Also can you please advice for diagnostics which integration (telegram /Facebook/....)is recommended

Thank you

Cyril

On Fri, 6/28/19, Ella Rohm-Ensing [e-mail] wrote:

Subject: Re: [RasaHQ/rasa] Duplicate messages in Slack channel (#3882)
To: "RasaHQ/rasa" [e-mail]
Cc: "cyrilthank" [e-mail], "Mention" [e-mail]
Date: Friday, June 28, 2019, 4:18 PM

Okay thats very helpful actually!
If it only happens when using the utter within actions.py,
then it is a duplicate of #3569.
Slack expects some sort of response within 3 seconds and if
it doesn't get it will re-send the request. As for

I can't bypass http_timeout with
errors_ignore_retry

Looks like it should be a tuple as of here: https://github.com/RasaHQ/rasa/blob/master/rasa/core/channels/slack.py#L156

But this is strange because it looks like we actually try in
our code to ignore the timeout error already.

—
You are receiving this because you were mentioned.
Reply to this email directly, view
it on GitHub, or mute
the thread.

[cyrilthank]

@erohmensing can you please share some workaround on this?
requesting since while rasa-x is good for feedback from restricted audience, this is handy to deploy on mobile platforms

@selphe i am sorry for hijacking your issue. Can you please share the workaround you are using?

[selphe]

@cyrilthank no problem, but i haven't workaround for this. I thought, this error could be bypassed somehow via settings in credentials.yml

@erohmensing sorry for delay.
You're right, the problem occurs when utter comes from action server (both: utter_message and utter_template).

[cyrilthank]

@selphe Thank you for your response

@erohmensing @selphe can you please confirm if this happens only if we call utter_message and/or utter_template from actions.py
requesting since i was using the utter_* from actions.py as a pre-cursor

Also can you please share the setting in credentials.yml to be used as workaround

[erohmensing]

@cyrilthank I do believe it only happens via actions.py, because it is taking too long to get to the actions server, process, and come back.

As @selphe has said, we assumed there was a workaround, but that workaround doesn't seem to work correctly. This is a bug that doesn't seem to have a workaround at the moment. We will work to get this fixed soon.

[cyrilthank]

Thank you @erohmensing for your response
can you please confirm if this situation is likely to be encountered any time any of the intent in actions.py fires utter_*

Requesting since we are trying to ascertain this before we go too far with the slack integration

Thank you

Cyril

[erohmensing]

Hi @cyrilthank, it doesn’t depend on whether the action does an utterance, just how long that action takes. I believe even an action with no utterance that takes long will lead to this message, and a short action that only utters something will probably be fine.

[cuongnda]

Hi, I'm facing very similar issue. After checking the source code, there is code for handle retry case but I spot this:
https://github.com/RasaHQ/rasa/blob/e002d095134d52afbd4474a77738739957263db6/rasa/core/channels/slack.py#L256
```
retry_reason = request.headers.get("HTTP_X_SLACK_RETRY_REASON")
retry_count = request.headers.get("HTTP_X_SLACK_RETRY_NUM")

When I read the request from slack, it turn out that slack sending the request with different headers ("x-slack-retry-reason" and "x-slack-retry-num") so I changed the code to:
 ```
 retry_reason = request.headers.get("x-slack-retry-reason")
 retry_count = request.headers.get("x-slack-retry-num")

And now it is working as expected. I'm not sure if we can set this text as variable instead of hardcoded string?

[erohmensing]

@cuongnda awesome debugging! It seems that maybe that header name was changed or something, and we didn't catch it.

I'm not sure if we can set this text as variable instead of hardcoded string?

I'm all for making hardcoded values more noticable by making them variables, but in any case, we have to check for a certain value that slack names. What do you have in mind for this?

And now it is working as expected.

Did you configure anything in your credentials to get this to work?

We'd love to accept a PR if you have a fix for this issue -- i think a lot of people have been having trouble with this!

[cuongnda]

What do you have in mind for this?

I think I will add these 2 header name as variable in the credentials configuration, then getting these value from credentials in the source code. If you think it is reasonable approach too, I'll create like that

[erohmensing]

I think know what you mean. Let's try it and I will review the PR :) thanks for working on this!

[tormath1]

Hi @erohmensing,

As you said

I do believe it only happens via actions.py, because it is taking too long to get to the actions server, process, and come back.

In my action server I have a long (~20s.) process, the action was triggered multiple times. With @cuongnda fix, the action is not triggered multiple times. However, my form seems to stay active and each new message send to the bot will trigger the form.

[erohmensing]

However, my form seems to stay active and each new message send to the bot will trigger the form.

Why is there more user input happening without any response from the bot? IMO this is probably a UX problem -- if you have an action that is going to take a long time, you should let the user know (e.g. "Sure, let me get that for you, this might take a few seconds..."

[erohmensing]

Hey @cuongnda, how is the PR going? Would really like to get this fix in ASAP, and am willing to take it over if you don't have the time to work on it.

[cuongnda]

Hi, I've created PR here, please take a look: https://github.com/RasaHQ/rasa/pull/4081

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. Without this, we don't have enough information to help you. Please comment below with the requested information if you still need help.

[playground]

I'm seeing the same issue, the bot is responding three times
'x-slack-retry-num': '3',", 'x-slack-retry-reason': 'http_timeout',",

How to fix this?