Botbuilder-js: Error: BotFrameworkAdapter.processActivity(): 401 ERROR

[janandreschweiger]

Versions

botbuilder: 4.9.2 (Latest)
node: 12.18.2 (Latest)
OS: Ubuntu (Docker-Container)

Describe the bug

One of a sudden adapter.processActivity fails with these two error messages when sending a chat-message to the bot:

1. Error Message:
JwtTokenExtractor.getIdentity:err! ReferenceError: XMLHttpRequest is not defined dev_server | at /var/huforce/node_modules/whatwg-fetch/dist/fetch.umd.js:484:17 ... dev_server | at /var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:14:71 dev_server | at new Promise (<anonymous>) dev_server | at __awaiter (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:10:12) dev_server | at OpenIdMetadata.getKey (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:30:16) dev_server | at JwtTokenExtractor.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/jwtTokenExtractor.js:86:56)

2. Error Message:
(node:18) UnhandledPromiseRejectionWarning: Error: BotFrameworkAdapter.processActivity(): 401 ERROR dev_server | ReferenceError: XMLHttpRequest is not defined dev_server | at /var/huforce/node_modules/whatwg-fetch/dist/fetch.umd.js:484:17 dev_server | at new Promise (<anonymous>) dev_server | at fetch (/var/huforce/node_modules/whatwg-fetch/dist/fetch.umd.js:477:12) dev_server | at OpenIdMetadata.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:54:31) dev_server | at Generator.next (<anonymous>) dev_server | at /var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:14:71 dev_server | at new Promise (<anonymous>) dev_server | at __awaiter (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:10:12) dev_server | at OpenIdMetadata.refreshCache (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:53:16) dev_server | at OpenIdMetadata.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:34:32) dev_server | at Generator.next (<anonymous>) dev_server | at /var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:14:71 dev_server | at new Promise (<anonymous>) dev_server | at __awaiter (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:10:12) dev_server | at OpenIdMetadata.getKey (/var/huforce/node_modules/botframework-connector/lib/auth/openIdMetadata.js:30:16) dev_server | at JwtTokenExtractor.<anonymous> (/var/huforce/node_modules/botframework-connector/lib/auth/jwtTokenExtractor.js:86:56) dev_server | at BotFrameworkAdapter.<anonymous> (/var/huforce/node_modules/botbuilder/lib/botFrameworkAdapter.js:674:27) dev_server | at Generator.throw (<anonymous>) dev_server | at rejected (/var/huforce/node_modules/botbuilder/lib/botFrameworkAdapter.js:12:65) dev_server | at processTicksAndRejections (internal/process/task_queues.js:97:5)

Details

I created my bot via the App-Studio. I don't have an azure subscription. I neither use the emulator. The bot runs locally. I am using ngrok to tunnel the requests to my PC.

What I have tried

• I tried various versions of your (awesome) package
• I checked the client-id & the client-secret
• I created a new client-secret
• I verified my client-id & my client-secret
• I checked my messaging endpoint
• I verified the system time
• I created a new bot via the App Studio and the https://dev.botframework.com
• Installing xmlhttprequest manually

Code:

`
...
const bot = new Bot();
export const adapter = new BotFrameworkAdapter({
appId: msteams.CLIENT_ID,
appPassword: msteams.CLIENT_SECRET
});
MicrosoftAppCredentials.trustServiceUrl(
"https://smba.trafficmanager.net/emea/"
);
...
expressApp.post("/bot", secure((req, res) => {
adapter.processActivity(req, res, async context => { // <-- error
await bot.run(context);
});
}));

`

I am desperate, because we need to deliver a new version of our bot. But, I just can't fix this issue.
I really appreciate any help.

[bug]

[compulim]

@anusharavindrar could you take a look at this?

[ace-kay-law-neo]

I have the same issue. Any ideas?

[anusharr]

@janandreschweiger, I will attempt a repro on my end and update the thread with my findings. Also, do you find any additional errors on the channels tab in dev.botframework.com?
@geckgecko, are you also getting the same error messages when trying to communicate with the bot created via App Studio?

[janandreschweiger]

@anusharavindrar No there are neither error messages nor warnings.
Thanks for your support!

[anusharr]

@janandreschweiger, thanks for the above info. Also, any specific reason behind using such a setup and not use something like offline-directline if you don't want to have an Azure subscription? Can you elaborate more on what you are trying to accomplish and are you following any specific sample or are building a new bot from scratch as per your requirements?

[janandreschweiger]

@anusharavindrar the bot is deployed to the servers of our clients. The bot is running inside a docker container and needs a public ip. In order to simulate a production environment we use ngrok to tunnel the requests to our local PC (private ip). Our clients have a public ip and want to host the bot themselves.

[janandreschweiger]

Today something strange happened. I have an old (cached) docker container from my bot. This version works well. But when running the same old code in a new docker the error occurs. I tried different variants of docker-images. I can't fix this issue. I tried different combinations of package versions and docker images (node versions). Installing the xmlhttprequest manually didn't solve the issue either. However, I now know that the key issue doesn't correlate with this one.

[ace-kay-law-neo]

@anusharavindrar yes I get the exact same error message. I have created the my bot via the botframework webpage. I experience this error after adding this package: https://www.npmjs.com/package/cld3-asm. However, removing the package doesn't resolve the issue.
I used these samples https://github.com/microsoft/BotBuilder-Samples/tree/master/samples/javascript_nodejs for creating my bot.

If you need any more information or any help just let me know. Thank you!!!

[ace-kay-law-neo]

Ok, I now know how to fix the error. But it is extremely strange.

Error:
...
const expressApp = express();

import botbuilder from "botbuilder";
const { BotFrameworkAdapter } = botbuilder;
...

No Error:
...
import botbuilder from "botbuilder";
const { BotFrameworkAdapter } = botbuilder;

const expressApp = express();
...

Why on earth do I have to import the botbuilder before initializing express? Can someone please explain this to me.

[janandreschweiger]

@geckgecko Thank you! This solved my issue.
@anusharavindrar Do you have any idea why botbuilder throws an error?

This should be fixed or at least mentioned in the docs.

[ace-kay-law-neo]

@janandreschweiger I totally agree on this.

[anusharr]

@geckgecko & @janandreschweiger, glad to hear that the above method helped to fix the issue.
@geckgecko, can you confirm that you didn't get any error before adding the cld3-asm package?
It does look like that botbuilder and express are having some kind of overlap or conflict which is throwing up these errors. It is recommended that you make use of just 'restify' like in our samples and give it a try. Also, can you try downgrading the botbuilder version and see if you still get the error?

[ace-kay-law-neo]

@anusharavindrar Yes I can confirm. However, I removed this package (also from the cache) and I still got the error.
I downgraded the botbuilder to 4.8.0, makes no difference. Unfortunately, we can't move to restify, as we have graphql (/graphql), file-serving (/serve) and our bot (/bot) on the same server. Can you have a look, why this error happens?

[janandreschweiger]

@geckgecko same. I use express, because I am hosting different services on the same server.
@anusharavindrar I'd appreciate it, if you could take a closer look on this. It is very counterintuitive that botbuilder needs to be imported before initializing express. Now I have to combine the initialization of express and the bot into a single file.

[anusharr]

@geckgecko @janandreschweiger, thanks for the above info. I will update the thread soon with my findings.

[v-kydela]

@janandreschweiger
@geckgecko

I can try to help determine the cause of this issue. I can't reproduce it yet so I will need your help with experimentation to narrow down the possible factors.

1. Does the error occur when you use Emulator instead of Teams?
2. If so, does the error occur when the bot has no app ID or password?
3. Does the error occur when you don't use a Docker container?
4. Does the error occur when you take secure out of your request handler?

Here is my index.js if you'd like to start there and see if you can get the error using the same code as me:

const dotenv = require('dotenv');
const path = require('path');
const express = require('express');
const app = express();

// Import required bot services.
// See https://aka.ms/bot-services to learn more about the different parts of a bot.
const { BotFrameworkAdapter } = require('botbuilder');

// This bot's main dialog.
const { EchoBot } = require('./bot');

// Import required bot configuration.
const ENV_FILE = path.join(__dirname, '.env');
const PORT = 3978;

dotenv.config({ path: ENV_FILE });

app.listen(PORT);

console.log(`Listening on ${PORT}...`);

// Create adapter.
// See https://aka.ms/about-bot-adapter to learn more about how bots work.
const adapter = new BotFrameworkAdapter({
    appId: process.env.MicrosoftAppId,
    appPassword: process.env.MicrosoftAppPassword
});

// Catch-all for errors.
adapter.onTurnError = async (context, error) => {
    // This check writes out errors to console log .vs. app insights.
    console.error(`\n [onTurnError]: ${error}`);
    // Send a message to the user
    await context.sendActivity(`Oops. Something went wrong!`);
};

// Create the main dialog.
const bot = new EchoBot();

// Listen for incoming requests.
app.post('/api/messages', (req, res) => {
    adapter.processActivity(req, res, async (context) => {
        // Route to main dialog.
        await bot.run(context);
    });
});

[v-kydela]

@janandreschweiger - Are you still working on this?

[v-kydela]

I am closing this issue due to inactivity. We are still happy to provide help if you need any, though it may be a good idea to make a new post if that's the case. Please have a look at this to find out the best places to ask for help, and consider asking questions on Stack Overflow.