Authelia: LDAPS backend can cause Authelia to hang

[rajha-korithrien]

When Authelia is configured to use an ldap backend that uses TLS, for example:

ldaps://ldap-server.backend.com

There are times that concurrent queries to the ldap database hang, with no timeout. The result of which is Node eventually kills Authelia. If running in a Kubernetes environment Kubernetes will redeploy Authelia.

If I switch to using ldap vs. ldaps the problem goes away, this shows the issue to be related to using TLS.

Through some debugging and logging, I have traced the problem to this code in LdapUserDatabase.ts

function getInfo(session: ISession) {
        return Bluebird.join(
          session.searchGroups(username),
          session.searchEmails(username)
        )
        .spread((groups: string[], emails: string[]) => {
          return { groups: groups, emails: emails };
        });
}

In my environment, the searchGroups call never succeeds, fails, nor timesout, even when I set a timeout: 1000 option for the ldapjs client. The searchEmails call always succeeds.

If I change the order of the promises to Bluebird.join, where we say

return Bluebird.join(
  session.searchEmails(username),
  session.searchGroups(username)
)

instead of:

return Bluebird.join(
  session.searchGroups(username),
  session.searchEmails(username)
)

Then the searchEmails call fails and the searchGroups call succeeds. If I put a timeout on the promises, I can get Authelia to gracefully recover from the problem, preventing it from hanging forever, but it doesn't solve the issue.

return Bluebird.join(
  session.searchEmails(username).timeout(2000),
  session.searchGroups(username).timeout(2000)
)

Results in the expected stack trace:

error: date='Sun Jun 23 2019 20:23:28 GMT+0000 (Coordinated Universal Time)' method='POST', path='/api/firstfactor' requestId='73...' sessionId='0S...' ip='w.x.y.z' message='Reply with error 200: request timeout (client interrupt)'
debug: date='Sun Jun 23 2019 20:23:28 GMT+0000 (Coordinated Universal Time)' method='POST', path='/api/firstfactor' requestId='73...' sessionId='0S...' ip='w.x.y.z' message='Error: request timeout (client interrupt)
at /usr/src/server/src/lib/authentication/backends/ldap/LdapUsersDatabase.js:38:36
at tryCatcher (/usr/src/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/usr/src/node_modules/bluebird/js/release/promise.js:517:31)
at Promise._settlePromise (/usr/src/node_modules/bluebird/js/release/promise.js:574:18)
at Promise._settlePromise0 (/usr/src/node_modules/bluebird/js/release/promise.js:619:10)
at Promise._settlePromises (/usr/src/node_modules/bluebird/js/release/promise.js:695:18)
at _drainQueueStep (/usr/src/node_modules/bluebird/js/release/async.js:138:12)
at _drainQueue (/usr/src/node_modules/bluebird/js/release/async.js:131:9)
at Async._drainQueues (/usr/src/node_modules/bluebird/js/release/async.js:147:5)
at Immediate.Async.drainQueues [as _onImmediate] (/usr/src/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)'

The problem is not with Authelia, but with the ldapjs client it is using. Some searching via google seems to indicate there is a problem in the ldapjs client that causes this issue:

• joyent/node-ldapjs#329
• joyent/node-ldapjs#424
• joyent/node-ldapjs#463

I am posting here so other folks can track the issue. Hopefully joyent will fix the problem. A workaround in the mean time is to change getInfo to make its calls serially instead of concurrently. It is ugly but I can confirm that this code (which forces a single query at a time) works:

async function getInfo(session: ISession) {
  that.logger.debug("LDAP Database: getInfo in checkUserPassword called.");
  const emails: string[] = await session.searchEmails(username).catch(function (err: Error) {
    that.logger.debug("LdapUserDatabase: getInfo: Failed to get e-mail addresses for user: %s due to %s", username, err);
    throw new AuthenticationError("Unable to get e-mail address for user: " + username);
  });
  const groups: string[] = await session.searchGroups(username).catch(function (err: Error) {
    that.logger.debug("LdapUserDatabase: getInfo: Failed to get groups for user: %s due to %s", username, err);
    throw new AuthenticationError("Unable to get groups for user: " + username);
  });
  that.logger.debug("LDAP Database: getInfo in checkUserPassword for user %s got groups: %s and emails: %s",
      username, groups, emails);
  return {groups: groups, emails: emails};
}

[clems4ever]

Hello @rajha-korithrien , thank you for reporting that issue. I discussed Ldapjs with @nightah earlier and he proposed to remove the dependency in favor of another LDAP library. In the meantime I rewrote almost entirely Authelia in Go so I think I'm not gonna bump but move forward with a Go implementation. Hopefully, it's gonna be better.
Regarding your work on LDAPS, I will include it in the new version, don't worry.

[naggie]

move forward with a Go implementation.

Great! This would presumably reduce the size of the dependency tree that npm currently brings in, as well as allowing Authelia to ship as one executable with bundled assets. Is there a document or issue somewhere to discuss the migration and goals?

[clems4ever]

Hello @naggie , there is no open documentation about the goals except the commit logs of the branch holding the dev (authelia-go). The goals are mainly:

• Improve development lifecycle using a typed language.
• Improve performance.
• I know more system developers doing Go than Typescript. So I expect more contributions.
• Be able to ship one binary instead of npm installing everytime.
• Package less (hidden) dependencies for security reasons.

[clems4ever]

Ldaps should be easy to implement now that the first release of the Go version is coming soon. Stay tuned.

[naggie]

I forgot you were developing a Go version. I'm looking forward to the release, go is much better fit for distribution and dependency management!

[clems4ever]

This should add LDAPS support in v4: https://github.com/clems4ever/authelia/pull/477. I won't fix the problem in v3 though, sorry.