Rocket.chat: [3.6.0] Regression : LDAP "User not found or incorrect password"

[didierm]

Description:

Upgrading from v3.5.4 to v3.6.0 results in LDAP users not able to log in : the error popup "User not found or incorrect password" is displayed.
Local users are still able to log in.

Steps to reproduce:

1. Go to login screen
2. Enter LDAP username and password
3. Click [Login] button

Expected behavior:

User should be logged in, as in all previous release versions up to v3.5.4.

Actual behavior:

Error popup : "User not found or incorrect password"

Server Setup Information:

• Version of Rocket.Chat Server: 3.6.0
• Operating System: CentOS 8.2.2004
• Deployment Method: tar
• Number of Running Instances: 1
• DB Replicaset Oplog: yes
• NodeJS Version: v12.18.3
• MongoDB Version: 4.0.3

Client Setup Information

• Desktop App or Browser Version: Firefox 80.0
• Operating System: Linux Fedora 32

Additional context

• Having successfully upgraded today through releases 2.4.11 -> 3.0.0 -> 3.0.12 -> 3.1.3 -> 3.2.2 -> 3.3.3 -> 3.4.3 -> 3.5.4, this issue exhibits itself exclusively on the latest release 3.6.0.
• The LDAP user/password lookup query returns no errors (cfr. logs).
• The LDAP Synchronisation yields an error (cfr. logs) : error: 'file-too-small'

Relevant logs:

Please find relevant logs (potentially sensitive data obfuscated) in attachment.
_(Log_level=2 , LDAP_Internal_Log_Level=info)_

Debug log sections :

• Admin > LDAP > [Test connection]
• Admin > LDAP > [Execute Synchronization Now]
• LDAP login (with valid credentials)

For comparison, the following debug log sections were also added :

• LDAP login (invalid password)
• LDAP login (invalid user)
• Local user login

RC-3.6.0_LDAP_login.txt

[arpsyapathy]

+1 !!!

[danielnachtrub]

Same issue here - Logon seems to be successful on ldap backend but doesn't pass on frontend.

[netpages]

+1 Same issue here!

[didierm]

As the confirmations are starting to drop in, should this not be considered a blocker ?
Did any of you @arpsyapathy @danielnachtrub @netpages install one of the 3.6.0rc's ?

[danielnachtrub]

We didn't install RC - we're only upgrading to release builds.

[didierm]

We not testing RC's is part of the problem, I guess. :)

[netpages]

No, we're also only upgrading to release builds...

[priethor]

Same issue here with multiple 3.6.0 docker images.

[Zuko]

No one installs RC's. Same issues over and over again. :disappointed:

[aleadco]

Hi, same problem with my instance. Manual installation and without going through RC, direct to the stable version

I20200831-14:34:00.972(-4) server.js:204 API ➔ debug POST: /api/v1/method.callAnon/login 
I20200831-14:34:00.973(-4) server.js:204 LDAPHandler ➔ info Init LDAP login usuario 
I20200831-14:34:00.974(-4) server.js:204 LDAP ➔ Connection.info Init setup 
I20200831-14:34:00.977(-4) server.js:204 LDAP ➔ Connection.info Connecting ldaps://hostname:636 
I20200831-14:34:00.978(-4) server.js:204 LDAP ➔ Connection.debug connectionOptions {   url: 'ldaps://hostname:636',   timeout: 60000,   connectTimeout: 1000,   idleTimeout: 1000,   reconnect: true,   log: Logger {     _events: [Object: null prototype] {},     _eventsCount: 0,     _maxListeners: undefined,     _level: 30,     streams: [ [Object] ],     serializers: null,     src: false,     fields: { name: 'ldapjs', component: 'client', hostname: 'SMI', pid: 7835 },     [Symbol(kCapture)]: false   },   tlsOptions: { rejectUnauthorized: false } } 
I20200831-14:34:01.104(-4) server.js:204 LDAP ➔ Connection.info LDAP connected 
I20200831-14:34:01.105(-4) server.js:204 LDAP ➔ Bind.info Binding UserDN [email protected] 
I20200831-14:34:01.159(-4) server.js:204 LDAP ➔ Search.info Searching user usuario 
I20200831-14:34:01.160(-4) server.js:204 LDAP ➔ Search.debug searchOptions {   filter: '(&(&(objectClass=user)(objectCategory=person)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))(|(sAMAccountName=fqestrella)(mail=fqestrella)))',   scope: 'sub',   sizeLimit: 1000,   paged: { pageSize: 250, pagePause: false } } 
I20200831-14:34:01.161(-4) server.js:204 LDAP ➔ Search.debug BaseDN dc=dominio,dc=ldap,dc=net 
I20200831-14:34:01.169(-4) server.js:204 LDAP ➔ Search.info Search result count 1 
I20200831-14:34:01.170(-4) server.js:204 LDAP ➔ Auth.info Authenticating CN=uid,OU=sub,DC=dominio,DC=ldap,DC=net 
I20200831-14:34:01.303(-4) server.js:204 LDAP ➔ Search.info Search result count 1 
I20200831-14:34:01.304(-4) server.js:204 LDAP ➔ Auth.info Authenticated CN=uid,OU=sub,DC=dominio,DC=ldap,DC=net 
I20200831-14:34:01.306(-4) server.js:204 LDAPHandler ➔ info Querying user 
I20200831-14:34:01.307(-4) server.js:204 LDAPHandler ➔ debug userQuery { 'services.ldap.id': 'f642db03e242194a9fcb855072c6c2b3' } 
I20200831-14:34:01.314(-4) server.js:204 LDAPHandler ➔ info Logging user 
I20200831-14:34:01.314(-4) server.js:204 LDAPSync ➔ info Syncing user data 
I20200831-14:34:01.316(-4) server.js:204 LDAPSync ➔ debug user { email: undefined, _id: 'hDtftw4hxjQvc4qNT' } 
I20200831-14:34:01.317(-4) server.js:204 LDAPSync ➔ debug ldapUser undefined 
I20200831-14:34:01.318(-4) server.js:204 LDAPSync ➔ debug not syncing user roles 
I20200831-14:34:01.319(-4) server.js:204 LDAPSync ➔ debug not syncing groups to channels 
I20200831-14:34:01.321(-4) server.js:204 LDAPSync ➔ info Syncing user avatar 
I20200831-14:34:01.324(-4) Failed login detected - Username[unknown] ClientAddress[10.8.160.24] ForwardedFor[undefined] XRealIp[10.8.160.24] UserAgent[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0] 
I20200831-14:34:01.326(-4) Exception while invoking method login Error: File size (size = 0) is too small (min = 1) [file-too-small]     at Object.fileTooSmallError (packages/jalik:ufs/ufs-filter.js:43:53)     at Filter.check (packages/jalik:ufs/ufs-filter.js:89:28)     at FileUploadClass.insert (app/file-upload/server/lib/FileUpload.js:586:11)     at DDPCommon.MethodInvocation.<anonymous> (app/ldap/server/sync.js:418:15)     at packages/dispatch_run-as-user.js:211:14     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object.Meteor.runAsUser (packages/dispatch_run-as-user.js:210:33)     at syncUserData (app/ldap/server/sync.js:417:11)     at MethodInvocation.<anonymous> (app/ldap/server/loginHandler.js:124:3)     at packages/accounts-base/accounts_server.js:462:31     at tryLoginMethod (packages/accounts-base/accounts_server.js:1291:14)     at AccountsServer._runLoginHandlers (packages/accounts-base/accounts_server.js:460:22)     at AccountsServer.Accounts._runLoginHandlers (app/lib/server/lib/loginErrorMessageOverride.js:7:35)     at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:520:31)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20200831-14:34:01.326(-4) server.js:204 API ➔ debug Success {   statusCode: 200,   body: {     message: '{"msg":"result","id":"7","error":{"isClientSafe":true,"error":"file-too-small","reason":"File size (size = 0) is too small (min = 1)","message":"File size (size = 0) is too small (min = 1) [file-too-small]","errorType":"Meteor.Error"}}',     success: true   } } 
I20200831-14:34:02.302(-4) server.js:204 LDAP ➔ Search.info Idle 
I20200831-14:34:02.303(-4) server.js:204 LDAP ➔ Connection.info Disconecting

[adgerth]

Same Issue here. I found the logins to be working again after disabling the avatar sync in
Administration --> LDAP --> Sync / Import --> Toggle 'Sync User Avatar' to off.

[aleadco]

Same Issue here. I found the logins to be working again after disabling the avatar sync in
Administration --> LDAP --> Sync / Import --> Toggle 'Sync User Avatar' to off.

Well, yes, I have checked in my instance and with deactivating the avatar synchronization you can now log in, a function that was executed correctly in previous versions.

[habnux]

This issue stops us from updating to 3.6.0. The mentioned workaround works but is something we don't want in production.

values.yaml from Helm deployment
OVERWRITE_SETTING_LDAP_Sync_User_Avatar: false

[netpages]

Same Issue here. I found the logins to be working again after disabling the avatar sync in
Administration --> LDAP --> Sync / Import --> Toggle 'Sync User Avatar' to off.

I can also confirm that the login will work again after deactivating the avatar sync.

[ankar84]

@pierre-lehnen-rc @sampaiodiego can you take a look at that critical issue?

[hpvb]

I have the same issue after upgrading to 3.6.0. Since I use LDAP sync in combination with oauth I didn't have the login issues but avatar synchronization is broken for me.

[capricorn72]

Same problem here after upgrade to 3.6.0
LDAP error is no LDAP error, this is a fail with the avatar:
When uploading an avatar manually an error occurs, too.

[r4nc0r]

This issue existed in 2017 #7405
Maybe the Problem is related?

[capricorn72]

No, nothing to do with it.
When disabling "thumbnail sync" for avatars, login via samaccountname works fine.
The error root is in saving avatar pics, browser throws error too when trying to upload an avatar pic manually.

[r4nc0r]

but this was also the fix for the last issue: https://github.com/RocketChat/Rocket.Chat/issues/7405#issuecomment-313373402

[masterbbgit]

+1 Same issue here!
When disabling "thumbnail sync" for avatars, login via samaccountname works fine.

[pbexiga]

same issue here - we have implemented the workaround - but eagerly anticipate a fix.

I can provide logs if needed.

Thanks!

[VIP-Gun]

Facing the same issue after a local install of 3.6.0 via Docker.
Currently using the workaround mentioned, i.e.:
Administration --> LDAP --> Sync / Import --> Toggle 'Sync User Avatar' to off.

[jtorrex]

Same issue for me after upgrading to the latest release via .tar.

[ClundXIII]

While having the described faulty configuration, we had a few new user being pulled from LDAP. They are now half-created in the database without any roles. Editing them via the Admin Panel shows a white page, using the API Leads to "error":"Cannot read property 'indexOf' of undefined". I cannot find any related logs. Logging in fails with User has no roles.

Is there a way to remove those user?

[seppultra]

They are now half-created in the database without any roles. Editing them via the Admin Panel shows a white page, using the API Leads to "error":"Cannot read property 'indexOf' of undefined". I cannot find any related logs. Logging in fails with User has no roles.

We faced the same issue. It seems this happens when a channel is set to "default" so it will assign to the user while or after sync. Disabling the default value on every channel, your user should be synced correctly.
I have to investigate further but i think it's worth a new bug report.

[hpvb]

The problem is still there in 3.6.1

[steffen-kdab]

Same problem here after upgrading to 3.6.0, and still a problem in 3.6.1.
Disabling "Sync User Avatar" fixes the problem, but curiously, users can still log in using their email address instead of username, even if "Sync User Avatar" is enabled.

[Skygge666]

I confirm the issue in 3.6.1
Users cannot login with "User not found or incorrect password" message.
But some users can login using old password, if they changed it meanwhile (me, for example).
I'm on 389-ds.

[bbrendon]

I had what I think is this issue. (Login only worked with email and no longer with username). I'm on AD. I changed my ldap filter to match more of what is in the docs and I was up and running. I didn't see this issue until after I resolve my problem so I'm not sure about the avatar thing.

[arpsyapathy]

I had what I think is this issue. (Login only worked with email and no longer with username). I'm on AD. I changed my ldap filter to match more of what is in the docs and I was up and running. I didn't see this issue until after I resolve my problem so I'm not sure about the avatar thing.

Hi! Can you show your filter?

[jtorrex]

I had what I think is this issue. (Login only worked with email and no longer with username). I'm on AD. I changed my ldap filter to match more of what is in the docs and I was up and running. I didn't see this issue until after I resolve my problem so I'm not sure about the avatar thing.

In my case I experienced the same, but users still logging by username and fail if they try to login by email. We are using LDAP, and why the Filters configuration changed? Maybe something has changed on a RocketChat release about LDAP/AD configuration?

[nguyenorlab]

I also had the same issue and cannot be resolved yet.. :(

[hpvb]

I'll just share my LDAP config here, perhaps it'll help:

LDAP Server: ip address
SSL: no (containers with direct connections)

BaseDN: ou=users,dc=company,dc=com
Username field: uid
Unique identifier field: entryUUID
Default domain: company.com
Merge existing users: on
Sync user data: on
User data field map: {"cn":"name", "mail":"email"}
Sync LDAP groups: on
Auto remove user roles: on
User group filter: (&(cn=#{groupName})(member=uid=#{username},ou=users,dc=company,dc=com))
LDAP group basedn: ou=groups,dc=company,dc=com
User Data Group Map: {"rocket.chat-admin": "admin"}
Auto sync ldap groups to channels: on
Channel admin: rocket.cat
LDAP Group Channel Map: {"several": "mappings"}
Auto remove users from channels: on
Sync user avatar: Off (normally on)
User Avatar Field: Default (also when turned on)
Background sync: on
Background sync interval: every 10 minutes
Background Sync Import New Users: on
Background Sync Update Existing Users: on
User Search Filter: (&(objectclass=inetOrgPerson)(memberOf=cn=rocket.chat,ou=apps,ou=groups,dc=company,dc=com))
scope: sub
search field: uid

[arpsyapathy]

3.6.2
The problem is still not resolved.
Please help me solve the problem with LDAP. This is very important
@sampaiodiego

Thanks in advanced

[ankar84]

3.6.2
The problem is still not resolved.
Please help me solve the problem with LDAP. This is very important
@sampaiodiego

Thanks in advanced

Totally agree with you!
That problem is really critically important for me too.
And now I can't upgrade to 3.6.x version from 3.1.1 because of that issue and pdf rendering problem.
@rodrigok @sampaiodiego please help us with that problem.

[pierre-lehnen-rc]

PR #18948 will prevent errors on the avatar from blocking the login.
I'm working on a fix for the avatar problem itself.

[ankar84]

PR #18948 will prevent errors on the avatar from blocking the login.
I'm working on a fix for the avatar problem itself.

Glad to hear that, @pierre-lehnen-rc
And thanks a lot for you great job!!

[Skygge666]

Still not resolved in 3.6.2.
Waiting for the fix.

[ankar84]

Still not resolved in 3.6.2.
Waiting for the fix.

I think it will be soon in 3.6.3 version.
Hope avatar bug itself will be fixed too.

[TwizzyDizzy]

Just rolled out Rocket.Chat 3.7.0-rc.0 to my development instance. Avatar sync is enabled. LDAP login is now possible again for me. Beware of https://github.com/RocketChat/Rocket.Chat/issues/18987 though - stumbled over this when rolling out the new version.

Cheers
Thomas

[damyan]

Still not being able to login after upgrading to 3.6.3....

[Skygge666]

3.6.3 - works for me. THX!

[VIP-Gun]

Version 3.6.3 fixed the login issue with LDAP avatar sync for me.

[jtorrex]

3.6.3 also works for me. Thank you all!

[hpvb]

Fixed for me also with 3.6.3

[sampaiodiego]

@damyan can you please provide more details? or maybe create another issue? thx

[damyan]

@damyan can you please provide more details? or maybe create another issue? thx

Sorry. Upgrading from 3.5.4. to the buggy version 3.6.1 must somehow have messed up my data (mongo). Neither upgrading to 3.6.2, nor to 3.6.3 or 3.7.0 helped - I kept getting the "LDAP user not found" message above. After wiping all the data and restoring it from a production instance, the upgrade to 3.7.0 went smoothly, I was able to login with my LDAP user.

Sorry about the noise and thank you for the good work!

--
Damyan

[Ramhm]

Hi all,
Rocket.chat version: 3.7.0
Rocket.Chat Config:

BaseDN: cn=users,cn=accounts,dc=company,dc=com
Username field: uid
Unique identifier field: entryUUID
Default domain: company.com
Sync user data: on
User data field map: {"cn":"name", "mail":"email"}
Sync LDAP groups: on
User group filter: (&(cn=#{groupName})(member=uid=#{username},ou=users,dc=company,dc=com))
LDAP group basedn: ou=groups,dc=company,dc=com
User Data Group Map:{"rocket-admin": "admin","tech-support": "support"}
Sync user avatar: Off (normally on)
Background sync: on
Background sync interval: every 5 minutes
Background Sync Import New Users: on
Background Sync Update Existing Users: on
User Search Filter: (&(objectclass=inetOrgPerson)(memberOf=,ou=groups,dc=company,dc=com))
scope: sub
search field: uid
Group ObjectClass:groupOfNames
Group ID Attribute: cn
Group Member Attribute:Member
Group Member Format:Member
Group name:rocket-admin

All FREIPA users are in the rocket-admin group.

I am using freeipa service for ldap and the error is displayed below.
Error: server.js:204 LDAPHandler ➔ error Error: User not Found

Does anyone have a config for freeipa?
Please check this item.

[Sx3]

Same issue here with snap version 3.6.2