Harbor: LDAP Search Operations Error

[cavemandaveman]

docker engine 1.11.2 and docker-compose 1.9.0
tried on both 0.4.5 and 0.5.0-rc1 of harbor

harbor.cfg

auth_mode = ldap_auth
ldap_url = ldap://example.com:389
ldap_searchdn = CN=powerusr,OU=Admin,DC=example,DC=com
ldap_search_pwd = secret
ldap_basedn = DC=example,DC=com
ldap_filter = (objectClass=person)
ldap_uid = sAMAccountName
ldap_scope = 3

When trying to login with the user myusr (or any AD user), Harbor throws an error

log

harbor-ui     | 2016-11-21T19:36:26Z [DEBUG] [authenticator.go:57]: Current AUTH_MODE is ldap_auth
harbor-ui     | 2016-11-21T19:36:26Z [DEBUG] [ldap.go:53]: ldapURL:ldap://example.com:389
harbor-ui     | 2016-11-21T19:36:26Z [DEBUG] [ldap.go:64]: baseDn:DC=example,DC=com
harbor-ui     | 2016-11-21T19:36:26Z [DEBUG] [ldap.go:68]: Search DN: CN=powerusr,OU=Admin,DC=example,DC=com
harbor-ui     | 2016-11-21T19:36:26Z [DEBUG] [ldap.go:84]: one or more filter(&(objectClass=person)(sAMAccountName=myusr))
harbor-ui     | LDAP::Search() error : 1 (Operations error)
harbor-ui     | 2016-11-21T19:36:26Z [ERROR] [base.go:162]: Error occurred in UserLogin: LDAP::Search() error : 1 (Operations error)

As far as I can tell, my config is correct, because I can do an ldapsearch with those same parameters and have it return the correct info.

[hainingzhang]

it the example.com configured in /etc/hosts file? this may not be recognized by the UI component in container.

[baoxingliu]

i have the same error, i use ip in ldap_url

[reasonerjt]

@cavemandaveman could you show us the ldapsearch cmd you used to successfully verify the credentials?

[baoxingliu]

@reasonerjt i can show you my command,could you help me
ldapsearch -x -D '[email protected]' -w 'xxxx' -b 'dc=even-inc,dc=com,dc=cn' -H 'ldap://192.168.100.15:389' '(sAMAccountName=baoxingliu)'
it return the correct info,but when i login from harbor's ui,harbor throws the same error

[baoxingliu]

and here is my configuration:
auth_mode = ldap_auth
ldap_url = ldap://192.168.100.15:389
ldap_searchdn = [email protected]
ldap_search_pwd = xxxxxx
ldap_basedn = dc=even-inc,dc=com,dc=cn
ldap_uid = sAMAccountName
ldap_scope = 3

[cavemandaveman]

@hainingzhang I do not need it in my hosts file, my AD host url is resolvable on the lan.

@reasonerjt here is the successful ldapsearch:
ldapsearch -D "CN=powerusr,OU=Admin,DC=example,DC=com" -w secret -p 389 -h example.com -b "DC=example,DC=com" -s sub "(&(objectClass=person)(sAMAccountName=myusr))"

I can exec into the ui container (docker exec -it harbor-ui sh) and run that cmd successfully

[knackaron]

The issue appears to be with using the base component of the LDAP hierarchy as the base DN and having the scope set to 2 or 3. The operations error is coming from the AD server after the search account successfully binds.

By changing the base DN to a top-level OU (e.g., ou=something,dc=example,dc=com), it works fine. I'm working around it now with this, but it's not ideal as Harbor will have a different LDAP config than everything else in my environments that query LDAP. Not sure why the behavior difference between the openldap utils (e.g., ldapsearch) and whichever LDAP library Harbor is using.

broken:

Nov 22 21:01:24 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:01:24Z [DEBUG] [authenticator.go:57]: Current AUTH_MODE is ldap_auth
Nov 22 21:01:24 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:01:24Z [DEBUG] [ldap.go:53]: ldapURL:ldap://ad01.example.com
Nov 22 21:01:24 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:01:24Z [DEBUG] [ldap.go:64]: baseDn:dc=example,dc=com
Nov 22 21:01:24 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:01:24Z [DEBUG] [ldap.go:68]: Search DN: cn=site01-harbor-bind,ou=service-accounts,ou=site01,ou=sites,dc=example,dc=com
Nov 22 21:01:24 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:01:24Z [DEBUG] [ldap.go:84]: one or more filter(&(objectClass=person)(sAMAccountName=aparsons))
Nov 22 21:01:24 dkr01.site01.example.com docker-current/ui[10726]: LDAP::Search() error : 1 (Operations error)
Nov 22 21:01:24 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:01:24Z [ERROR] [base.go:162]: Error occurred in UserLogin: LDAP::Search() error : 1 (Operations error)

working (note the difference in base DN):

Nov 22 21:05:55 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:05:55Z [DEBUG] [authenticator.go:57]: Current AUTH_MODE is ldap_auth
Nov 22 21:05:55 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:05:55Z [DEBUG] [ldap.go:53]: ldapURL:ldap://ad01.example.com
Nov 22 21:05:55 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:05:55Z [DEBUG] [ldap.go:64]: baseDn:ou=people,dc=example,dc=com
Nov 22 21:05:55 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:05:55Z [DEBUG] [ldap.go:68]: Search DN: cn=site01-harbor-bind,ou=service-accounts,ou=site01,ou=sites,dc=example,dc=com
Nov 22 21:05:55 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:05:55Z [DEBUG] [ldap.go:84]: one or more filter(&(objectClass=person)(sAMAccountName=aparsons))
Nov 22 21:05:55 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:05:55Z [DEBUG] [ldap.go:109]: found entry:{ 0 CN=Aron Parsons,OU=people,dc=example,dc=com [{cn [Aron Parsons]} {mail [[email protected]]}]}
Nov 22 21:05:55 dkr01.site01.example.com docker-current/ui[10726]: 2016-11-23T02:05:55Z [DEBUG] [ldap.go:132]: username:aparsons,email:[email protected]

[baoxingliu]

@knackaron great,it works for me too

[cavemandaveman]

@knackaron I can also confirm that your solution is working. Not ideal, but usable as a temporary fix.

[reasonerjt]

Seems the same issue as this one:
http://stackoverflow.com/questions/6222641/how-to-php-ldap-search-to-get-user-ou-if-i-dont-know-the-ou-for-base-dn

[hainingzhang]

We can probably allow users to set the option LDAP_OPT_REFERRALS for Active Directory.
https://github.com/mqu/openldap/blob/master/defines.go

More info on the LDAP_OPT_REFERRALS can be found at microsoft's page:
https://technet.microsoft.com/en-ca/library/cc755809(v=ws.10).aspx

[cavemandaveman]

I attempted to hardcode this option to test it out (https://github.com/cavemandaveman/harbor/commit/225d51fb69df3ab237384956e2136ce35bd947fb), however it is not working as expected. After building a dev image, I get the same results as when the option is not set.

[cavemandaveman]

One other workaround I have found is setting your ldap_url port to the Global Catalog port.

For example, mine would now be ldap_url = ldap://example.com:3268. If you use ldaps, you can use port 3269.

Here's some more context about the Global Catalog: https://technet.microsoft.com/en-us/library/cc978012.aspx

[hainingzhang]

@cavemandaveman Thanks for trying out the code. Maybe it is not actually set by the ldap library we use.

The global catalog seems a workaround, we will have more investigation to see if we should document this.

[yhua123]

@cavemandaveman
Your solution worked in my scenario. Thanks~
I feel there still remain some problems with Harbor. I thought that caused by search process of AD. Harbor client sent the search request for AD server, the respond returned overtime or broken by something. Why ldapsearch response was timely, I guess some default configurations is different. Harbor's client configuration defined in https://github.com/mqu/openldap/blob/master/defines.go, but I still didn't have any clue to this.
In my scenario have one AD server (win server 2012).
But I think there still remain some problems with Harbor. I thought that caused by search process of AD. Harbor client sent the search request for AD server, the respond returned overtime or broken by something. Why ldapsearch return very fast, I guess some default configurations are different. Harbor's client configuration defined in https://github.com/mqu/openldap/blob/master/defines.go, but I still didn't have any clue on this.
In my scenario have one AD server (win server 2012).
Using ldapsearch get response very fast:
ldapsearch -x -D 'username' -w 'password' -b 'dc=harbor,dc=com' -H 'ldap://ipaddress:389' -s sub '(sAMAccountName=username)'
But Harbor login response more than 6 minutes (found one UI issue, not handle LDAP response overtime, fixed in 0.5.0-rc2). I compiled the example code from https://github.com/mqu/openldap (Harbor references this package), meet same problem.
./ldapsearch sAMAccountName=username -base dc=harbor,dc=com -host ldap://ipaddress:389/ -passwd password -user username
I tried @knackaron solution, add OU to baseDN, response in 1 secound:
./ldapsearch sAMAccountName=username -base ou=site,dc=harbor,dc=com -host ldap://ipaddress:389/ -passwd password -user username
change the port also response very fast.
./ldapsearch sAMAccountName=username -base dc=harbor,dc=com -host ldap://ipaddress:3268/ -passwd password -user username

[yhua123]

I found the different between mqu/openldap and openldap/tools/ldapsearch:

• https://github.com/mqu/openldap/blob/master/openldap.go line 212:
  rv := int(C.ldap_search_s(self.conn, _base, C.int(scope), _filter, &_attributes[0], C.int(attrsonly), &msg))
• https://github.com/openldap/openldap/blob/master/clients/tools/ldapsearch.c line 1433:
  `rc = ldap_search_ext( ld, base, scope, filter, attrs, attrsonly, sctrls, cctrls, tv_timelimitp, sizelimit, &msgid );
  ...
  rc = ldap_result( ld, LDAP_RES_ANY, sortattr ? LDAP_MSG_ALL : LDAP_MSG_ONE, tvp, &res )'

When call "ldap_search_s(...)" (synchronous method) return need 6 minutes, call "ldap_search_ext(...)" (asynchronous method) and setting "ldap_result (...LDAP_MSG_ONE...)" return very fast.

For functions reference: https://msdn.microsoft.com/en-us/library/aa366112(v=vs.85).aspx
I modified harbor mqu/ldap's source code for testing (https://github.com/yhua123/harbor/blob/dev/src/vendor/github.com/mqu/openldap/openldap.go), and build new ui image (docker pull prjharbor/harbor-ui).
If you have time, please help me testing in your environment. Thanks~

[cavemandaveman]

This mqu/openldap library, which seems to have come to a standstill (see it's README: __I don't work any more with golang, so, please fork this project.__), uses a combination of deprecated, sync, and async functions:

function | status | type
--- | --- | ---
C.ldap_initialize | Non-deprecated | N/A
C.ldap_start_tls_s | Non-deprecated | Sync
C.ldap_bind_s | Deprecated | Sync
C.ldap_unbind | Deprecated | Async
C.ldap_search_s | Deprecated | Sync

I would recommend to go all the way and change it all to _non-deprecated, async_ functions to avoid any future issues:

function | status | type
--- | --- | ---
C.ldap_initialize | Non-deprecated | N/A
C.ldap_start_tls | Non-deprecated | Async
C.ldap_sasl_bind | Non-deprecated | Async
C.ldap_unbind_ext | Non-deprecated | Async
C.ldap_search_ext | Non-deprecated | Async

This is only the main class of the project; I haven't looked through the rest...

[yhua123]

@cavemandaveman
Thanks for your recommend~
We have 2 solution now:
1 Update all to non-deprecated, async functions for mqu lib (as cavemandaveman recommend)
Don't need to change a lot of code.
2 Update to new lib, maybe https://github.com/go-ldap/ldap
Need more test.
We will make a decision on next release.
Another question is "LDAP search operations error" caused by sync function "C.ldap_search_s" or not. I can't reproduce this issue in our scenario. Confirm this problem might have prevented other potential coding bugs. Can you help me reproduce and test again? Thanks~

[cavemandaveman]

@yhua123 I checked out the dev branch of your repo with the ldap async changes and built a test instance and I can confirm it is working as expected

[yhua123]

great, thanks a lot...
next step we will think about how to support LDAP better~

[rikatz]

@cavemandaveman Can you test again with the 'dev' branch. There is a merge upgrading the LDAP library to go-ldap, just to check if this is working now.

Hope that the guys from harbor merge this to master ASAP, and then release a new version with this library updated.

About the other directives (DEFER, and so), don't think it's hard to turn them configurable (as I did with the connection timeout from ldap). I'm just without time to make this now, but it's pretty easy ;)

[rikatz]

@cavemandaveman Did the PR #1277 solved your problem?

[cavemandaveman]

Sorry I have not had a chance to look at it till today.

I built off the current HEAD of the dev branch and it works just fine! The new library seems to have fixed the issue. Thanks!

[yhua123]

Hi Cavemandaveman and Rikatz,
Thanks for your hard work~
If no more question on this issue, I will help you close this in 24 hours.

[rikatz]

@yhua123 NP to me.

[cavemandaveman]

All good, thanks

[yhua123]

Thanks~