Android: LDAP users don't get listed when I try to share

[MarcoO74]

Expected behaviour

When I try to share a file I should be able to search for LDAP users.

Actual behaviour

I can only find users that do not come from LDAP.

Steps to reproduce

1. Install vanilla owncloud server
2. Configure LDAP
3. Install owncloud app client for android and connect
4. Try to share a file to a specific LDAP user
   

   Environment data

Android version: 5.1.1

Device model: LG Nexus 4

Stock or customized system: none

ownCloud app version: 1.9.1

ownCloud server version: 9.0.0 (stable)

Logs

Web server error log

no entries in the corresponding time frame

Insert your webserver log here
85.xxx.xxx.xxx - - [16/Mar/2016:18:10:59 +0000] "GET /owncloud/status.php HTTP/1.1" 200 1100 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - - [16/Mar/2016:18:10:59 +0000] "HEAD /owncloud/remote.php/webdav/ HTTP/1.1" 401 1015 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:14 +0000] "HEAD /owncloud/remote.php/webdav/ HTTP/1.1" 200 1035 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:24 +0000] "GET /owncloud/status.php HTTP/1.1" 200 1096 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:24 +0000] "GET /owncloud/ocs/v1.php/cloud/capabilities?format=json HTTP/1.1" 200 2406 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:25 +0000] "PROPFIND /owncloud/remote.php/webdav/ HTTP/1.1" 207 1937 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:25 +0000] "PROPFIND /owncloud/remote.php/webdav/ HTTP/1.1" 207 3621 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:26 +0000] "GET /owncloud/ocs/v1.php/apps/files_sharing/api/v1/shares?path=%2F&reshares=true&subfiles=true HTTP/1.1" 200 1953 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:28 +0000] "PROPFIND /owncloud/remote.php/webdav/Documents/ HTTP/1.1" 207 1944 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:28 +0000] "PROPFIND /owncloud/remote.php/webdav/Documents/ HTTP/1.1" 207 2844 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:28 +0000] "GET /owncloud/ocs/v1.php/apps/files_sharing/api/v1/shares?path=%2FDocuments%2F&reshares=true&subfiles=true HTTP/1.1" 200 1947 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:32 +0000] "GET /owncloud/ocs/v1.php/apps/files_sharing/api/v1/shares?path=%2FDocuments%2FExample.odt&reshares=false&subfiles=false HTTP/1.1" 200 1947 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:39 +0000] "GET /owncloud/ocs/v2.php/apps/files_sharing/api/v1/sharees?format=json&itemType=search&search=j&page=1&perPage=50 HTTP/1.1" 200 2150 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:42 +0000] "GET /owncloud/ocs/v2.php/apps/files_sharing/api/v1/sharees?format=json&itemType=search&search=jp&page=1&perPage=50 HTTP/1.1" 200 2141 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"
85.xxx.xxx.xxx - marco [16/Mar/2016:18:11:42 +0000] "GET /owncloud/ocs/v2.php/apps/files_sharing/api/v1/sharees?format=json&itemType=search&search=jpc&page=1&perPage=50 HTTP/1.1" 200 2146 "-" "Mozilla/5.0 (Android) ownCloud-android/1.9.1"

ownCloud log (data/owncloud.log)

[MarcoO74]

Tried both Ubuntu 14.04 LTS and CentOS 7 and both Owncloud Android app and other owncloud apps like Cirrus.
Desktop client works fine (can search and share with LDAP users). Web interface also works OK.

Something may be broken with Android app.

Anyone got it to work with LDAP users?

[MarcoO74]

I just confirmed it works OK with Apple iOS.
Seems to be an Android issue.

[blizzz]

Works for me.

Your OC log is odd, apparently an integer is passed as LDAP link? Did you apply any patches or modify the ownCloud code? Are there integritiy issues (you would see a notice on the admin page).

cc @rperezb can you or someone reproduce this?

[MarcoO74]

Thanks. It's good to know that it is supposed to work. It is a features we need much.

I did not apply any patches to Owncloud or modified the code. Admin page shows no issues.

What catches my attention is this line in the log (the oldest line):
Error OCPShare Sharing backend for search not found

I did a quick check in the share code and it seems it's not supposed to exist a Sharing backend for search(?)

Anyway, if someone can reproduce this it would be helpful.
Meanwhile I will start debugging through the LDAP PHP code.

[rperezb]

@blizzz we have done tests with LDAP and sharing and it works.

[MarcoO74]

Thank you for doing tests.
I have been debugging the server code (9.0.1), using xdebug and PhpStorm.
I managed to trace the failing call to the function invokeLDAPMethod in file owncloudappsuser_ldaplibldap.php
The ldap_search for a LDAP user (we want to share with) fails under the following conditions:
1-The request comes from an Android client;
2-memcache.local is configured (either APCu, Redis or Memcached)

If I omit memcache.local or if I use a desktop or iOS client then the problem doesn't show.

In your tests are you using memcache.local?

[AndyScherzinger]

Awesome bug hunt @MarcoO74! So looping in @DeepDiver1975 and @MorrisJobke for some server dev power and @davivel for the Android side.

[MarcoO74]

Just adding that I built the latest master branch Android apk (1.9.1) and the problem persists.
I can't get LDAP users to list for sharing using Android app if memcache.local is used.

Can anyone confirm they have LDAP users share functionality with Android client while using memcache.local?

Thank you.

[wimaerts]

I have the same problem. (with memcache.local)

'memcache.local' => 'OCMemcacheAPCu',

[davivel]

Thanks a lot, @MarcoO74.

I wonder how is

invokeLDAPMethod in file owncloudappsuser_ldaplibldap.php

aware that the call is triggered from an Android client. What are the input parameters of that method that differ from iOS or desktop clients?

@blizzz, any clue of what can be the problem in that method?

@rperezb , @SergioBertolinSG , @davitol , any bug you remember in the server that could be related?

Thanks to all in advance.

[SergioBertolinSG]

Perhaps this one https://github.com/owncloud/core/issues/23701

[MarcoO74]

Thank you all.

Since the desktop client works without any problem but the android doesn't (with memcache.local), I've been tracing the server code (xdebug in autostart mode + phpstorm) trying to find any differences.

I can trace the problem to a single web request, sent by both clients when we type part of the name of an LDAP user to share with in the search box. (note: ‘owncloud’ users do list)

From desktop client the request is:
/ocs/v1.php/apps/files_sharing/api/v1/sharees?search=Ma&itemType=file&page=1&perPage=50&format=json

And from Android (9.0.1) it is:
/ocs/v2.php/apps/files_sharing/api/v1/sharees?format=json&itemType=file&search=Ma&page=1&perPage=50

(previous Android client used to make itemType=search which threw the error I had in the log about OCPShare. That seems to have been fixed already)

The difference is the v1.php vs. v2.php. At some point I was getting errors (something like no Route to app) and even setup a reverse proxy to replace the v2.php part in Android requests to v1.php… but now I can’t replicate those errors and abandoned the proxy. V2.php seems to be OK.

But I did see something very different in the behaviour of the server code between the two clients. The Android client request triggers multiple LDAP calls in succession, related to the authenticated user (like search for the LDAP user using the app) and after those comes the call to LDAP search the string, but with a supposedly invalid link resource. On the other hand, the desktop client does not seem to query for the authenticated user and instead goes straight to the search string.

I’m not yet familiar with the LDAP and the Memcache server code to understand why this happens or what happens to the the ldap link resource to become screwed in the process.

If you hold the code on pause long enough, sometime before the relevant LDAP search, then it works. This probably means that Memcache expired and a lot of LDAP queries do have to be performed.

Conversely, if Memcache is not enabled, I can see a lot of LDAP calls in both cases, as expected, and Android client works.

Btw, the problem is present no matter we use APCu, Redis or Memcached.

There is something I didn’t explore yet that may be relevant (or not)… the difference in session parameters (eg. cookies) between the clients and the server.

[wimaerts]

I removed 'memcache.local' => 'OCMemcacheAPCu', from my config and did a reboot of my server(s) but sharing still doesnt work on android. I only get local users/groups, no ldap users.

[blizzz]

@rullzer any idea about https://github.com/owncloud/android/issues/1527#issuecomment-212075408 ?

[rullzer]

I don't see any reason why only the android app would be affected.

The v1.php vs v2.php is only about returned status codes. As you point out both use the same request otherwise.

It does look like the android app authenticates again. (that is most likely why you see the extra ldap calls). But that _should_ not cause any harm.

@blizzz this looks similar to the ussue with caching ldap stuff we have. Where we also have no idea why.

[blizzz]

@blizzz this looks similar to the ussue with caching ldap stuff we have. Where we also have no idea why.

what are you referring too?

[davivel]

I don't see any reason why only the android app would be affected.

This has to be related with sessions. Android app sends credentials in every request. Desktop and iOS client don't, they keep a session based in cookies. There have been some other errors recently related with this point.

[davivel]

But that should not cause any harm.

The keyword here is _should_ . Other bugs occurred due to this in the recent past, though affecting desktop and iOS and not Android.

[rperezb]

@davivel yes, sessions may be involved, however, there has to be other setting that makes this fails since we have tested here, sharing using the Android app with ldap users and it works

[davivel]

@rperezb , sure. From the comments before I'd say it's about sessions + memcache enabled.

[MarcoO74]

Thank you all for your attention.

Today I decided to be more methodical in tracing the code and finally found both the bug and a workaround!!!

Bug location (9.0.1): ./owncloud/apps/user_ldap/lib/access.php

…
        public function areCredentialsValid($name, $password) {
                $name = $this->DNasBaseParameter($name);
                $testConnection = clone $this->connection;
                $credentials = array(
                        'ldapAgentName' => $name,
                        'ldapAgentPassword' => $password
                );
                if(!$testConnection->setConfiguration($credentials)) {
                        return false;
                }
                $result=$testConnection->bind();
                $this->ldap->unbind($this->connection->getConnectionResource());
                return $result;
        }
…

Bug conditions:
-Android client trying to search for LDAP users to share;
-Some mechanism like memcache that greatly reduces the number of LDAP calls.

Bug trigger:
Android client, unlike other clients, triggers a call to isAuthorised() in ./owncloud/lib/private/api.php. This call eventually reaches areCredentialsValid() in ./owncloud/apps/user_ldap/lib/access.php which in turn calls ldap_unbind on the main LDAP connection resource, killing it.
Only after is ./owncloud/apps/files_sharing/api/sharees.php called but it never opens a new LDAP connection before doing an ldap_search and so it fails.

The sequence of LDAP calls is this:
1-ldap_connect
2-ldap_bind with user defined in LDAP configuration
3-ldap_search for login user (to check if exists)
4-ldap_bind with login user (to check if authorized)
5-ldap_unbind (kills the connection resource, but the code keeps thinking it is OK)
6-ldap_search for search string (too late!!!)

Bug workaround:
Comment out the line $this->ldap->unbind($this->connection->getConnectionResource());

According to LDAP documentation, ldap_unbind kills the connection. Also, sequential calls to ldap_bind are accepted and new replace the old. ldap_bind is just for authentication. ldap_unbind is used to close a connection (opened with ldap_connect) and not to close a “login” to LDAP.

There are also two other problems with the function areCredentialsValid():
1-By using PHP clone the connection resource stays the same so using $testConnection is the same as using $this->connection for LDAP;
2-unbind should act upon $testConnection, not $this->connection.

Q: Why did we think memcache.local was involved?
A: Without memcache.local a lot more LDAP calls are made and eventually one is ldap_connect. I did not trace under such conditions to confirm this. It’s a nightmare of LDAP calls lol

Q: Why do some have this problem even with memcache.local turned off?
Q: Why don’t some have this problem, even with memcache.local turned on?
Q: Why don’t I have this problem if I run the code slowly (trace)?
A: I don’t really know. Probably it has to do with LDAP itself. Just try to comment out that single line of code and see if it solves the problem.

It solved for me :P

[wimaerts]

@MarcoO74 :+1: solved it for me too. Great job !!

[davivel]

Awesome debugging you did, @MarcoO74 , thanks a lot for sharing your results here.

@blizzz , is the workaround applied by @MarcoO74 safe? Might be used as a bug fix in the server, or need to go through other way?

[blizzz]

@blizzz , is the workaround applied by @MarcoO74 safe? Might be used as a bug fix in the server, or need to go through other way?

Not entirely, just from quickly looking the connection would exists with the logged in user, not the configured agent which might end up in LDAP permission issues. I think I know when it was introduced and provide a final fix. So this is server, yes.

@MarcoO74 thank you very much for these awesome debugging efforts!

[MarcoO74]

@blizzz is right.
The connection (this->connection) exists with a bind to the configured agent and is tested in this function for the logged in user. Since the same connection is used, the bind with the configured agent is lost. So, I agree it may end up in LDAP permission issues since the next LDAP calls use the logged in user credentials.

I think the function should make a new connection entirely and then close it (unbind), just for the purpose of testing the logged in user credentials.

[MarcoO74]

Let me take this opportunity to thank you all and say how much we enjoy using Owncloud here at the Agency. It has recently become an important tool for us.

Our need to use Android clients and your words of appreciation fed my enthusiasm with trying to solve this issue.

I will remain an active user at Transifex translating Owncloud to our language.

Keep up the good work!

[rperezb]

@MarcoO74 impresive!

Thanks a lot for debugging this!

[rperezb]

@blizzz needed an issue on the core repo?

[blizzz]

@blizzz needed an issue on the core repo?

I just created a PR on it in core https://github.com/owncloud/core/pull/24214, technically speaking opening an issue there would be the right way, I am not sure it is worth the effort though.

[davivel]

I am not sure it is worth the effort though.

I agree, let's keep it here. Seems this will be fast now.

[rperezb]

@MarcoO74 it would be great if you may check whether next version fixes your issue.
If helpful we may provide to you a temporal test server, just send an email to [email protected]

[MarcoO74]

@rperezb I manually applied the exact changes to our server and it fixed the issue.
We use Ubuntu 14.04 LTS repository and the fixed version package is not yet available.
Yes, it would be great to have a test server. I'm sending an email to [email protected]

[rperezb]

@MarcoO74 i have not seen any mail

[MarcoO74]

@rperezb I did send an email Monday at 16:30 (UTC) to [email protected]

[rperezb]

@MarcoO74 do you mind sending it again? thx

[MarcoO74]

@rperezb , I sent it again now. It is originating from my workplace account.