Core: Unable to send a file with accents inside its name to an external OpenStack Object Storage mount [$10]

[bchaintreuil]

Steps to reproduce

1. Make a fresh install of Owncloud 8.2.1, with an external OpenStack Object Storage mount enabled
2. Try to send on it a file/folder with some specials characters (é,à,ç,è,°,etc...) inside its name
3. See the error message : "Client error response [status code] 400 [reason phrase] Bad request [url] https://....."
   

   Expected behaviour

Files should be send on the storage mount, like the others with no specials characters on their names

Actual behaviour

We are unable to send file on the storage mount when they have specials characters on their names

Maybe a way to solve the problem :

I think that the problem comes from the manner of which URLs requests to send files to the Openstack platform are made, more precisely of the encoding of them

A simple example :

When I send a file named "Hello World.txt", it works...
But when I send a file named "Cœur.txt", I get the following error message : "Client error response [status code] 400 [reason phrase] Bad request [url] https://storage.gra1.cloud.ovh.net/v1/AUTH_2fe4a0cb726c438ba4041626e316a36d/Cloud%20-%20Philippe/Cœur.txt" ("Cœur" == heart in french)

As you can see, the "œ" wasn't converted to "%C5%93" in the URL, but spaces in "Cloud - Philippe" was (%20)

Specials characters in filename aren't converted to HTML URL Encoding format, so requests can't be passed to the Openstack platform, and the send fail...

It's a big issue for people who don't speak english...

Server configuration

Operating system:
Debian 8

Web server:
Nginx

Database:
MariaDB

PHP version:
5.6.14

ownCloud version:
8.2.1

Updated from an older ownCloud or fresh install:
Fresh install

List of activated apps:

Enabled:
  - activity: 2.1.3
  - encryption: 1.1.0
  - files: 1.2.0
  - files_external: 0.3.0
  - files_pdfviewer: 0.7
  - files_sharing: 0.7.0
  - files_texteditor: 2.0
  - files_trashbin: 0.7.0
  - files_versions: 1.1.0
  - files_videoviewer: 0.1.3
  - gallery: 14.2.0
  - notifications: 0.1.0
  - provisioning_api: 0.3.0
  - templateeditor: 0.1
  - updater: 0.6
Disabled:
  - external
  - firstrunwizard
  - user_external
  - user_ldap

The content of config/config.php:

$CONFIG = array (
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'instanceid' => 'Never gonna give you up',
  'passwordsalt' => 'never gonna let you down',
  'secret' => 'Never gonna run around and desert you \o/...',
  'trusted_domains' => 
  array (
    0 => '****',
  ),
  'datadirectory' => '****',
  'overwrite.cli.url' => '****',
  'dbtype' => 'mysql',
  'version' => '8.2.1.4',
  'dbname' => '****',
  'dbhost' => '****',
  'dbtableprefix' => '****',
  'dbuser' => '****',
  'dbpassword' => '****',
  'logtimezone' => 'UTC',
  'mail_from_address' => '****',
  'mail_smtpmode' => 'php',
  'mail_domain' => '*****',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpauth' => 1,
);

Are you using external storage, if yes which one:
OpenStack Object Storage

Are you using encryption:
Yes

[karlitschek]

@butonic

[PVince81]

@karlitschek "app:objectstore" is for the SWIFT home object store mount.

This report is for the regular external storage mount. Tagging as "files_external".

Let me have a try, I'm pretty sure this worked before...

[PVince81]

ownCloud uses the php-opencloud library and doesn't generate URLs directly.
I'd expect the library to automatically encode file names in the URL.

@Xenopathic I'm having trouble with your docker instance, it doesn't seem to start the service (port 5001 is not open). So I'm unable to test this at the moment.

[karlitschek]

@PVince81 got it. Do we have an up to date list of the tags including meaning in the wiki?

[PVince81]

@karlitschek it's on my todo list, hopefully next week

[bchaintreuil]

So if i have understand, it will be fix in the 9.0 release?

[PVince81]

Not necessarily. First we need to be able to reproduce and understand the issue and find out if it's a bug in ownCloud, in the library or an environment issue. (ex: locale issue)

So far the unit tests against SWIFT storages work, and the unit tests do contain special characters, so it is not guaranteed to be an ownCloud issue.

[PVince81]

Does OVH provide SWIFT-like containers ? I noticed the URL in question points at OVH. Or did you install your own OpenStack container there ?

[bchaintreuil]

@PVince81 It's a container provide by OVH (https://www.ovh.co.uk/cloud/storage/object-storage.xml)
With the Horizon panel I can upload files/folders with accents, but with owncloud I can't

[bchaintreuil]

Files on the container are encrypted, so maybe the issue come from here ?

I use 3 containers, one per users, and each ones are concerned by this issue, and everythings is in green in the admin section

[DanielTosello]

I believe I also saw this issue at one point when working on swift interface stuff but had it evaporate when i rebuilt the server i was using & started using the swift-ceph image @Xenopathic provided.

From memory, I could temporarily resolve it by wrapping the urls opencloud returned with rawurlencode(), but this clearly isn't a good fix.

I suspect Guzzle was the culprit, but again it's purely from memory & this was months ago now.

[bchaintreuil]

@karlitschek @PVince81 @DanielTosello @cmonteroluque It's very weird... I've reinstalled my server twice, with a basic debian installation, installed Nginx, php5-fpm as in the documentation... Activated the External storage support, created a new openstack user and swift container named "test", connected it to owncloud, and the problem persist in 9.0

I can give you an access on a test server to help you find the problem, i'm sure is not a misconfiguration issue

[bchaintreuil]

@karlitschek @PVince81 @DanielTosello @cmonteroluque I've posted a bounty on this issue : https://www.bountysource.com/issues/28887419-unable-to-send-a-file-with-accents-inside-its-name-to-an-external-openstack-object-storage-mount
Just enough to pay you a big coffee, and thanks you if you could help me

[PVince81]

I added a unit test for special chars for file names here https://github.com/owncloud/core/pull/25352. There was already one for folders and they both run fine against the test Ceph SWIFT from CI.
Test can be run locally with: ./autotest-external.sh sqlite swift-ceph

So not sure, maybe OVH's container isn't 100% compatible with the way how it's done for Ceph or openStack SWIFT and might need special handling.

[bchaintreuil]

I notified OVH about this problem and they are on it

[nicoolas]

I would really appreciate seeing the issue fixed, whether from OVH or with some workaround in owncloud.
Just doubled (well, added 5$) to the bounty above.
Cheers!

[PVince81]

Any update from OVH ?

[PVince81]

Closing due to lack of feedback. Last time I tried this worked for me with an OpenStack server, so it's likely an environment issue.

[jonathantisseau]

@bchaintreuil hi, I just found this issue. I have it too. Did you find a solution ?

[nicoolas]

@jonathantisseau,
Only solution I found: switch to Amazon S3, that works fine.

[jonathantisseau]

@nicoolas Thanks for the feed back but I found the solution : the file's path needs to be encoded, but PHP urlencode() doesn't use the good encoding (spaces are translated to + instead of %20, ...). Instead, you have to use rawurlencode() to encode it correctly.

[tdont]

@jonathantisseau

Hello, I share my server with french people and they do have accent in folder and file names. Needless to say they can't upload some of their files. I do understand the problem but I do not know where to indicate to owncloud to use rawurlencode() instead of urlencode(). Can you indicate what kind of changes you have done ?

Thanks

[jonathantisseau]

Hi @tdont
For the record, I don't use owncloud to send request to OVH hubiC servers. But they are based on OpenStack API and I had the same issue, that's why I came here for help.

But this might work for you anyway : I made this function to prepare the file/folder path before sending the request

/**
 * Prepare a path to be sent
 * @param string $path
 * @param bool $url_encode
 * @return string
 */
private function _prepare_path($path, $url_encode)
{
    if (mb_detect_encoding($path, 'UTF-8', true) !== 'UTF-8')
        $path = utf8_encode($path);
    if ($url_encode)
        $path = implode('/', array_map('rawurlencode', explode('/', $path)));
    return $path;
}

You can then send the request by giving it the path like before. Here I'm downloading a file using \Guzzle\GuzzleClient ($this->session)

$path = $this->_prepare_path($path, true);
$result = $this->session->get($this->get_endpoint().'/default/'.$path, $this->get_headers())->send();
...

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.