Ejabberd: HTTP Upload Module: "Non-unicode filename" warning

[alexara]

Summary

If there's an umlaut like ä,ü in the uploaded file's name, ejabberd throws an error.

General Information

• Ejabberd 19.05
• Nginx
• Nginx HTTP Upload Perl Module

Config

    module: ejabberd_http
    request_handlers:
      "/upload": mod_http_upload
    tls: false

  mod_http_upload:
    access: acsupload
    put_url: "https://@HOST@/upload"
    docroot: "/path/upload"
    external_secret: ""

Warning in ejabberd log:

> 2019-05-30 14:57:29.523 [warning] <0.50.0> Non-unicode filename <<"KÃ_che.jpg">> ignored

[badlop]

When does the error appear: when uploading or when downloading?

What clients are you using? I tried with Gajim 1.1.2 for upload, and Tkabber for download in ejabberd 19.05.11, with a file named Testä,ü.jpg and the feature worked correctly. In the ejabberd log file I can see:

09:57:55.397 [info] Got HTTP upload slot for user1@localhost/gajim.XYW1Y7Z7 (file: Testä,ü.jpg, size: 577716)
09:57:55.420 [info] (<0.513.0>) Accepted connection 127.0.0.1:59056 -> 127.0.0.1:5444
...
09:58:03.703 [info] (<0.515.0>) Accepted connection 127.0.0.1:59058 -> 127.0.0.1:5444
09:58:03.825 [info] Serving /tmp/upload/83a4e1d220e89d9dada2f5ba18c86a576e03562e/wqrMMc7txe2Ptcv1wSoAl8MWHGnLvFbPLql9Os9C/Testä_ü.jpg to 127.0.0.1

[weiss]

When does the error appear: when uploading or when downloading?

The warning is issued by file:list_dir/1 which is called (several times) from mod_http_upload_quota.

the feature worked correctly.

Yeah, it's unclear to me how you'd get mod_http_upload to store a non-unicode filename. (I had no time to look into this yet, sorry.)

[alexara]

When does the error appear: when uploading or when downloading?

Neither. I think the warning only appears when upload quota tries to clean up old files?

What clients are you using?

The client which causes the issue is Gajim 1.1.3 with the HTTP upload plugin 2.6.29.

The original filename on the computer was Küche, the put url is domain/path/K%C3%BCche.jpg and on ejabberd's / nginx's side it is K�_che.jpg.

Another example is RÃ_ckfahrt instead of Rückfahrt.

[weiss]

The problem may well be related to Nginx and/or ngx_http_upload.

[alexara]

The problem may well be related to Nginx and/or ngx_http_upload.

In the hope to clarify that, I tried to debug it a little bit further:

With the current Gajim (as mentioned above) I sent myself a file called Test-ü-ä-ß-Test.txt.

The Ejabberd log (level 5) reads:

```(erlang)
2019-06-10 15:45:47.749 [debug] <0.2406.0>@mod_http_upload_quota:handle_cast:177 Enforcing quota for user1@domain/gajim.UWR6C06S
2019-06-10 15:45:47.750 [info] <0.2405.0>@mod_http_upload:create_slot:618 Got HTTP upload slot for user1@domain/gajim.UWR6C06S (file: Test-ü-ä-Ã<9f>-Test.txt, size: 2)

2019-06-10 15:45:47.755 [debug] <0.2405.0>@ejabberd_router:do_route:353 route:

iq{id = <<"baf3c101-3f75-451a-9e7e-61105680457c">>,type = result,

lang = <<"de">>,
from = #jid{user = <<>>,server = <<"upload.domain">>,
            resource = <<>>,luser = <<>>,
            lserver = <<"upload.domain">>,lresource = <<>>},
to = #jid{user = <<"user1">>,server = <<"domain">>,
          resource = <<"gajim.UWR6C06S">>,luser = <<"user1">>,
          lserver = <<"domain">>,
          lresource = <<"gajim.UWR6C06S">>},
sub_els = [#upload_slot_0{get = <<"https://domain/upload/path/Test-%C3%BC-%C3%A4-%C3%9F-Test.txt">>,
                          put = <<"https://domain/upload/path/Test-%C3%BC-%C3%A4-%C3%9F-Test.txt?v=(...)5a987">>,
                          xmlns = <<"urn:xmpp:http:upload:0">>}],
meta = #{ip => {0,0,0,0,0,65535,23753,60755}}}

2019-06-10 15:45:47.828 [warning] <0.1986.0> Non-unicode filename <<"Test-Ã_-Ã_-Ã_-Test.txt">> ignored

#### Nginx 1.17.0 with` charset utf-8;` in the config. The log reads:

118 127.0.0.1 - - [10/Jun/2019:15:45:48 +0200] "PUT /upload/path/Test-%C3%BC-%C3%A4-%C3%9F-Test.txt?v=(...)5a4d (...)5a987 HTTP/1.1" 201 0 "-" "Gajim 1.1.3"
119 127.0.0.1 - - [10/Jun/2019:15:45:50 +0200] "HEAD /upload/path/Test-%C3%BC-%C3%A4-%C3%9F-Test.txt HTTP/1.1" 200 0 "-" "Gajim 1.1.3
```

On the ejabberd server

LC_ALL=C ls --quoting-style=c gives: "Test-\303_-\303_-\303_-Test.txt"
ls gives: Test-?_-?_-?_-Test.txt or (after a second try) Test-�_-�_-�_-Test.txt

[badlop]

In my test with Erlang/OTP 21 I get info and debug lines with similar content and filename encoding
sub_els = [#upload_slot_0{get = <<"https..../Test-%C3%BC-%C3%A4-%C3%9F-Test.txt">>,
But I don't get the warning about non-unicode filename. The other difference is that the filename is correct in the local ext4 filesystem:

$ cd /tmp/upload/83a4e1d220e89d9dada2f5ba18c86a576e03562e/MjC39oUdHDdACMEg1qLjYs5YUF8unKZlGr7mWsfH/
$ LC_ALL=C ls --quoting-style=c
"Test-\303\274-\303\244-\303\237-Test.txt"
$ ls
Test-ü-ä-ß-Test.txt

[alexara]

@badlop have you testet it using @weiss https://github.com/weiss/ngx_http_upload/ perl library?

I now moved back from the perl library to my former Nginx http_upload_config and voilá, problem seems to be solved. Hence I think the issue is caused by the perl module. ;-)

[weiss]

I now moved back from the perl library to my former Nginx http_upload_config and voilá, problem seems to be solved. Hence I think the issue is caused by the perl module. ;-)

Thanks for tracking this down. It would be nice if you could post an issue against ngx_http_upload, then. (But I'll probably not forget about this anyway.)