Dokuwiki: File names in UTF-8 changed encoding method with PHP 7.1 & 7.2

[chang-zhao]

Previously I had file names in data/pages which looked like gibberish in Windows 7:

кулинария.txt

(php5 and php7.0).

Now I try to use DokuWiki with PHP 7.1 & 7.2

File names created by these new installations in Windows 7 look meaningfully -- like actual words in Russian or Chinese.

I don't know, why, perhaps they use non-UTF-8 locales encoding.

That's good for me; I would actually like that.

But the problem is that older file names are still in UTF-8, and DokuWiki doesn't see them.

It's like almost all the 3000 pages of my wiki disappeared.

Is there a way to convert the old file names to new encodings, or to revert DokuWiki with PHP 7.1 & 7.2 to the old way to use (or to recognize) file names?

[splitbrain]

UTF-8 file names were not support on Windows (we have a clear warning on https://www.dokuwiki.org/config:fnencode). It seems that PHP 7.1+ has finally implemented UTF-8 handling (seemingly by converting between UTF-8 and the local file system encoding). This needs to be investigated.

To fix your files you will need to convert them to the proper encoding. However I have no good idea how to do that right now.

[chang-zhao]

Indeed: https://secure.php.net/manual/en/migration71.windows-support.php

I think I could write a tool to convert file names (with manual checking/setting of proper codepages).

[chang-zhao]

OK, the simplest way to convert the filenames seems to be this:

• We read the "data" directory tree into a nested array (with an older PHP, 5 or 7.0). We save that array (into a file, for example).
• Then we read the same directory tree with a newer PHP; and we read the old array from the file. Now these two trees should be corresponding 1:1, so we just rename files with the names from an older array.

[chang-zhao]

Here is the tool to do it:

https://github.com/chang-zhao/encoding

I think the problem is solved.

[weltling]

@chang-zhao you don't need two versions of PHP for that. One option is to set the default_charset so then it corresponds to the local codepage, that would enforce the old behavior per fact and no files need to be touched. However, the good way to migrate would be of course to use the tool you wrote, but having two runs with 71+, where

• first run,default_charset equal to the local codepage
• second run, default charset equals to UTF-8

internal_encoding could server that way, too.

Thanks.

[chang-zhao]

@weltling, thank you. (I think setting default_charset would not be a good option, because files' contents is in UTF-8).

[chang-zhao]

Finally I refactored the code (it didn't handle files in dirs properly), so now the code works well, and gives proper info in case of problems. Better late than never.

[sdiebel]

@chang-zhao: Warning: Invalid argument supplied for foreach() in convert.php on line 92

[sdiebel]

I want to list some files f.e. "橡木桶+麻布袋Oak+barrels.skp" with information f.e. fileperms but php alerts that stat failed so that php cannot find or use the file. I use Win8 and as I think it's UTF-16. What do I have to do to use this file with all its original data?

[phy25]

@chang-zhao: Warning: Invalid argument supplied for foreach() in convert.php on line 92

I do see another issue opened by you at chang-zhao/encoding#1.

I want to list some files f.e. "橡木桶+麻布袋Oak+barrels.skp" with information f.e. fileperms but php alerts that stat failed so that php cannot find or use the file. I use Win8 and as I think it's UTF-16. What do I have to do to use this file with all its original data?

I believe that is still UTF-8. Are you doing an upgrade or it's just a new installation?

[chang-zhao]

@sdiebel, do you mean PHP doesn't get some file name as valid parameter?
I've no idea what to do. I just made a tool which worked for me (on Windws 7), but I know not much about encoding, and don't have Windows 8. I hope someone else could shed some light on the problem.

[sdiebel]

I will try to fix that problem this weekend.
Is it possible to edit comment? Cause I use Win8 not Win 7 I mistyped

[chang-zhao]

@sdiebel - It was me who mistyped; but then I clicked dots in the upper right corner and edited to correct that I use Windows 7 & have no idea about 8.