Fmt: how do we print u8 literals using fmt?

[DBJDBJ]

The issue is this

// ./fmt6/core.h(813,5): error : static_assert failed due to requirement '!sizeof(char8_t)'
//  "formatting of non-void pointers is disallowed"

fmt::print( "{}", u8"ВИКТОР ЗВЕРОВИЧ");

[vitaut]

{fmt} explicitly disallows mixing character types but you can write a function that works with char8_t yourself (which I wouldn't recommend because char8_t is broken). On sensible systems

fmt::print("{}", "ВИКТОР ЗВЕРОВИЧ");

just works.

[DBJDBJ]

I am now wondering what are those "sensible systems":
Using u8 + std::printf

1. currently g++ does segv -- Wandbox
2. clang does it ok -- VS2019 and Wandbox
3. mvc does it ok too -- VS2019

char8_t is indeed broken, to make it unbroken for C++20 release will be an interesting spectacle...
Committee will certainly claim the GNU, LLVM and MSFT are to implement it somehow.

Let just hope compiler vendors will agree on a common fix ...

[vitaut]

Passing u8string_view to printf is a UB, so GCC is correct to crash.

[DBJDBJ]

https://wandbox.org/permlink/1o4PAYT0tvDpBcPd

I was always wondering why letting UB to compile?

[risa2000]

{fmt} explicitly disallows mixing character types but you can write a function that works with char8_t yourself (which I wouldn't recommend because char8_t is broken).

Could you elaborate a bit more about the "broken" part? I was just thinking about going full std::u8string (with C++20 support in MSVC /std:c++latest), but basically got stuck at this:

#include <filesystem>
#include <fmt/format.h>

int main(int argc, char* argv[])
{
    auto modPath = std::filesystem::path(argv[0]);
    auto u8path = modPath.u8string();
    fmt::print(u8"module path = {:s}\n", u8path);

    return 0;
}

which complains about missing overload:

------ Build All started: Project: fmt_test, Configuration: x64-Debug ------
[1/2] C:\PROGRA~2\MICROS~2\2019\COMMUN~1\VC\Tools\MSVC\1425~1.286\bin\HostX64x64\cl.exe /nologo /TP -DFMT_LOCALE -ID:\Dev\include -ID:\Dev\crtdll\include /DWIN32 /D_WINDOWS /GR /EHsc /Zi /Ob0 /Od /RTC1 -MDd -std:c++latest /showIncludes /FoCMakeFiles\fmt_test.dir\fmt_test.cpp.obj /FdCMakeFiles\fmt_test.dir\ /FS -c ......\fmt_test.cpp
FAILED: CMakeFiles/fmt_test.dir/fmt_test.cpp.obj
C:\PROGRA~2\MICROS~2\2019\COMMUN~1\VC\Tools\MSVC\1425~1.286\bin\HostX64x64\cl.exe /nologo /TP -DFMT_LOCALE -ID:\Dev\include -ID:\Dev\crtdll\include /DWIN32 /D_WINDOWS /GR /EHsc /Zi /Ob0 /Od /RTC1 -MDd -std:c++latest /showIncludes /FoCMakeFiles\fmt_test.dir\fmt_test.cpp.obj /FdCMakeFiles\fmt_test.dir\ /FS -c ......\fmt_test.cpp
D:\Dev\crtdll\include\fmt\core.h(1781): error C2665: 'fmt::v6::vprint': none of the 2 overloads could convert all the argument types
D:\Dev\crtdll\include\fmt\core.h(1743): note: could be 'void fmt::v6::vprint(fmt::v6::string_view,fmt::v6::format_args)'
D:\Dev\crtdll\include\fmt\core.h(1779): note: while trying to match the argument list '(fmt::v6::basic_string_view, fmt::v6::format_arg_store>,char8_t>,std::basic_string,std::allocator>>)'
......\fmt_test.cpp(8): note: see reference to function template instantiation 'void fmt::v6::print,std::allocator>&,char8_t>(const S (&),std::basic_string,std::allocator> &)' being compiled
with
[
S=char8_t [20]
]
ninja: build stopped: subcommand failed.

Do you recommend using std::string with UTF-8 content instead? (I know this has worked so far, once the Windows console is set to CP_UTF8).

[vitaut]

char8_t will only be introduced in C++20 and will be a big breaking change for everyone using u8 literals. It will also be nonportable for a very long time and likely will never be supported by system APIs.

Do you recommend using std::string with UTF-8 content instead?

Yes.

[risa2000]

char8_t will only be introduced in C++20 and will be a big breaking change for everyone using u8 literals. It will also be nonportable for a very long time and likely will never be supported by system APIs.

One more thought though. At the moment, I have been using std::filesystem::path::u8string() as the most straightforward way to get the UTF-8 encoded paths on Windows. This was handy, as I could construct the path from std::wstring (native UCS-2/UTF-16LE). Until now (i.e. up to C++17), std::filesystem::path::u8string() returned std::string, which was kind of confusing, because the content was really UTF-8, but on the other hand, I could easily use it with fmtlib.

Once I switch to C++20 support, this function starts returning proper std::u8string and I am getting stuck in between. std::filesystem::path::string() does not return UTF-8, but some local encoding, which I cannot use.

So it does not seem very clear at the moment, how to best manage the switch to C++20, considering this context.

[vitaut]

Sadly path::u8string is another thing that will be broken in C++20 adding to the hot mess of std::filesystem. I recommend writing a helper function that takes a path and returns a UTF-8 encoded string. This function can be made to work portably with the forthcoming C++20 and earlier versions of the standard.

[DBJDBJ]

@risa2000

1. char8_t can not hold a full range of utf-8 glyphs. utf-8 glyphs can be up to 4 bytes. char8_t holds up to 1 byte.
2. char8_t stream out is expressly forbidden in C++20
3. for utf-8 compatibility while in VStudio just use char and /utf-8
4. in case you really want C++20 way of transforming to/from char8_t, you need to use <cuchar> ... AFAIK not yet fully implemented in any of the 3, as required by the C++20 standard

std::filesystem is a hot mess indeed. :)

[DBJDBJ]

OK, what point 3 above actually means is, one can print the utf8 by casting to char * and one can iterate over utf8 literals too. For example:

// note: in case of MSVC, cl.exe /utf-8  is in use
#define SPECIMEN u8"ひらがな"
 {
printf("\n\nIterating over u8 string literal: %s\n\n", (char const *)SPECIMEN);
unsigned  cnt_ = 1;
    // msvc c++20 , type of elem is int
    // same is for clang and g++
    // iteration is byte by byte, total of 14 times
    for ( auto const & elem : SPECIMEN )
    {
        cnt_ += 1;
        printf("%c", elem);
    }
    assert( cnt_ == 14);
}

printf("%c", elem); "knows" utf-8 is the source and executable character set, thus the expected appears, glyph by glyph:

ひらがな

ps: on windows be sure font can show those glyphs. and chcp is 65001 which on win10 by now is the cmd default.