Server: Conversion error and unresponsive channel when CG UPDATE contains special characters in JSON payload

[saltomodules]

Expected behaviour

CG UPDATE command can handle special characters without destabilizing the server.

Current behaviour

CaspsarCG 2.2 on Ubuntu 18.04 can't handle (common) special characters like é, á or ‘ in the JSON payload of a CG UPDATE command.

When the command is issued by a TCP client the result is a v2s_mt_posix::conversion_error.

For instance CG 5-100 UPDATE 0 "{\"testdata\":\"test á test\"}" results in:

[2018-11-08 15:39:40.090] [info]    Received message from 127.0.0.1: REQ 56qr3my CG 5-100 UPDATE 0 "{\"testdata\":\"test\\t? test\"}"\r\n
[2018-11-08 15:39:40.091] [error]   Exception: libs/log/src/code_conversion.cpp(130): Throw in function (unknown)
[2018-11-08 15:39:40.091] [error]   Dynamic exception type: boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::log::v2s_mt_posix::conversion_error> >
[2018-11-08 15:39:40.091] [error]   std::exception::what: Could not convert character encoding
[2018-11-08 15:39:40.091] [error]   
[2018-11-08 15:39:40.091] [error]    0# caspar::log::get_stack_trace[abi:cxx11]() in bin/casparcg
[2018-11-08 15:39:40.091] [error]    1# caspar::IO::connection::handle_read(boost::system::error_code const&, unsigned long) in bin/casparcg
[2018-11-08 15:39:40.091] [error]    2# boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, std::_Bind<void (caspar::IO::connection::*(caspar::spl::shared_ptr<caspar::IO::connection>, std::_Placeholder<1>, std::_Placeholder<2>))(boost::system::error_code const&, unsigned long)> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) in bin/casparcg
[2018-11-08 15:39:40.091] [error]    3# boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) in bin/casparcg
[2018-11-08 15:39:40.091] [error]    4# 0x000055D219947C39 in bin/casparcg
[2018-11-08 15:39:40.091] [error]    5# 0x00007FF921C60363 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[2018-11-08 15:39:40.091] [error]    6# 0x00007FF929E3E6DB in lib/libpthread.so.0
[2018-11-08 15:39:40.091] [error]    7# clone in /lib/x86_64-linux-gnu/libc.so.6
[2018-11-08 15:39:40.091] [error] 

When issuing the same command directly from the CasparCG console you don't get an error but JSON parsing breaks off when the special character is reached:

CG 5-100 UPDATE 0 "{\"testdata\":\"test  á test\"}"
[2018-11-08 16:09:21.602] [info]    Received message from Console: CG 5-100 UPDATE 0 "{\"testdata\":\"test  \r\n
[201#202 CG OK
8-11-08 16:09:21.602] [debug]   Executing command: CG UPDATE
[2018-11-08 16:09:21.603] [debug]   Executed command (0.001s): CG UPDATE
[2018-11-08 16:09:21.603] [info]    html[file:///opt/test.html] 1920 1080 50 Log: Uncaught SyntaxError: Unexpected end of JSON input

In both scenario's the channel in question becomes unresponsive and no longer reacts to new commands.

Best guess is that it are multibyte utf8 characters which are not handled well.

Issue is observed on the Linux build. The windows build replaces the special character with an ?, but the channel remains responsive:

---

Steps to reproduce

1. Issue a CG UPDATE command containing special (possibly only multibyte utf8) characters, like CG 1-100 UPDATE 0 "{\"testdata\":\"test á test\"}". Either from a TCP client or directly from the console.
2. Output is respectively an v2s_mt_posix::conversion_error or a log message with incomplete JSON
3. Channel becomes unresponsive to further commands.

When reproducing on Windows the result seems to be JSON data with question marks where the special character should be.

Environment

Issue is observed with some differences on both an Ubuntu and a Windows 7 system:

System 1

CasparCG Server version:
2.2.0 ec6d30dc7 Beta 10
video-mode 1080i5000    

Operating system:
Ubuntu 18.04

Graphics driver:
Nvidia 390.42  (for GTX1050)

Decklink drivers:
Blackmagic Design BlackmagicIO driver 10.10a3

System 2

CasparCG Server version:
######################
2.2.0 ec6d30dc Beta 10
video-mode 1080i5000    

Operating system:
Win 7 x64

Graphics driver:
Nvidia 391.01 (for GTX660)

Decklink drivers:
none

[Julusian]

Sending the cg update from casparcg client was working ok for me, and input on the console should now be fixed.

Can you retest to confirm both cases are working for you?

[saltomodules]

Retested and things are definitely a lot better!

On the linux build, for both console and client input, multibyte utf 8 characters now show up correctly in html5 templates. Internal parsing and sending works.

The characters are also printed correctly in the logs files. However on the terminal they still show as question marks:

This might be a local terminal issue or could it also be an issue with the functionality that prints lines to the terminal? Some sort of bottleneck perhaps?

Windows is not much different as before. No errors, but multibyte characters show up as question marks on the console or in the logs. They often do not show at all in rendered templates.

[Julusian]

Thats good to hear.

It looks like the questions marks int he console is 'expected' behaviour. https://github.com/CasparCG/server/blob/20fc0b90f32b1c5c7efc196d195b891d97090448/src/common/log.cpp#L155-L156
This definitely should be revisited as that filter dates back to 2012, and I expect modern oses should be able to handle printing the full unicode range