Defold: Project is not displayed in the list (at editor start)

[otsakir]

I confirmed the issue on Ubuntu 20.04 (1.2.170). I boils down to problem reading UTF-8 characters from a java configuration file (prefs.xml). Providing some more info:

prefs.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE map SYSTEM "http://java.sun.com/dtd/preferences.dtd">
<map MAP_XML_VERSION="1.0">
  <entry key="bundle-generate-build-report?" value="[&quot;~#'&quot;,true]"/>
  ...
  <entry key="open-project-directory" value="[&quot;~#'&quot;,&quot;/home/nando/tmp/ΔοκιμήTest&quot;]"/>
  <entry key="recent-project-entries" value="[&quot;^ &quot;,&quot;/home/nando/tmp/dSoundBufferFull/game.project&quot;,&quot;2021-01-14T21:57:08.059198Z&quot;,&quot;/home/nando/tmp/ΔοκιμήTest/game.project&quot;,&quot;2021-01-14T22:08:30.306786Z&quot;]"/>
</map>
  ...

Exception

2021-01-15 00:11:16.758 ERROR default    editor.prefs - {:line 66, :msg "failed to fetch preference for open-project-directory with value [\"~#'\",\"/home/nando/tmp/ΔοκιμήTest\"]"}
java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x94
 at [Source: java.io.StringBufferInputStream@a596bdd; line: 1, column: 26]
        at com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read(ReaderFactory.java:119)
        at cognitect.transit$read.invokeStatic(transit.clj:296)
        at editor.prefs$read_transit.invokeStatic(prefs.clj:38)
        at editor.prefs$fn__10076.invokeStatic(prefs.clj:62)
        at editor.prefs$fn__10076.invoke(prefs.clj:59)
        at editor.prefs$fn__10058$G__10036__10067.invoke(prefs.clj:42)
        at editor.welcome$last_opened_project_directory.invokeStatic(welcome.clj:134)
        at editor.welcome$show_welcome_dialog_BANG_.invokeStatic(welcome.clj:572)
        at editor.welcome$show_welcome_dialog_BANG_.invoke(welcome.clj:568)
        at editor.welcome$show_welcome_dialog_BANG_.invokeStatic(welcome.clj:570)
        at editor.boot$select_project_from_welcome$fn__89782.invoke(boot.clj:72)
        at clojure.lang.AFn.run(AFn.java:22)
        at com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
        at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
        at com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
        at com.sun.glass.ui.gtk.GtkApplication.lambda$runLoop$11(GtkApplication.java:277)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x94
 at [Source: java.io.StringBufferInputStream@a596bdd; line: 1, column: 26]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidInitial(UTF8StreamJsonParser.java:3542)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidChar(UTF8StreamJsonParser.java:3536)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2539)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2465)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:315)
        at com.cognitect.transit.impl.JsonParser.parseVal(JsonParser.java:60)
        at com.cognitect.transit.impl.JsonParser.parseArray(JsonParser.java:143)
        at com.cognitect.transit.impl.JsonParser.parseVal(JsonParser.java:56)
        at com.cognitect.transit.impl.JsonParser.parse(JsonParser.java:46)
        at com.cognitect.transit.impl.ReaderFactory$ReaderImpl.read(ReaderFactory.java:117)
        ... 18 common frames omitted

Note, the word 'Δοκιμή' present in prefs file, is represented like the following in hex:

CE 94 CE BF CE BA CE B9 CE BC CE AE 0A

The exceptions complains for starting byte '0x94'. That's the _second_ byte in the hex string above btw.

[otsakir]

I did some more investigation on the issue. Sharing some findings and thoughts.

Reading a preference is done in two steps.

1. First, the java preference string is read from the .java/.userPrefs/defold/prefs.xml file. As a string.
2. The string is parsed and assumed to contain something in json.

I logged the offending preference (containing greek/UTF8 chars) after step 1 but before step2.

FETHED PREFERENCE key open-project-directory and value ["~#'","/home/nando/τεστ2"]
2021-01-19 01:12:37.276 ERROR default    editor.prefs - {:line 67, :msg "failed to fetch preference for open-project-directory with value [\"~#'\",\"/home/nando/τεστ2\"]"}
java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 middle byte 0xc4

So, from the first line it seems that the greek characters are loaded ok. The error occurs afterwards at step 2, when reading the string and try to decode it as json. But why deal with UTF-8 at this later stage though? The string is already loaded and displayed correctly and i guess it should now be in java's internal format (utf16 or something like that).

Digging further, the offending clojure block that will try to read from the pref string and parse into json is this:

(defn- read-transit [s]
  (let [reader (transit/reader (StringBufferInputStream. s) :json {:handlers read-handlers})]
    (transit/read reader)))

(https://github.com/defold/defold/blob/dev/editor/src/clj/editor/prefs.clj#L39)

IntelliJ reports StringBufferInputStream as Deprecated. And javadoc says the same:

https://docs.oracle.com/javase/7/docs/api/java/io/StringBufferInputStream.html

I'm wondering if this bug is about that i.e. StringBufferInputStream being unable to properly handle such characters right.

I'll try to replace it with something else and see if this makes things any better.

[britzl]

Excellent detective work! It sounds like you're on the right track!

[britzl]

Fixed via #5490