Openrefine: Loss of expression history and language at restart

[wetneb]

Describe the bug

There are reports that OpenRefine 3.3 forgets language settings and expression history when restarted:
https://twitter.com/daieuxdailleurs/status/1248911911036628992

I am unable to reproduce this on 3.3. If anyone could get in touch and ask for more details (platform info, perhaps), it would potentially help to debug this.

Desktop (please complete the following information):

• OS: Windows 64bits [version not specified]
• Browser Version: FF ou Chrome [no versions]
• JRE or JDK Version: ?

OpenRefine:

• Version 3.3

[ettorerizza]

The original poster is on Windows 10 64 bits and uses Chrome or FF, like me. I installed the latest version of Java, JRE 1.8.0 241.

I can reproduce something similar if I abruptly close the OpenRefine console: all the actions including language settings done before the last automatic saving have disappeared. But looks like this doesn't happen if I close the console with CTRL+C.

If I remember correctly, closing the console saved the changes in the previous versions, didn't it?

[thadguidry]

What works:

• Expression History
• Change Language

What breaks:

• Star an expression in history

Details:

• OpenRefine win 3.3 release
• Windows 10
• JDK 8
• Firefox, Chrome, Edge
• The last part of workspace.json is formed as such:

            ],
            "ui.browsing.listFacet.limit": 22000,
            "scripting.starred-expressions": null,
            "userLang": "en",
            "scripting.expressions": {
                "top": 100,
                "list": ["grel:value.startsWith(\"http\")", "grel:value.replace(\"/wiki/Property:\",\"/entity/\")", "grel:value.contains(\"Property\")"],
                "class": "com.google.refine.preference.TopList"
            },
            "ui.show-control-characters": "true"
        }
    }

Error:

• Clicking on a Star next to a GREL in the Expression History does not work and throws an NPE here:

10:06:28.438 [          org.mortbay.log] /command/core/toggle-starred-expression (0ms)
java.lang.NullPointerException
        at com.google.refine.commands.expr.ToggleStarredExpressionCommand.doPost(ToggleStarredExpressionCommand.java:53)
        at com.google.refine.RefineServlet.service(RefineServlet.java:189)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
        at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
        at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
<snip>

Additional context:

Wondering if any leftover TODO's from this issue #2086 ?

[ettorerizza]

Clicking on a Star next to a GREL in the Expression History does not work and throws an NPE here:

I cannot reproduce this specific error. :/

[thadguidry]

@ettorerizza What does the last part of your workspace.json look like compared to mine? paste it for us.

[wetneb]

I can reproduce something similar if I abruptly close the OpenRefine console: all the actions including language settings done before the last automatic saving have disappeared. But looks like this doesn't happen if I close the console with CTRL+C.

If I remember correctly, closing the console saved the changes in the previous versions, didn't it?

Ah thanks, I think you are on the right track with this. It could be due to a change in Launch4j, that I updated just before 3.2 to solve a JRE detection issue. It would be useful to know if this behaviour also occurs with 3.2.

[ettorerizza]

         "scripting.starred-expressions":{
            "top":2147483647,
            "list":[
               "grel:value.toUppercase()",
               "grel:value.toLowercase()"
            ],
            "class":"com.google.refine.preference.TopList"
         },
         "userLang":"en",
         "scripting.expressions":{
            "top":100,
            "list":[
               "grel:value",
               "grel:value.toLowercase()",
               "grel:value.toUppercase()"
            ],
            "class":"com.google.refine.preference.TopList"
         }
      }
   }
}

[thadguidry]

@ettorerizza So this is 2 issues in reality:

1. Starring expressions not working because of null in workspace.json (let me open a new issue)
2. And this issue regarding Expression history and Language

[ettorerizza]

It's really weird. I have no problem to star formulas (even with JDK8) and, once checked, OR 3.2 behaved the same way when we close the console without typing CTRL+c...

[thadguidry]

@ettorerizza Its not weird. Its because you have this

         "scripting.starred-expressions":{
            "top":2147483647,
            "list":[
               "grel:value.toUppercase()",
               "grel:value.toLowercase()"
            ],
            "class":"com.google.refine.preference.TopList"
         },

and I only have this:

         "scripting.starred-expressions": null

It's a small corner case regression that we just need @wetneb to explain how to best provide users (such as me) a quick fix and then later provide a quality check for the future.

[wetneb]

@ettorerizza ok, if it behaves the same way with OR 3.2 then the bug might have been introduced between OR 3.1 and 3.2 by my update of Launch4j, which might behave differently with respect to its "stayAlive" configuration property (http://launch4j.sourceforge.net/docs.html).

[thadguidry]

2544 is related to loss of the Starring history.

[tfmorris]

I had refine crash yesterday when I changed branches underneath a running instance, which isn't a problem, but it corrupted my workspace.json file so that it now says "scripting.starred-expressions":null,"scripting.expressions":null in both workspace.old.json and workspace.json.

We need to make fixing this a high priority because it means users will be losing their entire expression histories. Note that this was running from the ./refine shell script, so no Windows or Launch4J involved.

[tfmorris]

Here's an extract of the server log.
refine-server-log-jackson-error.txt

It looks like perhaps after the server reload, it got an error deserializing workspace.json, but then wrote it back out again anyway, which is perhaps not the same scenario as some of the others, but we should still try to make this more bulletproof.

[grahamjevon]

I experienced this issue last week. When I returned to my laptop after an hour or so (with Open Refine v 3.2 left open), my expression history and starred expressions had all disappeared.

I have attached a copy of my workspace after the error. I continued to use OR before logging this issue, so it is not an example of the workspace immediately after the error. I have also attached a copy of my workspace from 2 months ago, in case this is useful for comparison.

I have just now saved my OR command history (not attached), but I'm not sure if I have closed and re-opened Open Refine since this error occurred.

workspaceAfterError.txt
workspace2mthsBeforeError.txt