Openrefine: Projects with timezone in modified time metadata are not displayed in Open Project

[ostephens]

This is on the current Master branch with new Tags functionality implemented. None of my projects created on a pre-tags version of OR display in the Open Project screen. However, projects created with the Tags version of OR do display even if no tags assigned.

The 'All' option should display all projects, no matter whether any Tags metadata exists or not.

[jackyq2015]

Not sure if it is caused by merge or not. let me check and update.

[jackyq2015]

@ostephens Did you see any exception at back end? could you please paste the console message while startup? I have no problem to show "All" projects. But I cannot add tags anymore. can somebody confirm?

[ostephens]

No exceptions or issues flagged in the console. However, checking I can see that get-all-project-metadata is coming back with an empty list:

{"projects":{}}

[ostephens]

OK, think I have tracked down the issue to the parsing of the "modified" date from the project metadata.json. This happens in ParsingUtilities.stringToLocalDate(String s)

static public LocalDateTime stringToLocalDate(String s) {
      System.err.println(LocalDateTime.parse(s, DateTimeFormatter.ISO_LOCAL_DATE_TIME));
      return LocalDateTime.parse(s, DateTimeFormatter.ISO_LOCAL_DATE_TIME);
    }

The string is going in, but nothing is being returned (which I think leads the creation of the projectmetadata to fail in ProjectMetadata.loadFromJson"

ProjectMetadata pm = new ProjectMetadata(JSONUtilities.getLocalDate(obj, "modified", LocalDateTime.now()));

The issue seems to be that the existing metadata.json records time with strings like:
"modified": "2014-01-15T21:46:25Z",

But LocalDateTime.parse(s, DateTimeFormatter.ISO_LOCAL_DATE_TIME); requires it to have no timezone information.

If I manually edit the metadata.json to read:
"modified": "2014-01-15T21:46:25",
The project displays OK in the UI

[jackyq2015]

@ostephens May I know your version of Java while you created those old projects? It is a regression introduced by local time stamp. I had a UT for testing the new API can parse the old project. but still missed something. will figure out and close it.

[ostephens]

@jackyq2015 I have projects created over a long period - I don't know which version of Java or OR I was using with them I'm afraid

[jackyq2015]

that is fine. no worry, Owen!

[ettorerizza]

@ostephens I tested yesterday the development version by adding tags to a project already created, and now I get this bug when I try to open the project in question in the stable version 2.8. Is it a related issue?

Note : I can open the project without problem in the development version.

java.lang.NullPointerException
        at java.util.Calendar.setTime(Unknown Source)
        at java.text.SimpleDateFormat.format(Unknown Source)
        at java.text.SimpleDateFormat.format(Unknown Source)
        at java.text.DateFormat.format(Unknown Source)
        at com.google.refine.util.ParsingUtilities.dateToString(ParsingUtilities.java:177)
        at com.google.refine.history.HistoryEntry.write(HistoryEntry.java:112)
        at com.google.refine.history.History.write(History.java:270)
        at com.google.refine.commands.Command.respondJSON(Command.java:311)
        at com.google.refine.commands.Command.respondJSON(Command.java:297)
        at com.google.refine.commands.history.GetHistoryCommand.doGet(GetHistoryCommand.java:54)
        at com.google.refine.RefineServlet.service(RefineServlet.java:170)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
        at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
        at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
22:46:21.738 [                   refine] GET /command/core/get-history (270ms)
22:46:21.742 [          org.mortbay.log] /command/core/get-history (4ms)
java.lang.NullPointerException
        at java.util.Calendar.setTime(Unknown Source)
        at java.text.SimpleDateFormat.format(Unknown Source)
        at java.text.SimpleDateFormat.format(Unknown Source)
        at java.text.DateFormat.format(Unknown Source)
        at com.google.refine.util.ParsingUtilities.dateToString(ParsingUtilities.java:177)
        at com.google.refine.history.HistoryEntry.write(HistoryEntry.java:112)
        at com.google.refine.history.History.write(History.java:270)
        at com.google.refine.commands.Command.respondJSON(Command.java:311)
        at com.google.refine.commands.Command.respondJSON(Command.java:297)
        at com.google.refine.commands.history.GetHistoryCommand.doGet(GetHistoryCommand.java:54)
        at com.google.refine.RefineServlet.service(RefineServlet.java:170)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
        at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
        at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

[jackyq2015]

I guess this project is created using the new version of refine. if so, the reason is that the old API in jdk 7 can not process the date time written from jdk8. but from vise versa is fine. since jdk 7 was dropped already, I will mark it as not fix if it is the case.

[ettorerizza]

@jackyq2015 Yes, looks like projects (or only some projects, don't know) created with the development version can not be opened in previous versions. This is a major change that will need to be reported in bold to users. For example, we can no longer advise those who want to use the RDF extension to export their project and re-import it into LODRefine. cc @ostephens

[jackyq2015]

@ettorerizza Do you have the instruction to do the export/import somewhere(include versions information)? I can take a look. The back capability is possible if there is a need. The thing is that every change to the core could potentially break other extensions. Since we don't have the full picture and usage of those extensions, it is a little bit hard to do the major change and also keep all the back capability at the same time.

[ostephens]

I think breaking backwards compatibility with previous versions of OpenRefine is a very bad idea. OpenRefine 2.5 no longer works on my Mac, so I can't go back and do an export/import for older projects.

I think definitely newer versions of OpenRefine should open projects created in older versions of OpenRefine.

I can see an argument for saying that projects created in more recent versions of OpenRefine may not open in older versions - that feels a less problematic breaking change, although still I think we should avoid this where possible