Openrefine: Enhancement : Add fields for projects metadata

[ettorerizza]

I have close to 300 projects in my workspace directory. Some have several years. The problem is I don't know where the data comes from, when exactly I processed them (two years ago, OK, but what month / day?), or sometimes simply what they correspond to. It should be recorded somewhere, for example, in metadata fields beside the project name.

I do not know if it's difficult to implement. If someone else is interested, maybe we could associate to launch a decent bounty? I already put 150 dollars on BountySource.

[wetneb]

That would make a lot of sense! The original source and format of the data could be tracked there. This might also be useful for rich format imports (such as https://specs.frictionlessdata.io/data-package/ ).

[ettorerizza]

@wetneb Interesting ! Open refine has become a sort of data warehouse for me, and for others also I guess. It is widely used in the information sciences community. For a librarian, not being able to add metadata is a torture.

[wetneb]

Some ideas here: #1045

[ettorerizza]

We have an offer for $ 250 (100 missing)

https://www.bountysource.com/issues/47809745-enhancement-add-fields-for-projects-metadata

[wetneb]

Hey @denim2x, welcome! Can you give us an idea of what your plans would be for this feature? There are various issues related to project metadata (see the tag: https://github.com/OpenRefine/OpenRefine/labels/metadata, although not all of them are relevant), so it would be good to think about a solution with these broad use cases in mind.

[magdmartin]

Here is a summary of the comments and suggestion from the different tickets together (adding few of my own). Feel free to comment and enrich.

List of meta to create:

• created (from metadata.json)
• modified (from metadata.json)
• encoding (from metadata.json)
• other the import option settings which were used (from #1045 - #182)
• file source (path or url?) (from #1045)
• free form of comment (from #1045)
• number of rows (from #657)
• number of columns (from #657)
• number of operation
• project id

Make the list of project sortable by metadata (from #657)

Meta information are available from metadata.json ; /command/core/get-all-project-metadata, /command/core/get-models and /command/core/get-columns-info.

[ettorerizza]

Dublin Core can be an inspiration for metadata fields, although not all of the fifteen elements are needed in this case.

Title
Creator
Subject
Description
Publisher
Contributor
Date
Type
Format
Identifier
Source
Language
Relation
Coverage
Rights

[wetneb]

Thanks a lot @magdmartin for this summary of the other issues! That's very convenient. I think it would also be good to have other generic metadata fields as @ettorerizza proposes.

We really should keep #778 and #1096 in mind for that. This document gives some ideas about what sort of metadata data packages provide: https://www.w3.org/TR/2015/PR-tabular-metadata-20151117/

If we want to support these sort of formats, I think it would be reasonable to designate a list of core metadata fields (things that apply to all OpenRefine projects and would be exposed in the UI) and also provide some support for custom metadata provided by the original import format. This extra metadata could be reused by exporters or extensions (so that, for instance, no metadata is lost if you import a data package, clean it, and export it to another data package).

[denim2x]

@wetneb PLease clarify: do you require PR-tabular-metadata conformance for this issue?

[wetneb]

@denim2x I think the Tabular Metadata specification goes way beyond what @ettorerizza had in mind when he started his issue and bounty. So I do not think implementing that should be required to close this issue.

However, solving this issue will probably involve some changes to the model, possibly impacting the format in which projects are persisted (if we want to add fields that cannot be stored yet). Doing this sort of change is slightly complicated: we might need to adapt the importers so that they fill the fields we introduce, ensure the existing workspaces will be migrated seamlessly to the format that supports the new fields, and so on.

So, this relatively heavy change should be made with the other use cases in mind: ideally, we should avoid further model changes and adopt a model that works for a wide range of metadata needs. That does not mean the UI should fully expose the flexibility of this new model, or anything. But maybe that will turn out to be too complicated and therefore out of the scope of this issue. I just want us to have that discussion and not rush too much on adding very particular fields without stepping back and looking at the bigger picture.

[fpompermaier]

Hi to all,
I'm happy to see OpenRefine regaining interest from mantainers!
I think that nowadays Refine should be able to load data from local filesystem (as usual) but it should also evolve to exchange data from/to external systems like databases or Hive/HBase tables or Parquet Directory, delegating transformation of those datasets to Big Data engine like Spark or Flink (if the required memory to process a dataset goes over a certain threshold).
In this direction, it should be nice to introduce into Refine the concept of Catalog, in order to be able to properly manage metadata and interface with other existing repositories (like HCatalog and similar).
In this direction, a change to Refine datamodel could be also useful (e.g Apache Arrow could be a perfect fit).

I know that comment s a little bit out of discussion but I'd like to stimulate some discussion in this sense..

[denim2x]

@wetneb Are you considering using Apache Arrow in OpenRefine?

[wetneb]

@denim2x I don't know! Do you think it would make sense to use it to solve this issue? I am not familiar at all with that.

[thadguidry]

Breathe everyone. :)

Don't worry about compatibility with external projects. Others outside of OpenRefine can take that task because its just simple JSON with our existing metadata format.

@fpompermaier We have a plan, long term for bigger data under Performance Improvements 2017

All -

1. What you want at the end of the day is reusing our simple key/value pair storage per project. We have that in place and it's stored like so in metadata.json

1. For those doing the work, this search actually shows quite well (using "last modified") all the parts of OpenRefine that get touched or will need to be touched or reviewed. https://github.com/OpenRefine/OpenRefine/search?utf8=%E2%9C%93&q=Last+Modified&type=

2. I'd prefer that the design allow the user with a button and simple dialog (reuse our existing dialogs in many places already to allow checkbox choice) to choose which keys are important to show from that metadata on the project open ui and show them as columns as @ettorerizza wants. Don't pick which keys for them, but having the initial default of created and notes would be a very wise choice, the other key/values should be allowed to be filled in by the author. Similar to the UI experience we have for our preferences.vt You could have a button next to each project name that opens the metadata table reading in the metadata.json and allowing editing in table form, like we do for preferences.vt
   See [1] https://github.com/OpenRefine/OpenRefine/blob/master/main/webapp/modules/core/scripts/preferences.js
   [2] https://github.com/OpenRefine/OpenRefine/blob/master/main/src/com/google/refine/ProjectMetadata.java#L108 , etc

[ettorerizza]

My initial idea was to have fields directly editable right next to the project name, but a key-value menu like preferences makes a lot of sense. I just want to mention a possible UI implementation already used by Google Fusion tables. A screencast will be clearer than my explanations.

[thadguidry]

@ettorerizza Yeap, that's the idea, an ABOUT or ? question mark button next to each project name that can open up a table of KEY / VALUE columns similar to our preferences.vt Also, being able to click checkboxes (up to perhaps max 4, or whatever works) those KEY's Values that you want to "SHOW ON PROJECT PAGE" as an additional column (space available) for easier instant viewing when the project page shows.

[denim2x]

@thadguidry @wetneb Any further information for working on this feature would be very appreciated

[denim2x]

@ettorerizza It'd be nice if you could provide some mockups with the improved UI (with steps)

[ettorerizza]

@denim2x Personally, I do not have any other expectations than those mentioned above: key-value fields that allow the user to add custom metadata to each project and a summary table of the metadata that allows to sorts and find project by date, by keywords, by title, etc. @thadguidry or @magdmartin probably have a more precise and more general view of what the enhancement should look like in practice.

[wetneb]

@denim2x any news about this? Let us know if you have any question about the feature.

[ettorerizza]

Is anyone able to connect to BountySource? https://salt.bountysource.com/teams/openrefine

[wetneb]

@ettorerizza it looks like their SSL certificate expired yesterday - I expect it will be renewed soon. I don't think they ran away with your bounty ^^

[jackyq2015]

I have a little concern. I haven't received my payment after more than 2 months. I was told that "the payment though cheque is rare".

[wetneb]

@jackyq2015 yeah, actually maybe we should have a discussion about Bountysource… it has a number of issues so we might want to switch to something else instead.

• there is a fairly high service fee
• github integration is broken
• the website is currently unavailable
• bounties take a long time to be paid out

I quite like Liberapay (https://liberapay.com/ ), which is actively maintained and has a smaller service fee (basically just the payment processing fees of their bank). It does not allow to put bounties on specific issues though.

[jackyq2015]

@denim2x any progress on this?

[ettorerizza]

@jackyq2015 He stopped working on the issue. I wonder why. Too difficult for the bounty amount ?

[jackyq2015]

@ettorerizza I think one of the reason could be the requirement is not so clear for him.I can take on this if you can narrow down the required fields. There are some dynamic data such as row number I don't think it's good to put as meta data. My thought is to pick some from:
Title
Creator
Subject
Description
Publisher
Contributor
Date
Type
Format
Identifier
Source
Language
Relation
Coverage
Rights

And we also store the "import options" as a json data filed to the meta data.

Layout could be adding the columns for some fields on the index page and also provide a "About" link to edit all the supported meta data.

[ettorerizza]

Thanks for your answer, jacky. If I had to choose ten metadata fields, putting aside the dynamic metadata, here is what my selection would be.

Identifier (To make sure everyone is working on the same version. MD5 ?)
Creator
Modified by/contributors (not essential for me)
Subject/project (Several refine projects may be linked)
Description (free form of comment)
Source (filepath or url)
Size (at the creation ? Essential for cleaning old projects too heavy)
Created (from metadata.json)
Modified (from metadata.json)
Encoding (from metadata.json)

Does anyone have any other suggestions?

[wetneb]

what do you mean by "identifier" (what would you take the MD5 of?)

[ettorerizza]

@wetneb I mean something that makes it possible to identify the project in an unambiguous way. Oddly, Open Refine allows two projects to have exactly the same name. But this could be just the ID under which the project is stored in the Workspace Directory (1464554182862.project, etc.) But I do not know how that number is created. May two people have two different Open Refine projects with the same ID?

[wetneb]

@ettorerizza oh, then we already have the project id for that (the one that you find in the URL of a project in OpenRefine).

[magdmartin]

I will prefer if we follow the proposal from @thadguidry in #1221#issuecomment-320259551 and provide key/value store where the user can define its own field.

Create new field / metadata

In preference.vt the user can:

• create the key he want to track
• define which keys are displayed on the homepage

We need to provide a link to the preference page from the homepage.

Display and edit the metavalue

As @jackyq2015 suggested: we can do it from the homepage by adding the columns for some fields on the index page and also provide a "About" link to biew and edit all the supported meta data.

We can also add a button next to Export to edit the metadata once someone opened the project.

Default metadata

I suggest we keep as default metadata information we already have available in OpenRefine . I compiled the following list of default key:

• created at (from metadata.json)
• modified at (from metadata.json)
• encoding (from metadata.json)
• other the import option settings which were used (from #1045 - #182)
• file source (path or url?) (from #1045)
• free form of comment (from #1045)
• number of rows (from #657)
• number of columns (from #657)
• number of operation
• project id

[thadguidry]

I agree with @magdmartin Let's have a "predefined" set of fields that the preferences.vt would have and from those, select few of the fields that are exposed and displayed on the Project homepage by default out-of-the-box installation of OpenRefine.

I have no opinion on what those "predefined" fields should be, but defer to the community to decide. After all, they will be able to change those fields at anytime with the "About" link or "Metadata" link, or whatever we decide to call the link.

Regarding the implementation, the preferences.vt dataset can be arrayed. The "preferences" for OpenRefine program itself and the "metadata" for a ProjectID. We want to ensure we keep those separate views or tabs or whatever. "preferences" affect ALL Projects. "metadata" is individual per ProjectID. However we want a single link or button for each ProjectID in the Project homepage listing to click on and takes our users to that ProjectID's "metadata" to edit. I do not see a need for a "preferences" for each ProjectID, but that could also be explored or warranted if needed. (By the way, Extensions have their own preferences setup already and its separate from this discussion.)

Hope that is clear and makes sense @wetneb @magdmartin @jackyq2015

[jackyq2015]

I finished most of the work as we discussed above. Those are some screenshots that I want to share with you firstly:

• Index page(client name is user defined)
  

• To display all the meta and edit them
  

• To define the user metadata from preference page
  

There is some minor issue to populate the data etc. will fix in few days. Also just a heads up, we plan to add columnar metadata to support further development. that will be added in near future as separate PR.

[ettorerizza]

@jackyq2015 At first glance, this really looks like what I had in mind! Good job ! Just a question for my information: I see that the metadata fields are not sortable. Is there any way to find a project by its metadata in this version?

[jackyq2015]

@ettorerizza Glad you like it. :) There is no support for "search project by metadata" yet. But I will try to add a function to sort by header. is that work for you?

[ettorerizza]

As far as I'm concerned, you can claim the bounty. I do not know what @magdmartin thinks about it.

[thadguidry]

@jackyq2015 what appears or happens when the users clicks on all those EDIT buttons ? Why have an EDIT button , when clickable input fields could be used and then a single cancel and save button ?

[jackyq2015]

@thadguidry when click edit, a popup box will show and you can enter the new value. It is the same way we edit the preference value. Ideally, we should use a pretty popup box including the height and other css. But I would like to tackle it as a separate task for all the prompt/alert function.

[thadguidry]

@jackyq2015 ok understood. 1 step at a time :)

[magdmartin]

It looks great @jackyq2015 thanks a lot. My only comment is to add a link to the preference page - see #1303

@ettorerizza ctrl+F also work to search for a specific meta on the page ;)

[jackyq2015]

@ettorerizza sort added
@magdmartin pref link added
done by PR #1301

[ettorerizza]

@jackyq2015 I can not launch the development version anymore.

Error in console

org.mozilla.javascript.EcmaError: TypeError: [JavaPackage com.google.refine.commands.project.SetProjectMetaDataCommand] n'est pas une fonction, est un object (file:/C:/Users/Boulot/Documents/GitHub/OpenRefine/main/webapp/modules/core/MOD-INF/controller.js#71)
        at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3654)
        at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3632)
        at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3660)
        at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3679)
        at org.mozilla.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3734)
        at org.mozilla.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3722)
        at org.mozilla.javascript.ScriptRuntime.newObject(ScriptRuntime.java:2324)
        at org.mozilla.javascript.gen.c22._c1(file:/C:/Users/Boulot/Documents/GitHub/OpenRefine/main/webapp/modules/core/MOD-INF/controller.js:71)
        at org.mozilla.javascript.gen.c22._c4(file:/C:/Users/Boulot/Documents/GitHub/OpenRefine/main/webapp/modules/core/MOD-INF/controller.js:304)
        at org.mozilla.javascript.gen.c22.call(file:/C:/Users/Boulot/Documents/GitHub/OpenRefine/main/webapp/modules/core/MOD-INF/controller.js)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:398)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3065)
        at org.mozilla.javascript.gen.c22.call(file:/C:/Users/Boulot/Documents/GitHub/OpenRefine/main/webapp/modules/core/MOD-INF/controller.js)
        at edu.mit.simile.butterfly.ButterflyModuleImpl.scriptInit(ButterflyModuleImpl.java:636)
        at edu.mit.simile.butterfly.ButterflyModuleImpl.init(ButterflyModuleImpl.java:94)
        at edu.mit.simile.butterfly.Butterfly.initializeModule(Butterfly.java:476)
        at edu.mit.simile.butterfly.Butterfly.initializeModule(Butterfly.java:470)
        at edu.mit.simile.butterfly.Butterfly.configure(Butterfly.java:451)
        at edu.mit.simile.butterfly.Butterfly.init(Butterfly.java:308)
        at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
        at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
        at com.google.refine.RefineServer.configure(Refine.java:296)
        at com.google.refine.RefineServer.init(Refine.java:208)
        at com.google.refine.Refine.init(Refine.java:114)
        at com.google.refine.Refine.main(Refine.java:108)

[jackyq2015]

@ettorerizza just put a fix. please try again.

[ettorerizza]

Same problem.

[jackyq2015]

After pull, did you run ant? I tried on Linux, windows and Mac. there is no issue to start

[ettorerizza]

@jackyq2015 Oh, gosh... Sorry for my stupidity, I'm tired. It works !

[ettorerizza]

Hmm, looks like the edit buttons (by clicking on "edit metadata") appear in the old projects, but not for those just created.

EDIT : OK, I see, the problem comes from the ImportOptionMetadata row, far longuer than the screen.

[jackyq2015]

Can you move the dialog to the left a little bit?

Also, when change the userMetaData, the server has to restart to load it since it is a singleton.

[ettorerizza]

"Also, when change the userMetaData, the server has to restart to load it since it is a singleton."

Yep. Clicking on the column name did not seem to have any effect, but the sort works after restarting. I will do other tests tomorrow. Thanks again for everything, @jackyq2015 !

[ettorerizza]

Just before I forget it: sorting by "last modified" does not work as one might expect. It does not sort values as dates, but as strings. If you click on it, all the chronological order of your projects is lost.

[jackyq2015]

@ettorerizza Thanks for the test. created issue #1306, #1307, #1308 to track them separately

[ettorerizza]

@jackyq2015 Wow, it looks like you did a lot of work last night! Thanks !

I did some tests again this morning and here are the few bugs I have identified so far:

• Sorting by date still does not work properly, even if the column now displays a date. I guess that projects are sorted by hour-minute-second rather than by day-month-year.

• Long free descriptions tend to mess up the display. Related to #1306

• There is a problem with user metadata. The new custom columns appear in the table, but they appear as "undefined" in the editing window and seems not editable. This is probably related to this error message in the console (unless I made a mistake somewhere by creating the metadata in preferences?):

11:59:16.032 [       FileProjectManager] Exception when mergeEmptyUserMetadata (831ms)
org.json.JSONException: JSONObject["name"] not found.
        at org.json.JSONObject.get(JSONObject.java:406)
        at org.json.JSONObject.getString(JSONObject.java:577)
        at com.google.refine.io.FileProjectManager.mergeEmptyUserMetadata(FileProjectManager.java:436)
        at com.google.refine.io.FileProjectManager.loadFromFile(FileProjectManager.java:377)
        at com.google.refine.io.FileProjectManager.load(FileProjectManager.java:338)
        at com.google.refine.io.FileProjectManager.<init>(FileProjectManager.java:92)
        at com.google.refine.io.FileProjectManager.initialize(FileProjectManager.java:79)
        at com.google.refine.RefineServlet.init(RefineServlet.java:129)
        at javax.servlet.GenericServlet.init(GenericServlet.java:241)
        at edu.mit.simile.butterfly.Butterfly.init(Butterfly.java:180)
        at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
        at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
        at com.google.refine.RefineServer.configure(Refine.java:296)
        at com.google.refine.RefineServer.init(Refine.java:208)
        at com.google.refine.Refine.init(Refine.java:114)
        at com.google.refine.Refine.main(Refine.java:108)
11:59:16.034 [       FileProjectManager] Exception when mergeEmptyUserMetadata (2ms)
org.json.JSONException: JSONObject["name"] not found.
        at org.json.JSONObject.get(JSONObject.java:406)
        at org.json.JSONObject.getString(JSONObject.java:577)
        at com.google.refine.io.FileProjectManager.mergeEmptyUserMetadata(FileProjectManager.java:436)
        at com.google.refine.io.FileProjectManager.loadFromFile(FileProjectManager.java:377)
        at com.google.refine.io.FileProjectManager.load(FileProjectManager.java:338)
        at com.google.refine.io.FileProjectManager.<init>(FileProjectManager.java:92)
        at com.google.refine.io.FileProjectManager.initialize(FileProjectManager.java:79)
        at com.google.refine.RefineServlet.init(RefineServlet.java:129)
        at javax.servlet.GenericServlet.init(GenericServlet.java:241)
        at edu.mit.simile.butterfly.Butterfly.init(Butterfly.java:180)
        at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
        at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
        at com.google.refine.RefineServer.configure(Refine.java:296)
        at com.google.refine.RefineServer.init(Refine.java:208)
        at com.google.refine.Refine.init(Refine.java:114)
        at com.google.refine.Refine.main(Refine.java:108)

• The row number column does not extract project metadata (even for a new project) and is not editable manually.

Note: I bothered every time to restart Open Refine to check if the bug was persistent or not.

• Detail: we can remove the small "sort triangles" in the first three columns (delete, rename, about). In addition, is it still necessary that the text of these cells appear only on mouse over? It's a question of aesthetics, but these three empty columns seem weird.

Here is a summary of these bugs in a walktrough screencast:

[jackyq2015]

@ettorerizza I fixed the bugs you mentioned above except "user metadata" one and the row number one. The row number one is related to the singleton refresh. It was populated but has to be reloaded.
I disabled some metadata which should NOT be changed manually.

could you please create a separate issue with detail to reproduce the user metadata issue? Thanks

One off the topic question, is there any way that I can drag back and forth a gif file?

[ettorerizza]

Great @jackyq2015 ! If we put aside the "user metadata", I think we are close to the end. I see that you have chosen as an option to keep a fixed size for the display of columns on the screen, and a fixed size for the columns. I think it's a good choice considering the limitations of the UI. Here is what it looks with the original metadata and three additional columns of user metadata.

With a 100% zoom in the browser (15-inch screen).

With a zoom of 80%.

It is therefore necessary to save space. The most obvious way seems to strongly reduce the first three columns (or eliminate them with a clever trick), as well as the column "row number" (it can be fixed at the width of 7 digits, I can not see how to manage more than 99 million lines in Open Refine) .

Everyone: What do you think about it?

[thadguidry]

@jackyq2015 @ettorerizza The idea was to have each column as optional to display on the project page. We just need a checkbox column (Show/Hide) next to each metadata column name on the edit dialog to handle this. The Project Name column should always show and not allow to hide. We'll provide resizeable columns throughout the UI's later on in further milestone after December 2017.

[jackyq2015]

The width css is not taken for some reason after apply of the table sorter. Will try to fix it to make it nicer. But as Thad said, the long term solution will be the resizeable table.

[ettorerizza]

The "edit cells" -> Transform" and "All -> transform" windows sometimes extend the entire screen (but not the "add column based on this column" window)

Not really a problem for me.

[jackyq2015]

@ettorerizza I fixed the table width issue. Not perfect as I said. but much better. :)

Also the refresh issue was fixed. Please have a try.

[wetneb]

@jackyq2015 thanks a lot for taking my rants into account! That was quick! I have just tried the new projects view and it looks great! Here are a few other things:

1. When your mouse is not positioned on any row of the table, the first three columns look a bit mysterious - maybe it would be worth having some labels for them too? Or not having empty headers at all?
   

2. I just tried creating a project from a URL (CSV, with default import options) and the row count in the project list was displayed as 0. It took me a hard refresh (Ctrl-F5) to see the correct count.

3. Do we store the importer used to create the project anywhere? I can see you store the importing options (that's great!) but I would find it useful to know which importer was used.

4. Quite a lot of comments about the metadata view:
   

[jackyq2015]

@wetneb Thanks for the feedback. I agreed all of them. I will be working on the improvement.

For that 3 mysterious link at left hand, I think we can just give them a label instead of hiding it. Also we can remove the header borders for those 3.

For the "about" view, are you suggesting to remove the background of "edit" button to make it fit?

Any thoughts?

[jackyq2015]

Pushed the code. It is much better now! cheers

[jackyq2015]

I would suggest to close this issue if there is no objection. Please raise issue if there is any.

[wetneb]

@jackyq2015 it looks much better now! I still see a few points that we can still discuss:

• do we really want to display the fields with raw JSON? Especially the one with column metadata (as it will currently remain empty and should be eventually manipulated from the project view anyway?)
• the "Create Date" is currently marked as editable but does not have any validation (it's possible to type in invalid datestamps)… I would either make it properly editable (with a stock date picker widget) or simply disable editing for that (why would you change that date anyway?)
• I like the light blue better in the metadata view (and it's more in line with the overall style of the UI) but I still think it's a bit too heavy - why don't we reuse the two colors used in the grid view for that? (#fff and #f2f2f2). That would be more consistent I think.
• the "About" cell is capitalized whereas "rename" isn't. I would change it to "about" for consistency. (Also, I think the other labels should only be capitalized for the first word - again for consistency with the rest of the UI - so "Project Name" -> "Project name")
• Do we still need the rename link now that the name can be edited in the metadata view?
• It's not clear to me that "about" is the think you should click if you want to edit what you see in the table. I would find it more intuitive with "edit", or even better: a pencil symbol (for consistency with the delete button on the left hand side).
• Would it make sense to have a metadata field that stores the original format (or name of the importer) of the dataset? It does not look like this is stored in the import metadata.

If you're fed up with these UI tweaks let me know, I can do them :) I just think it's important to keep the UI as clean as we can, especially for a part that is seen a lot.

One nice byproduct of this change is that the delete buttons are now always visible - I remember that it took me a while to figure out how to delete a project and that is now much clearer!

[wetneb]

Oh, and two other things (sorry):

• the delete button is placed weirdly with Chromium (not centered in its cell)
• it would be nice if the "last modified" could still be displayed in the nice human-readable form they had while preserving the sorting functionality… I have no idea how hard that would be. But in the current state it's kind of a regression in terms of user-friendliness.

[jackyq2015]

@weblate please see below:

• do we really want to display the fields with raw JSON? Especially the one with column metadata (as it will currently remain empty and should be eventually manipulated from the project view anyway?)
  A: For column metadata, I would imagine we may need an extra view just for handling its complexity. For the import option, since it's read only. It should be sufficient for now
  _THAD AGREES WITH ANSWER , WE WANT A DIFFERENT VIEW FOR COLUMNAR META EDITS_
• the "Create Date" is currently marked as editable but does not have any validation (it's possible to type in invalid datestamps)… I would either make it properly editable (with a stock date picker widget) or simply disable editing for that (why would you change that date anyway?)
  A: Will disable the edit for this field
• I like the light blue better in the metadata view (and it's more in line with the overall style of the UI) but I still think it's a bit too heavy - why don't we reuse the two colors used in the grid view for that? (#fff and #f2f2f2). That would be more consistent I think.
  A: Agree. will change style.
• the "About" cell is capitalized whereas "rename" isn't. I would change it to "about" for consistency. (Also, I think the other labels should only be capitalized for the first word - again for consistency with the rest of the UI - so "Project Name" -> "Project name")
  A: I can change the rename to Rename for consistence. But I think we should keep the "About" rather than "edit" since "edit" may means "edit the data". For the first upper case only, will make the change.
  _THAD AGREES WITH ANSWER_
• Do we still need the rename link now that the name can be edited in the metadata view?
  A: It is a shortcut. I would prefer to keep it
  _THAD AGREES WITH ANSWER_
• It's not clear to me that "about" is the think you should click if you want to edit what you see in the table. I would find it more intuitive with "edit", or even better: a pencil symbol (for consistency with the delete button on the left hand side).
  A: see above comments
• Would it make sense to have a metadata field that stores the original format (or name of the importer) of the dataset? It does not look like this is stored in the import metadata.
  A: There is key "source" in the import option. from that mostly likely you can tell the file format. Also the options are different from importer to importer.
• the delete button is placed weirdly with Chromium (not centered in its cell)
  A: will try to fix it if I can. :)
• it would be nice if the "last modified" could still be displayed in the nice human-readable form they had while preserving the sorting functionality… I have no idea how hard that would be. But in the current state it's kind of a regression in terms of user-friendliness.
  A: For me, the "human-readable form" is actually harder to read than the current way(at least for me). Especially given that it can sort asc and desc. For example, if I want to find a project I did at 2017/11/01, it is much easier to do a sort then I can locate it. For the old way, you have to calculate it's XX days ago then you can go get it. It is personal preference. If there are more person like the old way to present, it can be done.
  _THAD AGREES WITH ANSWER AND THIS HELPS WITH CONSISTENCY INTERNATIONALLY AS WELL_

[thadguidry]

@wetneb @jackyq2015 My Comments are in italics above.

[wetneb]

The "source" key in import metadata is not enough to tell which importer was used! For instance, if it's just from the clipboard (and in general not all file names indicate their format correctly, and the user might have chosen a different importer)… So I think it would really be worth storing that explicitly!

I'm curious to know if users want to see ISO timestamps rather than human-formatted dates… I think it looks quite hacky (especially with all the useless "000Z" at the end…). If you prefer to see dates rather than time deltas, then dates should be human-readable too. Think about our non-programmer users: they should not have to know about ISO timestamps to use OpenRefine!

[ettorerizza]

The display of these dates on a 15-inch screen is problematic. It might also be necessary to reduce a little the font of the words "rename" and "about".

A shorter timestamp, without the milliseconds, could solve the problem easily.

[thadguidry]

Jacky just need to make the fields selectable to display or not. That probably will solve your problem better.

[jackyq2015]

How about shorten the date to such as "2017-11-11 07:25 PM" ? The rename and about font is already small. Also I can reduce the width of "Row Count" a little bit but not too much(the header has to fit in)

[jackyq2015]

Above was done by PR #1323

[wetneb]

I agree "Rename" and "About" take a lot of space… That's why I think "Rename" should be removed. I don't see any reason why this shortcut should be kept:

• it only saves one click anyway
• it is not placed logically next to the title
• why provide a shortcut for the title (which is generally filled at project creation) while there are so many other empty fields that the user could want to fill?

[ettorerizza]

@wetneb @jackyq2015 If I rely on my own practices, it's true that I rename projects much more often while they are open. But I do not know if everyone does it that way.

[jackyq2015]

I will remove the "Rename" link from the index page if there is no objection by next Monday before release 2.7.2. All, please speak out if you want to keep it as is.

[jackyq2015]

"Rename link" was removed by PR #1324

[wetneb]

@jackyq2015 in the end do we have tests for this feature?

[ettorerizza]

Thanks a lot for all this work, @jackyq2015 ! As I said above, you already deserved the bounty with the first draft.

The system as it suits me perfectly. However, I wonder if it's ready for a public release. There are still some display issues that could frustrate users.

The dates display in Chrome or Edge on a 15-inch screen with a 100% zoom is perfectible, but that's just aesthetics:

Note that there is not this problem with Firefox:

But in every browsers, the metadata editing window comes out of the screen again. Each time you have to zoom out to access the "edit" buttons.

There is also a small problem of consistency between the "transform" window, which is displayed all along the screen, and other edit windows, which remained the same.

All this annoys me very little. I set Open Refine on a 90% screen zoom. But I'm afraid it looks a little buggy in an official version 2.7.2.

What do you think, everyone?

[thadguidry]

@ettorerizza Yes, these need to be cleaned up a bit in our CSS files.
@jackyq2015 It annoys me A LOT. Let's open additional issues to note the problems and then fix them. I also would like to see the Transform window back to the way it was as not automatically stretched 100% in width of window.

Next Release is blocked until these UI issues can be cleaned up.

[jackyq2015]

@ettorerizza I understand you concern. Do you mind open 2 issues(or more if you want) for metadata and the transform UI. In the meantime, I will take a look what we can do

[jackyq2015]

@ettorerizza @wetneb @thadguidry created PR #1331. Please try / review.
when doing the test for long string(for example description), please add while space in between like normally. that's how browser know where to wrap the word.