Dataverse: Dataset Metadata - allow one subfield to reference the value of another

[scolapasta]

For example, allow one subfield to reference the value of another

[posixeleni]

This feature will particularly be useful to not have to display the full URL (which can sometimes be very long and take up space in the UI) but instead have Link Text be hyperlinkable.

For example the Topic Class Vocabulary URL would not display but instead just show the Topic Class Vocabulary be hyperlinkable using the Topic Class Vocabulary URL.

[sbarbosadataverse]

@mcrosas @scolapasta @posixeleni
I'm going to ask to add this ticket to 4.4 is possible. I've been cleaning Gary's dataverse by removing the url display since there is no link and it's an eye sore without the linking ability, and Avahan dataverse, and now the Harvard Subscription dataverse--what would it take to activate the url field to create an actual link?

[bencomp]

May I suggest to split or rewrite the issue to describe the intended functionality from the user's point of view? @posixeleni and @sbarbosadataverse seem to want a field type that has a URL and accompanying link text, but

Improve display of compound fields in metadata tab

and

allow one subfield to reference the value of another

both have different meanings to me that do not include URL+link text.

[posixeleni]

@bencomp good point I will change the title of this issue to reflect what we are actually working on.

[posixeleni]

after speaking with @scolapasta and looking over the metadata fields with fieldtype URL this will need to be looked at by development since we should apply this code in a way that can be generally applied to any subfield but with the initial use case being for fields that have URLs.

[posixeleni]

Adding a note here since our work for the Helmsley grant will need this functionality

[bmckinney]

@posixeleni Ultimately we need to render several links for a single PDB ID. Here's our current hack in dataset.xhtml:
<!-- PDB ID --> <div id="pdbid" class="form-group" jsf:rendered="#{!empty DatasetPage.datasetVersionUI.datasetVersion.getDatasetField(datasetFieldServiceBean.findByName('refinedStructurePDBCode'))}"> <label class="col-sm-3 control-label"> <span data-toggle="tooltip" data-placement="auto right" class="tooltiplabel text-info" data-original-title="PDB Coordinates"> #{DatasetPage.datasetVersionUI.datasetVersion.getDatasetField(datasetFieldServiceBean.findByName('refinedStructurePDBCode')).value} Coordinates: </span> </label> <div class="col-sm-9"> <a href="http://www.rcsb.org/pdb/explore/explore.do?structureId=#{DatasetPage.datasetVersionUI.datasetVersion.getDatasetField(datasetFieldServiceBean.findByName('refinedStructurePDBCode')).value}" title="PDB" target="_blank">PDB</a>, <a href="http://www.ncbi.nlm.nih.gov/Structure/mmdb/mmdbsrv.cgi?uid=#{DatasetPage.datasetVersionUI.datasetVersion.getDatasetField(datasetFieldServiceBean.findByName('refinedStructurePDBCode')).value}" title="MMDB" target="_blank">MMDB</a> </div> </div>

[kcondon]

duplicate ticket, #2457 contains screen shot example of urls needing to be listed as text links for Murray datasets.

[bmckinney]

Here's the SBGrid use case for this:

As a Dataverse Admin, I’d like to apply display formats to any metadata value, not just compound metadata. For example, I’d like to display a Protein Database ID as links to multiple websites.

Example:

<a href="http://www.rcsb.org/pdb/explore/explore.do?structureId=#VALUE">PDB</a>, <a href="http://www.ncbi.nlm.nih.gov/Structure/mmdb/mmdbsrv.cgi?uid=#VALUE">MMDB</a>

[pdurbin]

@bmckinney good find. I just added the "SBGrid" label to this issue so we can find it again.

[pameyer]

I'm pretty sure that #3946 will provide the necessary functionality for external links (without subfield referencing).

[pdurbin]

@pameyer yes, I think so too. Thanks.

@sekmiller @scolapasta do you think the work in #3946 will mean that we could implement the request below (from earlier in this issue)?

"For example the Topic Class Vocabulary URL would not display but instead just show the Topic Class Vocabulary be hyperlinkable using the Topic Class Vocabulary URL."

If so, perhaps we have an example change to a tsv file we can include in the pull request, as I was asking for at standup yesterday.

[pdurbin]

Yesterday @sekmiller and I gathered around @dliburd 's desk to get a better understanding of this issue. We concluded that more code is needed and @sekmiller said he'd leave a note here to help a developer know what changes need to be made to the code.

[sekmiller]

What we would need is a DatasetFieldType that would be associated with a url such that the this new DFT would provide a link text for the underlying url to follow. There would need to be a way to associate the two separate fields (which in turn would probably be part of a compound DatasetField)
and to display only the text while having the link follow the url.

[mderuijter]

Here at DANS I have been assigned to work on a certain issue for DANS that is directly related to this issue.

The following screenshot illustrates the use case and shows that I have managed to allow one subfield to reference the value of another:

DANS would like to see the Author Identifier automatically turned into a hyper link associated with ORCID(https://orcid.org/), ISNI(http://isni.org/isni), etc.

To do this a reference to another field and a location to store a base URL for the Identifier Scheme's is necessary. To store the base URL I configured the citation.tsv file to use the Identifier column next to the normal Value column in the ControlledVocabularyValue table. This is a bit of a hack to illustrate what we would like to do here, I'm not sure if this column is suited for this.

Some advice would be welcome.

The reference I implemented by using a special tag in the citation.tsv in the same manner #VALUE, #NAME and #EMAIL can be used. The feature is programmed to use a # + nameOfMetadataField, where nameOfMetadatField corresponds to a field in the #datasetField block in citation.tsv

To get the result shown in the image above, I configured the authorIdentifier row in the citation.tsv as follows:
fieldType has value text
displayFormat has value <a href="#authorIdentifierScheme#VALUE" target="_blank">#VALUE</a>

Before I continue developing and testing we would like to know if this is an acceptable approach for this issue.

[pdurbin]

@mderuijter hi! Can you please provide your citation.tsv file?

[mderuijter]

@pdurbin
citation.tsv.zip

[pdurbin]

@mderuijter thanks. It sounds like you're blocked until you get feedback on your approach so I moved this issue to code review.

Recently we improved out "spin up Dataverse on EC2" scripts to allow for arbitrary custom metadata blocks to be loaded up: https://github.com/IQSS/dataverse-ansible/issues/133#issuecomment-555227955

Some day it would be nice if we could tell those scripts "go get the citation.tsv from this URL". 😄

Of course, you could go ahead and change the citation.tsv file in a branch. Then we could just spin up your branch. @mderuijter do you feel like doing that? You don't have to create a pull request for this (unless you want to). A branch would be enough.

[mderuijter]

@pdurbin Yes, I could that this Monday :)

[mderuijter]

@pdurbin So I think I managed to publish my current code to a branch on my fork.
I think it's publicly accessible, let me know if i missed something.

[pdurbin]

@mderuijter assuming all of the changes you want code review on are this single commit, this is fine for code review purposes. Thanks! Your commit: https://github.com/mderuijter/dataverse/commit/60fd279809ea978f5dee0ec4d80350320d8aae99

[mderuijter]

Yes they are in this single commit @pdurbin :)

[scolapasta]

Hi @mderuijter this is an interesting approach. I'll be discussing tomorrow with our team here at our weekly technical hours and then get back to you with some more detailed feedback.

[jggautier]

When people browsing for data see these metadata values displayed as links (instead of seeing the value and a URL next to the value), I wonder where they think clicking the link will take them.
For the Author Identifier ID, it's being assumed that people will want to go to the author's profile page, like 0000-0003-3053-5252, but that might not be the case for all values that we want to make clickable.

For example, for values like keywords and topic classifications, will they expect the link to take them to:

• the home page of the vocabulary that the term comes from (this is what we ask for in the "Vocabulary URL" fields, like http://id.loc.gov/authorities/subjects.html or http://aims.fao.org/vest-registry/vocabularies/agrovoc)?
• the URL for the term itself (like http://id.loc.gov/authorities/subjects/sh00005743.html or http://aims.fao.org/aos/agrovoc/c_47a98f68.html)?
• other datasets that use the same term (this is how the links in the facets work)

My hunch is that, at least for keywords and topic classifications, people will expect the link to take them to other datasets that use the same term. This is how these links usually work in other metadata catalogs.

After @scolapasta's review with us yesterday, it's clear @mderuijter's commit won't affect fields like keywords and topic classifications, which don't have controlled vocabularies (yet), so right now it doesn't address the use case of this GitHub issue. I chatted with colleagues here about this and they'll have more to say (didn't want it to seem like I was leaving a comment that didn't propose next steps :)

[scolapasta]

Thanks @jggautier, I think one other thing to consider is consistency, so it's clear to the user which stay on site and which go off. Maybe the display of off site ones should always display the full URL (that would be possible with the infrastructure provided by this code.

Do note that this idea could work for not CVV (display format referencing another field), the CVV (controlled vocabulary value) specific part, I believe, is just the url prefixes.

So on to the technical comments:

• The idea of using the #fieldName is nice. It would be better to make it ## for these dynamic ones, so in theory if there were ever, for example, a custom field called "NAME" , there would not be a clash. Also probably best to make this case insensitive. (though I don't know if our field type names are, they probably should be).

• Where to store the url prefix? Clearly the identifier field is not it, as this is meant for an external id for if / when we ever automate retrieval of these CVVs. One idea I had at first was to actually make it the value, since you can control the display value within the bundles. But that won't work either, as you need to actual value for export, plus for conditionals in the code sometimes. Also, could have some scenario with two different identifiers, but the same url prefix? I don't know maybe, or maybe for some other field using this infrastructure some day. Either way, we need a different storage.

• What would make the most sense then is another table. Not setting any of this stone, just typing through ideas, imagine a table called ControlledVocabularyValueDetail which references the CVV and has a name and value. Name could be url_prefix and then the value.

• A bigger issue is how to handle the case where some identifiers have a url prefix while others don't. Then we'd have to add conditionals to the displayformat and I'm not really sure we want to make this a whole "programming / markup" language. (if we really did, I think we'd want to use some existing language rather than continue to make our own)

[scolapasta]

While typing that response out, I actually had a possible solution that could work for all of the last three bullets. Still not sure exactly how it would work, but let me put it in here so others can think on it and see how it could work.

What if the new table, instead of storing the url_prefix stored the display format for a specific cvv? So for those with a url_prefix it could make it into a link, and for those without, it could just display.

The part I'm not sure of is that It would need to work the other way around, i.e. you would define the displayformat for the scheme, rather than the identifier.

Let me know if you need further explanation of how this could work and more importantly, your thoughts on this as a possible solution.

[mderuijter]

@scolapasta Let me start with the best wishes for 2020! :) I have been out of office for the Xmas holidays these past 17 days.

I agree on creating a new table to store the url_prefix or the display format as you describe in your previous comment. How would storing the display format there relate to the #referencedField tag in the .tsv file?

I might be answering my own question here:
So basically the information whether to show a url_prefix or just display normally would be stored in the new table, the #referenceField tag would first make clear which field to reference(so no extra markup to indicate url or normal value), afterwards the code would just fetch the value via the information of the display format. Do I understand correctly?

Could you help me visualise how these display formats would look like in the table?

I certainly agree that an approach that overcomplicates the .tsv file with conditionals and a sort of markup language should be avoided.

[pdurbin]

creating a new table to store the url_prefix

@mderuijter something I brought up in tech hours and that I've been meaning to mention is that once we have good place to store URL prefixes we should refactor the getIdentifierAsUrl method in DatasetAuthor.java to use the new place. Here's a screenshot of the code from https://github.com/IQSS/dataverse/blob/v4.18.1/src/main/java/edu/harvard/iq/dataverse/DatasetAuthor.java#L111-L145

getIdentifierAsUrl was implemented as part of the Schema.org JSON-LD metadata output used by Google Search, etc. Please see also http://guides.dataverse.org/en/4.18.1/api/native-api.html#schema-org-json-ld

[mderuijter]

@pdurbin hello :) I see, so the new table for the URL prefixes could be reused in other places in the code?

[mderuijter]

@pdurbin I believe i'm at the stage where I can create a Pull Request, I commited my new work to my fork. Would a repeat of the previous process be better or just do a PR?

[pdurbin]

@mderuijter a pull request is nice because we can can move it to the code review column of our project board: https://github.com/orgs/IQSS/projects/2#column-5298410

Thanks!

[scolapasta]

@mderuijter I took a look at the PR and it seems to go in the direction of of storing the url prefix in a column (named url prefix). I'm not sure if I'm missing something but it seems like it would cause issues for scheme's that did not have a url? Also, I still prefer a solution that is more scalable for other details besides url prefix (i.e. instead of a single column, do have name / value). Did you consider the idea above about actually storing the display format for each CVV, in that way if their is a url it can make it a link and if there's not, it can made it plain text (but possibly with some useful static prefix)?

[mderuijter]

@scolapasta I did actually consider it, and to be honest I had a different solution at first.
That solution had two columns: displayFormat and value.
displayFormat stored urlPrefix (or any other kind of display format, but if there is no display format in this table plain text will be used) and value had the actual url.
I changed that after some feedback from a colleague, who thought I shouldn't try to solve future problems. I don't think I made it clear enough what the idea was behind this implementation...
Looking back I should have left it the way it was...

Let me change it back to my idea of the implementation you suggested above.