Hyrax: Work creation fails with RDF was not parsable, Broken token (newline)

[conorom]

Descriptive summary

You can't create a Work with a field that contains a new line anywhere and ends in a double quote (matched or not).
All versions of Hyrax (FCREPO?) are affected. Tested today on Nurax with this version:
Hyrax Version: 3.0.0.pre.beta1
Hyrax Branch: master
Hyrax Revision: b775c5b25c9bf5afb300664195e9931f47cc110e

Also hit this today in our project (Fulcrum) on Hyrax 2.0.0

I'm hoping someone can take over and get the fix into dependencies as needed.

Rationale

This may seem like a real corner case but we've seen it twice in Fulcrum in a few months.
edit: actually I'm scripting Work creation just after opening this issue and I'm seeing dozens of these. We originally noticed it once using the UI.

Anyway, you can successfully edit a field to contain the text indicated. You should be able to create a Work with this entry in that field as well.

Expected behavior

The thing you want gets saved when you create a Work.

Actual behavior

It breaks and can't save the Work.

Steps to reproduce the behavior

Create a new Work using the UI. After filling in required metadata, edit a field that allows new lines (description, say). Type some text that contains a new line anywhere and ends in a double quote.
For example:

some text with a new line after
and some more text with the last character being a "double quote"

This will break character escaping in FCREPO with the following error shown:
RDF was not parsable: [line: 9, col: 1 ] Broken token (newline): ;

Related work

It seems the same (or similar) to these older FCREPO issues, but I wanted to start this as a Hyrax bug and let someone who knows FCREPO take over!
https://jira.duraspace.org/browse/FCREPO-1299
https://jira.duraspace.org/browse/FCREPO-2038

[no-reply]

This error seems to be coming from Fedora. I'm not sure whether problem is a malformed serialization on our side, or something on the other end.

I can reproduce (see screenshot attached) when creating (but not editing?) a work. A next step might be to reproduce minimally with a request directly to FCRepo.

cc: @escowles @barmintor

[conorom]

Thanks for taking a look @no-reply, and for commenting so soon. You rock!
I did add the "metadata" label hesitantly at the end, thinking "hmmm, are problems with specific characters in a metadata field a metadata issue?!". Not certain what tags mean, sometimes, even in my own team's project :)
Anyway... the FCRepo stuff is a little beyond my ken. The divergent behavior I mentioned between create/edit does seem a little unsettling but I guess there must be some separation somewhere, deep down.

[escowles]

I have looked into similar issues before, that involved literals that included a line break and ended with a quote. The basic issue was that to include a line break in an RDF literal, you have to use the triple quote escaping — where literally anything except three consecutive quotes can appear in the value. But then having a quote as the last value meant that the quote and the first two quotes of the final triple-quote counted as the literal terminator, and the extra quote resulted in a syntax error.

I think this could be avoided by stripping the line breaks, escaping all quotes in the literal, padding the end to make sure the final character wasn't a quote, etc.

[escowles]

Also, a script to reproduce this with just curl and Fedora: https://gist.github.com/escowles/14f460a9344b4dec86a2ce28f2cbdec5

Though as I commented on one of those issues, I think the root issue here is that ending a triple-quoted literal with four quotes is not legal turtle.

[conorom]

I hope this is fixable for those that stick with Fedora in the long term. At the very least it should be hidden from the user somehow. Like what I'm doing in my Work-creation script right now is simply appending a space to the field, knowing that it will be successfully edited the next time around (we're iteratively pulling metadata edits from another system running on a Windows server).
Could the code calling Fedora detect this problem, fix it in a similar manner and then immediately edit it to the value the user actually wanted?
Please forgive my complete ignorance of what is happening at the Fedora level. I guess a hack like that could go in Hyrax itself. Pretty horrible though [edit: putting the hack/workaround at the top level is horrible, I meant... obviously it should go in a client dependency]

[barmintor]

If that's the underlying problem, the fix should go into Hyrax. If you're using a backend server that expects data in a format, it's incumbent on the client to submit data in the format. The server is responsible for accepting data that is encoded in that format. That's the contract.

[conorom]

@escowles Do you know off the top of your head why an update with the same data succeeds?

[no-reply]

Thanks @escowles, that helps a ton.

It looks like a bug in the RDF.rb turtle serializer to me. I can produce malformed data with:

require 'rdf'
require 'rdf/turtle'

g << RDF::Statement.new(RDF::URI('http://example.com/moomin'), RDF.value, "has a newline\n and \"ends in a quote\"")

puts g.dump :ttl
# <http://example.com/moomin> <http://www.w3.org/1999/02/22-rdf-syntax-ns#value> """has a newline
# and "ends in a quote"""" .
# => nil

ntriples does not have the same problem, with output:

puts g.dump :ntriples
# <http://example.com/moomin> <http://www.w3.org/1999/02/22-rdf-syntax-ns#value> "has a newline\n and \"ends in a quote\"" .
# => nil

[no-reply]

I was about to open a ticket on https://github.com/ruby-rdf/rdf-turtle, but in doing my due diligence: it looks to me like the Turtle data above is allowed, and Fedroa should accept it. see: https://www.w3.org/TR/turtle/#sec-grammar-grammar and https://www.w3.org/TR/turtle/#turtle-literals

It seems multi-line literals delimited with """ can contain quotes anywhere internally, so long as they don't contain the """ sequence itself.

So """"this" ""is a"" ""literal""""" is allowed, but """this is not a """ literal""" is not.

Possibly a Jena bug, then?

[escowles]

@no-reply I don't know why an update with the same data would succeed. Maybe the normalization is different somehow, or the literal gets filtered out because it hasn't changed?

But my reading of https://www.w3.org/TR/turtle/#turtle-literals is that neither of those examples are acceptable because they both contain """ — that is, """-escaped literals can't end with four or more quotes, or otherwise contain """ if it's in the middle. Though confusingly, it does seem fine to include any number of quotes at the beginning. I see the same behavior locally with raptor, so I don't think this is a Jena bug.

[no-reply]

ah, you're quite right! I'll go ahead an open the RDF.rb ticket.

[no-reply]

This is related to https://github.com/ruby-rdf/rdf-turtle/issues/16.

There's a proposed fix at https://github.com/ruby-rdf/rdf-turtle/pull/17

[no-reply]

A release is pending for rdf-turtle.

Once it's out, bundle update rdf-turtle should fix the issue.