Node-solid-server: Persist Content-Type of resources without file extensions

[dmitrizagidulin]

(Creating a tracking issue for PRs https://github.com/linkeddata/ldnode/pull/159 and https://github.com/linkeddata/ldnode/pull/229)

Problem: When writing a resource without a file extension, there is currently no way for the server to remember what its content-type was. So, when it's next requested, the server doesn't know what that resource's content type is, and returns text/turtle by default.

Potential Solutions:

1. Add the content type to the resource's .meta file. Have the server check the .meta (if the resource doesn't have a file extension), discover the content type, and return it with the response.
2. (Route taken by PR #229) Append a transparent file extension to the resource, only when the resource does not include a file extension. For example, if you PUT a resource named article with an HTML content type, the file will be named article.$.html on disk. When reading the resource, ldnode detects the special extension (and filters it out), and returns the resource with the correct content-type.

[sandhawke]

Do we have a spec for how resources live on the filesystem? It seems like something a lot of tools might want to agree on, even though it's not the wire protocol, and not everyone has to use a filesystem, of course. My copy-web-to/from-filesystem tool needs the same thing.

[dmitrizagidulin]

@sandhawke - What about solid servers that don't use the filesystem at all, like Meccano?
There shouldn't really be a spec for that, since the only way apps (like the migrator app) should be accessing those files is through the API.

[sandhawke]

@dmitrizagidulin I'm just talking about the systems that happen to want to join in, in using the same filesystem conventions. Meccano may want to use it, too, for its export-in-a-zip-file and import-from-a-zip-file functions.

Other elements of this spec would be where metadata and acls go. I'm thinking ...$describedby.ttl and ...$acl.ttl (and this can be extended, where basically the word after the $ is the rel text)

So if I GET /foo and get html, and there's a rel=describedby to /foo,meta.ttl and a rel=acl to /foo,acl then I'll create three files

• foo$.html for the main data
• foo$describedby.ttl for the metadata (note that its name is NOT what the resource name was)
• foo$acl.ttl for the acl (again, this filename bears no relation to where the ACL was served from)

How's that sound, @timbl ?

[nicola]

I agree with @dmitrizagidulin. Although it is nice to have everyone being on the same page on how to represent data on the file system, this is beyond the solid spec - in other words, everyone can implement it the way they want, the _import/export_ that you mention should be done in a ldp way, exactly to avoid this problem, where all of this is abstracted.

[sandhawke]

I completely agree it wouldn't be part of solid, per se. Certainly it wouldn't affect normal clients at all. That's why I asked if we "have a spec". (Although I guess if you view solid as a collection of specs my phrasing is ambiguous.)

The "filename conventions" or whatever spec would be both useful and potentially confusing. As you guys suggest, people might think it's part of solid and somehow think they have to follow it. I would think the solid spec could be clear enough about that. Solid could point to it and explain what it's for. I'm sure you'd agree it would actually be useful if gold and ldnode and solid-copy (my current name for the command line copy client) all happily used the same filesystem conventions. At the same time, of course, when folks aren't sharing a filesystem between multiple people or multiple tools, they can do as they like, and even if they are they can use different filename conventions and still be solid.

[nicola]

One point @sandhawke, solid-copy because it is an "LDP" copy, should not know about how the server store files

[sandhawke]

Indeed, solid-copy uses solid to talk to the remote server. But it needs some convention for how it is going to store files on the local filesystem. And if that convention is the same as gold and ldnode, then it'll avoid some pain for people running their own servers.

[elf-pavlik]

IMO all the RDF Sources should allow content negotiation. How they get stored in filesystem, document database, quads store etc. should not concern clients and stay a choice of internal plumbing used by particular server.

I would recommend that ldnode simply stores all the RDF documents as Normalized RDF.
In my prototype of a server I pass documents through https://github.com/rdf-ext/rdf-normalize before storing them. Later I parse them again with https://github.com/rdf-ext/rdf-parser-n3 and depending on content negotiation response gets serialize with one of https://github.com/rdf-ext/rdf-ext#serializers

[elf-pavlik]

Problem: When writing a resource without a file extension, there is currently no way for the server to remember what its content-type was. So, when it's next requested, the server doesn't know what that resource's content type is, and returns text/turtle by default.

For RDF Source, shouldn't the server just store the named graph and don't give any importance to the content type of the payload which provided the g-text ?

Only Non-RDF Source seems to need a way to save information about their content type. Maybe RDF Source which they have _desribedby_ relation with, could include triple with predicate similar to as:mediaType

[dmitrizagidulin]

For RDF Source, shouldn't the server just store the named graph

How do you picture the server storing the named graph? For node-solid-server, it stores them as files. And files need extensions, otherwise how is the server supposed to know how to serve them back? (The server does not look inside the file, incidentally).

And yes, this is mainly a problem for non-RDF sources.

[elf-pavlik]

how is the server supposed to know how to serve them back?

For RDF Source server could have configured default media type e.g _text/turtle_. I assume that we talk about cases where client doesn't include _Accept_ header to content negotiate preferred representation.

And yes, this is mainly a problem for non-RDF sources.

I think RDF Source and Non-RDF Source deserve different treatment

[zoggy]

@elf-pavlik:

Only Non-RDF Source seems to need a way to save information about their content type. Maybe RDF Source which they have desribedby relation with, could include triple with predicate similar to as:mediaType

I just discover this issue and I this is the way I chose to store the content type of resources posted or put by clients: the contents of the content-typeheader of the POST or PUT request is added to the .meta associated to the new/modified resource. I use dc:format though, but without strong conviction about this property rather than another one.

[csarven]

There is one alternative approach. Leave the resource as is. Test different serializations. Check which succeeds:

Take the contents of the requested resource, then test a series of serializations to a fixed media type (doesn't matter which eg. text/turtle is fine). That's basically going through a list of types that the fixed media type is able to serialize to. If the from and the to media types are the same, return the data as is. Only one of the serializations will be a success, and that will be the media type to use in the response header.

This solution might come across like it is wasting CPU or something but it is safe. No need to touch the URI or what's on disk. I use it here https://github.com/csarven/mayktso/blob/master/index.js#L840

[RubenVerborgh]

MIME type sniffing doesn't seem like a good idea; it's a heuristic, not a final answer. Plus, if the client really wants to store an HTML-like document as text/plain, it should be allowed.

[csarven]

If the goal is to store that information; document contains this but we want it served like that, then there needs to be a separate meta to track that. An extension is as much of a heuristic. If anything, we've learned not to trust extensions, unless it is processed and verified.

A request may have an extension that portrays something it is not. What happens when it is served back out? Is it always the original creator consuming it knowing that case? I think not.

Certainly a client may want to store an HTML document to be served back out as text/plain, but I think that's an edge case. I'm not sure what "HTML-like document" would be. Example? Markdown? That'd have its own media type.

In any case, I think here we have a simple solution and over engineered ones.

[RubenVerborgh]

An extension is as much of a heuristic.

Not really, since the extension is determined from the content type in this case.

we've learned not to trust extensions, unless it is processed and verified.

That's what we do.

I'm not sure what "HTML-like document" would be.

A document that conforms to the HTML syntax (or at least passes a parser), but that a client wants to store as text. It was just an example though of how sniffing can fail.

[RubenVerborgh]

Fix for the POST use case in #584.

[csarven]

Not really, since the extension is determined from the content type in this case.

That's not verification. That assumes that the payload matches the content type. It at least opens up the possibility to assign incorrect extensions, and at worst case it'll be misused by consuming applications.

A document that conforms to the HTML syntax (or at least passes a parser), but that a client wants to store as text. It was just an example though of how sniffing can fail.

Is this an actual practice? How common is it? Or is it hypothetical?

[RubenVerborgh]

That assumes that the payload matches the content type.

It assumes the client sets the right Content-Type header. If not, the interaction is broken anyway. E.g., if a clients sends a JSON document under text/html, we might try to reject it if we notice the mistake, but we should not serve it as application/json.

How common is it? Or is it hypothetical?

Nah, it was a bad example I suppose, the bottomline is that it is up to the client to say what the interpretation is. A specific JSON-LD document could be either text/plain, application/json, application/ld+json, or application/activity+json. None of these are incorrect; some of these are just more specific. It depends on the interpretation the client allows on the document.

[dmitrizagidulin]

@csarven yeah, I agree with Ruben that content sniffing might not be the best strategy here. I understand clients wanting to POST resources with no file extensions, but there's no reason for them to not include a valid Content-Type.

[csarven]

It assumes the client sets the right Content-Type header. If not, the interaction is broken anyway. E.g., if a clients sends a JSON document under text/html, we might try to reject it if we notice the mistake, but we should not serve it as application/json.

"Trust, but verify". And, the only way to do that with certainty is to check the body and see if it can be parsed as proclaimed.

I still would like to know why on one hand the argument is made against serialization tests and deterministically (in fact) identifying the true media type of the body. How else can one truly know what the body really is? The original comment on the issue was about understanding the body of the request, not the UC about what the user may have intended.

there's no reason for them to not include a valid Content-Type.

If that's the case, the class of arguments for UCs like "HTML-like document" doesn't hold. An "HTML-like document" in most likely case - and the simplest explanation - is that it would have text/html.

So, looking at Content-Type and then slapping the extension is guess work. Again, not verification. Did the sender really want that, was it a mistake, was it used with malicious intention? See also https://mimesniff.spec.whatwg.org/

What I'm saying is that if you want to be certain of that, you have to check the body and not solely rely on the Content Type. Relying on the Content Type means that the server trusts the client to always do the right thing. That may very well be the case, and the Robustness principle very much applies here. I'm not objecting to the idea of having the extension in the path being only visible locally. Visible globally is another matter and I don't think that's a good idea. I suppose Cool URIs.. applies here. Having said that, Tim's points at https://github.com/solid/node-solid-server/pull/584#pullrequestreview-61383117 all sound reasonable to me.

Lastly, persisting "meta" information like the Content-Type is one of several things that the sender (as well as the server) may want to persist, orthogonally, what the client intended. The exact same argument holds for Content-Language, Content-Encoding, charset etc. Is there an extensible plan to include those under the same approach as Content-Type?

[RubenVerborgh]

As far as I'm concerned, the argument is really simple: HTTP specifies the allowed interpretation of a representation through the Content-Type header. Other interpretations fall outside of the spec.

If the server sends an HTML document with text/plain, your browser will render it as text. If the server sends it as application/javascript, your browser will report a syntax error. That's the way the contract is designed.

Furthermore, this is precisely the reason dokieli fails: the server sends HTML with text/turtle. Do I expect dokieli to know that it's Turtle? No, it's the responsibility of the sender to get the content type right. The exact same reasoning applies to the Solid server receiving a document from a client: just like dokieli, it is bound by the HTTP protocol.

The only allowed interpretation is specified by Content-Type, all the rest are (brittle) out-of-band contracts.

[RubenVerborgh]

Lastly, persisting "meta" information like the Content-Type is one of several things that the sender (as well as the server) may want to persist, orthogonally, what the client intended. The exact same argument holds for Content-Language, Content-Encoding, charset etc. Is there an extensible plan to include those under the same approach as Content-Type?

Replying to this separately, as this is another point.

You're very right, and I think that only an extension is not the sustainable path. Content-Encoding and charset I suppose not need to be preserved; the server is free to store it in any encoding, and agree at runtime. However, you have a point about Content-Language, and there are further points to be made about other dimensions of conneg, such as the upcoming profile negotiation work that we are doing (where a representation can have _multiple_ profiles even, so that is way more tricky).

I think having the extension on disk is nice, as this works together well with the filesystem (this is also @timbl's argument). But in the end, we might very well need a .meta file for every resource, that can track all those other dimensions, as well as authors, history, etc… (Which I would then serve directly in the main representation as separate graphs, but that is yet another discussion.)

[RubenVerborgh]

I'm not objecting to the idea of having the extension in the path being only visible locally. Visible globally is another matter and I don't think that's a good idea.

Note that #584 is an intermediary fix to ensure that commenting on dokieli documents at least _works_ (and doesn't break anything else). The server _will_ reply with xyz.html and the likes, but clients should interpet this as an opaque identifier (e.g., would have been the same as xyz-abcd). In the future, I think the _returned_ URL should become xyz, but this requires a totally different set of modifications to node-solid-server, since the assumption that "outside path = inside path" is hard-coded in many places.

[csarven]

Alright, sounds reasonable.

Aside: re HTML+RDFa documents (or dokieli-like thing if you will), we'll see how that works out. Its articles, annotations, notifications do use relative values for some of its subjects/objects. Normally, the base URL will resolve to the URL of the document. So, I think it is possible that .html for instance will leak out. That can be overridden of course when processed, but tools like dokieli can't arbitrarily - and in most cases shouldn't - make the call on what is the intended URL and what's preferable. Needless to say, this is not to suggest that seeing or using .html is an error in any way.

I'd like to know what happens with GET /foo (with foo.html on disk). Does it 301/302, or 200 with optionally Content-Location? If I understand correctly, GET /foo is a 404.. and the POST from earlier probably returned Content-Location: /foo.html.. so, only GET /foo.html is a 200?

[RubenVerborgh]

So, I think it is possible that .html for instance will leak out.

Yes, it will currently leak out. But that's not a problem if we treat URLs as opaque identifiers. I.e., the server used to return foo12345 and we did not break any of that apart, now the server will return foo12345.html and we still don't break it apart. To the client, the .html should not have any meaning, it might as well have been `foo12345-xyz' or anything else.

but tools like dokieli can't arbitrarily - and in most cases shouldn't - make the call on what is the intended URL and what's preferable.

Fully agree.

I'd like to know what happens with GET /foo (with foo.html on disk). Does it 301/302, or 200 with optionally Content-Location?

Nothing at the moment, because we still need a good mapping between URIs and the file system (#551).

[RubenVerborgh]

Implemented in #643, will be fixed by #662.