Specification: Content-Type MUST be used on all PUT/POST/PATCH requests?

[kjetilk]

The LDP spec says

5.2.3.6 LDP servers SHOULD use the Content-Type request header to determine the request representation's format when the request has an entity body.

I'm not sure how the SHOULD has been justified there, since I struggle to see any other valid way, so it seems we should have a MUST there.

[pmcb55]

Couldn't the server use content sniffing to 'guess' the content type? I'm not sure if that counts as a 'valid way', but it seems to have been common enough in the past I believe...

[RubenVerborgh]

Couldn't the server use content sniffing to 'guess' the content type?

Security issue; let's not do that. Agree on MUST.

[csarven]

[Saw this on my phone.. literally rushed to get my laptop to respond.. Ruben beat me to it by a second.]

https://tools.ietf.org/html/rfc7231#section-3.1.1.5 is fairly clear about potential issues around content sniffing.

[csarven]

MUST raises the bar - so to speak from, SHOULD - but it will remove potential issues and any ambiguity. That's a quality on its own and what to expect from clients. So, +1 to MUST.

[pmcb55]

Yeah, +1 to MUST from me too (I was only trying to point out that content sniffing was perhaps another 'valid way', I certainly wasn't trying to suggest that we ever rely on it, absolutely not :) !)

[csarven]

For the sake of documentation and to offer an answer for why SHOULD and to eventually resolve this issue:

The server may not want to completely rely on what a client claims in the request. It gives a way out I suppose to do some verification. For authenticated agents, this would be less of a concern.

[acoburn]

The SHOULD-level requirement in LDP is used in the context of clients sending POST requests -- requests with an entity body -- to a server. If this is increased to MUST, what are the implications if a client does not sent a Content-Type header?

For example, if a client attempts to create an LDP-NR and does not send a Content-Type, does that mean that the request must be rejected? If the request is accepted, does that mean that subsequent responses for that resource cannot use, for example, Content-Type: application/octet-stream?

[kjetilk]

For example, if a client attempts to create an LDP-NR and does not send a Content-Type, does that mean that the request must be rejected?

Yes, that has been my understanding. There is basically no legitimate way to tell what media type there is without the header, so a 400 response results. We should have structured error message bodies for those cases though.

[pmcb55]

So @csarven - could you elaborate a little on what you mean below: (as I don't understand what you mean at all):

The server may not want to completely rely on what a client claims in the request. It gives a way out I suppose to do some verification.

And @acoburn, when you say:

The SHOULD-level requirement in LDP is used in the context of clients sending POST requests -- requests with an entity body -- to a server.

...I think that's clear from @kjetilk's original comment that created this issue. The question is, does anyone know a good justification for the LDP guys deciding on the Content-Type header being a SHOULD here instead of a MUST, right?

'Cos I agree with @kjetilk that '_the implications if a client does not send a Content-Type header_' should be simply a 400 response. The only other option I see for processing an incoming request without a Content-Type header is content sniffing, which everyone (so far) agrees would be a really, really bad idea.

So again, are we missing something here that the LDP spec editors realised, and that justified their making the Content-Type header a SHOULD instead of a MUST?

[csarven]

If this is increased to MUST, what are the implications if a client does not sent a Content-Type header?

400.

Noting here that changing requirements around Content-Type will need to be factored in TSE (The Solid Ecosystem), especially the HTTP section.

[csarven]

So @csarven - could you elaborate a little on what you mean below: (as I don't understand what you mean at all):

The server may not want to completely rely on what a client claims in the request. It gives a way out I suppose to do some verification.

(I'm only speculating a possible reason because I don't have a citation.)

We can consider the public-write or append case where a client may end up using an incorrect Content-Type value - not matching the payload. Noting here that if a server wants to ensure that a resource is eventually served accurately, it may end up content-sniffing (whether Content-Type has a value and valid for the payload, or not). This is neither encouraged or part of the agreement for Content-Type use.

"[..] does anyone know a good justification [..]"

@TallTed @bblfish may want to chime in.

[bblfish]

Protocols have to say what must happen for things to work correctly. One can work one's way around bad clients, but then one is taking risks, which inevitably will be taken advantage of at some point by folks trying to break security.

I believe this is related to deontic logics, and at some point I'll see if I can find a good explanation of this in those terms. Until then I'd go with the above recipe.

[kjetilk]

We can consider the public-write or append case where a client may end up using an incorrect Content-Type value - not matching the payload.

Yup, but that's where we slap the $ from https://www.w3.org/DesignIssues/HTTPFilenameMapping.html on it, I think.

[csarven]

slap the $

Implementation detail... useful if there is a Content-Type.

[RubenVerborgh]

Implementation detail..

Not for all backends.

@timbl has indicated that he wants https://www.w3.org/DesignIssues/HTTPFilenameMapping.html to evolve into a specification that makes Solid also interoperable on a filesystem level.

[csarven]

I was saying that in context of TSE. As for it being part of "file-based-solid-spec", okay. Perhaps I shouldn't have used "..." because the "if" is important. I'm not sure how HTTPFilenameMapping is expected to work for the no Content-Type case, unless of course there is some fallback to .ttl and/or check Link header for rel RDFSource or something (but still wouldn't know if the payload is actually Turtle or something else). And, that brings things back to non-turtle-rdf.ttl being conneg'd for Turtle.. So, I think Content-Type is quite critical (MUST) for file-based-solid-spec.

[TallTed]

Forgive me for not tracking down all citations; I'm on a wet-string connection at the moment.

As noted above, the cited quote of 5.2.3.6 from LDP is within the 5.2.3 HTTP POST section (so, not a global LDP rule).

Note also, from 6.2 HTTP 1.1 --

6.2.6 When the Content-Type request header is absent from a request, LDP servers might infer the content type by inspecting the entity body contents ([RFC7231] section 3.1.1.5).

I believe that the RFCs and W3 specs upon which LDP was built -- and which inheritance should be persisted in Solid -- state that POST (and PUT) without a Content-Type at least MAY (and my personal feeling is SHOULD) be accepted and treated as if submitted with Content-Type: application/octet-stream. I don't understand the apparently perceived issue with so doing.

The server doesn't parse, transform, or otherwise manipulate the payload of such submissions. Turtle content is left as Turtle; it is not parsed and loaded to a back-end RDF store. Clients requesting Content-Type: text/turtle from that resource don't get that, even though the content of the file is structured as Turtle. Clients requesting Content-Type: text/ld+json (or whatever that MIME type is) don't get a JSON-LD transformation of the Turtle. Clients including */* or application/octet-stream in their GET request Accept: (or similar) list will get the raw resource, with Content-Type: application/octet-stream, and what they choose to do with that is their own lookout.

What am I missing?

[acoburn]

FWIW, the Trellis server does, effectively, what @TallTed describes:

A POST with an LDP-NR link header, an entity body and no content-type header will store the resource with Content-Type: application/octet-stream. Subsequent requests for that LDP-NR returns a resource with a Content-Type: application/octet-stream response header.

A POST with no LDP link header, an entity body and no content-type header is accepted as an LDP-NR and will respond with a Content-Type: application/octet-stream header.

A POST with an LDP-RS link header, an entity body and no content-type header is rejected immediately.

Nowhere in that process is there any content sniffing. LDP-NRs are always treated (internally) as opaque byte arrays, and LDP-RSs must be parseable (if an entity body is present), based on the provided content-type.

[csarven]

@TallTed Thanks for the feedback and giving the thread a bit more juice :)

I don't understand the apparently perceived issue with so doing.

AFAICT, the perceived issue wasn't particularly about application/octet-stream but given that it is a MAY and the alternative option that's mentioned in https://tools.ietf.org/html/rfc7231#section-3.1.1.5 :

or examine the data to determine its type

and the paragraph that discusses misconfigurations and potential risks.

What am I missing?

Nothing. The discussion just went in the direction to eliminate ambiguity by forcing clients to always indicate their intentions.

I now see that my comment in https://github.com/solid/specification/issues/70#issuecomment-535876142 wasn't adequate either because it was only in context of eliminating ambiguity as much as possible based on earlier discussion (re: "content sniffing"). So with that as a premise, MAY for application/octet-stream probably didn't matter. We need to revisit.

[csarven]

@acoburn thanks for sharing!

The first two cases leading to LDP-NR/application/octet-stream is already covered by LDP/RFC and so leave as is (as @TallTed and you already indicated).

As for:

A POST with an LDP-RS link header, an entity body and no content-type header is rejected immediately.

Is the reasoning based on https://www.w3.org/TR/ldp/#ldpc-post-createrdf :

If any requested interaction model cannot be honored, the server MUST fail the request.

Does Trellis consider Content-Type as a requirement to check if the interaction model can be honoured? Or is this separate?

We can acknowledge that RDF payload can make its way into a server but can only be reused via */* or application/octet-stream. Perhaps more specifically, I'd like to understand what are the use cases for @TallTed's remark about clients:

what they choose to do with that is their own lookout.

Put differently, why should RDF payload without indicating an interaction model through Link header or Content-Type be accepted whereas if the interaction model is specified (LDP-RS in Link header) gets a reject:

A POST with an LDP-RS link header, an entity body and no content-type header is rejected immediately.

What am I missing?

[acoburn]

A POST with an LDP-RS link header, an entity body and no content-type header is rejected immediately.

Actually I was incorrect about this. A POST with an LDP-RS link header and no content type is accepted as text/turtle.

Within Trellis, LDP-NRs and LDP-RSs follow entirely different code paths and are stored in different subsystems. LDP-NRs are just opaque byte streams while LDP-RSs need to be parsed and validated before being persisted: hence, there is a need to know the content type of the HTTP entity body.

Thus far, I have treated these "what to do if the client isn't completely explicit" heuristics as implementation decisions, but if the Solid specification chooses to weigh in on or clarify these behaviors, it would be pretty simple to make adjustments, if adjustments need to be made.

RDF payload without indicating an interaction model through Link header or Content-Type be accepted

In this case, the reason the request is accepted has to do with the fact that it is accepted _as an LDP-NR_. This also raises a slightly different issue: if a client does not supply an LDP interaction model, what interaction model (if the request is accepted) should be assigned to the resource? Here, Trellis makes a decision based on the Content-Type header, choosing either an LDP-NR or an LDP-RS (if there is no Content-Type header, as stated above, an LDP-NR is chosen).

[TallTed]

@acoburn --

Actually I was incorrect about this. A POST with an LDP-RS link header and no content type is accepted [by Trellis] as text/turtle.

That doesn't seem _quite_ the right action. An LDP-RS link header does give you a strong hint of the client's intent, but it would be equally valid attached to a JSON-LD payload as a Turtle payload -- and indeed, a payload in any other RDF serialization (though only JSON-LD and Turtle are required to be supported by LDP servers). Minimally, it seems that Trellis should test whether the payload is Turtle or JSON-LD (or neither), and take appropriate next steps...

[acoburn]

@TallTed the reasoning behind assuming that an LDP-RS is Turtle (rather than any other format) stems from an inversion of this statement:

4.3.2.1 LDP servers must respond with a Turtle representation of the requested LDP-RS when the request includes an Accept header specifying text/turtle, unless HTTP content negotiation requires a different outcome.

In other words, treat representations as Turtle unless there is a reason to think differently. I would also be _very_ cautious about any sort of content sniffing, as has already been pointed out above.

[TallTed]

@acoburn - Speaking as a participant in the LDP WG that produced that spec, I don't think any of us expected any of its statements to be inverted in such a way.

Further, I'm not understanding how telling Servers "you must deliver Turtle representation of LDP-RS when Turtle is requested (i.e., when request includes Accept: text/turtle), unless conneg indicates another representation of the LDP-RS is preferred" turns into telling Servers "every LDP-RS submitted by a Client must be (treated as) Turtle unless you're told otherwise by the Client" into nor telling Clients "every LDP-RS you submit must be (or will be treated as) Turtle unless you tell the Server otherwise" especially given that we explicitly required Servers to accept both JSON-LD (5.2.3.14) and TTL (5.2.3.5) if they accepted POST at all (5.2.3).

Still further, we didn't recommend content sniffing, though we said, in a non-normative section, that "When the Content-Type request header is absent from a request, LDP servers might infer the content type by inspecting the entity body contents ([RFC7231] section 3.1.1.5)." Note also that RFC7231 says in the third paragraph of 3.1.1.5 "If a Content-Type header field is not present, the recipient MAY either assume a media type of "application/octet-stream" ([RFC2046], Section 4.5.1) or examine the data to determine its type."

In other words -- the expectation is that you would either examine the data, or assume that it is application/octet-stream -- not that you would assume that it is text/turtle.

In sum -- it is dangerous to pluck any single statement from any spec, and more so to use your own interpretation of the inverse of such statement.

[csarven]

A POST with an LDP-RS link header and no content type is accepted as text/turtle.

If the payload is in JSON-LD, what does Trellis do as it accepts the request? What happens when the created resource is requested i) without an Accept header, ii) with Accept: application/ld+json ?

[acoburn]

@TallTed my position on this is that the LDP specification is silent on this matter. As a consequence, a server's behavior is an implementation decision. If another specification were to define what ought to be done in this case, I would be happy to follow that, and I believe that is the basic purpose of this issue. I really have no opinion on what _ought_ to be done in this case.

@csarven if the POST payload is JSON-LD, then the client-submitted request is parsed as JSON-LD (provided that the request includes a Content-Type header). Provided that the request succeeds, then subsequent GET requests (i) without an Accept header would return text/turtle while (ii) with an Accept: application/ld+json would return application/ld+json.

[csarven]

@acoburn

(provided that the request includes a Content-Type header)

is a different case. I'm trying to understand this better:

A POST with an LDP-RS link header and no content type is accepted as text/turtle.

If the request has an LDP-RS Link header and message body in JSON-LD, what does "accepted as text/turtle" entail? Can you expand on that process?

[acoburn]

Internally, for LDP-RSs, a create/update request involves a parsing stage before the data is persisted. That is, the byte stream (java.io.InputStream) that is an incoming RDF serialization (in this case, a JSON-LD document) is translated into Java objects, specifically a org.apache.commons.rdf.api.Dataset with some additional metadata.

That parsing stage must succeed before the RDF is persisted into the storage layer. In fact, between parsing and storage, there is also a validation step. For instance, LDP imposes certain rules, and if those rules are violated, the request will be rejected. (Shape validation of various sorts happens at this point)

But as for the parsing stage, the parser needs to be told exactly how to parse the InputStream: is it text/turtle or application/n-triples or application/ld+json? Is it an RDF serialization that is not supported (e.g. application/rdf+xml or application/trig)? If that parsing stage fails, the request fails.

So, when I write:

accepted as text/turtle

I mean that, in the absence of an explicit Content-Type from the client, the RDF parser uses a Turtle-based reader. So in this case, if a Turtle-based reader is used (because the client didn't supply a Content-Type) but the HTTP entity body is actually something else (e.g. JSON-LD), then the request will simply fail with a parsing error. A log entry will be generated and the server will respond with a 400 Bad Request.

[acoburn]

One thing to add as more of a pragmatic point is that, by having the RDF parser default to Turtle, that means that an empty InputStream can be treated as valid RDF, since an empty file is also a valid Turtle document. So a request with no body can be treated as a valid incoming RDF document.

[pmcb55]

Great discussion, but I remain a MUST-er (i.e. Content-Type MUST be used). That means a Solid server receiving a POST with an LDP-RS link header and no content type is rejected immediately (intuitively, I just don't like the implicit assumption of Turtle, as it feels (as with all implicit assumptions) 'dangerous' somehow).

@acoburn's point on being able to handle empty streams is interesting though, as it does seem a little strange to MUST provide a Content-Type if I _know_ I'm passing an empty body. But I still think just a non-normative note to suggest clients provide Content-Type: text/turtle in that (presumably edge-) case is fine too...

So @TallTed, given your experience on the LDP spec, do you personally think a MUST is justified here (i.e. in the Solid spec), or would we be losing something if we mandate that (apart from the obvious 'burden' it places on the client to be explicit in what it wants to actually do!)?

[csarven]

Clear. Thank you. I asked about this implementation detail because I wanted to get a sense of Trellis' approach re: "examine the data to determine its type" vs "content sniffing".

That parsing stage must succeed before the RDF is persisted into the storage layer.

is what I've suggested at https://github.com/solid/node-solid-server/issues/275 but it was deemed to be sniffing. In addition to that, at least from NSS's perspective, the thread worked with the idea that no Content-Type meant a client error, as opposed to a possibility permitted by LDP/RFC 7231.

In fact, it appears to be that https://github.com/csarven/mayktso follows the same steps as Trellis: parsing and validation. As a file-based server, mayktso just stores the payload as a file (after confirming that it is one of the accepted formats) without tracking its type. When the resource is requested, it'll try to serialise - success would be what's returned back to the client.

[csarven]

@pmcb55

I remain a MUST-er

Why?

POST with an LDP-RS link header and no content type is rejected immediately

In this case I think the interaction model via Link header is irrelevant as it just comes down to whether Content-Type is used.

I just don't like the implicit assumption of Turtle

It is not a rule. It is only Trellis' decision and doesn't conflict with LDP/RFC 7231.

Given:

If a Content-Type header field is not present, the recipient
MAY [..] examine the data to determine its type.

I actually don't see any issue or disagreement between what @TallTed and @acoburn is saying.

do you [..] think a MUST is justified

My understanding of the thread so far is that there is no strong argument in favour of it - the why - or at least the consequences of the change factored in. If I've overlooked or misinterpreted, please let me know.

On a related note, I think it is in TSE's interest to not diverge too far from LDP/RFC 7231 so that existing LDP implementations can pick up on Solid's requirements without breaking existing interop, ideally.

[csarven]

To clarify re:

5.2.3.6 LDP servers SHOULD use the Content-Type request header to determine the request representation's format when the request has an entity body.

• No Content-Type falls back to RFC 7231.
• No entity body is undefined and so depends on implementation (which may derive the validity of the representation based on format or other rules).

What for example might happen with MUST:

"Solid servers MUST use the Content-Type request header to determine the request representation's format when the request has an entity body."

Given that a client may end up requesting without Content-Type (re RFC 7231's SHOULD for senders), a server may still end up handling it as per RFC 7231's MAY scenario. I think this is permitted in that TSE says MUST for Content-Type and inherits RFC 7231's MAY if there is no Content-Type.

Aside:
In order to put the MUST into full effect - strictly based on what's discussed in this thread - it follows that the Solid spec has to say something about the MAY in RFC 7231. For example, "MAY NOT assume a media type of "application/octet-stream ([RFC2046], Section 4.5.1) or examine the data to determine its type." I think that may be one potential answer to @acoburn question: "what are the implications if a client does not sent a Content-Type header". Also mentioned about TSE's HTTP section at https://github.com/solid/specification/issues/70#issuecomment-535876142 . If that's an accurate reflection, I can't say that I like the solution. It is a breaking change.

[TallTed]

As I recall, the main LDP WG discussion around all this was wrapped up in Postel's Law (approximately, "Be liberal in what you accept, and conservative in what you send.") and focused on how an LDP server could allow older and/or imperfectly written general HTTP clients (of which there are many in the wild, and I believe there were one or two specifics in others' minds), which do not always include a Content-Type header on POST, to work with them.

Note that the sections which have been referenced are about how an LDP server handles input from clients, and NOT about what LDP clients should submit to LDP servers.

I believe that Trellis is acting within bounds, by treating POST without Content-Type as Turtle and producing errors when it's not (as this amounts to "examining the data to determine its type" and then acting based on determination that it is or is not Turtle).

That said, I do not believe this is the best pattern to follow. I believe that treating submissions without Content-Type as application/octet-stream is what should happen -- though the server may then choose to examine the data and take other action based on the results of that examination.

For all anyone here knows, a client making such a submission may intend for the Turtle document to be preserved as an application/octet-stream object, and for it not to be parsed -- which is the default pattern suggested by all these specs. The deployer of Trellis may understand full-well what is going to happen -- but the client interacting with Trellis will have no advance warning.

Anyway -- much of this is now going off on a tangent.

Overall, I believe MUST is stronger than necessary or appropriate for this; that SHOULD ought to be sufficient; and that clear guidance for both Client and Server is possible for those occasions when a SHOULD is ignored on either side, _which is entirely permissible._ Such guidance SHOULD describe best/recommended practices, and warn of the potential impacts, local and remote, of both best and less-than-best practice.

[pmcb55]

So first just to answer all @csarven's questions:

I remain a MUST-er

Why?

As I said, simply because forcing the client to be explicit is just simpler for everyone (assuming the client 'can' provide a Content-Type). That's literally my only justification, i.e. favoring explicit behaviour over implicit.

POST with an LDP-RS link header and no content type is rejected immediately

In this case I think the interaction model via Link header is irrelevant as it just comes down to whether Content-Type is used.

I totally agree if the Content-Type were a MUST (i.e. if it's not there, then the request is just rejected and the interaction model indeed becomes irrelevant). But if Content-Type is a SHOULD, then the interaction model could be used to implicitly apply text/turtle if the interaction model is LDP-RS, and application/octet-stream if the interaction model is LDP-NR.

I just don't like the implicit assumption of Turtle

It is not a rule. It is only Trellis' decision and doesn't conflict with LDP/RFC 7231.

Yep, I know it's not a rule. And I also know, and totally agree, that Trellis' decision is valid and totally within the current specs. I was just justifying my above MUST-er position by saying I don't 'like' implicit behaviour (i.e. generally, but also in this specific case) if making explicit behaviour mandatory was an option - that's all (i.e. I favor explicitness).

I actually don't see any issue or disagreement between what @TallTed and @acoburn is saying.

Neither do I.

do you [..] think a MUST is justified

So I think @TallTed just did a great job in justifying keeping it a SHOULD (i.e. keeping Solid in-line with the current specs, and maintaining support for cruddy old systems out there that can't provide a Content-Type (jeez!)), so I'm more than happy to go along with that and become a SHOULD-er.

Personally though, I do think this warrants a table of non-normative suggested guidance on what implementations SHOULD do (even just to help me get my head around the implications of allowing this 'crazy' implicit behaviour!), something like:

| HTTP Method | Request Interaction Model | Request Content-Type | Applied Interaction Model | Applied Content-Type |
| --- | --- | --- | --- | --- |
| POST | LDP-RS | NONE | LDP-RS | text/turtle |
| POST | LDP-NR | NONE | LDP-NR | application/octet-stream |
| POST | NONE | text/turtle | LDP-RS | text/turtle |
| POST | NONE | NONE | application/octet-stream | LDP-NR |
| etc. | etc. | etc. | etc. | etc. |

But such a table should, of course, be a completely separate issue (assuming we stick with SHOULD for Content-Type).

[TallTed]

Assuming POST with LDP-RS and no Content-Type means that the payload should be treated as RDF, that payload should be fed to a library which has appropriate routines to detect and translate all other known RDF serializations (or at least JSON-LD, since that's a 100% valid LDP serialization) to whatever the accepting server accepts -- Turtle, in this case. JSON-LD payloads should absolutely not be rejected because they're not Turtle, because JSON-LD is required by LDP, which is supposedly being inherited here. (And yes, I think this rejection by Trellis is a violation of the LDP spec.)

[csarven]

@TallTed Point re LDP's position on robustness is well taken. I would however question to what extent the Solid ecosystem ought to be concerned at the same level as LDP re "older and/or imperfectly written general HTTP clients" and at the very least in context of making POST requests. I don't mean to wave this concern/need. There are many specialised requirements for clients in the Solid ecosystem such that missing Content-Type in POST would relatively be a minor concern in my opinion. Nevertheless, acknowledging basic scenarios around that would be useful eg. what might "Solid-compatible" user-agents do if they can't determine the media type of what they want to send? I'd suppose application/octet-stream will be used as opposed to no Content-Type.

Note that the sections which have been referenced are about how an LDP server handles input from clients, and NOT about what LDP clients should submit to LDP servers.

Good point! Also why I said the following for the server:

I think this is permitted in that TSE says MUST for Content-Type and inherits RFC 7231's MAY if there is no Content-Type.

Re:

Anyway -- much of this is now going off on a tangent.

Yes, initially simple-looking issue exploded but I think that's part of the process. We need to have a good handle on the options and their consequences. Not just declare a solution and call it a day.

re:

clear guidance for both Client and Server [..] best/recommended practices

If I understand you correctly, you suggest not to introduce or change any normative requirements involving LDP/RFC 7231 but instead include non-normative text.

I'd like to understand what essentially changes or becomes a potential problem by forcing servers to use Content-Type? I ack the robustness argument/background or being backwards-compatible but are there others?

[csarven]

@pmcb55 I like the table but majority of the information is known (can be derived from the specs) eg. honouring client's interaction model in the LDP spec. If we have substantial amount of (non)normative text, we could introduce something like that. Let's revisit this once we have rough consensus on this and other LDP-related issues.

[csarven]

@TallTed re https://github.com/solid/specification/issues/70#issuecomment-537290739 , I generally agree with the approach that if LDP-RS is mentioned in Link and there is no Content-Type, then it is probably reasonable to expect that both Turtle and JSON-LD (lower-case) should at least be considered for tests, and practically any other RDF representation format that the server wants to check for. It is a bit of an odd case though.. I mean the client can manage to say LDP-RS but not the format?

I think this rejection by Trellis is a violation of the LDP spec.

I don't think so because the LDP spec does not prescribe what servers should do when there is no Content-Type in POST request from a sender. Trellis is falling back on the MAY from RFC 7231 re "examine the data to determine its type" - it is a bit fuzzy as to how much a server should examine, but I think perfectly reasonable to stop at one test eg. Turtle or not, or more.

[TallTed]

I think this rejection by Trellis is a violation of the LDP spec.

I don't think so because the LDP spec does not prescribe what servers should do when there is no Content-Type in POST request from a sender.

It's the rejection of JSON-LD payload as LDP-RS without Content-Type, but acceptance of Turtle payload as LDP-RS without Content-Type, to which I object.

It does appear to be a loophole in the letter of the spec, which I think is unfortunate. It definitely departs from the spirit of the spec.

[kjetilk]

Pretty long thread I set off and then just disappeared, eh? ;-) I have to admit that I now only skimmed the thread, so bear with me.

IIRC, @timbl expressed when we discussed https://github.com/solid/node-solid-server/issues/1256 that a request without Content-Type should be treated as invalid in Solid.

My personal take is that Solid clients with write capability will largely be written to the Solid specs, since they have to support Solid-specific authn anyway, and therefore this is a very little penalty for a client for good gain in lower complexity on the server side. I don't quite see what a fallback to an octet-stream would give us practically.

[TallTed]

LDP was written to allow generic HTTP clients to write to LDP servers, leading to (among other things) no requirement of Content-Type and the fallback to application/octet-stream.

At this point in the thread, the question seems to me to be whether writing to Solid Servers will require Solid Clients -- i.e., that generic HTTP clients may not be able to write to Solid servers.

[dmitrizagidulin]

@TallTed re generic HTTP clients -- so, this has always been a hard topic, in solid. On the one hand (at least so far), generic HTTP clients can always write to Solid. (I'm not sure why requiring Content-Type would change that).
On the other hand, it may make sense to have a helper Solid client library at some point, but what its functionality should be is unclear.

[TallTed]

@dmitrizagidulin - I was picking up from @kjetilk's "since they have to support Solid-specific authn anyway"...

[kjetilk]

Indeed... I mean, we should always strive for requiring as little as possible, and I have been advocating for that just a little above any Web server would be a good idea, but it seems to me that the authn sets requirements in both ends.

[dmitrizagidulin]

@TallTed Ah, got it. Yeah, definitely needs an auth client. (And I don't think that's against the spirit of the LDP spec, since, ahem, the LDP spec left authentication as out of scope :) )

[kjetilk]

Proposal following F2F meeting with @csarven , @timbl and @kjetilk present of 2019-10-30:

We found it is advantageous to avoid lack of clarity in content types, since a fallback to defaults like application/octet-stream would result in that apps cannot determine the content type, and therefore not present a suitable UX for it.

Users of basic UAs (e.g. curl) should be prevented from skipping the content type, because that may cause subsequent problems for apps using the data.

The cost of requiring clients to submit content type is thus much lower than cost of the requirement on servers and clients to deal with the consequences of wrong or useless content types.

This points towards a strict interpretation, i.e. MUST.

[TallTed]

Does this mean that the uploads must include content types that are known to the solid server? Or just ANY content type (which might well include application/octet-stream)?

How does the server test whether the content type a client specifies for a resource is appropriate to its content? If the server does not test such, how does this requirement prevent "wrong or useless content types"?

I do not see how simply forcing a content type to be present achieves the stated goals -- i.e., guaranteeing a suitable UX for that content, or that apps will know how to work with such content types.

[kjetilk]

Does this mean that the uploads must include content types that are known to the solid server?

No.

It can't prevent a bad client from adding bad data, if the client has no idea about the content-type, it will fall back to application/octet-stream, but then, the client needs to understand that other clients will not be able to use it for anything. I didn't write "guarantee", I wrote "advantageous" :-)

It is more about making the chance of breakage smaller, not prevent breakage entirely.

[TallTed]

"Other clients will not be able to use it for anything" overstates the impact. "Other clients" may inspect the payload of .ttl typed as application/octet-stream, discover that it's actually JSON-LD, and take appropriate action.

"Making the chance of breakage smaller" sounds to me much more like a SHOULD rule, than a MUST rule, as the latter will tend to lead to expectations that will not necessarily be satisfied.

This decision is not mine in the end, but I hope that the documentation around it will be clear about actual effect, as opposed to wishful effect.