Did-core: verification method IDs MUST contain a fragment

[llorllale]

Current text:

id
The value of the id property for a verification method MUST be a string that conforms to the rules in Section § 3.2 DID URL Syntax.

DID URL Syntax:

did-url = did path-abempty [ "?" query ] [ "#" fragment ]

Fragment is optional. Clearly the intent here is for verification method IDs to have fragments - I think this is just a simple oversight.

[msporny]

Fragment is optional. Clearly the intent here is for verification method IDs to have fragments - I think this is just a simple oversight.

Verification method IDs don't have to have fragments... they're optional, which is what the spec says. For example, this is valid:

{
  "@context": "https://www.w3.org/ns/did/v1",
  "id": "did:example:123456789abcdefghi",
  "authentication": [{
    "id": "did:example:123456789abcdefghi/keys/1", <--- legal verification method ID
    "type": "Ed25519VerificationKey2020",
    "controller": "did:example:123456789abcdefghi",
    "publicKeyMultibase": "zH3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  }]
}

I think the spec text is fine as is... but perhaps I'm missing something?

[peacekeeper]

I agree verification method IDs don't necessarily need to have a fragment.

Does this make sense @llorllale ? Or do you see some problem that would require introducing such a restriction?

[llorllale]

I guess it is _I_ who is missing _something_.

I've been operating under the understanding that a verification method's identifier is anchored by the enclosing DID.

I'd like to check my understanding...

Keeping with this example:

{
  "@context": "https://www.w3.org/ns/did/v1",
  "id": "did:example:123456789abcdefghi",
  "authentication": [{
    "id": "did:example:123456789abcdefghi/keys/1", <--- legal verification method ID
    "type": "Ed25519VerificationKey2020",
    "controller": "did:example:123456789abcdefghi",
    "publicKeyMultibase": "zH3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  }]
}

I assume the full DID URL to dereference that verification method is just did:example:123456789abcdefghi/keys/1. Is that correct? Therefore, to verify a VC with that method as proof, a Verifier would:

1. resolve did:example:123456789abcdefghi to a DID document
2. iterate through all verification methods of purpose assertionMethod looking for ID did:example:123456789abcdefghi/keys/1
3. verify

[llorllale]

I also note no text anchoring the verification method ID to the enclosing DID.

So for example - this appears to be legal:

{
  "@context": "https://www.w3.org/ns/did/v1",
  "id": "did:example:123456789abcdefghi",
  "authentication": [{
    "id": "did:example:XYZ/keys/1", <--- different DID
    "type": "Ed25519VerificationKey2020",
    "controller": "did:example:123456789abcdefghi",
    "publicKeyMultibase": "zH3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  }]
}

[llorllale]

Combining the above two is tricky. If the answer to this question:

I assume the full DID URL to dereference that verification method is just did:example:123456789abcdefghi/keys/1. Is that correct?

... is "Yes", then the example with the differing DIDs does not work.

If instead the answer is "No" - then please show what the full URL looks like - the one that needs to be handed out to Verifiers.

[dlongley]

@llorllale to verify an LD signature proof for purpose P (e.g., assertionMethod) that references verification method, VM:

1. Dereference VM to get its public crypto material and its controller, C.
2. Verify the signature (verify the crypto).
3. Dereference C (e.g., to get a DID Document).
4. Verify VM is referenced under C's verification relationship that matches P.

All of the URLs (what gets dereferenced) should be considered opaque here.

[llorllale]

Looking at the section on DID URL Dereferencing:

It seems the full DID URL to be handed out for the examples so far looks like: did:example:123456789abcdefghi#did:example:123456789abcdefghi/keys/1. Looks legal... but what if the verification method's ID has a fragment of its own? Doesn't look legal. Fragment always comes after the query. This example has a query inside the fragment. The grammar in RFC3986 seems to indicate that the fragment is percent-encoded.

Some examples would greatly help with clarity.

[llorllale]

@dlongley thanks for offering some of the nuances around the steps for verification.

This issue started with my [mis]understanding that verification method IDs MUST have fragments... but now it's turned into _how does dereferencing DID URLs work?_, ie. how to perform step 1 in your list.

[jricher]

Various properties that are defined as "Strings in the DID URL syntax" really should be defined as "DID URLs" which would branch to either an absolute DID URL or a relative DID URL. This goes beyond just the verification method property, anything that is allowed to be a DID URL should be allowed to be either.

[llorllale]

Here's something else from the section of relative URLs:

{
  "@context": "https://www.w3.org/ns/did/v1",
  "id": "did:example:123456789abcdefghi",
  "verificationMethod": [{
    "id": "did:example:123456789abcdefghi#key-1",
    "type": "Ed25519VerificationKey2018", 
    "controller": "did:example:123456789abcdefghi",
    "publicKeyBase58": "H3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  }, ...],
  "authentication": [
    "#key-1"
  ]
}

How can the authentication section include a relative URL referencing the verification method if the verification method's ID did not include a fragment?

[llorllale]

@jricher

Various properties that are defined as "Strings in the DID URL syntax" really should be defined as "DID URLs" which would branch to either an absolute DID URL or a relative DID URL. This goes beyond just the verification method property, anything that is allowed to be a DID URL should be allowed to be either.

Agreed. I would additionally think it makes sense to constrain the did portion of the DID URL to a value equal to the enclosing DID.

[clehner]

@llorllale
Building on @msporny's example, I think the verification method could be referenced with a relative DID URL starting with a slash (/):

{
  "@context": "https://www.w3.org/ns/did/v1",
  "id": "did:example:123456789abcdefghi",
  "verificationMethod": [{
    "id": "did:example:123456789abcdefghi/keys/1",
    "type": "Ed25519VerificationKey2020",
    "controller": "did:example:123456789abcdefghi",
    "publicKeyMultibase": "zH3C2AVvLMv6gmMNam3uVAjZpfkcJCwDwnZn6z3wXmqPV"
  }],
  "authentication": [
    "/keys/1"
  ]
}

Whether or not this example actually works I think depends on DID Resolution. I do not see it said in DID Core that a DID URL with a path must be dereferenced by searching for an object with that id embedded in the DID document resolved from the DID part of the DID URL. Using the fragment in this way works however because of the meaning of the fragment identifier in RFC 3986. DID Core says about Paths: "path semantics can be specified by DID Methods". DID Resolution currently leaves it up to the DID method, and/or the client/application, how to handle DID paths when dereferencing a DID URL (in Step 3 of Dereferencing the Primary Resource). So I would expect did:example to specify how to dereference did:example:123456789abcdefghi/keys/1.

DID Core says in 5.2.2 Referring to Verification Methods that verification methods may be included by reference even from other DID documents: "if the value is a URL string, the verification method has been included by reference and its properties will need to be retrieved from elsewhere in the DID document or from another DID document". Perhaps this could be changed to clarify that a verification method could also be retrieved from some other resource resulting from dereferencing a DID URL containing a path, assuming we want to allow that. Otherwise, if it is intended that verification methods can be defined only by embedding in DID documents, maybe we should mention that, and either disallow paths in verification method ids, or specify that dereferencing DID URLs containing paths must return their value from their DID document. I'm not sure what benefit there would be to allowing verification method ids with paths but requiring them to be defined by embedding in the DID document, vs. just using fragments (like this issue title suggests). But allowing verification methods to be defined outside DID documents (using DID paths), and referenced in DID documents, seems useful. Externally defining verification methods could reduce the size of DID documents, in case that is a concern.

About the example DID URL resolving (in the RFC 3986 sense) to did:example:123456789abcdefghi#did:example:123456789abcdefghi/keys/1, I don't think that would happen. DID Core - Relative DID URLs says to follow RFC3986 Section 5: Reference Resolution. Accordingly, resolving URI reference did:example:123456789abcdefghi/keys/1 (or /keys/1) with base URI did:example:123456789abcdefghi should return target URI did:example:123456789abcdefghi/keys/1.

[iherman]

The issue was discussed in a meeting on 2021-03-30

• no resolutions were taken

View the transcript

5.3. verification method IDs MUST contain a fragment

_See github issue #708._

Brent Zundel: manu you are assigned; do you have an update for us?

Manu Sporny: I do not, have not been tracking updates. Need to read. Charles...

Charles Lehner: I don't have anything to add right now.

Brent Zundel: Last activity was nearly three weeks ago. We will come back to it another time

[msporny]

@llorllale I'm trying to figure out if I need to write a PR for your issue. I note that your understanding is now different, and your request for a change was responded to by @clehner above (with whom I agree).

Would you like to see a concrete change to the specification that has a good chance of achieving consensus? Or do you think the spec is ok as is? Or something else?

[msporny]

@llorllale second ping. I'm going to mark this issue as pending close, which means it'll be closed in 7 days if you don't object before then. I don't think there is anything here for us to do (given the original issue you raised). Please let us know what you'd prefer to happen here.

[llorllale]

@msporny

I still think there is a gap and I'm not sure how to proceed.

This is what I see:

• the comments on dereferencing provided in this thread do not seem to address all scenarios. I am thinking of cases when the VM ID is just a DID - whether it's the same as the enclosing DID or not. However, this may be a topic to bring up in the did-resolution spec. Is that so?
• the other side of the coin: as a "prover", how do I construct the VM's DID URL? Does this question fall into this spec?

[msporny]

• the comments on dereferencing provided in this thread do not seem to address all scenarios. I am thinking of cases when the VM ID is just a DID - whether it's the same as the enclosing DID or not. However, this may be a topic to bring up in the did-resolution spec. Is that so?

Hmm, /what/ you're doing when you deference matters here (in order to answer your question)... and I expect many of those scenarios are going to be in the domain of the Linked Data Proof/Integrity/Signatures (or other more specific verification method) specifications.

For example, are you a software library that is dereferencing "the VM ID that is just a (bare) DID (URL)" in order to verify a digital signature? In that case, you dereference the DID URL and the thing you get back better be a Verification Method that you understand or you're going to return "false" when someone asks "is this a valid signature?"

Or, are you just a DID Document linter, in which case, a VM ID that is a bare DID URL is just fine... 'cause it's a DID URL and that's all you care about as a linter.

In any case, I don't think any of those questions should be answered in DID Core -- all DID Core cares to say about the topic is that the VM ID value must be a DID URL.

• the other side of the coin: as a "prover", how do I construct the VM's DID URL? Does this question fall into this spec?

It's not clear what you mean by "prover"... you could be saying "I need to create a proof", or "I need to prove that this digital signature is valid"... but in both cases, how you construct the VM's DID URL doesn't seem like it has anything to do with DID Core, unless I'm missing something.

[llorllale]

@msporny

how you construct the VM's DID URL doesn't seem like it has anything to do with DID Core, unless I'm missing something.

Where can we find that information?

What is the thought process here?

Combining both of your points, it sounds like the proof and the Verifier are coupled not only by @type, but also by the format of the DID URL on the verification method.

Can you please point to a concrete spec(s) where implementers can see how to priperly construct verificationMethod strings?

[mavarley]

but in both cases, how you construct the VM's DID URL doesn't seem like it has anything to do with DID Core, unless I'm missing something.

I believe the challenge may be more around the construction of a verificationMethod DID-URL so it can be consumed/resolved reliably. And I see a few options to handle this in the spec:

1. verificationMethod DID-URLs are constructed according to the relevant DID Method spec. (end statement)
2. (same as 1. plus) It is RECOMMENDED verificationMethod DID-URLs include a # fragment, as this provides a method independent reference as per https://w3c.github.io/did-core/#fragment

I also see the spec already describes (1) by referring to the DID-URL Syntax which in turn references the DID Method syntax, but an explicit callout to the best practice (like 2.) may help implementers build interoperable libraries. Or not, if # fragment support is already hit-or-miss in the current method specs.

Can the spec help the implementor "do the expected thing" here for verificationMethods, while allowing for cases where some methods/resolvers do not support fragments?

[iherman]

The issue was discussed in a meeting on 2021-05-25

• no resolutions were taken

View the transcript

3.3. verification method IDs MUST contain a fragment

_See github issue #708._

Markus Sabadello: Thanks, Shigeya, I will share my source docs and ask for help

Manu Sporny: George Aristy from SecureKey would like to have normative statements about the structure of DID URLs
… but I don't know as the WG could agree on anything more than a valid URI
… however it would be worth a call with George to understand

[msporny]

@llorllale, I'm trying to get you on the DID WG Agenda so we can hear your concerns here and do what we can to address them. Would you be available at 11am ET next Tuesday, 2021-06-01?

[llorllale]

Hi @msporny

@mavarley will handle this one.

[kdenhartog]

I've been working on our deference logic and I can say this would be a healthy constraint for id dereferencing.

Currently these are the code paths I am checking for:

did:example:123#fragment
#fragment <= Should use the @base property and requires that the content-type be JSON-LD
did:example:123?query
?query
did:example:123/path
did:example:123?service=fragment => did:example:123#fragment

And these are ones I think I need to check for, but have chosen to ignore:

did:example:123?service=query => did:example:123?query (I'm assuming it's the first query parameter only here. Trying to support anything else on this code path is unnecessarily complicated)
did:example:123?service=fragment => #fragment
did:example:123?service=query => ?query
did:example:123?service=path => did:example:123/path
did:example:123?service=path => /path

Is it really necessary to support every one of these code paths or can we reduce the amount of options?

Given these are relative to the did document in question I don't really see the value in all of this optionality. Especially since we're already constraining it to MUST be a didUrl.

If we constrain to only the fragment we reduce the number of supported query paths to this:

did:example:123#fragment
#fragment
did:example:123?service=fragment => did:example:123#fragment

We also make it simpler to support relativeRef pathing by making it:

did:example:123?service=fragment&relativeRef= => did:example:123#fragment => serviceEndpoint + relativePath

[msporny]

If we constrain to only the fragment we reduce the number of supported query paths

Just to be clear, adding that constraint would be a normative change to the specification and would kick us into a 3rd Candidate Recommendation, which would cause the WG to almost certainly fail to get any specification to the Recommendation phase before our charter expires. You should read that as: "The complete and utter failure of the DID WG to standardize anything." -- The W3C Membership does not look favorably on groups that fall into this category. Even if this is a good idea, I expect the WG to reject a normative change that we are not absolutely forced to do at this point in the process.

Now, is it a good idea? No, probably not -- DID URLs follow the URL/URI syntax, which has evolved over 25+ years. Arbitrarily limiting URL syntax just because it reduces code paths will surprise developers because they're probably expecting to be able to do everything they can with a URL with a DID URL. If you limit it, all of a sudden you have a new class of URL that developers need to learn the details about, that don't act like normal URLs... and that increases cognitive burden for developers (who already have to keep waaaay to many things in their heads while developing). So, on that basis, limiting DID URLs to a subset is problematic.

I'm also really confused by what you wrote above... for example:

fragment <= Should use the @base property and requires that the content-type be JSON-LD

No, fragment semantics are defined by the media type... no requirement for JSON-LD and @base... and even if it's JSON-LD, the thing that matters is the base URL for the document, which can be set via @base OR be set by some other mechanism (set via API call).

did:example:123?service=query => ?query

Hmm, this looks wrong, it should be: did:example:123#query ?

did:example:123?service=path => did:example:123/path

This looks wrong as well, and should be: did:example:123#path ?

What am I missing?

[kdenhartog]

If we constrain to only the fragment we reduce the number of supported query paths

Just to be clear, adding that constraint would be a normative change to the specification and would kick us into a 3rd Candidate Recommendation, which would cause the WG to almost certainly fail to get any specification to the Recommendation phase before our charter expires. You should read that as: "The complete and utter failure of the DID WG to standardize anything." -- The W3C Membership does not look favorably on groups that fall into this category. Even if this is a good idea, I expect the WG to reject a normative change that we are not absolutely forced to do at this point in the process.

Fair enough, no point throwing the baby out with the bath water.

Now, is it a good idea? No, probably not -- DID URLs follow the URL/URI syntax, which has evolved over 25+ years. Arbitrarily limiting URL syntax just because it reduces code paths will surprise developers because they're probably expecting to be able to do everything they can with a URL with a DID URL. If you limit it, all of a sudden you have a new class of URL that developers need to learn the details about, that don't act like normal URLs... and that increases cognitive burden for developers (who already have to keep waaaay to many things in their heads while developing). So, on that basis, limiting DID URLs to a subset is problematic.

In all of the implementations I've read through or played around with none of us are treating DID URLs like URLs in the classic sense nor the more modern WHATWG sense. Sure, they are by all means compliant and normatively defined as such, but the implementations I've seen to date don't treat them that way. So either our tribal knowledge has led us astray (which lead me to the same assumption as @llorllale) or we did a bad job conveying how this should work to support DID URLs other than fragments.

I'm also really confused by what you wrote above... for example:

fragment <= Should use the @base property and requires that the content-type be JSON-LD

No, fragment semantics are defined by the media type... no requirement for JSON-LD and @base... and even if it's JSON-LD, the thing that matters is the base URL for the document, which can be set via @base OR be set by some other mechanism (set via API call).

Yup looks like you're correct based on section 3.2.2. Thanks that helps with my implementation a bit.

did:example:123?service=query => ?query

Hmm, this looks wrong, it should be: did:example:123#query ?

did:example:123?service=path => did:example:123/path

This looks wrong as well, and should be: did:example:123#path ?

What am I missing?

Seems like we've got a lacking normative definition of the service parameter that's allowing us to interpret implementation in two different ways.

Identifies a service from the DID document by service ID. If present, the associated value MUST be an ASCII string.

I think our divergence may be the interpretation of how the service parameter should be mapped to a service.id property. Is there anywhere that we choose to normatively define this or is this just an area of assumed tribal knowledge?

As an example, what would you expect to produced by this DID URL

did:example:123?service=%3Fquery
did:example:123?service=%2Fpath

Similarly, how would I use the service parameter to point at a service with the id did:example:123/path or /path?

Given this, I'm inclined to say it would make sense to remove the service parameter (and the relativeRef parameter since it requires it's usage) from the core spec and move the text to a extension draft report so that we can properly define how implementors should use the property. I suspect @msporny would like that since we're already struggling to get implementations in the test suite, but curious if @peacekeeper would be opposed to that path. I'm pretty sure Indy was looking to use this as well for referencing ledger objects. @talltree what's your take on doing that as well?

[jandrieu]

Similarly, how would I use the service parameter to point at a service with the id did:example:123/path or /path?

This is partially a style issue, but I would recommend not using paths to identify services. Rather, I would use fragments.

For example in a service definition:

id: did:example:123#path

Then, I believe a DID-URL of did:example:123?service=path is what I would expect to use for syntax to access that service endpoint.

That is, the identifier identifies a node in the DID Document using fragment identifiers, effectively pointing to itself.

Then the DID-URL specifies a service parameter whose value is equal to fragment identifier of the service. So the resolver finds the node whose absolute form of the service parameter value matches the id and uses that for the service definition.

FWIW, there are implementations using /path and #fragment semantics in the did-url to distinguish between referring to a node in the graph (with #fragment) and retrieving a specific resource with the path property == path parameter. This is our approach to avoiding the HttpRange14 problem. DID-URL with #fragment points to a node in the DID Document graph as a reference. DID-URL with a /path points to an associated resource to be downloaded. This approach is still in development, so we haven't socialized it beyond our immediate teams, but it seems to be a cleaner separation of concerns, and since the semantics of the path parameter are deferred to the DID Method, this will work with those methods that support it.

[kdenhartog]

partially a style issue

Yup I think we're on the same page. This is where I was going with tribal knowledge. We could have constrained here further to make writing an implementation easier, but by leaving more things unstated we leave room for lots of different optional paths which have to be considered in order to produce robust code. Right now by intentionally choosing to support only a limited subset of paths we're limiting interoperability over valid patterns no matter the semantic reasoning behind them.

To be clear, my assumption about using fragments is what I'm implementing, but it's by way of knowing I'm intentionally not supporting additional code paths that could be supported based on the current normative language. Hence the feeling that I'm forced to make a tradeoff between writing robust interoperable code and writing code that gets enough done for what I'm trying to achieve in the moment. If the spec was more specific here I could have the best of both worlds. Such as knowing if did:example:123?service=%2Fpath maps to did:example:123#/path, did:example:123/path or did:example:123#path. All are valid styles, but the question is which ones should I actually support to be interoperable with people.

FWIW, there are implementations using /path and #fragment semantics in the did-url to distinguish between referring to a node in the graph (with #fragment) and retrieving a specific resource with the path property == path parameter. This is our approach to avoiding the HttpRange14 problem. DID-URL with #fragment points to a node in the DID Document graph as a reference. DID-URL with a /path points to an associated resource to be downloaded. This approach is still in development, so we haven't socialized it beyond our immediate teams, but it seems to be a cleaner separation of concerns, and since the semantics of the path parameter are deferred to the DID Method, this will work with those methods that support it.

Sure that makes sense and seems like a good semantic differentiation to solve a real problem. Unfortunately that's one example in a sea of many of good patterns that aren't normatively defined so it's not a common shared pattern used by compliant implementations that lead to interoperable implementations.

[msporny]

Yup I think we're on the same page. This is where I was going with tribal knowledge. We could have constrained here further to make writing an implementation easier, but by leaving more things unstated we leave room for lots of different optional paths which have to be considered in order to produce robust code.

Right, and this isn't a good place to be (the tribal knowledge bit). What we can do is write some non-normative text stating "It is expected that DID URL Dereferencers will interpret the service DID Parameter value as a fragment identifier in the DID Document that identifies a service description." ... there is plenty we can do here to nudge implementations towards doing the right thing w/o blowing our timeline out of the water.

The exchange between @kdenhartog and @jandrieu was enlightening as I had not thought that someone might try to interpret the service DID Parameter value as a URL path... so, we should do what we can to ensure that people don't try to do that, even if it's non-normative language in this version of the spec.

[kdenhartog]

non-normative language seems like a good way to go for this. It's achieving the same thing as a draft report out of the CCG but allows us to keep the two parameters within a commonly found document. To address @llorllale original concern as well I think it would be useful to add some additional non-normative language about some commonly found patterns used for DID URLs when I'm back to work on my Tuesday (Monday is off here for a holiday) so we can resolve this issue.

[mavarley]

For the record, I feel @kdenhartog accurately captured and described the issue, and I feel it has been adequately discussed here. I am fully aligned with providing non-normative language that can describe/capture some of the tribal knowledge around current implementations w/o disrupting the progress of the spec. Thank-you.

[kdenhartog]

Got some initial text in for this. @mavarley or @llorllale can you make sure to review #764? I wasn't able to add either of you to the reviewers for that PR.

[peacekeeper]

At some point we decided that for resolution and dereferencing, DID Core would define the "contract", i.e. the interfaces with their inputs and outputs, but that their concrete implementations were out of scope.

In other words, we would define the syntax of DID URLs and some parameters as well as their meanings, but we would not define the concrete dereferencing logic. This also felt in line with the URI principle of Separating Identification from Interaction.

The DID Resolution specification defines a more concrete algorithm for implementing the dereference() function, including how to treat the service DID URL parameter: https://w3c-ccg.github.io/did-resolution/#dereferencing-algorithm-primary

So I'm not strongly opposed to Kyle's PR, but I'd actually prefer to not add more language to DID Core, and instead point to DID Resolution in a non-normative way and improve the content there, when it comes to details of dereferencing that go beyong the "contract".

[msporny]

@kdenhartog, can you please address @peacekeeper's concern in your PR?

[iherman]

The issue was discussed in a meeting on 2021-06-08

• no resolutions were taken

View the transcript

4. verification method IDs MUST contain a fragment (issue did-core#708)

_See github issue #708._

_See github pull request #764._

Brent Zundel: there has been a long discussion in the issue
… Mike indicated to me he'd be on the meeting today
… but perhaps someone can speak on his behalf?

Justin Richer: your friendly local deputy here to speak. Mike couldn't make it
… he told me that having looked over where things landed, secure key is happy with it being a note to implementers
… Markus made a good point that the details of this kind of thing belong in resolution
… the details of resolution
… and kyle's proposed text does seem to put things in the right framing
… by the way you can expect something like this to happen but dont' count on it, is what i might personally add
… but that's implied
… mike said that as far as secure key is concerned the issue can be closed with or without that PR, though my recommendation is we take kyle's PR

Brent Zundel: thanks
… sounds like that PR will resolve that issue