Manifest: Describe the manifest data structure

[mgiuca]

There isn't really a formal specification of the manifest data structure. Instead, each member has its own section which describes the format of that member in prose. This makes it hard to discuss (e.g., in Ken's shortcut proposal, we'd like to be able to say "sequence<IconInfo> icons;" but there's no name for "the dictionary that describes a Manifest icon" because it's described in prose).

I think we should add a WebIDL section that contains a dictionary definition for the full manifest, as a first step (I can do it).

As a second step, I think this would allow us to simplify the language in each individual member. The "steps for processing a manifest" algorithm could be replaced with "Parse the JSON into an IDL dictionary value, then convert it into an ECMAScript object" (which automatically applies all of the IDL type conversion rules).

Then, for example, this text:

The steps for processing the short_name member is given by the following algorithm. The algorithm takes a manifest as an argument. This algorithm returns a string or undefined.

1. Let value be the result of calling the [[GetOwnProperty]] internal method of manifest with argument "short_name".
2. If Type(value) is not "string":
   
   
   
   1. If Type(value) is not "undefined", issue a developer warning that the type is not supported.
   
   
   2. Return undefined.
   
   
3. Otherwise, Trim(value) and return the result.

could be replaced by this (assuming short_name is a DOMString):

The steps for processing the short_name member is given by the following algorithm. The algorithm takes a manifest [referring to the processed ECMAScript object] as an argument. This algorithm returns a string or undefined.

1. Let value be the result of calling the [[GetOwnProperty]] internal method of manifest with argument "short_name".
2. If Type(value) is not "string", return value.
3. Otherwise, Trim(value) and return the result.

Were it not for the call to Trim, this would make almost all the processing steps trivial. Unfortunately, almost all of them do Trim their strings, which makes things more complicated. Perhaps we could make a global rule that Manifest strings are trimmed, then we wouldn't really need any processing steps at all.

[mgiuca]

@marcoscaceres What do you think?

[marcoscaceres]

Agree. This would be nice.

[kenchris]

I am totally pro this, I have ran into this problem myself a couple of times

[marcoscaceres]

Perhaps we could make a global rule that Manifest strings are trimmed, then we wouldn't really need any processing steps at all.

We could also not trim and be more strict.

[kenchris]

I don't think that having a few extra line per entry is that much of a deal, editorial wise

[kenchris]

I did a stab at this https://github.com/w3c/manifest/pull/612 (defining the IDL)

Would be great with some initial comments from @marcoscaceres to know if I am doing this right before continuing :)

Then after landing that change we can simplify the prose.

Guess I need to add [NoInterfaceObject] to all of these right?

[marcoscaceres]

Pasting what I wrote in the PR, so we can try to reconcile this...

So... although we can define processing in IDL, we need to be careful if we are going to expose these on the web. In particular, “foo_bar” property names would make this API inconsistent with the rest of the platform.

It’s a bit similar (but obviously not the same) as CSS in JS, where “foo-bar” becomes “fooBar”. See casing rules https://w3ctag.github.io/design-principles/#casing-rules

[kenchris]

I don't think we actually need to expose the types, these are mostly for being able to easily refer to them and use them potentially in other specs. That is why I was originally using [NoInterfaceObject] but I guess I understood that wrong.

[tobie]

So this is a tough call. My understanding is this won't actually ever be exposed as such, right? If so, it seems OK, but others more knowledgeable than I might disagree.

wrt to the USVString vs. DOMString issue, again, I'm of little help. Sorry. Clarifying the use cases in WebIDL is on my plate and I should get to it shortly, but until then, you probably want to ask @annevk.

[kenchris]

No the idea is not to expose these. There is no way for web devs to obtain the manifest info without fetching it manually, in which they will get JSON back.

The IDL helps defining the spec in a less verbose way (we can simplify algorithms), and it gets types we can refer to in incubation specs or documents before integrating new features in the spec itself. It also makes the spec more structured and gives better overview (IDL snippets and IDL index).

[tobie]

No the idea is not to expose these. There is no way for web devs to obtain the manifest info without fetching it manually, in which they will get JSON back.

Maybe add a pretty large advisement about that?

[annevk]

What exactly do you get from IDL and what would this look like? As far as I can tell what you need to be doing is UTF-8 decode, followed by JSON parse, followed by operating on the resulting JSON value. What am I missing?

[kenchris]

I am fine with doing that. Any suggestion on the best way to do that?

"This spec uses Web IDL to define the structure and processing model of the JSON document, which is internal to the User Agent, and thus not exposed to developers directly"

@annevk My PR is here: https://github.com/w3c/manifest/pull/613/files

[mgiuca]

Note there is a preview here. In particular, the WebAppManifest dictionary.

So is the issue just the naming discrepancy between the manifest fields and normal web attributes ("short_name" vs "shortName")? Or is it more fundamentally that we shouldn't create an IDL that isn't actually exposed on the web?

What exactly do you get from IDL and what would this look like?

The reason for this is mostly for readability. It's becoming increasingly harder to read the manifest spec because there are so many fields. I would like to be able to see at a glance what all the fields are and their types. WebIDL is a convenient language to describe it succinctly instead of loads of prose algorithms.

Another advantage is that an implementation could potentially just take the IDL, parse it, and generate manifest parser code. Obviously we can already do that today (but we don't do this today in Chrome), but it would make sense to have "official" IDL for the manifest, rather than implementations essentially writing their own IDL to match the prose text in this document.

Having said that, we do have to be careful because we might want to expose some of these in JavaScript APIs. The original motivation was Ken's shortcuts proposal, which has a ShortcutInfo dictionary that would be both the type of a field in the manifest _and_ the type of parameters to a JavaScript API. That ShortcutInfo dictionary itself has an icons field which has the same type as icons in the manifest (so that means the dictionary Ken calls ImageResource in his PR would also be exposed to JavaScript APIs). And Google's proposed getInstalledRelatedApps API would also return instances of the related_applications manifest member (which Ken calls ExternalApplicationResource in his PR).

So I think in the future, there _will_ be some of these dictionaries exposed to web APIs (and that is part of the motivation behind this change). Questions for the community:

1. Is it inappropriate to have underscores in dictionary fields that are exposed on the web?
2. If so, if we expose manifest members to JavaScript APIs, will we need to expose them under camelCase names instead?
3. If so, should we specify the manifest version and the JS version as two different IDL dictionaries, or have some kind of automated renaming mechanism like Marcos mentioned CSS does.

[marcoscaceres]

So is the issue just the naming discrepancy between the manifest fields and normal web attributes ("short_name" vs "shortName")? Or is it more fundamentally that we shouldn't create an IDL that isn't actually exposed on the web?

It's mostly the naming.

There are advantages (and nice security properties) to having the manifest described in IDL, even if only passed internally. We've had a bug open about this for a couple of years in Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1176442

1. Is it inappropriate to have underscores in dictionary fields that are exposed on the web?

IMO, yes. This violates rules in tools like ESLint and makes for inconsistent coding styles. It's not very idiomatic to have underscores in name in JS (or on the web).

Having said that... it's not like anything would actually break.

If so, if we expose manifest members to JavaScript APIs, will we need to expose them under camelCase names instead?

We don't _have to_ ... but we may choose to.

If so, should we specify the manifest version and the JS version as two different IDL dictionaries, or have some kind of automated renaming mechanism like Marcos mentioned CSS does.

I don't know how the magic mapping works for CSS, but I'm sure @annevk does (as he worked on CSSOM for a few years)... but there is also data-foo-bar attribute in HTML that undergo similar conversion: where that becomes dataset.fooBar.

[marcoscaceres]

Oh, I'd just add that we shouldn't assume the dictionaries we define in the IDL will necessarily end up being dictionaries in an API... some may be generic and useful enough to become proper interfaces. We don't know yet, however. We will need to work that out as we go.

[annevk]

If you just want an overview of all JSON members and their corresponding types, have you considered creating a table?

If you actually want to use IDL, how exactly would that work? You would first convert bytes to code points. Then code points to JSON values. Then JSON values to ECMAScript values and then feed that to the dictionary parser to get an IDL dictionary?

I don't see any such processing model defined at the URL you pointed to.

[mgiuca]

If you just want an overview of all JSON members and their corresponding types, have you considered creating a table?

That would be preferable to what we have now, but not convey as much information as IDL. A couple of points:

1. IDL not only gives us syntax for describing the type of a bag of named fields, but it also gives us semantics for how to process it. For example, how to convert values that are not of the expected type, default values, etc. We could certainly specify all that in prose (mentioning the table), but why do that when the IDL processing model is already well understood?
2. It's possible that manifest parsing code (in the implementation) would be much more robust, standards-compliant and simpler, if it was based on existing IDL parsers rather than hand-coded logic. Of course, the implementor is free to do it this way regardless of whether it's specced as IDL, but it would be easier to build and make sure you had it right if you could just copy+paste IDL from the spec into the implementation (as we do with other APIs).

If you actually want to use IDL, how exactly would that work? You would first convert bytes to code points. Then code points to JSON values. Then JSON values to ECMAScript values and then feed that to the dictionary parser to get an IDL dictionary?

Yes, I think so. Implementations could cut out some of those steps but I think that's how the spec would go.

I don't see any such processing model defined at the URL you pointed to.

Not yet. Kenneth just wanted to capture the IDL initially, and in a follow-up work on rewriting the parsing logic to use the IDL primitives instead of manually parsing. What is your overall opinion on whether this is worth it?

[marcoscaceres]

I'll note that there are a few breaking changes if we do this:

1. enums are case sensitive, so orientation and display modes values could break in web content.
2. We wouldn't trim anymore (as you noted earlier) - I think that's fine, IMO.

We should check if the above would break a lot of web content. I doubt it will.

FWIW, I think it's worth doing this - if only because it gets rid of so much redundancy. But then again... I'm not sure how we reconcile the JSON naming vs DOM API naming issues.

[annevk]

What is your overall opinion on whether this is worth it?

It seems reasonable as long as it's well-defined. And as long as you stick to dictionaries there's not much room for collateral damage. If we get more consumers (e.g., Origin Policy would be a good candidate) we should maybe uplift this to IDL somehow.

I'm not sure how we reconcile the JSON naming vs DOM API naming issues.

Nothing blocks you from using the JSON names, but I'm also not sure we actually want to use that naming convention for all JSON... Certainly reporting infrastructure uses different names and I hope Origin Policy would too.

[tobie]

FWIW, I think it's worth doing this - if only because it gets rid of so much redundancy. But then again... I'm not sure how we reconcile the JSON naming vs DOM API naming issues.

We could add JSON bindings (I'm only half joking).

[marcoscaceres]

We could add JSON bindings (I'm only half joking).

Let's play this out as a straw person... we might need it to do the conversion dance that Anne described in https://github.com/w3c/manifest/issues/611#issuecomment-332452821 ... what are you thinking @tobie?

[kenchris]

Yes I was discussing some of that with @tobie yesterday. I might give it a shot today, but will probably need some help :-)

[annevk]

I guess the other thing I wasn't sure of is whether you actually need all the IDL processing here. Why would we convert numbers to strings for instance if we want strings? There's far less types in JSON and it's much easier to deal with JSON values than arbitrary JavaScript input. So where does all the complexity you allude to come from?

[annevk]

It does make sense for JSON to be consistently processed though, so something like IDL makes sense, but I'm not sure we'd want the same kind of ToString() all the things behavior and such.

[marcoscaceres]

@annevk agree. Thankfully there are not too many of those... @kenchris, can you evaluate how much breakage there will be if we make better use of the types?

[kenchris]

We we are to use parts of the manifest format for other web APIs, like say a "addShortcut({ // using same icon format here }) then we would be dealing with arbitrary JS

[annevk]

At that point @marcoscaceres's concern about naming becomes a real issue (which as I said you might want to revisit given precedent in reports and such).

[tobie]

Let's play this out as a straw person... we might need it to do the conversion dance that Anne described in #611 (comment) ... what are you thinking @tobie?

So @marcoscaceres, currently, if we want to turn JSON into WebIDL, we have to do JSON.parse(json) -> ES -> WebIDL. We could instead describe bindings for each of JSON 7(?) types, including camelCasing snake_case object properties. This could even nicely mirror the toJSON stuff we recently added.

That said, it's only really worth doing if there are other parts of the platform that would benefit.

[kenchris]

Could allowing both forms in the JSON be an option? ie. specify that _ turns the following character into uppercase?

[kenchris]

@tobie like this? https://w3c.github.io/payment-method-manifest/

[tobie]

@kenchris right, though I have no idea whether WebIDL bindings would help, here.

[kenchris]

the toJSON being discussed: https://heycam.github.io/webidl/#idl-tojson-operation

[mgiuca]

This is done, right?

[annevk]

No, see https://github.com/whatwg/infra/issues/159. We haven't really agreed upon a shared processing model for all JSON formats in the web platform, as far as I can tell.

[mgiuca]

I discussed with @dominickng and we have a vague plan for this.

We do want to keep defining the manifest in terms of IDL. It's just _so_ much more readable than explaining the types of all the fields by algorithms that check the types and reject if they don't match. I recall @marcoscaceres and @annevk had a few issues with this but the main issue is that we have no way of defining how much of the manifest to invalidate when there is a type error in some member (technically speaking, due to IDL, we should be invalidating the entire manifest if there is any type error, but that isn't our intention and I don't think any browser does this).

This is becoming an actual problem for us; see this Web Share Target code review on Chromium, where we have to unilaterally decide how much of the share target member to invalidate when an accept is invalid, because we don't want to invalidate the entire manifest, as the spec implicitly tells us to do.

The proposed solution is this:

1. Add to Web IDL (either directly in the Web IDL spec, or by monkey-patching from the Manifest spec) a new ECMAScript-specific extended attribute for types [CatchTypeErrors] (name can be debated, my personal preference is [TheBuckStopsHere] 😜). (There are a lot of these already, maybe there's already one that does this.)
2. Define ECMAScript-to-IDL conversion for [CatchTypeErrors] as "If during the conversion of the inner type to IDL, an exception was thrown of type TypeError, let the result be undefined." Essentially, this means if you put [CatchTypeErrors] on an optional dictionary member, any type errors inside that member will cause the member to take its default value, rather than propagating the TypeError upwards.
3. In Manifest, put [CatchTypeErrors] on most of the top-level members, so if any of them are invalid, they don't break the whole manifest. In the case of some more complex members, [CatchTypeErrors] may be put on the sub-members instead.

The behaviour I want (but haven't thought of how to define yet) is for sequence types, if you put [CatchTypeErrors] on the type of the sequence elements, any failing element simply drops the element from the resulting list, not putting undefined in the list. That's pretty key since I'd want to define icons as sequence<[CatchTypeErrors] ImageResource>, so if any icon is invalid, we drop that icon rather than dropping the whole icons member.

Does this sound like a reasonable approach, @marcoscaceres @annevk ?

[mgiuca]

I went ahead and specced this extended type attribute in WebIDL to see what it would be like. heycam/webidl#597

See https://pr-preview.s3.amazonaws.com/mgiuca/webidl/pull/597.html#CatchTypeErrors

[domenic]

So the plan would still be that, e.g., you could do "short_name": { "foo: "bar" } and get a short name of "[object Object]"? That's what Web IDL implies at least...

[mgiuca]

@domenic Yeah I guess so. 😕 I'm not trying to fix the bad but valid conversions. I'm trying to stop the entire manifest from being invalidated if any element actually throws a type error.

I'm not sure if the previous (pre-WebIDL) algorithms applied string conversions or if they just more strictly checked for the exact string type. If so, maybe there is a case to having that strictness also (which maybe warrants another extended attribute, for string values, to say "this actually has to be a string, not a random object that gets stringified".

[domenic]

I'd always assumed we would do actual type checks; that's my current plan for import maps. No spec yet but I did write up a reference implementation which is pretty easy to translate into spec-ese.

[mgiuca]

@domenic That sounds great :) but it seems like a separate feature than catching TypeErrors.

[domenic]

Sure, it's just unclear whether you're using Web IDL at that point, or just something that shares the syntax with completely different semantics.

[mgiuca]

Hmm. I mean, it sure is useful to be able to reuse the syntax and all of the semantics except for 2 type conversion rules that we want to be less strict (in one case) and more strict (in another case).

Arguably these rule changes could be useful in more places than just the manifest spec. Like, we have extended attributes to change how other fields are processed (e.g., [Clamp], [EnforceRange]) which seem to be defined because the old way was too lax, but we can't change the default. It seems that [StrictString] for strings would be no stranger than [EnforceRange] for integers. So I wouldn't say it's a different language, just a slightly different requirement for how to process some types. And that's exactly what extended IDL attributes are for.

[mgiuca]

Update: You can see what the old (pre-WebIDL) behaviour used to be by checking out e5536520. Looks like all the string properties were defined like this:

1. Let value be the result of calling the [[GetOwnProperty]] internal method of manifest with argument "name".
2. If Type(value) is not "string":
   a. If Type(value) is not "undefined", issue a developer warning that the type is not supported.
   b. Return undefined.
3. Otherwise, Trim(value) and return the result.

(I think we deliberately got rid of the Trim behaviour at the same time.) I think that is strict about it being a string.

[domenic]

Well, I think it's not quite that amount of reuse. You're only using a small subset of Web IDL, i.e. the dictionaries/basic data types/sequences subset, ignoring the large swathes of the spec devoted to defining interfaces, global variables, methods, class hierarchies, etc. And the semantic content of that subset consists entirely of type coercion rules. But, you're not actually using any of the type coercion rules; you're instead using other rules that do not share very much of the logic at all. And you're using them on a very different set of inputs, since the output space of JSON.parse is a small subset of the total set of JS types, so whatever logic is left over, is not a great match.

As an example, on a page where someone runs Object.prototype.short_name = "foo", using Web IDL would imply that parsing {} gives back a short name of "foo", not undefined. Again, you could add yet another extended attribute to fix that, but...

I understand the motivation of wanting a readable, declarative syntax. I just think Web IDL, subsetted to a small piece of its semantics, with extended attributes to change all remaining behaviors of that subset, is ... not great. Maybe it's better than inventing your own thing, or reusing something like JSON schema. Indeed, maybe it's the best choice. But it's still not great.

[annevk]

I think it would be better if we started with the high-level goals for this syntax and then see what kind of schema language would be the most appropriate fit.

• Parsing always uses https://infra.spec.whatwg.org/#parse-json-from-bytes. (Otherwise the processor returns failure.)
• A https://mimesniff.spec.whatwg.org/#json-mime-type is required. (Otherwise the processor returns failure.)
• There needs to be a way to declare types for fields. (It probably makes sense for these types to match IDL types.)
• A type-mismatch is equal to the field being set to undefined.

Things I'm unclear on:

• Is the top-level value always an object?
• Do we need to be able to require fields?
  
  
  
  • Only top-level or also nested fields?
  
  
  • Is such a field is undefined, does it mean the processor returns failure?
  
  

Are there other requirements here?

[marcoscaceres]

I've become increasingly uncomfortable with using WebIDL to spec the manifest format. I think it seemed like a great idea at the time, but for the reasons @domenic mentioned, and the issues we've hit in practice, we need to seriously re-evaluate what we have done here. I'm also super worried that other groups have started imitating us, and those groups have far less experience with WebIDL than we do (they are going to get themselves into a world of hurt).

As @annevk points out, we have a few options. When we started this project, I deliberately chose not to use any schema language because we'd had such a terrible experience with XML/XML Schema, XHTML+DTDs, RDF etc. that it seemed pointless to define yet another schema language. Rather, it seemed like just writing our own processing rules using prose would make more sense - even if it turned out to be more long winded.

In implementation, the parsing rules were supposed to be consistent. Consider, Gecko's "ValueExtractor" that pulls any value from the manifest using the same algorithm (19 lines of code).

And each one of our members gets extracted consistently too, for example, each rule looks like this:

    function processOrientationMember() {
      const spec = {
        objectName: 'manifest',
        object: rawManifest,
        property: 'orientation',
        expectedType: 'string',
        trim: true
      };
      const value = extractor.extractValue(spec);
      if (value && typeof value === "string" && this.orientationTypes.has(value.toLowerCase())) {
        return value.toLowerCase();
      }
      return undefined;
    }

We could go back and generalize the processing rules to make something simple and consistent like the above - which handles special case processing for particular members (basically @annevk's requirements). Or we could look at Chrome and WebKit's implementation, to see if either team has come up with something clever, concise, and consistent.

If we end up with a generalized model for the Web, then that's great (and we can push it to Infra or make a new spec)... but I don't know if that should be a goal just yet.

[annevk]

I think it's worth looking at schema languages for JSON, but I suspect we need a custom one, since we want forward compatibility. I.e., ignore optional fields that do not match the schema and ignore fields outside the scope of the schema.

If we need something custom, taking inspiration from dictionaries in IDL does seem appropriate, something nearly identical might even be acceptable, but I'm not sure it's worth trying to create a shared abstraction there as the goals are rather different and it'll be a hassle to maintain for either side.

I guess if I were to work on this I'd copy the IDL dictionary work and simplify it to fit the needs of what we need for all these JSON resources.

[tobie]

JSON Schema is forward compatible in that regard.

[marcoscaceres]

@tobie wrote:

JSON Schema is forward compatible in that regard.

As an implementer, I want to avoid implementing JSON Schema from scratch if I can avoid it. Unless we agree that that is what we are going to do for a substantial number of specs.

@annevk wrote:

I guess if I were to work on this I'd copy the IDL dictionary work and simplify it to fit the needs of what we need for all these JSON resources.

That might be reasonable. We have a fairly good idea what kinds of behaviors the processor should exhibit. From IDL dictionaries, I think we are only missing trimming and normalization behavior.

[g-ortuno]

FWIW, Chrome already has an implementation of JSON Schema. We currently use it to parse Policies but in the past we used it to parse Extension Manifests haha.

[mgiuca]

I really don't want to go down the route of either a) inventing a new schema language, b) forking a subset of WebIDL or another schema language, or c) using a schema language that isn't already part of the web platform. All three of those seem like a tremendous amount of work compared to just tweaking WebIDL for our needs.

FYI currently Chromium parses the manifest with a hand-written algorithm which has all of the same issues as the old Manifest spec (very hard to read and easy to make mistakes). I would be hopeful that we can modify Chromium's manifest parser to use WebIDL just like we did in the spec, but this needs these fixes first.

OK, here are all the features of WebIDL that we use in the Manifest spec, or Web Share Target (which is an extension of Manifest):

• Simple types DOMString, USVString, boolean.
• enum
• sequence
• dictionary
• partial dictionary (for easily extending in other specs)
• Optional and required dictionary members.
• Default values for dictionary members.
• "or" type clauses.
• Extended attributes / annotated types (to declaratively specify the extent of invalidation caused by type errors).
• Conversion from an ECMAScript value to an IDL value, of all of the above types. Includes a lot of type checking logic.

That's a huge chunk of WebIDL.

There's also quite a lot of functionality in WebIDL that we _don't_ need to define the Web App Manifest. A lot of the above discussion focuses on the fact that we aren't using the entire WebIDL spec, just the type conversion rules. I don't see this as any sort of argument to not use WebIDL. Specs depend on other specs all the time and use only a tiny fraction of the dependent spec's functionality. Pick any given web API and you'll probably find it uses a small subset of the WebIDL features. I don't see that as a reason for each web API to invent its own specification language with just the features it needs.

@domenic :

As an example, on a page where someone runs Object.prototype.short_name = "foo", using Web IDL would imply that parsing {} gives back a short name of "foo", not undefined. Again, you could add yet another extended attribute to fix that, but...

Yeah, as I said above, I think adding a [StrictString] extended type attribute would not be any different to the existing [EnforceRange] attribute for integers. TAG advice is to always use [EnforceRange] for integers, so why not add [StrictString] and recommend that all new interfaces use that? A desire to not convert {} to "[object Object]" is not unique to the manifest JSON file, and seems orthogonal to the suitability of WebIDL as a format for validating JSON.

Something I want to clear up with what I'm trying to do here:

@annevk :

• A type-mismatch is equal to the field being set to undefined.

No, not always. At the top level, we want each member to only invalidate itself, but at lower levels, we have the flexibility to decide (note in #750 I don't add [CatchTypeErrors] to every member in every dictionary, just the top-level members).

I want to be able to annotate dictionary members and sequence elements on a case-by-case basis, so that we have the flexibility of deciding the granularity of errors (when a type error occurs, what level of the manifest does it invalidate?).

For example, in Web Share Target, we are currently debating whether an error in a files.accept element should invalidate a) just that accept type, b) the entire accept array, c) the entire file, d) the entire files array, or e) the entire share target.

With [CatchTypeError], we can easily (and readably) specify one of those five options by placing the [CatchTypeError] annotation on a) the USVString inside the accept sequence, b) the accept member of ShareTargetFiles, c) the ShareTargetFile inside the files sequence, d) the files member of ShareTargetParams, or e) just the share_target member of WebAppManifest.

So I'm not trying to universally change how type checking works in WebIDL. I'm trying to add a feature to WebIDL to let us override type checking behaviour as desired.

I think maybe @domenic you and I could have a VC offline to sync on this instead of a long public thread, then we can summarize here.

[annevk]

I think we agree on reusing IDL types and perhaps on using most of the dictionary syntax. The question with respect to the syntax is whether it's best to abstract that somehow or fork it. Where we might disagree is conversion.

I think you're equating JSON with APIs a bit too much. [StrictString] makes no sense for APIs (using ToString on arguments is expected and used throughout the ecosystem), but makes perfect sense for a JSON resource. Trying to shoehorn those different validation expectation into the same declarative system is not a good idea as they might leak. A shared abstraction might be warranted though.

[mgiuca]

OK. I can agree in principle to factoring out the type definitions and ECMAScript conversion logic modulo strictness requirements into a shared abstraction. It sounds like a huge amount of work though.

This is a fairly urgent issue for Manifest since right now, we are not properly specifying how errors are checked. What I might try and do in the mean time is add some hand-wavy text to the processing steps like "During IDL conversion, if the conversion of any top-level member of the WebAppManifest throws a TypeError, set that member to its default value." (And so on.)

This isn't terribly formal, but at least it will unambiguously describe the expected error handling.

[annevk]

The thing is that if something expects a string and you give it a number or an array, IDL conversion wouldn't throw, whereas I think that's what we want for JSON conversion. For JSON I'd imagine we don't want much conversion at all so that's why I thought specifying that bit again wouldn't be too much effort.

The other problem with using IDL directly I think is that these dictionaries shouldn't be in the same namespace.

[marcoscaceres]

I'm still unsure what the problem is with just having a single property extraction algorithm as per
https://github.com/mozilla/gecko-dev/blob/master/dom/manifest/ValueExtractor.jsm#L34? And the example I gave. They object definitions are concise and get the job done in the same way that that defining an extended attribute would.

As @annevk mentioned, if we add things like [StrictString] and [CatchTypeError], they are going to leak into everyday WebIDL interfaces. I really don't think we want a special set of WebIDL things for a JSON format VS having a thing just Web APIs.

[annevk]

@marcoscaceres how would you envision that working? Basically an algorithm, that given a name and a type, returns the value? It seems you'd still need to describe the general JSON format, but that might work if flushed out a bit more.

[marcoscaceres]

Basically what we originally had, just tightened up a bit more that we understand what we want for each member type (and how to get values from each property).

So:

• Use just the JSON types - not the Web IDL types. And just hook directly into ES6 and skip WebIDL entirely.
• define a ExtractValue(object, property, expectedType, [trim, normalization]) abstract operation in terms of ES6. It throws TypeErrors that the caller then deals with. Optionally, normalization can be "lcase" or whatever - or we can do a breaking change and stop normalizing stuff.
  
  
  
  • We can even do specialized extractors: ExtractStringValue() if we want even more consistency with type checks and coercion (e.g., ExtractColorValue()).
  
  

[annevk]

I like that approach as a low-level building block. It shouldn't be as much work as IDL given that the space is much more constrained (i.e., we only have object, array, number, string, boolean, or null, and we can trust the input to not cause side effects). And you basically want the same kind of algorithm for each (as @marcoscaceres's code also demonstrates):

1. Let _value_ be the result of getting _property_ from _object_.
2. If _value_ is undefined, then return undefined.
3. If _value_'s _type_ is not _expectedType_, then return undefined.
4. Return _value_.

And then you can define higher-level helpers around that, which perform whitespace normalization, range checks, lone surrogate replacement, array to list concept mapping, etc.

[mgiuca]

I added a new attempt to fix this in #753 (This is the "low tech" approach of just adding some fairly non-formal language around behaving differently in manifest parsing than the normal WebIDL spec allows).

I'm less in favour of this than #750 (which would require upstream changes to WebIDL). Because if we did heycam/webidl#597 + #750, then we implementations could easily follow along with these changes and the manifest parser implementations could literally use the IDL parser. With #753, I think it specifies what we want, but it's impractical to implement it without changing the implementation's WebIDL parser.

(Of course, there are other discussions about reverting the use of WebIDL altogether, but in my opinion, I would rather keep it.)

[marcoscaceres]

753 still does the IDL conversion tho, which still leaves us in a strange place. I'd prefer we just go back to defining this in terms of pure ECMAScript and not do any IDL at all.

[marcoscaceres]

I'd like to pick this up again and rewrite the spec in terms of ECMAScript.

@mgiuca, would you be ok with that?

[marcoscaceres]

Ok, we now have a means to convert JSON into infra types:
https://github.com/whatwg/infra/issues/159#issuecomment-491338205

My plan is to rewrite the spec to use the above and operate on the infra types.

[kenchris]

Can we still get something like the IDL overview at the end?

[marcoscaceres]

No, but we can just improve the TOC.

[marcoscaceres]

(alternative, we have a non-normative JSON Schema we can link to... I think we already have that)

[dominickng]

It looks like there are more use cases which want an ImageResource IDL dictionary (e.g. Content Indexing, see https://github.com/rayankans/content-index/issues/2, and Background Fetch, see https://wicg.github.io/background-fetch/#background-fetch-manager). Maybe that should be spun out of Manifest, which then reduces the need to define Manifest itself in terms of IDL?

There's also discussion on the image selection front in https://github.com/w3c/manifest/pull/710 (@jakearchibald). Is there a way to move forward here @marcoscaceres @mgiuca?

@rayankans FYI

[mgiuca]

@marcoscaceres Do you think you'll proceed with converting this to a JSON schema type thing (and not IDL)? We have clients trying to use our ImageResource dict which means now is a good inflection point to decide either a) it's OK to use dicts from Manifest in other parts of the web platform, or b) no, these are specific to the Manifest format and can't be used in JS APIs.

I still think that being able to use them in other JS APIs is a good thing (reuse of those definitions, instead of having to define parallel dicts in two different languages), but if you want to change over to another language, we should do so before others start depending on it.

(I'm also hesitant about this specific usage, in Background Fetch, since the ImageResources used there are unrelated to the ones we use in Manifest.)

[marcoscaceres]

@marcoscaceres Do you think you'll proceed with converting this to a JSON schema type thing (and not IDL)?

Yes

We have clients trying to use our ImageResource dict which means now is a good inflection point to decide either a) it's OK to use dicts from Manifest in other parts of the web platform, or b) no, these are specific to the Manifest format and can't be used in JS APIs.

We should keep ImageResource as a thing for APIs tho (we've tried to do this previously also with @jakearchibald). But we can pull that into a new spec if needed.

[marcoscaceres]

Ok, first step: I've sent a PR to include the JSON schema into the spec https://github.com/w3c/manifest/pull/928

[marcoscaceres]

In 32b497c4ca34c9069aa5e545126c252724f370fe, I (finally) got rid of the IDL and converted the "processed manifest" to an (Infra) ordered map.