Amphtml: Discussion - do we need a breaking changes policy?

[jpettitt]

Since changes to the AMP runtime can have a profound impact of existing content there should be policy around breaking changes. Specifically:

1. How is a breaking change defined?
2. What is the process for notification?

With that in mind here is a straw man policy for discussion

A breaking change is any change that will cause a existing AMP pages to render in way that is one of:

1. Noticeably, visually, different,
2. Where non visual functionality will not longer produce the same external action.

Example of case 1 would be Issue #5152. An example of case 2 would be changing validation to make amp-ad.js mandatory which would break all existing ads. Another example of case 2: changes to amp-analytics that causes existing scripts to no longer report the same data.

Breaking changes should be subject to a notification policy (beyond the normal github PR) email. This should include:

1. An email "announce" list that all content developers can subscribe to.
2. At least 14 days notice of a change.
3. Validator warnings for a similar period.
4. A clear, scheduled, deployment date to both experiment and prod.
5. Post deployment confirmation notification.

Exception - any change that fixes a bug to bring behavior into conformance with existing documentation or restore existing behavior in response to browser bugs is, at the discretion of the Tech Lead, not subject to the 14 days advance notice requirement.

[erwinmombay]

/to @cramforce @rudygalfi

[cramforce]

This sounds good in general. One device that we have, which is not mentioned here, is to increase the major version of an extension. In general that is what should be used for true breaking changes. Of course, a downside (and upside) of doing this is that existing pages do not get upgraded.

It is sometimes hard to anticipate whether a change is breaking. We are planning to use automated visual diffing in the future to get more confidence.

[Gregable]

This does sound like a good policy to flush out.

I don't think validator warnings make sense for visual changes such as your case 1. We also have a mechanism for validator changes where we can assess the impact of the changes in terms of percentage of affected documents on the web, and we use that to inform how aggressive we can be with the timeline.

I will say that the current warnings regarding amp-ad.js are obviously affecting a large number of docs, and we won't make any changes without that story changing or far more notification. Right now, Google Search Console does not display warnings, only errors, so that's another issue to address along these notification lines.

[jpettitt]

I like using versioning to manage major transitions. Bumping a version lets the publisher decide when to cut over, gives an opportunity to do QA. Pretty much every developer understands the semver model. Then the validator could start to warn about end of life for older versions. As @Gregable notes it would be good if search console reflected the warnings.

[jpettitt]

Any update on making this (or something like it) into an official policy?

[cramforce]

Hey All,

I just posted a proposal in #6094. Please take a look and comment away!