Wpt: SignedExchange tests failing in wpt.fyi

[irori]

Currently, Signed Exchange tests are not passing in wpt.fyi, because pre-generated test certs / signed exchanges are expired.

To make them pass, we wave two options:

• Run Chrome with command-line flags that allows to ignore certificate / signature expiration errors, or
• Re-generate test certs and signed-exchanges before running the tests:
  
  
  
  • Signed Exchanges must be regenerated at least once per 7-days
  
  
  • Certificate must be regenerated at least once per 90-days
  
  
  • Generated certificate must be added as a trusted certificate in OS/browser
  
  

Which is more preferable?

[stephenmcgruer]

Run Chrome with command-line flags that allows to ignore certificate / signature expiration errors

We can definitely do this; we already pass --ignore-certificate-errors (https://github.com/web-platform-tests/wpt/blob/master/tools/wptrunner/wptrunner/browsers/chrome.py#L70). What flag is needed for signed-exchange?

Re-generate test certs and signed-exchanges before running the tests:

• Signed Exchanges must be regenerated at least once per 7-days
• Certificate must be regenerated at least once per 90-days
• Generated certificate must be added as a trusted certificate in OS/browser

We do have https://github.com/web-platform-tests/wpt/issues/21968 open already to setup a cron-job to regenerate the certs we use today, so I think we could do that for this too. Should be as simple as a weekly (or twice-weekly for safety?) GitHub Action that runs generate-test-sxgs.sh?

[stephenmcgruer]

Note: When I run that script myself, I get:

$ ./generate-test-sxgs.sh 
Warning: ocsp is not a correct DER-encoded OCSP response.
Warning: Neither cert nor OCSP have embedded SCT list. Use -sctDir flag to add SCT from files.

Expected warnings?

[irori]

Thanks for the reply @stephenmcgruer!

$ ./generate-test-sxgs.sh 
Warning: ocsp is not a correct DER-encoded OCSP response.
Warning: Neither cert nor OCSP have embedded SCT list. Use -sctDir flag to add SCT from files.

Expected warnings?

This is expected, but it reminds me that we use dummy OCSP / SCT for the test SignedExchange cert. So we have to ignore cert errors by command-line flags anyway. :confused:

To ignore SignedExchange certificate errors in Chrome, two flags are needed:

• --ignore-certificate-errors-spki-list=<SPKI fingerprints>
• --user-data-dir=<directory> (this is required not to persist dangerous changes in the user's regular profile.)

Specifically:

$ chrome --user-data-dir=/tmp/udd \
  --ignore-certificate-errors \
  --ignore-certificate-errors-spki-list='0Rt4mT6SJXojEMHTnKnlJ/hBKMBcI4kteBlhR1eTTdk=' \
  http://web-platform.test:8000/signed-exchange/sxg-location.tentative.html

0Rt4... is the SPKI fingerprint of signed-exchange/resources/127.0.0.1.sxg.pem.

Another issue is that currently the SignedExchange tests use hard-coded URLs embedded in the .sxg files, with the port number 8444 which is used by Chromium's test runner, instead of the WPT's default HTTPS port 8443.
So I had to use this config.json when running wpt serve:

{
  "ports": {
    "http": [8000, "auto"],
    "https":[8444]
  }
}

With this and the above Chrome command line, the SignedExchange tests passed locally in my environment.

[stephenmcgruer]

Ok, it seems like we should maybe not be too worried about re-generating the SignedExchange certs automatically for now (though I'm still happy to do so if that would be useful), and should use the command-line flags instead.

Another issue is that currently the SignedExchange tests use hard-coded URLs embedded in the .sxg files, with the port number 8444 which is used by Chromium's test runner, instead of the WPT's default HTTPS port 8443.

So this seems like a larger issue. Is there a path here to not hard-code the URLs?

[stephenmcgruer]

Ping @irori for the question in my comment above :)

[sleevi]

@jyasskin as FYI

[irori]

Sorry for the silence.

So this seems like a larger issue. Is there a path here to not hard-code the URLs?

Ideally, we should be able to generate SignedExchanges dynamically in tests, so that WPT server address can be embedded into SignedExchanges.

Currently there's no Python implementation of SignedExchange generator (Go and C implementations are available), so we'll need to check in generator binaries (for each platform) to WPT repository. Then, tests can use a wptserve's Python handler which generates SignedExchange on the fly, by running the generator binary as a subprocess.

[stephenmcgruer]

(cc @web-platform-tests/wpt-core-team )

Currently there's no Python implementation of SignedExchange generator (Go and C implementations are available), so we'll need to check in generator binaries (for each platform) to WPT repository. Then, tests can use a wptserve's Python handler which generates SignedExchange on the fly, by running the generator binary as a subprocess.

We've generally avoided including binaries in the WPT project (e.g. for quic we ended up agreeing on a Py3-only server) Is there a feasible path to a Python SignedExchange generator?

[irori]

Thanks for the pointer, I think we can take a similar approach to quic. SignedExchange generation requires cryptography support, but it seems quic depends on openssl via pica/cryptography so SignedExchange generator could do the same.

[LukeZielinski]

Periodic ping - are there plans or desires to work on this in the next couple months?