Lighthouse-ci: Questions on Assert Behaviour

[GuruCharan94]

The assert configuration is as below

"assert": {
  "preset": "lighthouse:no-pwa",
  "assertions": {      
      "categories:performance": ["warn", {"aggregationMethod": "optimistic", "minScore": 0.70}],
       "categories:accessibility": ["warn", {"aggregationMethod": "optimistic", "minScore": 0.90}]
    }
}

and from the Assertion Levels docs

If an audit is not found in the assertions object, it is assumed to be off. The audit result will not be checked.

Yet, when I run lhci assert I get the below output.

 ×  unused-css-rules failure for maxLength assertion
     Remove unused CSS
     Documentation: https://web.dev/unused-css-rules
        expected: <=0
           found: 1
      all values: 1, 1, 1

  ×  uses-optimized-images failure for maxLength assertion
     Efficiently encode images
     Documentation: https://web.dev/uses-optimized-images
        expected: <=0
           found: 1
      all values: 1, 1, 1

  ×  uses-responsive-images failure for maxLength assertion
     Properly size images
     Documentation: https://web.dev/uses-responsive-images
        expected: <=0
           found: 1
      all values: 1, 1, 1

  ⚠️  render-blocking-resources warning for maxLength assertion
     Eliminate render-blocking resources
     Documentation: https://web.dev/render-blocking-resources
        expected: <=0
           found: 2
      all values: 2, 2, 2

  ⚠️  uses-long-cache-ttl warning for maxLength assertion
     Serve static assets with an efficient cache policy
     Documentation: https://web.dev/uses-long-cache-ttl
        expected: <=0
           found: 7
      all values: 7, 7, 7

  ⚠️  uses-webp-images warning for maxLength assertion
     Serve images in next-gen formats
     Documentation: https://web.dev/uses-webp-images
        expected: <=0
           found: 1
      all values: 1, 1, 1

Assertion failed. Exiting with status code 1.

What I expected to happen ?
That these rules are not checked because they are not in the assertion object and more importantly that the assertion result does not fail because of this.

[patrickhulce]

Thanks for filing @GuruCharan94! It's failing because you're using the preset lighthouse:no-pwa which has assertions on all of the non-pwa audits. If you don't want to extend a preset and just have your assertions run, eliminate the preset property from your config.

[GuruCharan94]

Thanks @patrickhulce. Assert works as expected when I remove the preset from the config.

[waniparag]

@patrickhulce : My lighthouse.rc file has just
"assert": {
"matchingUrlPattern": "https://[^/]",
"preset": "lighthouse:recommended"
}, But i still get the same error

[waniparag]

This is my lighthouse.rc file to be precise

{
  "ci": {
    "collect": {
      "url": [
        "https://www.xxx.com",
        "https://www.xxx.com",
        "https://www.xxx.com.au/"
      ],
     "numberOfRuns" : 1,
     "ignoreDuplicateBuildFailure": true,
     "allowOverwriteOfLatestBranchBuild": true,
     "settings": {

     }
    },
    "assert": {
      "matchingUrlPattern": "https://[^/]",
      "preset": "lighthouse:recommended"    
    },
    "upload": {
      "target" : "lhci",
      "serverBaseUrl" : "http://localhost:9001/app/projects",

    },
    "wizard" :{
      "serverBaseUrl": "http://localhost:9001/app/projects",
      "extraHeaders": {"Authorization": "Basic noContent"}
    }
  }
}

[patrickhulce]

@waniparag the same thing applies here.

If you don't want to extend a preset and just have your assertions run, eliminate the preset property from your config.

[waniparag]

@patrickhulce : That's what was done. Still get the error and builds fail.

[patrickhulce]

That's what was done. Still get the error and builds fail.

But you just told me your configuration was

{
  "ci": {
    "assert": {
      "matchingUrlPattern": "https://[^/]",
      "preset": "lighthouse:recommended"    
    }
  }
}

so you're using a preset and it wasn't done.