K6: Thresholds aren't triggered when custom metrics aren't used

[na--]

This script should always fail, but instead it only fails roughly 50% of the time, when the counter not used branch is executed:

import { Counter } from "k6/metrics";

var myCounter = new Counter("my_counter");

export let options = {
   thresholds: {
       my_counter: ["count>500"],
   }
};

export default function () {
   if (Math.random() > 0.5) {
       console.log("counter used");
       myCounter.add(1);
   } else {
       console.log("counter not used");
   }
};

[ben-foster-cko]

This is also an issue for us as it's causing our tests to always fail when the counter has no value. Is it possible to initialise the counter with a 0 value instead?

Our test configuration

export let Errors = new Rate("errors");

export let options = {
  thresholds: {
    "errors": [{
      threshold: "rate<0.1" // <10% errors
    }],
    "http_req_duration": [{
      threshold: "p(99)<1200",
      delayAbortEval: "10s"
    }]
  }
};

export default function (data) {
  var number = data.numbers[Math.floor(Math.random() * data.numbers.length)];

  group("Request payment", function () {
    const paymentResponse = payments.requestAsync(helper.createPaymentRequest(number));

    check(paymentResponse, {
      "status is 201 (Created)": (r) => r.status == 201,
      "payment is approved": (r) => JSON.parse(paymentResponse.body).approved == true
    }) || Errors.add(1);
  });
}

[na--]

@ben-foster-cko, while there's a bug with the way thresholds are evaluated when no metrics are emitted, in your case the problem is that you're using the Rate custom metric incorrectly.

Rate is not meant to be add-ed to only when there's an error :slightly_smiling_face: A Rate metric represents "X out of Y events", so you should add to that rate on every iteration (or whatever you want to track). But you should add true if the event that you want to track (in this case, an error) has happened and false if that event hasn't happened.

Then the threshold "fail if there were errors in more than 10% _(of the total number of events)_" makes sense. Or, put another way, if you're only adding to the metric when there's an error, then you have an error counter, not a rate - you can't know in how many cases there wasn't an error.

In your case, since check() returns true when all checks pass (i.e. no errors) and false when one of the checks doesn't pass (i.e. there's an error). So, if you want to track the number of errors, that means basically tracking (i.e. adding to the Rate) the reverse of the check() result. You should be able to do something like this (note the ! before check):

Errors.add(!check(paymentResponse, {
  "status is 201 (Created)": (r) => r.status == 201,
  "payment is approved": (r) => JSON.parse(paymentResponse.body).approved == true
}));

[ben-foster-cko]

Thanks, I figured this out last night and was just about to post with an update :)

[MStoykov]

I did just take a few hours trying to figure out why a threshold I added wasn't showing up ... the answer: I defined it depending on the "vu" tag and it isn't enabled by default.

I do think though that just like it should be triggered it should always be shown and in this case possibly include a warning "No data was received for this threshold through the execution of the script"

[na--]

Closing in favor of https://github.com/loadimpact/k6/issues/1346