K6: Test suites / execute multiple scripts with k6

[na--]

I'm creating this issue as a place to discuss possible ways for k6 to execute multiple distinct scripts in a single run. This is prompted by https://community.k6.io/t/running-10-test-cases-out-of-100/448, and other similar previous user inquiries.

Something like this can currently be achieved by a simple shell script that sequentially calls k6 run "$script_path" for all files in a folder, but that workaround is not very convenient without an external output like InfluxDB. It also doesn't work well with k6 cloud, and isn't efficient in general (each separate k6 run has to re-initialize its VUs). GitLab’s performance testing tool is probably also worth investigating...

I also have a vague idea how we can implement something like "test suites" natively in k6 after #1007 is done, and do it relatively easily (:tm: :sweat_smile: ). To get 80% of the way there, we just need to implement one minor (sort-of-already-planned) improvement on the currently planned #1007 feature set :tada:. Instead of specifying startTime: "30s" for an executor, users should be able to specify startAfter: "someOtherExecutorName". This will be very easy to implement (minor changes in the config validation, here and here).

This will add a lot of convenience in general, so it's probably worth doing just for that. But it will also, when used with the exec option for each executor (to run some non-default function), probably satisfy the "test suite" requirements of the majority of our users, albeit with some manual configuration work (which we can ameliorate with some JS helper functions).

At a later point, to satisfy the remaining minority of more complex use cases, and to achieve something similar to k6 run test1.js test2.js test3.js (and, crucially, k6 cloud test1.js test2.js test3.js), we can make another (somewhat more tricky) change to k6. We need to transform the current local.ExecutionScheduler into just another executor. That is, we will add a new composite executor type that will be able to contain other executors. I may be wrong, but because of the new executor architecture in k6, and because we already have 95% of the required code in local.ExecutionScheduler, I think this will actually be very straightforward to do :tada: :smile: At least on the execution side, there will be some complexity elsewhere.

To illustrate, for example, if test1.js has vus: 20, duration: "30s" in its exported options, test2.js has iterations: 1000, vus: 20, and test3.js has something like this:

export let options = {
  execution: {
    constant_arr_rate: {
      type: "constant-arrival-rate",
      rate: 20,
      duration: "1m",
      preAllocatedVUs: 10,
    },
    per_vu_iters: {
      type: "per-vu-iterations",
      vus: 10,
      iterations: 100,
      maxDuration: "1m"
    },
  },
  // ...
}

then, when you run k6 run test1.js test2.js test3.js, it should be relatively easy to produce something like this as the final _derived_ options.execution value:

{
  "test1.js": {
    type: "composite",
    exec: "test1.js/default", // this is actually the tricky bit :D 
    options: {
      vus: 20,
      duration: "30s",
    },
  },
  "test2.js": {
    type: "composite",
    exec: "test2.js/default", 
    startAfter: "test1.js",
    options: {
      iterations: 1000,
      vus: 20,
    },
  },
  "test3.js": {
    type: "composite",
    exec: "test3.js/default", 
    startAfter: "test2.js",
    options: {
      execution: {
        constant_arr_rate: {
          type: "constant-arrival-rate",
          rate: 20,
          duration: "1m",
          preAllocatedVUs: 10,
        },
        per_vu_iters: {
          type: "per-vu-iterations",
          vus: 10,
          iterations: 100,
          maxDuration: "1m"
        },
      },
    },
  },
}

Notice how, with a few minor changes, we have a whole new _layer_ of flexibility - test suites, for lack of a better term. All the wile, we're preserving the nice new properties of being able to tell precisely how many VUs our test suite will use at most (20 in this case) and what the max test suite duration will be, giving us predictability and easy billing.

And, as I mention in the code comment above, the tricky bit isn't actually the execution. It would actually be things like figuring out how to reference the different default functions in the config. I don't think it's a huge issue, but we should be careful how we structure the archive bundles and the related things.

Also, because of the super-global k6 configuration (the curse of #883 strikes again :man_facepalming: ), I'm not sure how we should handle non-execution options that are different between scripts. We can probably refactor that at a later point though. And things like the planned new HTTP API (issue TBD :blush: ) would greatly reduce the issue... Still, some refactoring is likely required, even for a MVP version of the "full test suites 100% version"...

So, yeah, this seems reasonable, after we release #1007:
1) Add a small PR that implements startAfter, cover (hopefully) 80% of use cases
2) Add a medium-sized PR that implements the composite executor. No support for k6 run script1.js script2.js at this point, users have to manually configure the hierarchical config). Hopefully that covers 90% of use cases.
3) Figure out the tricky bits of implementing k6 run script1.js script2.js. This will give us great UX and hopefully 100% coverage of use cases. We can do this step much, much later than steps 1 and 2. Probably smart to do it after we have the new HTTP API.

[na--]

I'll give an example why I think points 1 and 2 above cover 80-90% of the use cases. Here's how, if we add the startAfter property and the composite executor, you can make an entry file test-suite.js that would allow you to run multiple independent tests:

import test01 from "./test01.js";
import test02 from "./test02.js";
import test03 from "./test03.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: "test01.default",
      options: test01.options,
    },
    "test2": {
      type: "composite",
      exec: "test01.default",
      options: test02.options,
      startAfter: "test1",
    },
    "test3": {
      type: "composite",
      exec: "test02.default",
      options: test02.options,
      startAfter: "test2",
    },
  }
}

You can execute any of the individual scripts like k6 run test01.js, and you can also execute all of them via test-suite.js. Of course, test-suite.js can be much more dynamic, with some environment variables and helper functions you can build much more flexible and complex execution pipelines, if that's required.

And you'd be able to do something similar even without the composite executor, though it will be a bit fiddly to construct the new options and you'd be restricted to only having a single executor in each sub-test.

[imiric]

I like this proposal, especially passing multiple scripts to k6 run. Being able to do k6 run tests/* would be great.

The idea for the composite executor makes sense, though I'm also worried about the complexity.

One improvement might be to pass actual functions to "exec" instead of string, so that one could do:

import * as test1 from "./test1.js";
import * as test2 from "./test2.js";
import * as test3 from "./test3.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: test1.default,
      options: test1.options,
    },
    ...

This seems intuitive and would avoid the parsing/matching unpleasantries if it were a string.

[na--]

The idea for the composite executor makes sense, though I'm also worried about the complexity.

I may be wrong, but I think this refactoring will actually reduce the complexity somewhat, at least when it comes to the script execution. We'll see when we come to it.

One improvement might be to pass actual functions to "exec" instead of string

Unfortunately, this probably wouldn't be possible. In the init context, k6 has to get the exported script options in a format that Go (and things like the cloud backend) understand. We can't just get a reference to a function in a specific goja VM (init context), because that reference won't be valid across the other goja VMs (VUs).

[MStoykov]

One improvement might be to pass actual functions to "exec" instead of a string

Unfortunately, this probably wouldn't be possible. In the init context, k6 has to get the exported script options in a format that Go (and things like the cloud backend) understand. We can't just get a reference to a function in a specific goja VM (init context), because that reference won't be valid across the other goja VMs (VUs).

While that is true I wonder if it is worth the trouble and (somewhat) inconsistency of having the go code figure out that test1.default is actually "test1.default" so that people can actually use the functions as are ... I also have a slight feeling that will be ... impossible but this depends on goja a lot, and I am not about to try to figure it out :). (also won't work with lambdas ... so maybe not such a great idea on my part).

In the same vein, I would like to point out that I am not certain we can just call test1.default if it isn't exported (as that exact name) by the actual main script ...
So there are some tricky stuff even with the "easy" part IMO, but let us leave it for when we actually can work on it.

[na--]

Thinking about an unrelated issue, I realized that my initial plan above has a major problem - startAfter wouldn't be as simple to implement as I thought :disappointed:

Most executors have a specified duration that they can't deviate from. For constant-looping-vus, variable-looping-vus, externally-controlled, constant-arrival-rate, and variable-arrival-rate, startAfter can be a simple shortcut option that will save users from the effort of manually adding up times to calculate startTime values.

But iteration-based executors, currently only shared-iterations and per-vu-iterations, don't have a constant execution duration, they only have a _maximum_ one. That is, they can finish early. So, any VU requirements of follow-up executors have to account for that. Which is problematic, since executors like the variable-looping-vus have very time-dependent VU requirements that aren't easy to calculate...

There are solutions to this, like reserving the max potentially required VUs for the overlapping periods, but they add significant complexity that probably isn't worth it at this point. Instead, I propose that we either initially implement startAfter as a simple shortcut option to automatically calculate the startTime value, or we don't implement it at all (i.e. rely only on startTime for the composite executor).

[na--]

Thinking some more, VU requirement calculations for a variable-looping-vus executor with a startAfter: "someIterationBasedExecutor" aren't a problem. They are complicated, but we already have the code to do them... :sweat_smile:

Essentially, the VU requirements for a variable-looping-vus executor with an uncertain start time can be calculated by just "pretending" its gracefulStop value is larger :tada: If, only for the purposes of GetExecutionRequirements(), we adjust its gracefulStop value exactly by how much the potential overlap with any previous executors is, we'll make sure that we always have sufficient VUs to run the executor, whenever it starts.

So, while I now have a good idea how to solve the startAfter issues, it adds sufficient complexity that I'd still advocate that we either start this issue without it, or to initially treat it as a startTime alias and do the optimizations later.

[Nicole1991]

I'll give an example why I think points 1 and 2 above cover 80-90% of the use cases. Here's how, if we add the startAfter property and the composite executor, you can make an entry file test-suite.js that would allow you to run multiple independent tests:

import test01 from "./test01.js";
import test02 from "./test02.js";
import test03 from "./test03.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: "test01.default",
      options: test01.options,
    },
    "test2": {
      type: "composite",
      exec: "test01.default",
      options: test02.options,
      startAfter: "test1",
    },
    "test3": {
      type: "composite",
      exec: "test02.default",
      options: test02.options,
      startAfter: "test2",
    },
  }
}

You can execute any of the individual scripts like k6 run test01.js, and you can also execute all of them via test-suite.js. Of course, test-suite.js can be much more dynamic, with some environment variables and helper functions you can build much more flexible and complex execution pipelines, if that's required.

And you'd be able to do something similar even without the composite executor, though it will be a bit fiddly to construct the new options and you'd be restricted to only having a single executor in each sub-test.

Hi, I'm wondering how can I run test-suite.js with k6 run test-suite.js?
I tried to define test-suite.js as below:

import test01 from "./test01.js";
import test02 from "./test02.js";
import test03 from "./test03.js";

export let options = {
  execution: {
    "test1": {
      type: "composite",
      exec: "test01.default",
      options: test01.options,
    },
   ...
  }
}

export default function() {
  test1.default();
  test2.default();
...
}

but it's not works as expected, could you please help me? Or can you give an example about how to run test1 & test2 & test3 together in test-suite.js?
Thanks.

[na--]

@Nicole1991, this is an _open_ issue that discusses ways to implement test suites. Neither type: "composite", startAfter: "X" nor anything like them has been implemented yet.

The only way you can currently implement something like a test suite in k6 is if you manually calculate and specify the startTime property of each scenario. Depending on your use case, it might be easier to simply have a shell script that calls k6 run scriptN.js sequentially for your scripts.

[Nicole1991]

@Nicole1991, this is an _open_ issue that discusses ways to implement test suites. Neither type: "composite", startAfter: "X" nor anything like them has been implemented yet.

The only way you can currently implement something like a test suite in k6 is if you manually calculate and specify the startTime property of each scenario. Depending on your use case, it might be easier to simply have a shell script that calls k6 run scriptN.js sequentially for your scripts.

Got it.
Thanks for your help.

[daniel-kun]

I'm also facing this issue, I want to run multiple tests sequentially after each other so that testing one API does not influence the results of the other API. scenarios with startAfter: "someOtherExecutorName" would totally solve my issue!
Or, alternatively, scenarios could optionally be an array instead of an object, which automatically runs the scenarios sequentially in the provided order.