Web-frameworks: Display frameworks version on benchmarks

[waghanza]

Hi,

I think it'll be much appreciable / accurate to display version of frameworks in use.

Some version could lead to completely different results, due to the awesome (:heart:) done by _contributors_.

To specify this, I think a good idea is :

• To fix version in dependency files (package.json, Gemfile, requirements.txt ...)
• To add version (fixed) in frameworks list.

For me this way is very strict, but ensure that result are running specifically a version (without a patch that could break everything).

What do you think ?

Related to #242

Regards,

[aichholzer]

Showing the version, I believe, is a must for any benchmark since different versions can make big differences and not providing this information might confuse people if and when they run the tests for themselves.

[waghanza]

Sure, the question is : _how_ ?

As @OvermindDL1 saied in #242, it can be done by a shell script, but for me it SHOULD be hard-coded (I mean written in a file by hand even if it's now of two places)

[aichholzer]

@waghanza -Have a look at this implementation. I wrote that to address a similar problem. In that case, the test will run against all packages defined in that file and use whatever properties each item has to include that in the results.

The second part:

Object.keys(packages).forEach((pkg) => {
  if (!packages[pkg].version) {
    const version = dependencies[pkg] || dependencies[packages[pkg].package] || ''
    packages[pkg].version = version.replace('^', '')
  }
  choices.push(pkg)
})

reads the version of each module from the global package.json file and assigns it to be used to render the results. This would work, since you would have a package-lock.json so you can lock the versions and when people run npm i they would have the same versions as locked.

I am sure you could come up with something similar to work across your wider board of frameworks. Let me know if you need any help to get the final implementation done.

[waghanza]

@aichholzer I see ... Very interesting approach then

However, this project has multiple languages, so not only package.json.

I think an approach like this will be very hard to maintain here

Any ideas are :heart:

[aichholzer]

Hencewhy i mentioned that a "similar approach to work across your wider range of frameworks" could be achieved.

[aichholzer]

Having that said, you can still rely on a "config.json" file (all languages can read .json) and you could even read it using bash script. This file would sort of act like the lock file where you define which version to use and run.

Another, perhaps more complicated approach, would be to use something like chef or puppet to orchestrate the build and the download of specific versions.

[waghanza]

or having an internal wrapper that read the package.json, Gemfile ....

[OvermindDL1]

That is why I keep saying use a shell script, because it can acquire the hard-coded dependency versions for whatever language/framework it is using and return it in a standard way. :-)

@aichholzer Actually I don't think Rust or C++'s build systems (which is where the versions are defined) can read json (Rust reads TOML, C++'s reads, well, custom turing-complete horror). Nor can puppet or chef read the dependency setup for C++ either (unsure about rust).

[waghanza]

so probably hard-code the version number on dependency file AND framework list ;-)

[OvermindDL1]

Sounds easy to get out of sync. You know what they say about duplication... ;-)

[waghanza]

yeah sure, but the only durable solution ...

after, their is some dependency update solution to help doing so

[waghanza]

Hi,

We deal about display version in benchmark results, which is a valuable information.

However, there is a question of what version is ?

Most implementation here use semver, so X.Y.Z, but is this very interesting to display Z in results ?

I mean using ruby (2.4) instead of ruby (2.4.3) seems to be clearer for me ?

What do you think ?

Regards,

[doanguyen]

Hi,

@waghanza omitting z seems reasonable since only LTS is worth to perform the benchmark, so I support.