Nixpkgs: nixos/tests: Leftover tasks towards unified Python integration tests

[tfc]

Thanks to https://github.com/NixOS/nixpkgs/pull/71684 being merged recently, we do now have a test driver implementation in Python that allows for writing the imperative part of NixOS integration tests in Python instead of Perl.

Thanks @flokli @edolstra @Lassulus @adisbladis @domenkozar @garbas @jonringer for the support and @blitz @jtraue @marijanp for helping me porting the remaining Perl tests to Python.

In order to finish this project, we are going to perform the following tasks over the next weeks/months:

• [x] Perl and Python Test driver feature-parity
  
  
  
  • [x] implement dumpTTYContents #72943
  
  
  • [x] implement forwardPort #72943
  
  
  • [x] implement getWindowNames #72835
  
  
  • [x] implement requireActiveUnit #72835
  
  
  • [x] implement waitForWindow
  
  
  • [x] implement waitUntilTTYMatches
  
  
  • [x] fix emptyDiskImages behaviour #73559
  
  
  • [ ] Collect kernel coverage data after test finished (as per https://github.com/NixOS/nixpkgs/pull/72354#discussion_r451031449), removal PR #97231
  
  
• [ ] Document all Python functions with docstrings
• [ ] Remove all global state/singletons from Python test driver (e.g. global log variable)
• [x] Port all remaining Perl tests
  
  
  
  • [x] cockroachdb #73934 #80047
  
  
  • [x] docker-containers broken: port #80049 (was renamed to oci-containers in the meantime and fixed)
  
  
  • [x] docker-preloader broken: #74143, port #80051
  
  
  • [x] docker-tools broken: #75081, port #80048
  
  
  • [x] ec2 #79696
  
  
  • [x] hardened #76708
  
  
  • [x] krb5/deprecated-config #80130
  
  
  • [x] krb5/example-config #80130
  
  
  • [x] kubernetes/*
  
  
  • [x] openstack-image #79696
  
  
  • [x] partition (test deleted in #87084)
  
  
  • [x] redmine (fixed: #80061) #85747
  
  
  • [x] rsyslogd #80096
  
  
  • [x] run-in-machine #80099
  
  
  • [x] systemd-confinement #80103
  
  
  • [x] taskserver #93413
  
  
  • [x] virtualbox #94858
  
  
  • [x] os-prober https://github.com/NixOS/nixpkgs/issues/72828#issuecomment-605658186
  
  
  • [x] acme
  
  
  • [x] ammonite
  
  
  • [x] atd #72833
  
  
  • [x] automysqlbackup
  
  
  • [x] avahi #72833
  
  
  • [x] babeld #72833 #72834
  
  
  • [x] bcachefs #72833
  
  
  • [x] beanstalkd #72833
  
  
  • [x] bees #74001
  
  
  • [x] bind #72833
  
  
  • [x] bittorrent
  
  
  • [x] blivet (broken, see #33496)
  
  
  • [x] boot-stage1 #72833
  
  
  • [x] borgbackup #72833
  
  
  • [x] buildbot #78556
  
  
  • [x] caddy #72935
  
  
  • [x] cadvisor #72935
  
  
  • [x] cassandra#72897
  
  
  • [x] ceph-multi-node #73190
  
  
  • [x] ceph-single-node #73190
  
  
  • [x] certmgr #72935
  
  
  • [x] cfssl #72935
  
  
  • [x] chromium #79352
  
  
  • [x] cjdns #72935
  
  
  • [x] clickhouse #73131
  
  
  • [x] cloud-init #72935
  
  
  • [x] codimd #74036
  
  
  • [x] colord #72935
  
  
  • [x] containers-bridge #74218
  
  
  • [x] containers-ephemeral #74196
  
  
  • [x] containers-extra_veth#74761
  
  
  • [x] containers-hosts #74196
  
  
  • [x] containers-imperative #74218
  
  
  • [x] containers-ipv4 #74218
  
  
  • [x] containers-ipv6 #74218
  
  
  • [x] containers-macvlans #74761
  
  
  • [x] containers-physical_interfaces #74761
  
  
  • [x] containers-portforward #74761
  
  
  • [x] containers-reloadable #74196
  
  
  • [x] containers-restart_networking #74761
  
  
  • [x] containers-tmpfs #74196
  
  
  • [x] couchdb #72935
  
  
  • [x] deluge #73131
  
  
  • [x] dhparams #75084
  
  
  • [x] dnscrypt-proxy #72935
  
  
  • [x] docker-edge
  
  
  • [x] docker-registry #74033
  
  
  • [x] docker-tools-overlay #75084
  
  
  • [x] docker #72935
  
  
  • [x] documize #72935
  
  
  • [x] dovecot #74004
  
  
  • [x] ecryptfs #75084
  
  
  • [x] emacs-daemon
  
  
  • [x] env #75084
  
  
  • [x] etcd-cluster #74005
  
  
  • [x] etcd #74005
  
  
  • [x] fancontrol #73131
  
  
  • [x] ferm #74513
  
  
  • [x] firefox #72835
  
  
  • [x] firewall #73131
  
  
  • [x] fish #73131
  
  
  • [x] flannel (broken #74941, PR #85252)
  
  
  • [x] flatpak-builder
  
  
  • [x] flatpak
  
  
  • [x] fluentd #73131
  
  
  • [x] fontconfig-default-fonts
  
  
  • [x] fsck
  
  
  • [x] fwupd
  
  
  • [x] gdk-pixbuf
  
  
  • [x] gitlab #73939
  
  
  • [x] gitolite #74063
  
  
  • [x] gjs
  
  
  • [x] glib-networking
  
  
  • [x] glusterfs #74003
  
  
  • [x] gnome-photos
  
  
  • [x] gnome3-xorg https://github.com/NixOS/nixpkgs/pull/73938
  
  
  • [x] gnome3 #74943
  
  
  • [x] gocd-agent #74082
  
  
  • [x] gocd-server #74082
  
  
  • [x] google-oslogin/default #74002
  
  
  • [x] gotify-server #74034
  
  
  • [x] grafana #72904
  
  
  • [x] graphene
  
  
  • [x] graphite #76562 (python-twisted dependencies fail already in Perl)
  
  
  • [x] graylog #74040
  
  
  • [x] hadoop/hdfs
  
  
  • [x] hadoop/yarn
  
  
  • [x] haka #76707
  
  
  • [x] handbrake #73131
  
  
  • [x] haproxy #75459 #75695
  
  
  • [x] hibernate #73131
  
  
  • [x] hitch/default
  
  
  • [x] home-assistant #74126
  
  
  • [x] hound #73131
  
  
  • [x] hydra/default
  
  
  • [x] i3wm #74938
  
  
  • [x] icingaweb2 #73131
  
  
  • [x] iftop
  
  
  • [x] ihatemoney #78556
  
  
  • [x] incron #73131
  
  
  • [x] influxdb #74069
  
  
  • [x] initrd-network-ssh/default #72904
  
  
  • [x] initrd-network #75695
  
  
  • [x] installer#78670
  
  
  • [x] ipv6 #74758
  
  
  • [x] jackett #73131
  
  
  • [x] jellyfin
  
  
  • [x] jenkins #74074
  
  
  • [x] kerberos/heimdal #73961
  
  
  • [x] kerberos/mit #73961
  
  
  • [x] kernel-latest
  
  
  • [x] kernel-lts
  
  
  • [x] kernel-testing
  
  
  • [x] kexec #76560 (hangs forever already in Perl)
  
  
  • [x] keymap #78556
  
  
  • [x] knot #72834
  
  
  • [x] ldap #74851
  
  
  • [x] leaps #75695
  
  
  • [x] libgdata
  
  
  • [x] libxmlb
  
  
  • [x] lidarr #75695
  
  
  • [x] lightdm #73135
  
  
  • [x] limesurvey (attempted in #75695, broke with python error on decoding incomplete utf-8 string), reattempt in #78555
  
  
  • [x] login
  
  
  • [x] loki #72904
  
  
  • [x] magnetico
  
  
  • [x] mailcatcher #75695
  
  
  • [x] matrix-synapse #72835
  
  
  • [x] mediawiki
  
  
  • [x] memcached #74312
  
  
  • [x] metabase #72857
  
  
  • [x] minidlna #73131
  
  
  • [x] miniflux #73131
  
  
  • [x] minio #74070
  
  
  • [x] misc #79064
  
  
  • [x] mongodb #73933
  
  
  • [x] moodle #72887
  
  
  • [x] morty #72887
  
  
  • [x] mosquitto #73827
  
  
  • [x] mpd #73935
  
  
  • [x] mumble #76410
  
  
  • [x] munin #74935
  
  
  • [x] mutable-users #75703
  
  
  • [x] mxisd #75703
  
  
  • [x] mysql-backup #73992
  
  
  • [x] mysql-replication #73992
  
  
  • [x] mysql #73992
  
  
  • [x] nat #74754
  
  
  • [x] ndppd a9a271792d59aa15b6cc4bc5dfafde9fd9529815
  
  
  • [x] neo4j #73991
  
  
  • [x] nesting #75703
  
  
  • [x] netdata #74078
  
  
  • [x] networkingProxy#78239
  
  
  • [x] networking #75721
  
  
  • [x] nextcloud/basic
  
  
  • [x] nextcloud/with-mysql-and-memcached
  
  
  • [x] nextcloud/with-postgresql-and-redis
  
  
  • [x] nexus
  
  
  • [x] nfs #73989
  
  
  • [x] nghttpx #75703
  
  
  • [x] nginx-sso #74072
  
  
  • [x] nginx #74072
  
  
  • [x] nix-ssh-serve #73994
  
  
  • [x] nixos-generate-config
  
  
  • [x] novacomd #75703
  
  
  • [x] nsd #80065
  
  
  • [x] nzbget #75703
  
  
  • [x] openarena
  
  
  • [x] openldap #74851
  
  
  • [x] opensmtpd #72390
  
  
  • [x] openssh
  
  
  • [x] orangefs #75703
  
  
  • [x] osquery #74073, broken: #74081
  
  
  • [x] osrm-backend #75703
  
  
  • [x] ostree
  
  
  • [x] overlayfs #75703
  
  
  • [x] packagekit
  
  
  • [x] pam-oath-login #74898
  
  
  • [x] pam-u2f #75180
  
  
  • [x] pantheon #73140
  
  
  • [x] paperless #75701
  
  
  • [x] pdns-recursor #75701
  
  
  • [x] peerflix #75701
  
  
  • [x] pgjwt #72904
  
  
  • [x] pgmanage #75701
  
  
  • [x] php-pcre #75701
  
  
  • [x] plasma5 #73134
  
  
  • [x] plotinus #80067
  
  
  • [x] postgis #75701
  
  
  • [x] postgresql-wal-receiver #80060 #96347
  
  
  • [x] powerdns #73059
  
  
  • [x] pppd #73056
  
  
  • [x] printing #79327
  
  
  • [x] prometheus-exporters #72904
  
  
  • [x] prometheus #74055
  
  
  • [x] proxy #75462 #78555
  
  
  • [x] quagga #75701
  
  
  • [x] quake3
  
  
  • [x] rabbitmq #74123
  
  
  • [x] radarr #72887
  
  
  • [x] radicale #74764
  
  
  • [x] redis #72887
  
  
  • [x] riak #74124
  
  
  • [x] roundcube #72887 #73131
  
  
  • [x] rspamd #75464
  
  
  • [x] rss2email #72887
  
  
  • [x] rxe #73568
  
  
  • [x] samba #73080
  
  
  • [x] shiori
  
  
  • [x] signal-desktop #72887
  
  
  • [x] simple
  
  
  • [x] slurm #73179
  
  
  • [x] smokeping #73057
  
  
  • [x] snapper #72887
  
  
  • [x] solr #80063
  
  
  • [x] sonarr #75701
  
  
  • [x] strongswan-swanctl #72887
  
  
  • [x] sudo #74076
  
  
  • [x] switch-test #75701
  
  
  • [x] syncthing-init #74114
  
  
  • [x] syncthing-relay #74114
  
  
  • [x] systemd-networkd-wireguard #73699
  
  
  • [x] systemd-nspawn #73990
  
  
  • [x] systemd-timesyncd #75701
  
  
  • [x] systemd #78241
  
  
  • [x] telegraf #72887
  
  
  • [x] tiddlywiki #74866
  
  
  • [x] timezone #74858
  
  
  • [x] tinydns #73059
  
  
  • [x] tor
  
  
  • [x] trac #72857
  
  
  • [x] transmission
  
  
  • [x] trezord #72857
  
  
  • [x] trickster #72887
  
  
  • [x] udisks2 https://github.com/NixOS/nixpkgs/pull/72860
  
  
  • [x] upnp #72887
  
  
  • [x] uwsgi #74061
  
  
  • [x] vault
  
  
  • [x] wireguard/default
  
  
  • [x] wireguard/generated
  
  
  • [x] wireguard/namespaces #75695
  
  
  • [x] wordpress #73993
  
  
  • [x] xautolock #72887
  
  
  • [x] xdg-desktop-portal
  
  
  • [x] xfce4-14 #73138
  
  
  • [x] xfce #73138
  
  
  • [x] xmonad #73137
  
  
  • [x] xmpp/ejabberd #74254
  
  
  • [x] xmpp/prosody-mysql #74031
  
  
  • [x] xmpp/prosody #74031
  
  
  • [x] xrdp #74854
  
  
  • [x] xss-lock
  
  
  • [x] yabar #72887
  
  
  • [x] yggdrasil #73411
  
  
  • [x] zfs #78670
  
  
  • [x] zookeeper #72887
  
  
  • [x] beegfs (broken & removed, see #73998)
  
  
• broken, fix/discuss what to do:
  
  
  
  • [x] hocker-fetchdocker/default #95517, PR: #96076
  
  
  • [x] ipfs (broken, see #74000)
  
  
  • [x] mathics #80055, PR: #95505
  
  
  • [x] mesos #78557, PR: #95507
  
  
• [x] Deprecate perl test driver
• [x] Drop Perl test driver PR: #96396

Additional work that would be awesome:

• [x] Add type annotations
• [x] Perform mypy checks
• [x] Drop XML output, use Python-native logging (but it would be worthwhile to keep the nesting approach in the log) #96254
• [x] Make output more readable by adding colors PR: #96152

[flokli]

some tests updated in https://github.com/NixOS/nixpkgs/pull/72810, which was merged.
Updated the checkboxes above.

[prusnak]

cc @mmahut : jormungandr metabase trezord
cc @1000101: trickster

[worldofpeace]

See https://github.com/NixOS/nixpkgs/pull/72835#issuecomment-549972297.

With getWindowNames and waitForWindow I will begin porting the tests mentioned there.

[jtojnar]

It would be nice if the driver had type annotations and was checked with mypy.

[jonringer]

would also help with IDE experience, as it will warn you of type mismatches. At least pycharm will ;)

[worldofpeace]

Did we get an equivalent in the python driver for perl's mustFail?

Edit: it's fail.

[JohnAZoidberg]

Why is it called differently?

[tfc]

Why is it called differently?

# nixos/lib/test-driver/Machine.pm

sub mustFail {
    fail @_;
}

sub mustSucceed {
    succeed @_;
}

They are synonyms of each other, so i dropped the mustXXX counterparts.

[Mic92]

Mypy support for the driver has been added here https://github.com/NixOS/nixpkgs/pull/73033
However the tests itself are not type checked yet. What makes this difficult is also the fact that dynamic variables are magically added when tests are run.

[FRidh]

Packaged as a Python package and using Sphinx for docs in https://github.com/NixOS/nixpkgs/pull/73496.

[worldofpeace]

Having a hard time with my gnome3-xorg port because it fails on waiting for the first call to a user unit

73938.

error: unit "[email protected]" is inactive and there are no pending jobs.

[nh2]

The new Python test functions made it easy for me to add some tests for consul as well:

74006

:+1:

[rnhmjoj]

I fixed magnetico in #72012.

[kampka]

It's not checked in the list, but ndppd is already ported it seems.

[rnhmjoj]

networking is not listed but it's not ported and is also a quite large test.

[kampka]

@rnhmjoj The size of the tests is rarely an issue once you have a set of search and replace regex's handy. You spend most of the time trying to figure out why tests do not work, mostly it seems because they didn't work to begin with, as is the case with networking.

[rnhmjoj]

The size of the tests is rarely an issue once you have a set of search and replace regex's handy.

Ah, nice.

mostly it seems because they didn't work to begin with, as is the case with networking.

This is quite alarming. I've assumed people would regularly check the tests...

[nh2]

This is quite alarming. I've assumed people would regularly check the tests...

We should have automatic CI on them. Don't we have that already?

[jonringer]

on PRs, we definitely don't. Ofborg is able to perform a test by doing @GrahamOfBorg test <test>, but it's up to someone with ofborg permissions to call it, and know that the package change relates to that tests.

[tfc]

on PRs, we definitely don't. Ofborg is able to perform a test by doing @GrahamOfBorg test <test>, but it's up to someone with ofborg permissions to call it, and know that the package change relates to that tests.

When a package's derivation on a PR changes, then it would only transitively change the tests' derivations that it really affects, right? Couldn't ofborg automatically determine this and then rerun these tests? Or has this already been tried and there were problems that my naive assumption does not reflect?

[rnhmjoj]

It looks like the situation with networking.networkd.virtual is known. See the commit message of a3a441cd870e29ec63b41b3fbedb9ccd9c81a69a.

[NinjaTrappeur]

When a package's derivation on a PR changes, then it would only transitively change the tests' derivations that it really affects, right? Couldn't ofborg automatically determine this and then rerun these tests? Or has this already been tried and there were problems that my naive assumption does not reflect?

Summoning @Ekleog who worked on that in the past.

See https://github.com/NixOS/nixpkgs/pull/44439 for the RFC, https://github.com/NixOS/ofborg/pull/410 for the ofborg associated PR.

[jonringer]

could and does are two separate things. I don't have context on why the decisions were made, but I'm assuming it was a compromise in compute resources vs validity guarantees.

There may be some history in there as well, as nixos modules used to be a separate repo altogether

[7c6f434c]

I'm assuming it was a compromise in compute resources vs validity guarantees.

Initial version was run on weakly managed and non-exclusive-use infrastructure without a way to filter by features… So an alive human was expected to stand behind each build (and there are still timeouts)

Note that a change that touches gtk can force a chromium rebuild if you ask for all tests, which will timeout after consuming a lot of build resources. So the proposed rules sometimes tries to build more than the current deployment finds acceptable.

There may be some history in there as well, as nixos modules used to be a separate repo altogether

I think Nixpkgs-NixOS re-merge is older than ofBorg

[nh2]

on PRs, we definitely don't.

But what about post-merge on Hydra? The way I understand it is that the perl networking tests were also broken for a while. Is there anything that runs tests post-merge?

If yes, how would one get notified by a failure there?

[edolstra]

I'm not sure what makes people think the networking tests aren't executed. Most of them are, see nixos/release-combined.nix:

        (all nixos.tests.networking.scripted.loopback)
        (all nixos.tests.networking.scripted.static)
        (all nixos.tests.networking.scripted.dhcpSimple)
        (all nixos.tests.networking.scripted.dhcpOneIf)
        (all nixos.tests.networking.scripted.bond)
        (all nixos.tests.networking.scripted.bridge)
        (all nixos.tests.networking.scripted.macvlan)
        (all nixos.tests.networking.scripted.sit)
        (all nixos.tests.networking.scripted.vlan)

However some tests (e.g. virtual and privacy) are not listed there so they are not executed. Also we only test scripted networking, not networkd-based networking.

[jonringer]

My comments were in response to:

We should have automatic CI on them. Don't we have that already?

It's very possible right now to have a PR which affects networking to be merged without the tests being ran.

From my point of view, running the networking tests before a release falls more into the category of CD (continuous delivery); as the CI part is done as the code already been merged, and now you have to git bisect to see which change broke the related test.

[kampka]

jormungandr was removed in aa98348f880052547f4035c6ce1f5403e6a91d15

[kampka]

Out of curiosity, do you guys see any downside of not inlineing larger tests suits?
I think that placing the test code into a dedicated python file and then reading it into the test string can provide a better experience with regards to python specific content assist tools like auto-formatting, making the whole process of writing tests more pleasant over all.

[7c6f434c]

Out of curiosity, do you guys see any downside of not inlineing larger tests suits?

That sounds dependent on whether you need variable expansion inside the test code. But maybe functions could still be defined and then called

[flokli]

Me and @tfc spoke about refactoring larger parts, and making the test driver more something like a library (which could also use other virtualization/isolation platforms except qemu).
Currently, we have some back and forth between who's rendering what, and how the testScript is being passed etc, which is a bit spaghetti-esque.

I don't really want to touch that as long as the perl test driver is still around, but happy to tackle that afterwards.

[kampka]

That sounds dependent on whether you need variable expansion inside the test code. But maybe functions could still be defined and then called

Of course I don't want to generallize this, but in practice a lot of tests don't interpolate anything major, so might be worth considering.

[minijackson]

Would it be possible to "forward" attributes as variables in Python, like it is done in derivations? Would it be a good idea? IMHO you would get the best of both worlds ^^

[kampka]

Would it be possible to "forward" attributes as variables in Python, like it is done in derivations? Would it be a good idea? IMHO you would get the best of both worlds ^^

From the top of my head, this should be possible:

testScript = ''
   PYTHON_VAR=${nix_interpolation}

  ${lib.readFile ./testScript.py}
'';

and then use PYTHON_VAR in your testScript.py as if it were defined there.

[minijackson]

I meant in an automated way, as in modifying the testing infrastructure so that all attributes in the returned attribute set passed to make-test-python.nix are available as variable in the python script. But this could be a close enough temporary alternative

[nh2]

Would it be possible to "forward" attributes as variables in Python, like it is done in derivations? Would it be a good idea? IMHO you would get the best of both worlds ^^

My opinion: It's not so good because it makes things more implicit and less clear when variables appear out of thin air in the Python file.

It's a write-time optimisation at the cost of readability, and I consider readability more important.

I would rather recommend a stand-alone test to be a function that takes arguments (passing them in as a dict via a single parameter, e.g. env. is also fine, because it creates lexical clarity).

[Ekleog]

@tfc @NinjaTrappeur

When a package's derivation on a PR changes, then it would only transitively change the tests' derivations that it really affects, right? Couldn't ofborg automatically determine this and then rerun these tests? Or has this already been tried and there were problems that my naive assumption does not reflect?

I haven't checked the full context of the discussion, but I remember a friend (@symphorien, I think? I'm not 100% sure) tried to work on re-evaluating all tests to check which ones changed, and evaluation was way too long to be reasonably doable, hence my not trying this way and just adding manual passthru.tests :)

[symphorien]

https://github.com/madjar/nox/pull/89

[nh2]

Can anybody explain me what sendKeys did before in perl?

It has no Python equivalent, there is only send_key().

It did:

sub sendKeys {
    my ($self, @keys) = @_;
    foreach my $key (@keys) {
        $key = $charToKey{$key} if exists $charToKey{$key};
        $self->sendMonitorCommand("sendkey $key");
    }
}

sub sendChars {
    my ($self, $chars) = @_;
    $self->nest("sending keys ‘$chars’", sub {
        $self->sendKeys(split //, $chars);
    });
}

What does the split // do here?

[7c6f434c]

Is it used in any remaining tests? // looks like an empty regular expression, splitting by empty separator should probably convert a string to the list of component characters, and it would make sense to then send each character as a key (to type a command, for example)

[nh2]

Is it used in any remaining tests?

I don't think so.

splitting by empty separator should probably convert a string to the list of component characters, and it would make sense to then send each character as a key (to type a command, for example)

I don't get it though: Wouldn't the foreach my $key (@keys) by itself already do that?

[nh2]

Great progress so far, only 18 tests remaining without PR from the issue description (at the time of post, may checkmark them once new updates come in below):

• [x] gnome3-xorg
• [x] haka
• [x] hardened
• [ ] hocker-fetchdocker/default
• [ ] krb5/deprecated-config
• [ ] krb5/example-config
• [ ] mathics
• [ ] mesos
• [x] misc
• [x] networking-proxy
• [ ] nsd
• [ ] os-prober
• [ ] partition
• [ ] plotinus
• [ ] printing
• [ ] systemd-confinement
• [x] systemd
• [ ] taskserver

[worldofpeace]

gnome3-xorg has had a PR for a while https://github.com/NixOS/nixpkgs/pull/73938

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/steps-towards-even-more-pr-automation/5634/1

[aanderse]

Somehow solr never showed up on this list... :thinking:

[tfc]

Somehow solr never showed up on this list... 🤔

I found some more missing items, refreshed the list and moved all undone items to the top

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-20-03-feature-freeze/5655/1

[aszlig]

Drop Perl test driver

Maybe not drop, but deprecate it with a warning for 20.09 or something, because when you have a whole lot of Perl tests in CI, updating to 20.03 means rewriting all those tests...

[worldofpeace]

@aszlig :+1: to this. If nixosTest was an internal API this would be fine. But it's public and I don't think it's reasonable to have such a change to hand off one release.

[tfc]

Maybe not drop, but deprecate it with a warning for 20.09 or something, because when you have a whole lot of Perl tests in CI, updating to 20.03 means rewriting all those tests...

@aszlig yes, totally makes sense.

[worldofpeace]

https://github.com/NixOS/nixpkgs/pull/79064

[grahamc]

For what it is worth, I think it would be best to keep the Perl driver around for 1 release, since it is a pretty coollibrary and I'm aware of a bunch of users of that library who would probably appreciate having a few months of transition time.

[flokli]

@grahamc sure, no reason to nuke it for 20.03, some tests are still in Perl.

However, we should show a warning in 20.03 every time it's used, so people are aware it's deprecated and will be removed for 20.09. Some of the planned refactorings are too big to do in both test drivers, so we should be able to break the perl variant during the next cycle.

[worldofpeace]

I'm going to unpin this, since it's basically 80+% complete.

[mawis]

It seems there is no progress on this. What will happen with the remaining broken software?

[flokli]

I assume broken tests will be marked as broken (if they aren't already), and eventually removed.
I'm not sure if it makes sense to mechanically translate them to python - as we can't verify they still work

Currently working tests that are still in perl need to be changed to python, before we can remove the perl driver.

[symphorien]

Here is my attempt at porting the os-prober test to python. https://github.com/symphorien/nixpkgs/tree/os-prober-python
When I run it, qemu simply hangs. Does anyone have an idea ?

[tfc]

It seems there is no progress on this. What will happen with the remaining broken software?

The broken tests are a thing that indeed needs to be discussed - what shall be done with them?

The non-broken tests will eventually be ported and are not forgotten (i am discussing this with colleagues currently), but the urgency dropped a bit after the 20.03 branch-off.

[worldofpeace]

@tfc I thought of the same thing as @flokli as what to do with the broken tests. A test being broken and that can't be translated to the python driver isn't very useful. There's not many, and they will continue to exist in the git history, so we can just remove them if we have to. A final issue can be opened so people can be aware of this.

[domenkozar]

16 to go where most are broken :)

[blitz]

I'm going to take care of the VirtualBox test soonish, because we are using this internally and I'd want this to stay working. :)

[worldofpeace]

I'm thinking it's a good idea to have the perl driver removed on unstable very soon.

[jonringer]

agreed

[flokli]

Here is my attempt at porting the os-prober test to python. https://github.com/symphorien/nixpkgs/tree/os-prober-python
When I run it, qemu simply hangs. Does anyone have an idea ?

@symphorien I rebased your branch onto latest master, and got further until this:

machine: must succeed: nixos-rebuild boot >&2
machine # error: The option `virtualisation.qemu.consoles' defined in `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/testing/test-instrumentation.nix' does not exist.
machine # (use '--show-trace' to show detailed location information)
machine # building Nix...
machine # error: The option `virtualisation.qemu.consoles' defined in `/nix/var/nix/profiles/per-user/root/channels/nixos/nixos/modules/testing/test-instrumentation.nix' does not exist.

The machine doesn't hang anymore, and the failure also exists like this in master.

Can you take another look, and maybe open a PR to carry the discussion over there?

[symphorien]

Not in the coming days.

[blitz]

If someone has bandwidth, we could use some eyes on #94858 to remove quite a big piece of perl test code. :)

[flokli]

With all remaining tests ported over, I opened #96396 removing the (now unused) perl code.

Thanks a lot for everyone doing the porting! :-)

[worldofpeace]

Soo proud that we did this :sparkles: