Nixpkgs: Home-assistant package integration test failure

[Mindavi]

Describe the bug

Related to ZHF: #80379

It seems this test is broken because attribute sets in nix are sorted by default.

I've done some tests and found out that not specifying --preserve-key-order changes the order of the keys in the json and thus also breaks the conversion back to yaml, because homeassistant expects the yaml to be sorted.

$ nix-shell -p remarshal
$ cat tmp.yaml 
homeassistant:
  auth_providers:
    - type: legacy_api_password
      api_password: http_password

$ yaml2json tmp.yaml | json2yaml
homeassistant:
  auth_providers:
  - api_password: http_password
    type: legacy_api_password

$ yaml2json --preserve-key-order tmp.yaml | json2yaml --preserve-key-order
homeassistant:
  auth_providers:
  - type: legacy_api_password
    api_password: http_password

If the - type: ... is not the first key, this behavior breaks.
Because the builtins.toJSON builtin is used to create the first JSON and I already tested with setting --preserve-key-order, I think the nix attribute sets are ordered and that's causing this issue.

To Reproduce
Steps to reproduce the behavior:

1. Build a VM with the following file:

# vm.nix
{ pkgs, lib, config, ... }:
{
  services.home-assistant = {
    enable = true;
    config = {
      homeassistant = {
        name = "Home";
        time_zone = "UTC";
        latitude = "0.0";
        longitude = "0.0";
        elevation = 0;
        auth_providers = [
          {
            type = "legacy_api_password";
            api_password = "test1234";
          }
        ];
      };
    };
  };

  users.users.root.initialPassword = "root";
}

1. Run cat /var/lib/hass/configuration.yaml
2. See the type and api_password in the incorrect order.

Expected behavior

The original order as specified in the nix file is preserved, instead of ordering alphabetically (?).

Other information

Maybe #75584 will solve this or help fix this by allowing flags or something.
Possibly related to #81986.

Metadata

• system: "x86_64-linux"
• host os: Linux 5.5.7-hardened, NixOS, 20.09pre215947.82b54d49066 (Nightingale)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.3
• channels(rick): ""
• channels(root): "nixos-20.09pre215947.82b54d49066, nixpkgs"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

Ping @dotlambda

[matt-snider]

I also came across this since I'm trying to fix another bug in the home-assistant module. I can confirm that setting --preserve-key-order does not work, and that the file generated by builtins.toJSON already has incorrect ordering.

[Mic92]

Mhm. I am not quite sure how to solve this.

[Mic92]

Is this only a problem for auth_providers or are other parts also affected?

[Infinisil]

From the YAML spec:

3.2.2.1. Keys Order

In the representation model, mapping keys do not have an order. To serialize a mapping, it is necessary to impose an ordering on its keys. This order is a serialization detail and should not be used when composing the representation graph (and hence for the preservation of application data). In every case where node order is significant, a sequence must be used. For example, an ordered mapping can be represented as a sequence of mappings, where each mapping is a single key: value pair. YAML provides convenient compact notation for this case.

So this is a problem with home-assistant not being compliant with the YAML spec. This should ideally be fixed upstream.

[Mic92]

I reported this upstream as well: https://github.com/home-assistant/core/issues/33466

[Mindavi]

Ok, so probably it hasn't got anything to do with the ordering of the keys. I've tried it in docker just now (I don't know why I didn't think about that earlier...), but in both cases I get 'Unauthorized' responses. I've made a bug report against home-assistant to request more information, because I'm not sure why it's not working as expected. Issue is linked in.

[Mindavi]

I've created a minimal 'config' directory which can be included in the home-assistant test (but not sure how to do so).
config.zip

Credentials for the accounts:

username: testuser
password: testing
long-lived access token (valid for 10yrs from yesterday): eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiIxOTgxYzBiMDUyY2M0NGY5YmZkMWJiMTE3ZDAwMTYzMCIsImlhdCI6MTU4NTg2MDM1NSwiZXhwIjoxOTAxMjIwMzU1fQ.wDx84NmvlfJjyCE9mK70Njx-1luavPECcPk6mU-a1xM

Instead of the password argument in hass-cli, the token argument should be used with the given token. Not sure about curl, probably the Authentication: Bearer header shall be used.

[Mic92]

Thanks for the research. If we would not need the token, we could just copy the onboarding file from .storage:

{
    "data": {
        "done": [
            "user",
            "core_config",
            "integration"
        ]
    },
    "key": "onboarding",
    "version": 3
}

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[Mindavi]

Still important to me, but unsure how to solve this. When I find the time and feel like it I'll have an IRC discussion or something like this.