Aws-parallelcluster: seven_zip error

[mukulagrawal78]

I am using cfn cluster 1.4.2. Cookbook version in 1.4.1.
It was working all good until yesterday. Today I am seeing cfn-init started throwing following error:

Cookbook 'seven_zip' version '3.1.0' depends on chef version [">= 13.0"], but the running chef version is 12.19.36

Latest version of chef for ubuntu 16.04 is actually 12.19.36. There is no 13.0 version available.

Any idea what is going on?

[direwolf79]

Same problem here. I currently have a SLURM cluster limping along with a single compute node, and multiple failing nodes.

[breezescreamer]

Same issue here with version 1.5.x of cfncluster and cookbook

[direwolf79]

Does anyone know how to force a full chef/cfn-init run on a running compute node? I grabbed the 14.x latest and updated one of my running hosts that has the issue, but am not sure what to kick off. In theory it would work, and we should just need to use an AMI with a proper chef version. Alternatively I would suppose they could simply call an earlier 7zip cookbook that does not require Chef >= 13.

[tkuzuhara]

FWIW we had all our clusters die a miserable slow death yesterday due to this, while auto-scaling groups continued to scale up until reaching the maximum number of nodes, where each compute node had failed to initialize due to the seven_zip dependency. We did an emergency update to v1.6.1 (which appears to run chef 14.2.0) and finally got clusters working again late into the evening.

I would be very interested in knowing if there is a way to eliminate having these dependencies needing to be installed on the fly for every node that comes up. It increases the time to get a node spun up, and can result in disasters like this. I'm wondering if it's possible to bake these packages into the AMI so that we aren't dependent on package availability that can be outside of our control.

[sean-smith]

Hi all,

We're working on a fix. The issue is seven_zip ~> 3.0.0, dropped support for chef 12. We'll release a patched version of the cookbook that you can use in the older versions of cfncluster.

[SunPj]

Updating to 1.6.1 resolved the issue

[breezescreamer]

1.6.1 does not appear to resolve the issue on centos 7 systems

[sean-smith]

Between cfncluster 1.5.1 and 1.5.4 we updated chef to version 14. This should only effect customers using cfncluster <= 1.5.1

[clintval]

I'm hesitant to upgrade (I am using cfncluster <= 1.5.1) for fear of making other breaking changes to our clusters. Will a patch be eminent or is the best recommendation be to upgrade?

[sean-smith]

I'm working on a patch, it'll be up shortly. Keep in mind you'll have to run a cfncluster update with the latest cookbook to fix it.

[clintval]

Thanks! I ended up biting the bullet and trying out v1.6.1. So far so good on an Amazon Linux AMI.

[pro100frol]

I can approve that upgrading to the v1.6.1 helped. But if you use your own AMI - you'll need to upgrade it from here:

"ap-northeast-1": {
"alinux": "ami-0775ab7ad13c0e74f",
"centos6": "ami-0f676dc2bebea5b1a",
"centos7": "ami-0b13eb5fa94691483",
"ubuntu1404": "ami-05b67b05deba61e8c",
"ubuntu1604": "ami-0232ea8f3f82712a2"
},
"ap-northeast-2": {
"alinux": "ami-09fdf91b4a93730f0",
"centos6": "ami-0adf39100d009ad31",
"centos7": "ami-015bf65572df2d94d",
"ubuntu1404": "ami-06aa825c33309fdfd",
"ubuntu1604": "ami-08d64fa4a89f85427"
},
"ap-northeast-3": {
"alinux": "ami-03a0e4ad407dd7f55",
"centos6": "ami-0da12c87d2aee0ef5",
"centos7": "ami-0479721fa287d3cf7",
"ubuntu1404": "ami-0c03a23da1606fa3a",
"ubuntu1604": "ami-0cfdcf1603db9a672"
},
"ap-south-1": {
"alinux": "ami-008d194619d177765",
"centos6": "ami-05a8a28e5c8b74642",
"centos7": "ami-087bda89547e089dc",
"ubuntu1404": "ami-096a418446cd1eeb1",
"ubuntu1604": "ami-009f971485befe66a"
},
"ap-southeast-1": {
"alinux": "ami-08eef0c7d23753d49",
"centos6": "ami-02ce4e46ca873697e",
"centos7": "ami-0d339c926f9d35ba3",
"ubuntu1404": "ami-0d413135984f7bd97",
"ubuntu1604": "ami-02ab2d97bea65a826"
},
"ap-southeast-2": {
"alinux": "ami-08d6ef8690d843ceb",
"centos6": "ami-013201a7fda637e83",
"centos7": "ami-0ac15e60210a10310",
"ubuntu1404": "ami-079737fc9c635c09a",
"ubuntu1604": "ami-0fc63c095a0c31f0a"
},
"ca-central-1": {
"alinux": "ami-00f519fdeb5feb438",
"centos6": "ami-0ee6c95ad927b2da4",
"centos7": "ami-068d6a6fc5e7987f3",
"ubuntu1404": "ami-00cec9c492f236616",
"ubuntu1604": "ami-0705c0b15666e7d60"
},
"eu-central-1": {
"alinux": "ami-07fa07aba1f6fc571",
"centos6": "ami-0de3eb16b173ea6b3",
"centos7": "ami-0f73d5d5bb22f7609",
"ubuntu1404": "ami-0d6242ced07309e7b",
"ubuntu1604": "ami-08ab664e40e331729"
},
"eu-west-1": {
"alinux": "ami-09cbb5230da5b6cc2",
"centos6": "ami-0c8eca7deb52169cc",
"centos7": "ami-05479870b31fa535e",
"ubuntu1404": "ami-06581a7f68c539440",
"ubuntu1604": "ami-096183a7e31fc4e35"
},
"eu-west-2": {
"alinux": "ami-006cd2956238a6dd7",
"centos6": "ami-013d7f40fc791f845",
"centos7": "ami-08c5c6f990f8253e1",
"ubuntu1404": "ami-00621b5f273783bce",
"ubuntu1604": "ami-0a4fbb1dd660013b5"
},
"eu-west-3": {
"alinux": "ami-0fce6e16d9970d71b",
"centos6": "ami-0cac2d11252d18c27",
"centos7": "ami-0d6c4b2dde29e341f",
"ubuntu1404": "ami-07da72e5a66a93c71",
"ubuntu1604": "ami-0fcf3de0f1913a190"
},
"sa-east-1": {
"alinux": "ami-0165d34de1afd1618",
"centos6": "ami-074dbf929dc0c8de0",
"centos7": "ami-00cfa447d16329380",
"ubuntu1404": "ami-0f4a298763a012d8e",
"ubuntu1604": "ami-06fa4a172ebe11a10"
},
"us-east-1": {
"alinux": "ami-0fd8b1245d1435bf3",
"centos6": "ami-044fd430f1fd3b163",
"centos7": "ami-0fb2dca992af734f2",
"ubuntu1404": "ami-07df96c1d8e62161d",
"ubuntu1604": "ami-0b9750a5856891754"
},
"us-east-2": {
"alinux": "ami-0cb3d04ae4b4ca8db",
"centos6": "ami-03c508534d6f54859",
"centos7": "ami-03421fa006b964767",
"ubuntu1404": "ami-0f7c7f69478093925",
"ubuntu1604": "ami-0a2c6075e91d8bbc3"
},
"us-gov-west-1": {
"alinux": "ami-e0fd6781",
"ubuntu1404": "ami-38f36959",
"ubuntu1604": "ami-3bf3695a"
},
"us-west-1": {
"alinux": "ami-02f24f766b5b7502f",
"centos6": "ami-05ee27a886e16ba39",
"centos7": "ami-012c5239be2beb7f1",
"ubuntu1404": "ami-077e3b1efa9443476",
"ubuntu1604": "ami-00b0c33ebc9576bf8"
},
"us-west-2": {
"alinux": "ami-01e7f12fc989e7004",
"centos6": "ami-08675896633a1d9fb",
"centos7": "ami-0053bef6747ca58bf",
"ubuntu1404": "ami-0581577b81f865ae9",
"ubuntu1604": "ami-00e5e4cc71cf05d6e"

[mukulagrawal78]

FWIW we had all our clusters die a miserable slow death yesterday due to this, while auto-scaling groups continued to scale up until reaching the maximum number of nodes, where each compute node had failed to initialize due to the seven_zip dependency. We did an emergency update to v1.6.1 (which appears to run chef 14.2.0) and finally got clusters working again late into the evening.

I would be very interested in knowing if there is a way to eliminate having these dependencies needing to be installed on the fly for every node that comes up. It increases the time to get a node spun up, and can result in disasters like this. I'm wondering if it's possible to bake these packages into the AMI so that we aren't dependent on package availability that can be outside of our control.

I also need reccomendation for doing this. We really need to make setup/installation absolutely static with no update checks and live downloading of libraries. Any reccomendation on how to do that?

[sean-smith]

Hi,

We've fixed the issue.

This bug effects versions of cfncluster < 1.5.2. We've published patched cookbooks for the following versions:

cfncluster 1.5.1
cfncluster 1.4.2
cfncluster 1.4.1
cfncluster 1.4.0

Clusters running with these versions will automatically get the latest cookbook when new compute nodes are launched. No action is needed to apply the patch.

For those interested, the updated code is in these PR's:
1.5.1 https://github.com/awslabs/cfncluster-cookbook/pull/203
1.4.1 https://github.com/awslabs/cfncluster-cookbook/pull/204
1.4.0 https://github.com/awslabs/cfncluster-cookbook/pull/205

While we recommend everyone to update to the latest version 1.6.1, we realize that the upgrade process is painful and this isn't an option for some people.

Please note that versions of cfncluster < 1.4.0 will not have the patched code.