Kops: After passing additional user-data instance won't start

Created on 16 Feb 2018 · 22Comments · Source: kubernetes/kops

What kops version are you running? The command kops version, will display
this information.

$ kops version
Version 1.8.1

What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-09T21:51:06Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

What cloud provider are you using?
AWS
What commands did you run? What is the simplest way to reproduce this issue?
I tried passing in additional user data to a instance group by following this doc
This is how my ig manifest looks like

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-01-30T19:38:43Z
  labels:
    kops.k8s.io/cluster: test.mycluster.xyz.com
  name: master-us-east-1b
spec:
  additionalUserData:
  - name: myscript.sh
    type: text/x-shellscript
    content: |
      #!/bin/sh
      echo "Hello World.  The time is now $(date -R)!" | tee /root/output.txt
  image: 595879546273/CoreOS-stable-1576.5.0-hvm
  machineType: t2.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-east-1b
  role: Master
  subnets:
  - us-east-1b

What happened after the commands executed?
When I update the cluster, it does launch the new instance. However, the instance stays in initializing state. I'm not able to ssh into the instance either. In system logs from AWS console getting following:
```[ 3.241788] systemd[1]: ignition-disks.service: Main process exited, code=exited, status=1/FAILURE
[ 3.247631] ignition[393]: no config URL provided
1: C
^
invalid character 'C' looking for beginning of value
````

lifecyclrotten

Source

aikram24

All 22 comments

When I'm passing debain image instance comes back fine. So, I think the problem is with coreos image not with kops.

aikram24 on 20 Feb 2018

/cc @KashifSaadat @gambol99

Adding some folks that know CoreOS

chrislovecnm on 20 Feb 2018

I don't believe this is supported for CoreOS I'm afraid: https://github.com/coreos/bugs/issues/912

You should be able to get around this by use of FileAssets & Hooks.

KashifSaadat on 23 Feb 2018

Since the release of CoreOS 1367.5.0, cloudinit was migrated to ignition. additionalUserData should be in Container Linux Config transplier yaml format which in turn, gets consumed by ignition during loading of initramfs.

You can see it in the output log which states "systemd: failed start Ignition".

Take a look at https://github.com/coreos/container-linux-config-transpiler/blob/master/doc/configuration.md for details.

layer3switch on 23 Feb 2018

Unfortunately, additionalUserData is completely unusable with CoreOS, because ignition and/or coreos-cloudinit do not support the multipart mime user data format, which is the format that Kops ends up using when additionalUserData is specified.

kops hooks are the correct workaround - assuming you can get them to work, given how limited they are. (You're going to append ".service" to the end of my .swap unit file? Really?)

geekofalltrades on 23 Mar 2018

😕1

kops hooks and file assets seem like a poor fit for managing /usr/share/oem/grub.cfg, or more generally finding a good way to set audit=1 on the kernel command line, or other kernel command line parameters. Is something else intended?

joeldeaguero on 27 Apr 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 26 Jul 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 25 Aug 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 24 Sep 2018

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot on 24 Sep 2018

Did anyone figure out what to do with this? Or is the solution for now the fileAssets and hooks?

sean-abbott on 26 Sep 2018

/reopen

eedgar on 2 Nov 2018

@eedgar: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

k8s-ci-robot on 2 Nov 2018

I am having this issue. I want to set the ntp time and install ssm on the bastion where hooks don't work either.

Is there a way to set a raw type that doesn't do the mime wrapping on coreos?

eedgar on 2 Nov 2018

/reopen

mikesplain on 2 Nov 2018

@mikesplain: Reopening this issue.

In response to this:

/reopen

k8s-ci-robot on 2 Nov 2018

I have the same issue when installing ssm on the coreos instances with additionalUserData. Some nodes, for instance, two masters, have started successfully, but most of them were restarting themselves constantly.

ziwon on 5 Nov 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 5 Dec 2018

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close