Aks-engine: CoreOs Support

[visitpay]

Is this a request for help?:

No

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

What version of aks-engine?:

v0.31.1

Used the standard template from https://github.com/Azure/aks-engine/blob/master/examples/coreos/kubernetes-coreos.json

{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "kubernetesConfig": { "networkPlugin": "kubenet" } }, "masterProfile": { "count": 1, "dnsPrefix": "{DNS_PREFIX}", "vmSize": "{K8_CLUSTER_MACHINESIZE}", "distro": "coreos" }, "agentPoolProfiles": [ { "name": "agentpool1", "count": 3, "vmSize": "{K8_CLUSTER_MACHINESIZE}", "availabilityProfile": "AvailabilitySet", "distro": "coreos" } ], "linuxProfile": { "adminUsername": "azureuser", "ssh": { "publicKeys": [ { "keyData": "{K8_CLUSTER_PUBLICKEY}" } ] } }, "servicePrincipalProfile": { "clientId": "{K8_Client_ID}", "secret": "{K8_Client_Secret}" } } }

I replace the things with curly brackets with values. Machine size: "Standard_B2ms"

Kubernetes version:
Ive tried 1.13 and 1.10

What happened:
Fails when executing the cse-agent-x and cse-master-x.

Deployment failed. Correlation ID: f370801c-da9e-4871-91f8-5cb54bdbf725. { "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'.", "details": [ { "code": "VMExtensionProvisioningError", "message": "VM has reported a failure when processing extension 'cse-agent-0'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\nti meout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to r un command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': N o such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or dir ectory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: f ailed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run comma nd 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such f ile or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\n timeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or d irectory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run com mand 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory \ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\ntimeout: failed to run command 'nc': No such file or directory\n\"." } ] } }

What you expected to happen:
I would expect a cluster to be built and not error.

How to reproduce it (as minimally and precisely as possible):
Use config from above, generate the arm template:
aks-engine generate configfromabove.json -o "_output/testdeploy/"
then execute it:
(must az login)
az group deployment create \ --name "MyDeploy" --resource-group "yourRG" \ --template-file "_output/testdeploy/azuredeploy.json" \ --parameters "_output/testdeploy/azuredeploy.parameters.json"

Anything else we need to know:
I dont think so.

Looks to be the same issue as:
https://github.com/Azure/aks-engine/issues/575

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[CecileRobertMichon]

Hi @visitpay. Thanks for reporting. CoreOS support was added as an experimental feature and is unfortunately not well tested/maintained at the moment. Was this apimodel previously working for you (ie. is it a recent regression)? If not, let's work together to get coreos support across the finish line and add documentation + tests for long term maintainability.

[visitpay]

@CecileRobertMichon I just recently tried it, it's never worked for me. So I cant comment if it's a recent regression or not. I definitely saw that it's experimental...

As for figuring out how to fix it, I'd be interested, however I'm not going to have time to start looking too deeply into this until mid next month.

[dwhanger]

@CecileRobertMichon I reported the same issue in #575 which was closed in favor of this one. I would be willing to pitch in and help....I will need to get a couple of things in place first though. I was simply trying to run the kubernetes-coreos.json example previously.

[alexeldeib]

I took a stab at this, not super pretty but I have a mostly-functional CoreOS Kubernetes cluster. Scribbled some notes in a gist if anyone wants to join the effort.

[DavidHache]

I will track the same issue as I run into it as well.

This is my Json file:

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.12",
      "kubernetesConfig": {
        "networkPlugin": "kubenet",
        "privateCluster": {
          "enabled": true
          }
        }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "",
      "vmSize": "Standard_DS2_v2",
      "distro": "coreos"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool1",
        "count": 2,
        "vmSize": "Standard_DS2_v2",
        "distro": "coreos"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": ""
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    }
  }
}

This is my SH to run it.

##!/usr/bin/env bash
echo "Getting latest version of AKS engine."
AKS_ENGINE="$(curl https://github.com/Azure/aks-engine/releases/latest | grep -o -e 'v[0-9]\.[0-9][0-9]*\.[0-9][0-9]*')"
apk add --no-cache -u ca-certificates
curl -LsS "https://github.com/Azure/aks-engine/releases/download/${AKS_ENGINE}/aks-engine-${AKS_ENGINE}-linux-amd64.tar.gz" | tar -zx

set -e

source $(dirname "$0")/login.sh
login

CLUSTER_DEFINITION="$(dirname "$0")/../../concourse_create_liferay_infra_aks_engine/tasks/aks-engine/djd_aks_engine.json"

FIRSTCONSECUTIVESTATICIP="$(echo ${SUBNET_CIDR}| sed 's/0\/[0-9][0-9]*/20/g')"

#### Modified code from : https://github.com/Azure/aks-engine/blob/master/examples/vnet/k8s-vnet-predeploy.sh
tempfile="$(mktemp)"
trap "rm -rf \"${tempfile}\"" EXIT

jq ".properties.masterProfile.vnetSubnetId = \"/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP_SUBNET}/providers/Microsoft.Network/virtualNetworks/${VNET}/subnets/${SUBNET}\"" ${CLUSTER_DEFINITION} > $tempfile && mv $tempfile ${CLUSTER_DEFINITION}
# Adding FistConsecutiveStaticIP
jq ".properties.masterProfile.firstConsecutiveStaticIP = \"${FIRSTCONSECUTIVESTATICIP}\"" ${CLUSTER_DEFINITION} > $tempfile && mv $tempfile ${CLUSTER_DEFINITION}

# Adding vnerCidr

jq ".properties.masterProfile.vnetCidr = \"${VNET_CIDR}\"" ${CLUSTER_DEFINITION} > $tempfile && mv $tempfile ${CLUSTER_DEFINITION}


indx=0
for poolname in `jq -r '.properties.agentPoolProfiles[].name' "${CLUSTER_DEFINITION}"`; do
  jq ".properties.agentPoolProfiles[$indx].vnetSubnetId = \"/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP_SUBNET}/providers/Microsoft.Network/virtualNetworks/${VNET}/subnets/${SUBNET}\"" ${CLUSTER_DEFINITION} > $tempfile && mv $tempfile ${CLUSTER_DEFINITION}
  indx=$((indx+1))
done

aks-engine-${AKS_ENGINE}-linux-amd64/aks-engine deploy \
    --subscription-id $SUBSCRIPTION_ID \
    --client-id $SERVICE_PRINCIPAL \
    --client-secret $SERVICE_SECRET \
    --dns-prefix liferay-test \
    --resource-group liferay_non_persistant \
    --location canadaeast \
    --api-model $CLUSTER_DEFINITION \
    --set servicePrincipalProfile.clientId=$SERVICE_PRINCIPAL \
    --set servicePrincipalProfile.secret=$SERVICE_SECRET



# Router fix from modified from: https://github.com/Azure/aks-engine/blob/master/examples/vnet/k8s-vnet-postdeploy.sh
rt=$(az network route-table list -g ${RESOURCE_GROUP_SUBNET} | jq -r '.[].id')
az network vnet subnet update -n ${SUBNET} -g ${RESOURCE_GROUP_SUBNET} --vnet-name ${VNET} --route-table $rt

I will investigate more into the code of aks-engine with you guys.

[alexeldeib]

@DavidHache if you are interested in making this work, check out some of the changes I made over here which helped get a functional cluster. Let's do it 🤓

[alexeldeib]

You may want to use make dev with the provided Makefile, it will provide
a ready to go Docker image, volume mount and exec into bash in the
container for you. Since you are already mounting a docker image, it will
likely simplify the process. If you're on windows that of course means you
need make, I installed it via scoop (package manager) but you can also
grab it from e.g. http://gnuwin32.sourceforge.net/packages/make.htm

Looks like you are hitting GOPATH issues. Try this: docker run -v /c/Users/dave/alex-aks-engine:/go/src/github.com/DavidHache/aks-engine -it --rm golang bash

Go is a bit finnicky about source code locations; there are some recent
developments (go modules) which break this pattern but it is not widespread
yet. Typically go code must be inside your GOPATH under the full repository
name. In the case of the docker golang images the GOPATH is /go/ and so
source code goes in /go/src/, installed binaries go in /go/bin/ (which
should be on your path), and 3rd party code ends up in /go/pkg (if only
that were the whole story). go will also look in the repo directory under
./vendor/ for 3rd party code.

Ace

On Fri, Mar 29, 2019 at 11:48 AM David Haché [e-mail]
wrote:

hi @alexeldeib https://github.com/alexeldeib , I got your repo and am
trying to build it but I get a weird message. (I am not familiar with go)
docker run -v /c/Users/dave/alex-aks-engine:/alex-aks-engine -it --rm
golang bash

cd /alex-aks-engine
make

go get -u github.com/golang/dep/cmd/dep
go get -u github.com/mitchellh/gox
go install ./vendor/github.com/go-bindata/go-bindata/...
vendor/github.com/go-bindata/go-bindata/go-bindata/main.go:15:2: cannot find package "github.com/go-bindata/go-bindata" in any of:
/usr/local/go/src/github.com/go-bindata/go-bindata (from $GOROOT)
/go/src/github.com/go-bindata/go-bindata (from $GOPATH)
Makefile:158: recipe for target 'bootstrap' failed
make: * [bootstrap] Error 1

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/aks-engine/issues/584#issuecomment-478109654,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGfF2TPYsco0HHva8bTlDtkB19TW1J4_ks5vbl_0gaJpZM4bS4UE
.