Kops: Kops does not create EC2 instances

Created on 11 Oct 2017 · 16Comments · Source: kubernetes/kops

What kops version are you running? use kops version

Version 1.7.0 (git-e04c29d)

What Kubernetes version are you running? use kubectl version

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Unable to connect to the server: dial tcp 203.0.113.123:443: i/o timeout

What cloud provider are you using?

aws

What commands did you execute (Please provide cluster manifest kops get --name my.example.com, if available) and what happened after commands executed?

kops create cluster \
    --zones eu-central-1a,eu-central-1b,eu-central-1c \
    --node-count=2 \
    --node-size=t2.small \
    --master-size=t2.small \
    ${NAME}

NAME            CLOUD   ZONES
kubernetes.bloxy.ru aws eu-central-1a,eu-central-1b,eu-central-1c

Instance Groups
NAME            ROLE    MACHINETYPE MIN MAX SUBNETS
master-eu-central-1a    Master  t2.small    1   1   eu-central-1a
nodes           Node    t2.small    2   2   eu-central-1a,eu-central-1b,eu-central-1c

What you expected to happen:

Amazon should create EC2 instances, volumes, DNS records and secrets.
And it creates everything. But not EC2 instances! So i can't connect to the cluster and setup anything.

How can we to reproduce it (as minimally and precisely as possible):

Try to setup kops as in tutorial

Source

lastonoga

Most helpful comment

@lastonoga In your aws console, goto the autoscaling groups:https://console.aws.amazon.com/ec2/autoscaling/home. Select the autoscaling groups created by kops, then look in activity history to see if there are any details.

mikesplain on 11 Oct 2017

👍2

All 16 comments

Hi @lastonoga, have you tried a larger master-size? Quite a few containers need to run so I would suggest using a larger instance like an m3.medium & m4.large for master.

mikesplain on 11 Oct 2017

Also are instances launching in your AWS account? Can you ssh into the instances?

mikesplain on 11 Oct 2017

@mikesplain I tried with any types of instances, anyway they don't appear.
There is no instances. Kops does not create them, but it creates everything else like volumes, dns records

lastonoga on 11 Oct 2017

@lastonoga can you add -v=10 to your create command and provide the output?

mikesplain on 11 Oct 2017

@mikesplain

I1011 18:03:30.835940   47617 s3context.go:159] Found bucket "kubernetes-example" in region "eu-central-1"
I1011 18:03:30.836034   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/config"
I1011 18:03:31.217172   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/instancegroup/"
I1011 18:03:31.293522   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/instancegroup: [s3://kubernetes-example/kubernetes.example.ru/instancegroup/master-eu-central-1a s3://kubernetes-example/kubernetes.example.ru/instancegroup/nodes]
I1011 18:03:31.293568   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/instancegroup/master-eu-central-1a"
I1011 18:03:31.385055   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/instancegroup/nodes"
I1011 18:03:31.517402   47617 channel.go:91] resolving "stable" against default channel location "https://raw.githubusercontent.com/kubernetes/kops/master/channels/"
I1011 18:03:31.517435   47617 channel.go:96] Loading channel from "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable"
I1011 18:03:31.517450   47617 context.go:132] Performing HTTP request: GET https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable
I1011 18:03:32.066655   47617 channel.go:105] Channel contents: spec:
  images:
    # We put the "legacy" version first, for kops versions that don't support versions ( < 1.5.0 )
    - name: kope.io/k8s-1.4-debian-jessie-amd64-hvm-ebs-2017-07-28
      providerID: aws
      kubernetesVersion: ">=1.4.0 <1.5.0"
    - name: kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-07-28
      providerID: aws
      kubernetesVersion: ">=1.5.0 <1.6.0"
    - name: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-07-28
      providerID: aws
      kubernetesVersion: ">=1.6.0 <1.7.0"
    - name: kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28
      providerID: aws
      kubernetesVersion: ">=1.7.0"
    - providerID: gce
      name: "cos-cloud/cos-stable-60-9592-90-0"
  cluster:
    kubernetesVersion: v1.5.7
    networking:
      kubenet: {}
  kubernetesVersions:
  - range: ">=1.7.0"
    recommendedVersion: 1.7.4
    requiredVersion: 1.7.0
  - range: ">=1.6.0"
    recommendedVersion: 1.6.7
    requiredVersion: 1.6.0
  - range: ">=1.5.0"
    recommendedVersion: 1.5.7
    requiredVersion: 1.5.1
  - range: "<1.5.0"
    recommendedVersion: 1.4.12
    requiredVersion: 1.4.2
  kopsVersions:
  - range: ">=1.7.0-alpha.1"
    recommendedVersion: 1.7.0
    #requiredVersion: 1.7.0
    kubernetesVersion: 1.7.4
  - range: ">=1.6.0-alpha.1"
    #recommendedVersion: 1.6.0
    #requiredVersion: 1.6.0
    kubernetesVersion: 1.6.7
  - range: ">=1.5.0-alpha1"
    recommendedVersion: 1.5.1
    #requiredVersion: 1.5.1
    kubernetesVersion: 1.5.7
  - range: "<1.5.0"
    recommendedVersion: 1.4.4
    #requiredVersion: 1.4.4
    kubernetesVersion: 1.4.12
I1011 18:03:32.067898   47617 populate_cluster_spec.go:369] Defaulted KubeControllerManager.ClusterCIDR to 100.96.0.0/11
I1011 18:03:32.067911   47617 populate_cluster_spec.go:376] Defaulted ServiceClusterIPRange to 100.64.0.0/13
I1011 18:03:32.091461   47617 aws_utils.go:38] Querying EC2 for all valid regions
I1011 18:03:33.073994   47617 aws_cloud.go:637] Querying EC2 for all valid zones in region "eu-central-1"
I1011 18:03:33.074119   47617 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I1011 18:03:33.400612   47617 subnets.go:48] All subnets have CIDRs; skipping asssignment logic
I1011 18:03:33.400663   47617 aws_cloud.go:637] Querying EC2 for all valid zones in region "eu-central-1"
I1011 18:03:33.400769   47617 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I1011 18:03:33.471788   47617 utils.go:121] Querying for all DNS zones to find match for "kubernetes.example.ru"
I1011 18:03:33.501780   47617 route53.go:50] AWS request: route53 ListHostedZones
I1011 18:03:34.312023   47617 populate_cluster_spec.go:249] Defaulting DNS zone to: Z3CEXJAAQ67LHN
I1011 18:03:34.312052   47617 tagbuilder.go:102] tags: [_aws _k8s_1_6 _networking_kubenet]
I1011 18:03:34.320980   47617 tree_walker.go:97] visit "config/_aws"
I1011 18:03:34.321054   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/_aws"
I1011 18:03:34.321072   47617 tree_walker.go:97] visit "config/_aws/defaults.options"
I1011 18:03:34.321201   47617 tree_walker.go:97] visit "config/_gce"
I1011 18:03:34.321217   47617 tree_walker.go:120] Skipping directory "config/_gce" as tag "_gce" not present
I1011 18:03:34.321225   47617 tree_walker.go:97] visit "config/components"
I1011 18:03:34.321236   47617 tree_walker.go:97] visit "config/components/docker"
I1011 18:03:34.321251   47617 tree_walker.go:97] visit "config/components/docker/_e2e_storage_test_environment"
I1011 18:03:34.321264   47617 tree_walker.go:120] Skipping directory "config/components/docker/_e2e_storage_test_environment" as tag "_e2e_storage_test_environment" not present
I1011 18:03:34.321277   47617 tree_walker.go:97] visit "config/components/docker/_networking_cni"
I1011 18:03:34.321289   47617 tree_walker.go:120] Skipping directory "config/components/docker/_networking_cni" as tag "_networking_cni" not present
I1011 18:03:34.321296   47617 tree_walker.go:97] visit "config/components/docker/_networking_kubenet"
I1011 18:03:34.321307   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/components/docker/_networking_kubenet"
I1011 18:03:34.321319   47617 tree_walker.go:97] visit "config/components/docker/_networking_kubenet/kubenet.options"
I1011 18:03:34.321388   47617 tree_walker.go:97] visit "config/components/docker/docker.options"
I1011 18:03:34.321462   47617 tree_walker.go:97] visit "config/components/kube-apiserver"
I1011 18:03:34.321486   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_vsphere"
I1011 18:03:34.321499   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_vsphere" as tag "_vsphere" not present
I1011 18:03:34.321506   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_aws"
I1011 18:03:34.321516   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_aws"
I1011 18:03:34.321525   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_aws/kube-apiserver.aws.options"
I1011 18:03:34.321573   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_gce"
I1011 18:03:34.321588   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_gce" as tag "_gce" not present
I1011 18:03:34.321596   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_4"
I1011 18:03:34.321607   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_4" as tag "_k8s_1_4" not present
I1011 18:03:34.321615   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_6"
I1011 18:03:34.321624   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_k8s_1_6"
I1011 18:03:34.321633   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_6/kube-apiserver.options"
I1011 18:03:34.321718   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_3"
I1011 18:03:34.321736   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_3" as tag "_k8s_1_3" not present
I1011 18:03:34.321743   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5"
I1011 18:03:34.321755   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_5" as tag "_k8s_1_5" not present
I1011 18:03:34.321762   47617 tree_walker.go:97] visit "config/components/kube-apiserver/kube-apiserver.options"
I1011 18:03:34.322017   47617 options_loader.go:101] executing template components/docker/docker.options (tags=[])
I1011 18:03:34.322130   47617 options_loader.go:101] executing template components/kube-apiserver/kube-apiserver.options (tags=[])
I1011 18:03:34.329173   47617 options_loader.go:101] executing template _aws/defaults.options (tags=[_aws])
I1011 18:03:34.329238   47617 options_loader.go:101] executing template components/docker/_networking_kubenet/kubenet.options (tags=[_networking_kubenet])
I1011 18:03:34.329277   47617 options_loader.go:101] executing template components/kube-apiserver/_aws/kube-apiserver.aws.options (tags=[_aws])
I1011 18:03:34.329328   47617 options_loader.go:101] executing template components/kube-apiserver/_k8s_1_6/kube-apiserver.options (tags=[_k8s_1_6])
I1011 18:03:34.329398   47617 options_loader.go:128] executing builder *components.DefaultsOptionsBuilder
I1011 18:03:34.329405   47617 options_loader.go:128] executing builder *components.KubeAPIServerOptionsBuilder
I1011 18:03:34.329421   47617 apiserver.go:100] Enabling apiserver insecure port, for healthchecks (issue #43784)
I1011 18:03:34.329425   47617 options_loader.go:128] executing builder *components.DockerOptionsBuilder
I1011 18:03:34.329438   47617 options_loader.go:128] executing builder *components.NetworkingOptionsBuilder
I1011 18:03:34.329446   47617 options_loader.go:128] executing builder *components.KubeDnsOptionsBuilder
I1011 18:03:34.329458   47617 options_loader.go:128] executing builder *components.KubeletOptionsBuilder
I1011 18:03:34.329475   47617 kubelet.go:139] Cloud Provider: aws
I1011 18:03:34.329485   47617 options_loader.go:128] executing builder *components.KubeControllerManagerOptionsBuilder
I1011 18:03:34.329493   47617 kubecontrollermanager.go:74] Kubernetes version "1.7.4" supports AttachDetachReconcileSyncPeriod; will configure
I1011 18:03:34.329504   47617 kubecontrollermanager.go:79] AttachDetachReconcileSyncPeriod is not set; will set to default 1m0s
I1011 18:03:34.329513   47617 options_loader.go:128] executing builder *components.KubeSchedulerOptionsBuilder
I1011 18:03:34.329518   47617 options_loader.go:128] executing builder *components.KubeProxyOptionsBuilder
I1011 18:03:34.336065   47617 options_loader.go:101] executing template components/docker/docker.options (tags=[])
I1011 18:03:34.336168   47617 options_loader.go:101] executing template components/kube-apiserver/kube-apiserver.options (tags=[])
I1011 18:03:34.336294   47617 options_loader.go:101] executing template _aws/defaults.options (tags=[_aws])
I1011 18:03:34.336337   47617 options_loader.go:101] executing template components/docker/_networking_kubenet/kubenet.options (tags=[_networking_kubenet])
I1011 18:03:34.336379   47617 options_loader.go:101] executing template components/kube-apiserver/_aws/kube-apiserver.aws.options (tags=[_aws])
I1011 18:03:34.336429   47617 options_loader.go:101] executing template components/kube-apiserver/_k8s_1_6/kube-apiserver.options (tags=[_k8s_1_6])
I1011 18:03:34.336512   47617 options_loader.go:128] executing builder *components.DefaultsOptionsBuilder
I1011 18:03:34.336523   47617 options_loader.go:128] executing builder *components.KubeAPIServerOptionsBuilder
I1011 18:03:34.336530   47617 apiserver.go:100] Enabling apiserver insecure port, for healthchecks (issue #43784)
I1011 18:03:34.336534   47617 options_loader.go:128] executing builder *components.DockerOptionsBuilder
I1011 18:03:34.336543   47617 options_loader.go:128] executing builder *components.NetworkingOptionsBuilder
I1011 18:03:34.336550   47617 options_loader.go:128] executing builder *components.KubeDnsOptionsBuilder
I1011 18:03:34.336558   47617 options_loader.go:128] executing builder *components.KubeletOptionsBuilder
I1011 18:03:34.336567   47617 kubelet.go:139] Cloud Provider: aws
I1011 18:03:34.336575   47617 options_loader.go:128] executing builder *components.KubeControllerManagerOptionsBuilder
I1011 18:03:34.336583   47617 kubecontrollermanager.go:74] Kubernetes version "1.7.4" supports AttachDetachReconcileSyncPeriod; will configure
I1011 18:03:34.336593   47617 options_loader.go:128] executing builder *components.KubeSchedulerOptionsBuilder
I1011 18:03:34.336597   47617 options_loader.go:128] executing builder *components.KubeProxyOptionsBuilder
I1011 18:03:34.341254   47617 spec_builder.go:68] options: {
  "channel": "stable",
  "configBase": "s3://kubernetes-example/kubernetes.example.ru",
  "cloudProvider": "aws",
  "kubernetesVersion": "1.7.4",
  "subnets": [
    {
      "name": "eu-central-1a",
      "zone": "eu-central-1a",
      "cidr": "172.20.32.0/19",
      "type": "Public"
    },
    {
      "name": "eu-central-1b",
      "zone": "eu-central-1b",
      "cidr": "172.20.64.0/19",
      "type": "Public"
    },
    {
      "name": "eu-central-1c",
      "zone": "eu-central-1c",
      "cidr": "172.20.96.0/19",
      "type": "Public"
    }
  ],
  "masterPublicName": "api.kubernetes.example.ru",
  "masterInternalName": "api.internal.kubernetes.example.ru",
  "networkCIDR": "172.20.0.0/16",
  "topology": {
    "masters": "public",
    "nodes": "public",
    "dns": {
      "type": "Public"
    }
  },
  "secretStore": "s3://kubernetes-example/kubernetes.example.ru/secrets",
  "keyStore": "s3://kubernetes-example/kubernetes.example.ru/pki",
  "configStore": "s3://kubernetes-example/kubernetes.example.ru",
  "dnsZone": "Z3CEXJAAQ67LHN",
  "clusterDNSDomain": "cluster.local",
  "serviceClusterIPRange": "100.64.0.0/13",
  "nonMasqueradeCIDR": "100.64.0.0/10",
  "sshAccess": [
    "0.0.0.0/0"
  ],
  "kubernetesApiAccess": [
    "0.0.0.0/0"
  ],
  "etcdClusters": [
    {
      "name": "main",
      "etcdMembers": [
        {
          "name": "a",
          "instanceGroup": "master-eu-central-1a"
        }
      ]
    },
    {
      "name": "events",
      "etcdMembers": [
        {
          "name": "a",
          "instanceGroup": "master-eu-central-1a"
        }
      ]
    }
  ],
  "docker": {
    "bridge": "",
    "logLevel": "warn",
    "ipTables": false,
    "ipMasq": false,
    "logDriver": "json-file",
    "logOpt": [
      "max-size=10m",
      "max-file=5"
    ],
    "storage": "overlay,aufs",
    "version": "1.12.6"
  },
  "kubeDNS": {
    "image": "gcr.io/google_containers/kubedns-amd64:1.3",
    "replicas": 2,
    "domain": "cluster.local",
    "serverIP": "100.64.0.10"
  },
  "kubeAPIServer": {
    "image": "gcr.io/google_containers/kube-apiserver:v1.7.4",
    "logLevel": 2,
    "cloudProvider": "aws",
    "securePort": 443,
    "insecurePort": 8080,
    "address": "127.0.0.1",
    "etcdServers": [
      "http://127.0.0.1:4001"
    ],
    "etcdServersOverrides": [
      "/events#http://127.0.0.1:4002"
    ],
    "admissionControl": [
      "NamespaceLifecycle",
      "LimitRanger",
      "ServiceAccount",
      "PersistentVolumeLabel",
      "DefaultStorageClass",
      "DefaultTolerationSeconds",
      "ResourceQuota"
    ],
    "serviceClusterIPRange": "100.64.0.0/13",
    "allowPrivileged": true,
    "apiServerCount": 1,
    "anonymousAuth": false,
    "kubeletPreferredAddressTypes": [
      "InternalIP",
      "Hostname",
      "ExternalIP"
    ],
    "storageBackend": "etcd2",
    "authorizationMode": "AlwaysAllow"
  },
  "kubeControllerManager": {
    "logLevel": 2,
    "image": "gcr.io/google_containers/kube-controller-manager:v1.7.4",
    "cloudProvider": "aws",
    "clusterName": "kubernetes.example.ru",
    "clusterCIDR": "100.96.0.0/11",
    "allocateNodeCIDRs": true,
    "configureCloudRoutes": true,
    "leaderElection": {
      "leaderElect": true
    },
    "attachDetachReconcileSyncPeriod": "1m0s",
    "useServiceAccountCredentials": true
  },
  "kubeScheduler": {
    "logLevel": 2,
    "image": "gcr.io/google_containers/kube-scheduler:v1.7.4",
    "leaderElection": {
      "leaderElect": true
    }
  },
  "kubeProxy": {
    "image": "gcr.io/google_containers/kube-proxy:v1.7.4",
    "cpuRequest": "100m",
    "logLevel": 2,
    "clusterCIDR": "100.96.0.0/11",
    "hostnameOverride": "@aws"
  },
  "kubelet": {
    "kubeconfigPath": "/var/lib/kubelet/kubeconfig",
    "requireKubeconfig": true,
    "logLevel": 2,
    "podManifestPath": "/etc/kubernetes/manifests",
    "hostnameOverride": "@aws",
    "podInfraContainerImage": "gcr.io/google_containers/pause-amd64:3.0",
    "allowPrivileged": true,
    "enableDebuggingHandlers": true,
    "clusterDomain": "cluster.local",
    "clusterDNS": "100.64.0.10",
    "networkPluginName": "kubenet",
    "cloudProvider": "aws",
    "cgroupRoot": "/",
    "nonMasqueradeCIDR": "100.64.0.0/10",
    "networkPluginMTU": 9001,
    "evictionHard": "memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"
  },
  "masterKubelet": {
    "kubeconfigPath": "/var/lib/kubelet/kubeconfig",
    "requireKubeconfig": true,
    "logLevel": 2,
    "podManifestPath": "/etc/kubernetes/manifests",
    "hostnameOverride": "@aws",
    "podInfraContainerImage": "gcr.io/google_containers/pause-amd64:3.0",
    "allowPrivileged": true,
    "enableDebuggingHandlers": true,
    "clusterDomain": "cluster.local",
    "clusterDNS": "100.64.0.10",
    "networkPluginName": "kubenet",
    "cloudProvider": "aws",
    "cgroupRoot": "/",
    "registerSchedulable": false,
    "nonMasqueradeCIDR": "100.64.0.0/10",
    "networkPluginMTU": 9001,
    "evictionHard": "memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"
  },
  "networking": {
    "kubenet": {}
  },
  "api": {
    "dns": {}
  },
  "authorization": {
    "alwaysAllow": {}
  }
}
I1011 18:03:34.342426   47617 channel.go:156] RecommendedVersion="1.7.0", Have="1.7.0".  No upgrade needed.
I1011 18:03:34.342488   47617 channel.go:184] VersionRecommendationSpec does not specify RequiredVersion
I1011 18:03:34.342499   47617 channel.go:136] RecommendedVersion="1.7.4", Have="1.7.4".  No upgrade needed.
I1011 18:03:34.342507   47617 channel.go:176] RequiredVersion="1.7.0", Have="1.7.4".  No upgrade needed.
I1011 18:03:34.342571   47617 apply_cluster.go:221] Adding default kubelet release asset: https://storage.googleapis.com/kubernetes-release/release/v1.7.4/bin/linux/amd64/kubelet
I1011 18:03:34.342581   47617 context.go:132] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.7.4/bin/linux/amd64/kubelet.sha1
I1011 18:03:34.537115   47617 apply_cluster.go:803] Found hash "7bf3fda43bb8d0a55622ca68dcbfaf3cc7f2dddc" for "https://storage.googleapis.com/kubernetes-release/release/v1.7.4/bin/linux/amd64/kubelet"
I1011 18:03:34.537531   47617 apply_cluster.go:232] Adding default kubectl release asset: https://storage.googleapis.com/kubernetes-release/release/v1.7.4/bin/linux/amd64/kubectl
I1011 18:03:34.537551   47617 context.go:132] Performing HTTP request: GET https://storage.googleapis.com/kubernetes-release/release/v1.7.4/bin/linux/amd64/kubectl.sha1
I1011 18:03:34.583075   47617 apply_cluster.go:803] Found hash "819010a7a028b165f5e6df37b1bb7713ff6d070f" for "https://storage.googleapis.com/kubernetes-release/release/v1.7.4/bin/linux/amd64/kubectl"
I1011 18:03:34.583126   47617 networking.go:120] Adding default CNI asset: https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz
I1011 18:03:34.583138   47617 urls.go:40] Using default base url: "https://kubeupv2.s3.amazonaws.com/kops/1.7.0/"
I1011 18:03:34.583153   47617 apply_cluster.go:258] Using default utils.tar.gz location: "https://kubeupv2.s3.amazonaws.com/kops/1.7.0/linux/amd64/utils.tar.gz"
I1011 18:03:34.583166   47617 context.go:132] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.7.0/linux/amd64/utils.tar.gz.sha1
I1011 18:03:35.206568   47617 apply_cluster.go:803] Found hash "5d95d64d7134f202ba60b1fa14adaff138905d15" for "https://kubeupv2.s3.amazonaws.com/kops/1.7.0/linux/amd64/utils.tar.gz"
I1011 18:03:35.206612   47617 urls.go:64] Using default nodeup location: "https://kubeupv2.s3.amazonaws.com/kops/1.7.0/linux/amd64/nodeup"
I1011 18:03:35.206636   47617 aws_cloud.go:637] Querying EC2 for all valid zones in region "eu-central-1"
I1011 18:03:35.206775   47617 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I1011 18:03:35.275871   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/ssh/public/admin/"
I1011 18:03:35.351805   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/ssh/public/admin: [s3://kubernetes-example/kubernetes.example.ru/pki/ssh/public/admin/76a1ab596eb55144ca36882187267d5a]
I1011 18:03:35.351828   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/ssh/public/admin/76a1ab596eb55144ca36882187267d5a"
I1011 18:03:35.441045   47617 route53.go:50] AWS request: route53 ListHostedZones
I1011 18:03:35.604550   47617 dns.go:101] Doing DNS lookup to verify NS records for "kubernetes.example.ru"
I1011 18:03:35.616393   47617 dns.go:118] Found NS records for "kubernetes.example.ru": [ns-1331.awsdns-38.org. ns-1801.awsdns-33.co.uk. ns-414.awsdns-51.com. ns-641.awsdns-16.net.]
I1011 18:03:35.616457   47617 tagbuilder.go:102] tags: [_aws _k8s_1_6 _networking_kubenet]
I1011 18:03:35.616600   47617 templates.go:87] loading resource "addons/core.addons.k8s.io/v1.4.0.yaml"
I1011 18:03:35.616647   47617 templates.go:87] loading resource "addons/core.addons.k8s.io/addon.yaml"
I1011 18:03:35.625802   47617 templates.go:79] loading (templated) resource "addons/kube-dns.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:35.625983   47617 templates.go:79] loading (templated) resource "addons/kube-dns.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:35.626022   47617 templates.go:87] loading resource "addons/limit-range.addons.k8s.io/addon.yaml"
I1011 18:03:35.626053   47617 templates.go:87] loading resource "addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I1011 18:03:35.626655   47617 templates.go:79] loading (templated) resource "addons/networking.projectcalico.org.canal/pre-k8s-1.6.yaml"
I1011 18:03:35.626803   47617 templates.go:79] loading (templated) resource "addons/networking.projectcalico.org.canal/k8s-1.6.yaml"
I1011 18:03:35.626885   47617 templates.go:79] loading (templated) resource "addons/networking.weave/k8s-1.6.yaml"
I1011 18:03:35.626955   47617 templates.go:79] loading (templated) resource "addons/networking.weave/pre-k8s-1.6.yaml"
I1011 18:03:35.627224   47617 templates.go:87] loading resource "addons/authentication.kope.io/k8s-1.6.yaml"
I1011 18:03:35.627296   47617 templates.go:79] loading (templated) resource "addons/dns-controller.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:35.627341   47617 templates.go:79] loading (templated) resource "addons/dns-controller.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:35.627631   47617 templates.go:87] loading resource "addons/networking.kope.io/k8s-1.6.yaml"
I1011 18:03:35.627688   47617 templates.go:87] loading resource "addons/networking.kope.io/pre-k8s-1.6.yaml"
I1011 18:03:35.627727   47617 templates.go:87] loading resource "addons/storage-aws.addons.k8s.io/v1.6.0.yaml"
I1011 18:03:35.637295   47617 templates.go:79] loading (templated) resource "addons/external-dns.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:35.637722   47617 templates.go:79] loading (templated) resource "addons/external-dns.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:35.638400   47617 templates.go:79] loading (templated) resource "addons/networking.kuberouter/k8s-1.6.yaml"
I1011 18:03:35.638802   47617 templates.go:87] loading resource "addons/networking.flannel/k8s-1.6.yaml"
I1011 18:03:35.639108   47617 templates.go:87] loading resource "addons/networking.flannel/pre-k8s-1.6.yaml"
I1011 18:03:35.639564   47617 templates.go:79] loading (templated) resource "addons/networking.projectcalico.org/k8s-1.6.yaml"
I1011 18:03:35.640013   47617 templates.go:79] loading (templated) resource "addons/networking.projectcalico.org/pre-k8s-1.6.yaml"
I1011 18:03:35.640165   47617 tree_walker.go:97] visit "config/_aws"
I1011 18:03:35.640186   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/_aws"
I1011 18:03:35.640196   47617 tree_walker.go:97] visit "config/_aws/defaults.options"
I1011 18:03:35.640209   47617 tree_walker.go:97] visit "config/_gce"
I1011 18:03:35.640228   47617 tree_walker.go:120] Skipping directory "config/_gce" as tag "_gce" not present
I1011 18:03:35.640236   47617 tree_walker.go:97] visit "config/components"
I1011 18:03:35.640252   47617 tree_walker.go:97] visit "config/components/docker"
I1011 18:03:35.640272   47617 tree_walker.go:97] visit "config/components/docker/_networking_cni"
I1011 18:03:35.640286   47617 tree_walker.go:120] Skipping directory "config/components/docker/_networking_cni" as tag "_networking_cni" not present
I1011 18:03:35.640294   47617 tree_walker.go:97] visit "config/components/docker/_networking_kubenet"
I1011 18:03:35.640307   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/components/docker/_networking_kubenet"
I1011 18:03:35.640318   47617 tree_walker.go:97] visit "config/components/docker/_networking_kubenet/kubenet.options"
I1011 18:03:35.640332   47617 tree_walker.go:97] visit "config/components/docker/docker.options"
I1011 18:03:35.640344   47617 tree_walker.go:97] visit "config/components/docker/_e2e_storage_test_environment"
I1011 18:03:35.640407   47617 tree_walker.go:120] Skipping directory "config/components/docker/_e2e_storage_test_environment" as tag "_e2e_storage_test_environment" not present
I1011 18:03:35.640417   47617 tree_walker.go:97] visit "config/components/kube-apiserver"
I1011 18:03:35.640448   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_3"
I1011 18:03:35.640463   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_3" as tag "_k8s_1_3" not present
I1011 18:03:35.640472   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_5"
I1011 18:03:35.640485   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_5" as tag "_k8s_1_5" not present
I1011 18:03:35.640494   47617 tree_walker.go:97] visit "config/components/kube-apiserver/kube-apiserver.options"
I1011 18:03:35.640507   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_aws"
I1011 18:03:35.640520   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_aws"
I1011 18:03:35.640530   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_aws/kube-apiserver.aws.options"
I1011 18:03:35.640544   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_gce"
I1011 18:03:35.640562   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_gce" as tag "_gce" not present
I1011 18:03:35.640570   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_4"
I1011 18:03:35.640584   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_k8s_1_4" as tag "_k8s_1_4" not present
I1011 18:03:35.640592   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_6"
I1011 18:03:35.640604   47617 tree_walker.go:124] Descending into directory, as tag is present: "config/components/kube-apiserver/_k8s_1_6"
I1011 18:03:35.640615   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_k8s_1_6/kube-apiserver.options"
I1011 18:03:35.640629   47617 tree_walker.go:97] visit "config/components/kube-apiserver/_vsphere"
I1011 18:03:35.640642   47617 tree_walker.go:120] Skipping directory "config/components/kube-apiserver/_vsphere" as tag "_vsphere" not present
I1011 18:03:35.640654   47617 tree_walker.go:97] visit "cloudup/resources"
I1011 18:03:35.640668   47617 tree_walker.go:97] visit "cloudup/resources/addons"
I1011 18:03:35.640712   47617 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io"
I1011 18:03:35.640731   47617 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io/addon.yaml"
I1011 18:03:35.640910   47617 loader.go:319] loading resource "addons/core.addons.k8s.io/addon.yaml"
I1011 18:03:35.640923   47617 tree_walker.go:97] visit "cloudup/resources/addons/core.addons.k8s.io/v1.4.0.yaml"
I1011 18:03:35.641358   47617 loader.go:319] loading resource "addons/core.addons.k8s.io/v1.4.0.yaml"
I1011 18:03:35.641370   47617 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io"
I1011 18:03:35.641391   47617 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io/k8s-1.6.yaml.template"
I1011 18:03:35.641638   47617 loader.go:311] loading (templated) resource "addons/kube-dns.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:35.641741   47617 tree_walker.go:97] visit "cloudup/resources/addons/kube-dns.addons.k8s.io/pre-k8s-1.6.yaml.template"
I1011 18:03:35.642135   47617 loader.go:311] loading (templated) resource "addons/kube-dns.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:35.642164   47617 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io"
I1011 18:03:35.642193   47617 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io/addon.yaml"
I1011 18:03:35.642242   47617 loader.go:319] loading resource "addons/limit-range.addons.k8s.io/addon.yaml"
I1011 18:03:35.642253   47617 tree_walker.go:97] visit "cloudup/resources/addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I1011 18:03:35.642302   47617 loader.go:319] loading resource "addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I1011 18:03:35.642312   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org.canal"
I1011 18:03:35.642334   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org.canal/k8s-1.6.yaml.template"
I1011 18:03:35.642529   47617 loader.go:311] loading (templated) resource "addons/networking.projectcalico.org.canal/k8s-1.6.yaml"
I1011 18:03:35.642550   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org.canal/pre-k8s-1.6.yaml.template"
I1011 18:03:35.642690   47617 loader.go:311] loading (templated) resource "addons/networking.projectcalico.org.canal/pre-k8s-1.6.yaml"
I1011 18:03:35.642765   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave"
I1011 18:03:35.642789   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave/k8s-1.6.yaml.template"
I1011 18:03:35.642905   47617 loader.go:311] loading (templated) resource "addons/networking.weave/k8s-1.6.yaml"
I1011 18:03:35.642925   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.weave/pre-k8s-1.6.yaml.template"
I1011 18:03:35.643093   47617 loader.go:311] loading (templated) resource "addons/networking.weave/pre-k8s-1.6.yaml"
I1011 18:03:35.643108   47617 tree_walker.go:97] visit "cloudup/resources/addons/authentication.kope.io"
I1011 18:03:35.643131   47617 tree_walker.go:97] visit "cloudup/resources/addons/authentication.kope.io/k8s-1.6.yaml"
I1011 18:03:35.643291   47617 loader.go:319] loading resource "addons/authentication.kope.io/k8s-1.6.yaml"
I1011 18:03:35.643302   47617 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io"
I1011 18:03:35.643330   47617 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/k8s-1.6.yaml.template"
I1011 18:03:35.643422   47617 loader.go:311] loading (templated) resource "addons/dns-controller.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:35.643436   47617 tree_walker.go:97] visit "cloudup/resources/addons/dns-controller.addons.k8s.io/pre-k8s-1.6.yaml.template"
I1011 18:03:35.643590   47617 loader.go:311] loading (templated) resource "addons/dns-controller.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:35.643622   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.kope.io"
I1011 18:03:35.643641   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.kope.io/k8s-1.6.yaml"
I1011 18:03:35.643716   47617 loader.go:319] loading resource "addons/networking.kope.io/k8s-1.6.yaml"
I1011 18:03:35.643731   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.kope.io/pre-k8s-1.6.yaml"
I1011 18:03:35.643914   47617 loader.go:319] loading resource "addons/networking.kope.io/pre-k8s-1.6.yaml"
I1011 18:03:35.643926   47617 tree_walker.go:97] visit "cloudup/resources/addons/storage-aws.addons.k8s.io"
I1011 18:03:35.643940   47617 tree_walker.go:97] visit "cloudup/resources/addons/storage-aws.addons.k8s.io/v1.6.0.yaml"
I1011 18:03:35.643979   47617 loader.go:319] loading resource "addons/storage-aws.addons.k8s.io/v1.6.0.yaml"
I1011 18:03:35.643988   47617 tree_walker.go:97] visit "cloudup/resources/addons/external-dns.addons.k8s.io"
I1011 18:03:35.644002   47617 tree_walker.go:97] visit "cloudup/resources/addons/external-dns.addons.k8s.io/k8s-1.6.yaml.template"
I1011 18:03:35.644064   47617 loader.go:311] loading (templated) resource "addons/external-dns.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:35.644074   47617 tree_walker.go:97] visit "cloudup/resources/addons/external-dns.addons.k8s.io/pre-k8s-1.6.yaml.template"
I1011 18:03:35.644126   47617 loader.go:311] loading (templated) resource "addons/external-dns.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:35.644139   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.kuberouter"
I1011 18:03:35.644152   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.kuberouter/k8s-1.6.yaml.template"
I1011 18:03:35.644241   47617 loader.go:311] loading (templated) resource "addons/networking.kuberouter/k8s-1.6.yaml"
I1011 18:03:35.644256   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.flannel"
I1011 18:03:35.644270   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.flannel/k8s-1.6.yaml"
I1011 18:03:35.644343   47617 loader.go:319] loading resource "addons/networking.flannel/k8s-1.6.yaml"
I1011 18:03:35.644353   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.flannel/pre-k8s-1.6.yaml"
I1011 18:03:35.644414   47617 loader.go:319] loading resource "addons/networking.flannel/pre-k8s-1.6.yaml"
I1011 18:03:35.644425   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org"
I1011 18:03:35.644439   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org/k8s-1.6.yaml.template"
I1011 18:03:35.644581   47617 loader.go:311] loading (templated) resource "addons/networking.projectcalico.org/k8s-1.6.yaml"
I1011 18:03:35.644596   47617 tree_walker.go:97] visit "cloudup/resources/addons/networking.projectcalico.org/pre-k8s-1.6.yaml.template"
I1011 18:03:35.644739   47617 loader.go:311] loading (templated) resource "addons/networking.projectcalico.org/pre-k8s-1.6.yaml"
I1011 18:03:35.647666   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.0.0"
I1011 18:03:35.648142   47617 visitor.go:39] float64 value at spec.strategy.rollingUpdate.maxUnavailable: %!s(float64=0)
I1011 18:03:35.648445   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/kubedns-amd64:1.9"
I1011 18:03:35.648471   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.httpGet.port: %!s(float64=8080)
I1011 18:03:35.648478   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.initialDelaySeconds: %!s(float64=60)
I1011 18:03:35.648483   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.successThreshold: %!s(float64=1)
I1011 18:03:35.648488   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.648493   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.failureThreshold: %!s(float64=5)
I1011 18:03:35.648499   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].ports.[0].containerPort: %!s(float64=10053)
I1011 18:03:35.648506   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].ports.[1].containerPort: %!s(float64=10053)
I1011 18:03:35.648512   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].ports.[2].containerPort: %!s(float64=10055)
I1011 18:03:35.648519   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].readinessProbe.httpGet.port: %!s(float64=8081)
I1011 18:03:35.648525   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].readinessProbe.initialDelaySeconds: %!s(float64=3)
I1011 18:03:35.648530   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].readinessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.648546   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/kube-dnsmasq-amd64:1.4"
I1011 18:03:35.648552   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.failureThreshold: %!s(float64=5)
I1011 18:03:35.648557   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.httpGet.port: %!s(float64=8080)
I1011 18:03:35.648562   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.initialDelaySeconds: %!s(float64=60)
I1011 18:03:35.648567   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.successThreshold: %!s(float64=1)
I1011 18:03:35.648571   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.648584   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].ports.[0].containerPort: %!s(float64=53)
I1011 18:03:35.648591   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].ports.[1].containerPort: %!s(float64=53)
I1011 18:03:35.648600   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/dnsmasq-metrics-amd64:1.0"
I1011 18:03:35.648606   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.failureThreshold: %!s(float64=5)
I1011 18:03:35.648612   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.httpGet.port: %!s(float64=10054)
I1011 18:03:35.648617   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.initialDelaySeconds: %!s(float64=60)
I1011 18:03:35.648622   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.successThreshold: %!s(float64=1)
I1011 18:03:35.648627   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.648633   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].ports.[0].containerPort: %!s(float64=10054)
I1011 18:03:35.648643   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/exechealthz-amd64:1.2"
I1011 18:03:35.648650   47617 visitor.go:39] float64 value at spec.template.spec.containers.[3].ports.[0].containerPort: %!s(float64=8080)
I1011 18:03:35.649752   47617 visitor.go:39] float64 value at spec.ports.[0].port: %!s(float64=53)
I1011 18:03:35.649765   47617 visitor.go:39] float64 value at spec.ports.[1].port: %!s(float64=53)
I1011 18:03:35.651634   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.2-r2"
I1011 18:03:35.652031   47617 visitor.go:39] float64 value at spec.strategy.rollingUpdate.maxUnavailable: %!s(float64=0)
I1011 18:03:35.652050   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].readinessProbe.httpGet.port: %!s(float64=8081)
I1011 18:03:35.652057   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].readinessProbe.initialDelaySeconds: %!s(float64=3)
I1011 18:03:35.652062   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].readinessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.652070   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4"
I1011 18:03:35.652078   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.initialDelaySeconds: %!s(float64=60)
I1011 18:03:35.652084   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.successThreshold: %!s(float64=1)
I1011 18:03:35.652089   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.652094   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.failureThreshold: %!s(float64=5)
I1011 18:03:35.652100   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].livenessProbe.httpGet.port: %!s(float64=10054)
I1011 18:03:35.652106   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].ports.[0].containerPort: %!s(float64=10053)
I1011 18:03:35.652112   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].ports.[1].containerPort: %!s(float64=10053)
I1011 18:03:35.652119   47617 visitor.go:39] float64 value at spec.template.spec.containers.[0].ports.[2].containerPort: %!s(float64=10055)
I1011 18:03:35.652130   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4"
I1011 18:03:35.652138   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.failureThreshold: %!s(float64=5)
I1011 18:03:35.652143   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.httpGet.port: %!s(float64=10054)
I1011 18:03:35.652149   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.initialDelaySeconds: %!s(float64=60)
I1011 18:03:35.652154   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.successThreshold: %!s(float64=1)
I1011 18:03:35.652159   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].livenessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.652165   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].ports.[0].containerPort: %!s(float64=53)
I1011 18:03:35.652172   47617 visitor.go:39] float64 value at spec.template.spec.containers.[1].ports.[1].containerPort: %!s(float64=53)
I1011 18:03:35.652182   47617 images.go:56] Consider image for re-mapping: "gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4"
I1011 18:03:35.652188   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.failureThreshold: %!s(float64=5)
I1011 18:03:35.652194   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.httpGet.port: %!s(float64=10054)
I1011 18:03:35.652199   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.initialDelaySeconds: %!s(float64=60)
I1011 18:03:35.652204   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.successThreshold: %!s(float64=1)
I1011 18:03:35.652209   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].livenessProbe.timeoutSeconds: %!s(float64=5)
I1011 18:03:35.652215   47617 visitor.go:39] float64 value at spec.template.spec.containers.[2].ports.[0].containerPort: %!s(float64=10054)
I1011 18:03:35.652223   47617 visitor.go:34] string value at spec.template.spec.volumes.[0].configMap.optional: %!s(bool=true)
I1011 18:03:35.653344   47617 visitor.go:39] float64 value at spec.ports.[0].port: %!s(float64=53)
I1011 18:03:35.653357   47617 visitor.go:39] float64 value at spec.ports.[1].port: %!s(float64=53)
I1011 18:03:35.654533   47617 visitor.go:39] float64 value at spec.replicas: %!s(float64=1)
I1011 18:03:35.654550   47617 images.go:56] Consider image for re-mapping: "kope/dns-controller:1.7.1"
I1011 18:03:35.654559   47617 visitor.go:34] string value at spec.template.spec.hostNetwork: %!s(bool=true)
I1011 18:03:35.655464   47617 visitor.go:39] float64 value at spec.replicas: %!s(float64=1)
I1011 18:03:35.655477   47617 visitor.go:34] string value at spec.template.spec.hostNetwork: %!s(bool=true)
I1011 18:03:35.655488   47617 images.go:56] Consider image for re-mapping: "kope/dns-controller:1.7.1"
I1011 18:03:35.675178   47617 s3fs.go:110] Writing file "s3://kubernetes-example/kubernetes.example.ru/cluster.spec"
I1011 18:03:35.675218   47617 s3fs.go:130] Calling S3 PutObject Bucket="kubernetes-example" Key="kubernetes.example.ru/cluster.spec" SSE="AES256" ACL="" BodyLen=4406
I1011 18:03:35.823442   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/instancegroup/master-eu-central-1a"
I1011 18:03:35.897983   47617 s3fs.go:110] Writing file "s3://kubernetes-example/kubernetes.example.ru/instancegroup/master-eu-central-1a"
I1011 18:03:35.898014   47617 s3fs.go:130] Calling S3 PutObject Bucket="kubernetes-example" Key="kubernetes.example.ru/instancegroup/master-eu-central-1a" SSE="AES256" ACL="" BodyLen=347
I1011 18:03:36.007354   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/instancegroup/nodes"
I1011 18:03:36.081144   47617 s3fs.go:110] Writing file "s3://kubernetes-example/kubernetes.example.ru/instancegroup/nodes"
I1011 18:03:36.081174   47617 s3fs.go:130] Calling S3 PutObject Bucket="kubernetes-example" Key="kubernetes.example.ru/instancegroup/nodes" SSE="AES256" ACL="" BodyLen=365
I1011 18:03:36.217766   47617 topological_sort.go:62] Dependencies:
I1011 18:03:36.217787   47617 topological_sort.go:64]   Secret/system:scheduler:    []
I1011 18:03:36.217796   47617 topological_sort.go:64]   kubernetes.example.ru-addons-dns-controller.addons.k8s.io-k8s-1.6:  []
I1011 18:03:36.217803   47617 topological_sort.go:64]   kubernetes.example.ru-addons-limit-range.addons.k8s.io: []
I1011 18:03:36.217809   47617 topological_sort.go:64]   IAMInstanceProfileRole/nodes.kubernetes.example.ru: [IAMInstanceProfile/nodes.kubernetes.example.ru IAMRole/nodes.kubernetes.example.ru]
I1011 18:03:36.217817   47617 topological_sort.go:64]   VPC/kubernetes.example.ru:  []
I1011 18:03:36.217823   47617 topological_sort.go:64]   RouteTableAssociation/eu-central-1b.kubernetes.example.ru:  [RouteTable/kubernetes.example.ru Subnet/eu-central-1b.kubernetes.example.ru]
I1011 18:03:36.217830   47617 topological_sort.go:64]   Keypair/kube-proxy: []
I1011 18:03:36.217836   47617 topological_sort.go:64]   IAMRolePolicy/masters.kubernetes.example.ru:    [IAMRole/masters.kubernetes.example.ru DNSZone/Z3CEXJAAQ67LHN]
I1011 18:03:36.217843   47617 topological_sort.go:64]   IAMInstanceProfile/masters.kubernetes.example.ru:   []
I1011 18:03:36.217849   47617 topological_sort.go:64]   IAMRolePolicy/nodes.kubernetes.example.ru:  [IAMRole/nodes.kubernetes.example.ru DNSZone/Z3CEXJAAQ67LHN]
I1011 18:03:36.217855   47617 topological_sort.go:64]   Keypair/kops:   []
I1011 18:03:36.217861   47617 topological_sort.go:64]   Keypair/master: []
I1011 18:03:36.217867   47617 topological_sort.go:64]   IAMRolePolicy/additional.nodes.kubernetes.example.ru:   [IAMRole/nodes.kubernetes.example.ru]
I1011 18:03:36.217874   47617 topological_sort.go:64]   SecurityGroupRule/all-master-to-master: [SecurityGroup/masters.kubernetes.example.ru SecurityGroup/masters.kubernetes.example.ru]
I1011 18:03:36.217881   47617 topological_sort.go:64]   SecurityGroupRule/ssh-external-to-master-0.0.0.0/0: [SecurityGroup/masters.kubernetes.example.ru]
I1011 18:03:36.217888   47617 topological_sort.go:64]   EBSVolume/a.etcd-main.kubernetes.example.ru:    []
I1011 18:03:36.217894   47617 topological_sort.go:64]   Secret/system:controller_manager:   []
I1011 18:03:36.217899   47617 topological_sort.go:64]   SecurityGroupRule/all-node-to-node: [SecurityGroup/nodes.kubernetes.example.ru SecurityGroup/nodes.kubernetes.example.ru]
I1011 18:03:36.217906   47617 topological_sort.go:64]   kubernetes.example.ru-addons-dns-controller.addons.k8s.io-pre-k8s-1.6:  []
I1011 18:03:36.217912   47617 topological_sort.go:64]   Secret/system:logging:  []
I1011 18:03:36.217917   47617 topological_sort.go:64]   Subnet/eu-central-1a.kubernetes.example.ru: [VPC/kubernetes.example.ru]
I1011 18:03:36.217938   47617 topological_sort.go:64]   Secret/system:dns:  []
I1011 18:03:36.217945   47617 topological_sort.go:64]   IAMRole/nodes.kubernetes.example.ru:    []
I1011 18:03:36.217951   47617 topological_sort.go:64]   Keypair/kubelet:    []
I1011 18:03:36.217956   47617 topological_sort.go:64]   Secret/kube:    []
I1011 18:03:36.217962   47617 topological_sort.go:64]   SecurityGroup/nodes.kubernetes.example.ru:  [VPC/kubernetes.example.ru]
I1011 18:03:36.217969   47617 topological_sort.go:64]   SecurityGroup/masters.kubernetes.example.ru:    [VPC/kubernetes.example.ru]
I1011 18:03:36.217975   47617 topological_sort.go:64]   Secret/kube-proxy:  []
I1011 18:03:36.217981   47617 topological_sort.go:64]   SecurityGroupRule/master-egress:    [SecurityGroup/masters.kubernetes.example.ru]
I1011 18:03:36.217987   47617 topological_sort.go:64]   SSHKey/kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a:    []
I1011 18:03:36.217993   47617 topological_sort.go:64]   Subnet/eu-central-1c.kubernetes.example.ru: [VPC/kubernetes.example.ru]
I1011 18:03:36.218362   47617 topological_sort.go:64]   Keypair/kube-scheduler: []
I1011 18:03:36.218376   47617 topological_sort.go:64]   DHCPOptions/kubernetes.example.ru:  []
I1011 18:03:36.218382   47617 topological_sort.go:64]   IAMInstanceProfile/nodes.kubernetes.example.ru: []
I1011 18:03:36.218389   47617 topological_sort.go:64]   kubernetes.example.ru-addons-storage-aws.addons.k8s.io: []
I1011 18:03:36.218395   47617 topological_sort.go:64]   VPCDHCPOptionsAssociation/kubernetes.example.ru:    [VPC/kubernetes.example.ru DHCPOptions/kubernetes.example.ru]
I1011 18:03:36.218403   47617 topological_sort.go:64]   Secret/kubelet: []
I1011 18:03:36.218409   47617 topological_sort.go:64]   Keypair/kube-controller-manager:    []
I1011 18:03:36.218414   47617 topological_sort.go:64]   AutoscalingGroup/master-eu-central-1a.masters.kubernetes.example.ru:    [Subnet/eu-central-1a.kubernetes.example.ru LaunchConfiguration/master-eu-central-1a.masters.kubernetes.example.ru]
I1011 18:03:36.218421   47617 topological_sort.go:64]   AutoscalingGroup/nodes.kubernetes.example.ru:   [Subnet/eu-central-1a.kubernetes.example.ru Subnet/eu-central-1b.kubernetes.example.ru Subnet/eu-central-1c.kubernetes.example.ru LaunchConfiguration/nodes.kubernetes.example.ru]
I1011 18:03:36.218430   47617 topological_sort.go:64]   SecurityGroupRule/all-master-to-node:   [SecurityGroup/nodes.kubernetes.example.ru SecurityGroup/masters.kubernetes.example.ru]
I1011 18:03:36.218438   47617 topological_sort.go:64]   kubernetes.example.ru-addons-kube-dns.addons.k8s.io-k8s-1.6:    []
I1011 18:03:36.218444   47617 topological_sort.go:64]   Secret/admin:   []
I1011 18:03:36.218450   47617 topological_sort.go:64]   LaunchConfiguration/nodes.kubernetes.example.ru:    [SSHKey/kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a SecurityGroup/nodes.kubernetes.example.ru IAMInstanceProfile/nodes.kubernetes.example.ru]
I1011 18:03:36.218459   47617 topological_sort.go:64]   Keypair/kubecfg:    []
I1011 18:03:36.218466   47617 topological_sort.go:64]   SecurityGroupRule/ssh-external-to-node-0.0.0.0/0:   [SecurityGroup/nodes.kubernetes.example.ru]
I1011 18:03:36.218704   47617 topological_sort.go:64]   SecurityGroupRule/node-to-master-tcp-1-4000:    [SecurityGroup/masters.kubernetes.example.ru SecurityGroup/nodes.kubernetes.example.ru]
I1011 18:03:36.218713   47617 topological_sort.go:64]   kubernetes.example.ru-addons-bootstrap: []
I1011 18:03:36.218719   47617 topological_sort.go:64]   Subnet/eu-central-1b.kubernetes.example.ru: [VPC/kubernetes.example.ru]
I1011 18:03:36.218726   47617 topological_sort.go:64]   SecurityGroupRule/https-external-to-master-0.0.0.0/0:   [SecurityGroup/masters.kubernetes.example.ru]
I1011 18:03:36.218732   47617 topological_sort.go:64]   RouteTableAssociation/eu-central-1c.kubernetes.example.ru:  [RouteTable/kubernetes.example.ru Subnet/eu-central-1c.kubernetes.example.ru]
I1011 18:03:36.218739   47617 topological_sort.go:64]   IAMRole/masters.kubernetes.example.ru:  []
I1011 18:03:36.218746   47617 topological_sort.go:64]   kubernetes.example.ru-addons-core.addons.k8s.io:    []
I1011 18:03:36.218752   47617 topological_sort.go:64]   IAMInstanceProfileRole/masters.kubernetes.example.ru:   [IAMInstanceProfile/masters.kubernetes.example.ru IAMRole/masters.kubernetes.example.ru]
I1011 18:03:36.218758   47617 topological_sort.go:64]   RouteTableAssociation/eu-central-1a.kubernetes.example.ru:  [RouteTable/kubernetes.example.ru Subnet/eu-central-1a.kubernetes.example.ru]
I1011 18:03:36.218765   47617 topological_sort.go:64]   RouteTable/kubernetes.example.ru:   [VPC/kubernetes.example.ru]
I1011 18:03:36.218771   47617 topological_sort.go:64]   DNSZone/Z3CEXJAAQ67LHN: []
I1011 18:03:36.218777   47617 topological_sort.go:64]   kubernetes.example.ru-addons-kube-dns.addons.k8s.io-pre-k8s-1.6:    []
I1011 18:03:36.218783   47617 topological_sort.go:64]   InternetGateway/kubernetes.example.ru:  [VPC/kubernetes.example.ru]
I1011 18:03:36.218789   47617 topological_sort.go:64]   LaunchConfiguration/master-eu-central-1a.masters.kubernetes.example.ru: [SSHKey/kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a SecurityGroup/masters.kubernetes.example.ru IAMInstanceProfile/masters.kubernetes.example.ru]
I1011 18:03:36.219015   47617 topological_sort.go:64]   SecurityGroupRule/node-egress:  [SecurityGroup/nodes.kubernetes.example.ru]
I1011 18:03:36.219030   47617 topological_sort.go:64]   Route/0.0.0.0/0:    [RouteTable/kubernetes.example.ru InternetGateway/kubernetes.example.ru]
I1011 18:03:36.219038   47617 topological_sort.go:64]   EBSVolume/a.etcd-events.kubernetes.example.ru:  []
I1011 18:03:36.219044   47617 topological_sort.go:64]   SecurityGroupRule/node-to-master-tcp-4003-65535:    [SecurityGroup/masters.kubernetes.example.ru SecurityGroup/nodes.kubernetes.example.ru]
I1011 18:03:36.219052   47617 topological_sort.go:64]   IAMRolePolicy/additional.masters.kubernetes.example.ru: [IAMRole/masters.kubernetes.example.ru]
I1011 18:03:36.219059   47617 topological_sort.go:64]   Secret/system:monitoring:   []
I1011 18:03:36.219064   47617 topological_sort.go:64]   SecurityGroupRule/node-to-master-udp-1-65535:   [SecurityGroup/masters.kubernetes.example.ru SecurityGroup/nodes.kubernetes.example.ru]
I1011 18:03:36.219135   47617 executor.go:91] Tasks: 0 done / 67 total; 34 can run
I1011 18:03:36.220128   47617 executor.go:157] Executing task "Secret/kubelet": *fitasks.Secret {"Name":"kubelet","Lifecycle":"Sync"}
I1011 18:03:36.220078   47617 executor.go:157] Executing task "Secret/system:scheduler": *fitasks.Secret {"Name":"system:scheduler","Lifecycle":"Sync"}
I1011 18:03:36.219249   47617 executor.go:157] Executing task "Secret/kube-proxy": *fitasks.Secret {"Name":"kube-proxy","Lifecycle":"Sync"}
I1011 18:03:36.220080   47617 executor.go:157] Executing task "Secret/system:controller_manager": *fitasks.Secret {"Name":"system:controller_manager","Lifecycle":"Sync"}
I1011 18:03:36.220092   47617 executor.go:157] Executing task "Secret/system:dns": *fitasks.Secret {"Name":"system:dns","Lifecycle":"Sync"}
I1011 18:03:36.220211   47617 executor.go:157] Executing task "Secret/system:monitoring": *fitasks.Secret {"Name":"system:monitoring","Lifecycle":"Sync"}
I1011 18:03:36.220185   47617 executor.go:157] Executing task "Keypair/kops": *fitasks.Keypair {"Name":"kops","Lifecycle":"Sync","subject":"o=system:masters,cn=kops","type":"client","alternateNames":null,"alternateNameTasks":null}
I1011 18:03:36.220244   47617 executor.go:157] Executing task "Keypair/kube-scheduler": *fitasks.Keypair {"Name":"kube-scheduler","Lifecycle":"Sync","subject":"cn=system:kube-scheduler","type":"client","alternateNames":null,"alternateNameTasks":null}
I1011 18:03:36.219226   47617 executor.go:157] Executing task "Keypair/kube-controller-manager": *fitasks.Keypair {"Name":"kube-controller-manager","Lifecycle":"Sync","subject":"cn=system:kube-controller-manager","type":"client","alternateNames":null,"alternateNameTasks":null}
I1011 18:03:36.220264   47617 executor.go:157] Executing task "Secret/admin": *fitasks.Secret {"Name":"admin","Lifecycle":"Sync"}
I1011 18:03:36.219261   47617 executor.go:157] Executing task "Secret/system:logging": *fitasks.Secret {"Name":"system:logging","Lifecycle":"Sync"}
I1011 18:03:36.220298   47617 executor.go:157] Executing task "Keypair/master": *fitasks.Keypair {"Name":"master","Lifecycle":"Sync","subject":"cn=kubernetes-master","type":"server","alternateNames":["kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster.local","api.kubernetes.example.ru","api.internal.kubernetes.example.ru","100.64.0.1","127.0.0.1"],"alternateNameTasks":null}
I1011 18:03:36.220254   47617 executor.go:157] Executing task "Secret/kube": *fitasks.Secret {"Name":"kube","Lifecycle":"Sync"}
I1011 18:03:36.220091   47617 executor.go:157] Executing task "Keypair/kubelet": *fitasks.Keypair {"Name":"kubelet","Lifecycle":"Sync","subject":"o=system:nodes,cn=kubelet","type":"client","alternateNames":null,"alternateNameTasks":null}
I1011 18:03:36.220175   47617 executor.go:157] Executing task "Keypair/kubecfg": *fitasks.Keypair {"Name":"kubecfg","Lifecycle":"Sync","subject":"o=system:masters,cn=kubecfg","type":"client","alternateNames":null,"alternateNameTasks":null}
I1011 18:03:36.220125   47617 executor.go:157] Executing task "DHCPOptions/kubernetes.example.ru": *awstasks.DHCPOptions {"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":null,"DomainName":"eu-central-1.compute.internal","DomainNameServers":"AmazonProvidedDNS"}
I1011 18:03:36.220286   47617 executor.go:157] Executing task "IAMInstanceProfile/masters.kubernetes.example.ru": *awstasks.IAMInstanceProfile {"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":null}
I1011 18:03:36.220276   47617 executor.go:157] Executing task "IAMRole/masters.kubernetes.example.ru": *awstasks.IAMRole {"ID":null,"Lifecycle":"Sync","Name":"masters.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"masters"}
I1011 18:03:36.220164   47617 executor.go:157] Executing task "IAMInstanceProfile/nodes.kubernetes.example.ru": *awstasks.IAMInstanceProfile {"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":null}
I1011 18:03:36.219224   47617 executor.go:157] Executing task "Keypair/kube-proxy": *fitasks.Keypair {"Name":"kube-proxy","Lifecycle":"Sync","subject":"cn=system:kube-proxy","type":"client","alternateNames":null,"alternateNameTasks":null}
I1011 18:03:36.220287   47617 executor.go:157] Executing task "EBSVolume/a.etcd-main.kubernetes.example.ru": *awstasks.EBSVolume {"Name":"a.etcd-main.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"AvailabilityZone":"eu-central-1a","VolumeType":"gp2","SizeGB":20,"KmsKeyId":null,"Encrypted":false,"Tags":{"k8s.io/etcd/main":"a/a","k8s.io/role/master":"1"}}
I1011 18:03:36.220229   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-bootstrap": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-bootstrap","Lifecycle":"Sync","Location":"addons/bootstrap-channel.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.220102   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-limit-range.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-limit-range.addons.k8s.io","Lifecycle":"Sync","Location":"addons/limit-range.addons.k8s.io/v1.5.0.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.220117   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-core.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-core.addons.k8s.io","Lifecycle":"Sync","Location":"addons/core.addons.k8s.io/v1.4.0.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.220215   47617 executor.go:157] Executing task "SSHKey/kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a": *awstasks.SSHKey {"Name":"kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a","Lifecycle":"Sync","PublicKey":{"Name":"","Resource":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs1NT4mr0LJcDQ9wu4PbWphFIF9m1u0rXq2u5ZGJZjVSxIolS2/AuLYDhNjNEcJLtPOMKUvGof4A3ywL1WTV5ABGxS6EzQ5Wb2FOLgqqOcoMYdH4YkpPHUMr3PNTJT8/jKcjryxAndkstQpskVQ61DznDKBIRIMSnV7/I6IxqibBWzM1jJ+8ANAmEnRZlXBZlqJzp8RgiRPvXd9PiO0jJKPpOBl7YFW9D7HWKAOZzk82jMSGoT9QGvVka6cvJG1SPqMbKpyk/Qd8owAr1E5E291Q/E+1eyTxbVvvT8xvtPQN5CcskeVMe7ZAmPTkEo4ZngUKnzNhKfPmieZv/vO7bhWn0nly9 [email protected]\n"},"KeyFingerprint":null}
I1011 18:03:36.220304   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-storage-aws.addons.k8s.io": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-storage-aws.addons.k8s.io","Lifecycle":"Sync","Location":"addons/storage-aws.addons.k8s.io/v1.6.0.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.220162   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-kube-dns.addons.k8s.io-pre-k8s-1.6": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-kube-dns.addons.k8s.io-pre-k8s-1.6","Lifecycle":"Sync","Location":"addons/kube-dns.addons.k8s.io/pre-k8s-1.6.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.220314   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-dns-controller.addons.k8s.io-pre-k8s-1.6": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-dns-controller.addons.k8s.io-pre-k8s-1.6","Lifecycle":"Sync","Location":"addons/dns-controller.addons.k8s.io/pre-k8s-1.6.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.220197   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-dns-controller.addons.k8s.io-k8s-1.6": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-dns-controller.addons.k8s.io-k8s-1.6","Lifecycle":"Sync","Location":"addons/dns-controller.addons.k8s.io/k8s-1.6.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.219227   47617 executor.go:157] Executing task "VPC/kubernetes.example.ru": *awstasks.VPC {"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":null,"CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:36.219275   47617 executor.go:157] Executing task "DNSZone/Z3CEXJAAQ67LHN": *awstasks.DNSZone {"Name":"Z3CEXJAAQ67LHN","Lifecycle":"Sync","DNSName":null,"ZoneID":"Z3CEXJAAQ67LHN","Private":null,"PrivateVPC":null}
I1011 18:03:36.220179   47617 executor.go:157] Executing task "IAMRole/nodes.kubernetes.example.ru": *awstasks.IAMRole {"ID":null,"Lifecycle":"Sync","Name":"nodes.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"nodes"}
I1011 18:03:36.220294   47617 executor.go:157] Executing task "kubernetes.example.ru-addons-kube-dns.addons.k8s.io-k8s-1.6": *fitasks.ManagedFile {"Name":"kubernetes.example.ru-addons-kube-dns.addons.k8s.io-k8s-1.6","Lifecycle":"Sync","Location":"addons/kube-dns.addons.k8s.io/k8s-1.6.yaml","Contents":{"Name":"","Resource":{}}}
I1011 18:03:36.220103   47617 executor.go:157] Executing task "EBSVolume/a.etcd-events.kubernetes.example.ru": *awstasks.EBSVolume {"Name":"a.etcd-events.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"AvailabilityZone":"eu-central-1a","VolumeType":"gp2","SizeGB":20,"KmsKeyId":null,"Encrypted":false,"Tags":{"k8s.io/etcd/events":"a/a","k8s.io/role/master":"1"}}
I1011 18:03:36.221056   47617 sshkey.go:210] Computed SSH key fingerprint as "93:a9:a5:f0:77:55:30:2e:d2:16:bd:c5:10:3b:fc:cc"
I1011 18:03:36.241015   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/kube-proxy"
I1011 18:03:36.241040   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/system:scheduler"
I1011 18:03:36.241078   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/system:logging"
I1011 18:03:36.241111   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/system:dns"
I1011 18:03:36.241085   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/system:controller_manager"
I1011 18:03:36.241421   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/kubelet"
I1011 18:03:36.241531   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/kube"
I1011 18:03:36.241546   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/system:monitoring"
I1011 18:03:36.241044   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/admin"
I1011 18:03:36.242507   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kube-controller-manager/"
I1011 18:03:36.242791   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/bootstrap-channel.yaml"
I1011 18:03:36.242990   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kube-proxy/"
I1011 18:03:36.243220   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/limit-range.addons.k8s.io/v1.5.0.yaml"
I1011 18:03:36.243416   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/core.addons.k8s.io/v1.4.0.yaml"
I1011 18:03:36.243620   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/storage-aws.addons.k8s.io/v1.6.0.yaml"
I1011 18:03:36.243827   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kubelet/"
I1011 18:03:36.244040   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/dns-controller.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:36.244221   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/dns-controller.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:36.244274   47617 request_logger.go:45] AWS request: ec2/DescribeVolumes
I1011 18:03:36.244700   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kube-scheduler/"
I1011 18:03:36.244758   47617 request_logger.go:45] AWS request: ec2/DescribeVpcs
I1011 18:03:36.245034   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kops/"
I1011 18:03:36.245266   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kubecfg/"
I1011 18:03:36.264302   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/kube-dns.addons.k8s.io/k8s-1.6.yaml"
I1011 18:03:36.264417   47617 request_logger.go:45] AWS request: iam/GetRole
I1011 18:03:36.264440   47617 request_logger.go:45] AWS request: iam/GetRole
I1011 18:03:36.265311   47617 request_logger.go:45] AWS request: iam/GetInstanceProfile
I1011 18:03:36.265432   47617 request_logger.go:45] AWS request: ec2/DescribeVolumes
I1011 18:03:36.265567   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/addons/kube-dns.addons.k8s.io/pre-k8s-1.6.yaml"
I1011 18:03:36.265584   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/master/"
I1011 18:03:36.265773   47617 request_logger.go:45] AWS request: iam/GetInstanceProfile
I1011 18:03:36.266239   47617 request_logger.go:45] AWS request: route53/GetHostedZone
I1011 18:03:36.266306   47617 request_logger.go:45] AWS request: ec2/DescribeDhcpOptions
I1011 18:03:36.266540   47617 request_logger.go:45] AWS request: ec2/DescribeKeyPairs
I1011 18:03:36.366469   47617 ebsvolume.go:93] found existing volume
I1011 18:03:36.366514   47617 changes.go:148] comparing maps: Name a.etcd-main.kubernetes.example.ru a.etcd-main.kubernetes.example.ru
I1011 18:03:36.366526   47617 changes.go:148] comparing maps: k8s.io/etcd/main a/a a/a
I1011 18:03:36.366532   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:36.366538   47617 changes.go:148] comparing maps: k8s.io/role/master 1 1
I1011 18:03:36.563973   47617 ebsvolume.go:93] found existing volume
I1011 18:03:36.564009   47617 changes.go:148] comparing maps: k8s.io/etcd/events a/a a/a
I1011 18:03:36.564019   47617 changes.go:148] comparing maps: k8s.io/role/master 1 1
I1011 18:03:36.564025   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:36.564032   47617 changes.go:148] comparing maps: Name a.etcd-events.kubernetes.example.ru a.etcd-events.kubernetes.example.ru
I1011 18:03:36.571621   47617 sshkey.go:95] SSH key fingerprints match; assuming public keys match
I1011 18:03:36.653904   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-controller-manager: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-controller-manager/6475652273487221349398232036.crt]
I1011 18:03:36.653939   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-controller-manager/6475652273487221349398232036.crt"
I1011 18:03:36.663678   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-proxy: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-proxy/6475652273243797550239666047.crt]
I1011 18:03:36.663703   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-proxy/6475652273243797550239666047.crt"
I1011 18:03:36.669110   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg/6475652274088147560683131170.crt]
I1011 18:03:36.669128   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg/6475652274088147560683131170.crt"
I1011 18:03:36.671877   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-scheduler: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-scheduler/6475652277046713874923325193.crt]
I1011 18:03:36.671894   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kube-scheduler/6475652277046713874923325193.crt"
I1011 18:03:36.671939   47617 vpc.go:84] found matching VPC *awstasks.VPC {"Name":"kubernetes.example.ru","Lifecycle":null,"ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":null,"EnableDNSSupport":null,"Shared":null,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:36.676556   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/master: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/master/6475652273911880246728739255.crt]
I1011 18:03:36.676583   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/master/6475652273911880246728739255.crt"
I1011 18:03:36.678776   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubelet: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubelet/6475652273855331210403533624.crt]
I1011 18:03:36.678793   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubelet/6475652273855331210403533624.crt"
I1011 18:03:36.715664   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kops: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kops/6475652273262567690391194902.crt]
I1011 18:03:36.715691   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kops/6475652273262567690391194902.crt"
I1011 18:03:36.728959   47617 request_logger.go:45] AWS request: ec2/DescribeVpcAttribute
I1011 18:03:36.729088   47617 dhcp_options.go:70] found existing DhcpOptions
I1011 18:03:36.730300   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:36.730409   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/kube-controller-manager/"
I1011 18:03:36.744969   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:36.745054   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/kube-proxy/"
I1011 18:03:36.745510   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:36.745612   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/kubecfg/"
I1011 18:03:36.749054   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:36.749139   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/kube-scheduler/"
I1011 18:03:36.750628   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:36.750718   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/kubelet/"
I1011 18:03:36.767076   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:36.767171   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/master/"
I1011 18:03:36.805723   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:36.805836   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/kops/"
I1011 18:03:36.822528   47617 request_logger.go:45] AWS request: ec2/DescribeVpcAttribute
I1011 18:03:36.823827   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-controller-manager: [s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-controller-manager/6475652273487221349398232036.key]
I1011 18:03:36.823847   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/kubelet: [s3://kubernetes-example/kubernetes.example.ru/pki/private/kubelet/6475652273855331210403533624.key]
I1011 18:03:36.823866   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/kubelet/6475652273855331210403533624.key"
I1011 18:03:36.823848   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-controller-manager/6475652273487221349398232036.key"
I1011 18:03:36.832860   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-scheduler: [s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-scheduler/6475652277046713874923325193.key]
I1011 18:03:36.832879   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-scheduler/6475652277046713874923325193.key"
I1011 18:03:36.835678   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-proxy: [s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-proxy/6475652273243797550239666047.key]
I1011 18:03:36.835695   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/kube-proxy/6475652273243797550239666047.key"
I1011 18:03:36.842817   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/kubecfg: [s3://kubernetes-example/kubernetes.example.ru/pki/private/kubecfg/6475652274088147560683131170.key]
I1011 18:03:36.842832   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/kubecfg/6475652274088147560683131170.key"
I1011 18:03:36.854771   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/master: [s3://kubernetes-example/kubernetes.example.ru/pki/private/master/6475652273911880246728739255.key]
I1011 18:03:36.854795   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/master/6475652273911880246728739255.key"
I1011 18:03:36.895054   47617 changes.go:148] comparing maps: kubernetes.io/cluster/kubernetes.example.ru owned owned
I1011 18:03:36.895082   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:36.895090   47617 changes.go:148] comparing maps: Name kubernetes.example.ru kubernetes.example.ru
I1011 18:03:36.910548   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:36.912502   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:36.915254   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:36.920270   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:36.923735   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:36.929597   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:36.929975   47617 changes.go:173] comparing slices: 0 100.64.0.1 100.64.0.1
I1011 18:03:36.929989   47617 changes.go:173] comparing slices: 1 127.0.0.1 127.0.0.1
I1011 18:03:36.929995   47617 changes.go:173] comparing slices: 2 api.internal.kubernetes.example.ru api.internal.kubernetes.example.ru
I1011 18:03:36.930002   47617 changes.go:173] comparing slices: 3 api.kubernetes.example.ru api.kubernetes.example.ru
I1011 18:03:36.930007   47617 changes.go:173] comparing slices: 4 kubernetes kubernetes
I1011 18:03:36.930013   47617 changes.go:173] comparing slices: 5 kubernetes.default kubernetes.default
I1011 18:03:36.930019   47617 changes.go:173] comparing slices: 6 kubernetes.default.svc kubernetes.default.svc
I1011 18:03:36.930025   47617 changes.go:173] comparing slices: 7 kubernetes.default.svc.cluster.local kubernetes.default.svc.cluster.local
I1011 18:03:37.076969   47617 iamrole.go:101] actual RolePolicyDocument was json-equal to expected; returning expected value
I1011 18:03:37.076988   47617 iamrole.go:109] found matching IAMRole "AROAJ7UNLOO6IV3MAAAZA"
I1011 18:03:37.097016   47617 iamrole.go:101] actual RolePolicyDocument was json-equal to expected; returning expected value
I1011 18:03:37.097041   47617 iamrole.go:109] found matching IAMRole "AROAI5YUX6GTAMWTAJ67A"
I1011 18:03:37.394655   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/kops: [s3://kubernetes-example/kubernetes.example.ru/pki/private/kops/6475652273262567690391194902.key]
I1011 18:03:37.394682   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/kops/6475652273262567690391194902.key"
I1011 18:03:37.470743   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:37.471139   47617 executor.go:91] Tasks: 34 done / 67 total; 14 can run
I1011 18:03:37.471238   47617 executor.go:157] Executing task "IAMInstanceProfileRole/masters.kubernetes.example.ru": *awstasks.IAMInstanceProfileRole {"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","InstanceProfile":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"AIPAJIL5CNLVETH4PZEA2"},"Role":{"ID":"AROAJ7UNLOO6IV3MAAAZA","Lifecycle":"Sync","Name":"masters.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"masters"}}
I1011 18:03:37.471305   47617 executor.go:157] Executing task "IAMInstanceProfileRole/nodes.kubernetes.example.ru": *awstasks.IAMInstanceProfileRole {"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","InstanceProfile":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"AIPAJRVDU7OS5T3PYYYHI"},"Role":{"ID":"AROAI5YUX6GTAMWTAJ67A","Lifecycle":"Sync","Name":"nodes.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"nodes"}}
I1011 18:03:37.471320   47617 executor.go:157] Executing task "Subnet/eu-central-1c.kubernetes.example.ru": *awstasks.Subnet {"Name":"eu-central-1c.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1c","CIDR":"172.20.96.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1c.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:37.471388   47617 executor.go:157] Executing task "InternetGateway/kubernetes.example.ru": *awstasks.InternetGateway {"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":null,"VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"Shared":false}
I1011 18:03:37.471178   47617 executor.go:157] Executing task "IAMRolePolicy/nodes.kubernetes.example.ru": *awstasks.IAMRolePolicy {"ID":null,"Lifecycle":"Sync","Name":"nodes.kubernetes.example.ru","Role":{"ID":"AROAI5YUX6GTAMWTAJ67A","Lifecycle":"Sync","Name":"nodes.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"nodes"},"PolicyDocument":{"Builder":{"Cluster":{"metadata":{"name":"kubernetes.example.ru","creationTimestamp":"2017-10-11T13:57:53Z"},"spec":{"channel":"stable","configBase":"s3://kubernetes-example/kubernetes.example.ru","cloudProvider":"aws","kubernetesVersion":"1.7.4","subnets":[{"name":"eu-central-1a","zone":"eu-central-1a","cidr":"172.20.32.0/19","type":"Public"},{"name":"eu-central-1b","zone":"eu-central-1b","cidr":"172.20.64.0/19","type":"Public"},{"name":"eu-central-1c","zone":"eu-central-1c","cidr":"172.20.96.0/19","type":"Public"}],"masterPublicName":"api.kubernetes.example.ru","masterInternalName":"api.internal.kubernetes.example.ru","networkCIDR":"172.20.0.0/16","topology":{"masters":"public","nodes":"public","dns":{"type":"Public"}},"secretStore":"s3://kubernetes-example/kubernetes.example.ru/secrets","keyStore":"s3://kubernetes-example/kubernetes.example.ru/pki","configStore":"s3://kubernetes-example/kubernetes.example.ru","dnsZone":"Z3CEXJAAQ67LHN","clusterDNSDomain":"cluster.local","serviceClusterIPRange":"100.64.0.0/13","nonMasqueradeCIDR":"100.64.0.0/10","sshAccess":["0.0.0.0/0"],"kubernetesApiAccess":["0.0.0.0/0"],"etcdClusters":[{"name":"main","etcdMembers":[{"name":"a","instanceGroup":"master-eu-central-1a"}]},{"name":"events","etcdMembers":[{"name":"a","instanceGroup":"master-eu-central-1a"}]}],"docker":{"bridge":"","logLevel":"warn","ipTables":false,"ipMasq":false,"logDriver":"json-file","logOpt":["max-size=10m","max-file=5"],"storage":"overlay,aufs","version":"1.12.6"},"kubeDNS":{"image":"gcr.io/google_containers/kubedns-amd64:1.3","replicas":2,"domain":"cluster.local","serverIP":"100.64.0.10"},"kubeAPIServer":{"image":"gcr.io/google_containers/kube-apiserver:v1.7.4","logLevel":2,"cloudProvider":"aws","securePort":443,"insecurePort":8080,"address":"127.0.0.1","etcdServers":["http://127.0.0.1:4001"],"etcdServersOverrides":["/events#http://127.0.0.1:4002"],"admissionControl":["NamespaceLifecycle","LimitRanger","ServiceAccount","PersistentVolumeLabel","DefaultStorageClass","DefaultTolerationSeconds","ResourceQuota"],"serviceClusterIPRange":"100.64.0.0/13","allowPrivileged":true,"apiServerCount":1,"anonymousAuth":false,"kubeletPreferredAddressTypes":["InternalIP","Hostname","ExternalIP"],"storageBackend":"etcd2","authorizationMode":"AlwaysAllow"},"kubeControllerManager":{"logLevel":2,"image":"gcr.io/google_containers/kube-controller-manager:v1.7.4","cloudProvider":"aws","clusterName":"kubernetes.example.ru","clusterCIDR":"100.96.0.0/11","allocateNodeCIDRs":true,"configureCloudRoutes":true,"leaderElection":{"leaderElect":true},"attachDetachReconcileSyncPeriod":"1m0s","useServiceAccountCredentials":true},"kubeScheduler":{"logLevel":2,"image":"gcr.io/google_containers/kube-scheduler:v1.7.4","leaderElection":{"leaderElect":true}},"kubeProxy":{"image":"gcr.io/google_containers/kube-proxy:v1.7.4","cpuRequest":"100m","logLevel":2,"clusterCIDR":"100.96.0.0/11","hostnameOverride":"@aws"},"kubelet":{"kubeconfigPath":"/var/lib/kubelet/kubeconfig","requireKubeconfig":true,"logLevel":2,"podManifestPath":"/etc/kubernetes/manifests","hostnameOverride":"@aws","podInfraContainerImage":"gcr.io/google_containers/pause-amd64:3.0","allowPrivileged":true,"enableDebuggingHandlers":true,"clusterDomain":"cluster.local","clusterDNS":"100.64.0.10","networkPluginName":"kubenet","cloudProvider":"aws","cgroupRoot":"/","nonMasqueradeCIDR":"100.64.0.0/10","networkPluginMTU":9001,"evictionHard":"memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"},"masterKubelet":{"kubeconfigPath":"/var/lib/kubelet/kubeconfig","requireKubeconfig":true,"logLevel":2,"podManifestPath":"/etc/kubernetes/manifests","hostnameOverride":"@aws","podInfraContainerImage":"gcr.io/google_containers/pause-amd64:3.0","allowPrivileged":true,"enableDebuggingHandlers":true,"clusterDomain":"cluster.local","clusterDNS":"100.64.0.10","networkPluginName":"kubenet","cloudProvider":"aws","cgroupRoot":"/","registerSchedulable":false,"nonMasqueradeCIDR":"100.64.0.0/10","networkPluginMTU":9001,"evictionHard":"memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"},"networking":{"kubenet":{}},"api":{"dns":{}},"authorization":{"alwaysAllow":{}}}},"Role":"Node","Region":"eu-central-1","HostedZoneID":""},"DNSZone":{"Name":"Z3CEXJAAQ67LHN","Lifecycle":"Sync","DNSName":"kubernetes.example.ru","ZoneID":"Z3CEXJAAQ67LHN","Private":null,"PrivateVPC":null}}}
I1011 18:03:37.471785   47617 request_logger.go:45] AWS request: ec2/DescribeInternetGateways
I1011 18:03:37.471523   47617 request_logger.go:45] AWS request: iam/GetInstanceProfile
I1011 18:03:37.471208   47617 executor.go:157] Executing task "RouteTable/kubernetes.example.ru": *awstasks.RouteTable {"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":null,"VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}}
I1011 18:03:37.471572   47617 request_logger.go:45] AWS request: ec2/DescribeSubnets
I1011 18:03:37.471439   47617 request_logger.go:45] AWS request: iam/GetInstanceProfile
I1011 18:03:37.471219   47617 executor.go:157] Executing task "IAMRolePolicy/masters.kubernetes.example.ru": *awstasks.IAMRolePolicy {"ID":null,"Lifecycle":"Sync","Name":"masters.kubernetes.example.ru","Role":{"ID":"AROAJ7UNLOO6IV3MAAAZA","Lifecycle":"Sync","Name":"masters.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"masters"},"PolicyDocument":{"Builder":{"Cluster":{"metadata":{"name":"kubernetes.example.ru","creationTimestamp":"2017-10-11T13:57:53Z"},"spec":{"channel":"stable","configBase":"s3://kubernetes-example/kubernetes.example.ru","cloudProvider":"aws","kubernetesVersion":"1.7.4","subnets":[{"name":"eu-central-1a","zone":"eu-central-1a","cidr":"172.20.32.0/19","type":"Public"},{"name":"eu-central-1b","zone":"eu-central-1b","cidr":"172.20.64.0/19","type":"Public"},{"name":"eu-central-1c","zone":"eu-central-1c","cidr":"172.20.96.0/19","type":"Public"}],"masterPublicName":"api.kubernetes.example.ru","masterInternalName":"api.internal.kubernetes.example.ru","networkCIDR":"172.20.0.0/16","topology":{"masters":"public","nodes":"public","dns":{"type":"Public"}},"secretStore":"s3://kubernetes-example/kubernetes.example.ru/secrets","keyStore":"s3://kubernetes-example/kubernetes.example.ru/pki","configStore":"s3://kubernetes-example/kubernetes.example.ru","dnsZone":"Z3CEXJAAQ67LHN","clusterDNSDomain":"cluster.local","serviceClusterIPRange":"100.64.0.0/13","nonMasqueradeCIDR":"100.64.0.0/10","sshAccess":["0.0.0.0/0"],"kubernetesApiAccess":["0.0.0.0/0"],"etcdClusters":[{"name":"main","etcdMembers":[{"name":"a","instanceGroup":"master-eu-central-1a"}]},{"name":"events","etcdMembers":[{"name":"a","instanceGroup":"master-eu-central-1a"}]}],"docker":{"bridge":"","logLevel":"warn","ipTables":false,"ipMasq":false,"logDriver":"json-file","logOpt":["max-size=10m","max-file=5"],"storage":"overlay,aufs","version":"1.12.6"},"kubeDNS":{"image":"gcr.io/google_containers/kubedns-amd64:1.3","replicas":2,"domain":"cluster.local","serverIP":"100.64.0.10"},"kubeAPIServer":{"image":"gcr.io/google_containers/kube-apiserver:v1.7.4","logLevel":2,"cloudProvider":"aws","securePort":443,"insecurePort":8080,"address":"127.0.0.1","etcdServers":["http://127.0.0.1:4001"],"etcdServersOverrides":["/events#http://127.0.0.1:4002"],"admissionControl":["NamespaceLifecycle","LimitRanger","ServiceAccount","PersistentVolumeLabel","DefaultStorageClass","DefaultTolerationSeconds","ResourceQuota"],"serviceClusterIPRange":"100.64.0.0/13","allowPrivileged":true,"apiServerCount":1,"anonymousAuth":false,"kubeletPreferredAddressTypes":["InternalIP","Hostname","ExternalIP"],"storageBackend":"etcd2","authorizationMode":"AlwaysAllow"},"kubeControllerManager":{"logLevel":2,"image":"gcr.io/google_containers/kube-controller-manager:v1.7.4","cloudProvider":"aws","clusterName":"kubernetes.example.ru","clusterCIDR":"100.96.0.0/11","allocateNodeCIDRs":true,"configureCloudRoutes":true,"leaderElection":{"leaderElect":true},"attachDetachReconcileSyncPeriod":"1m0s","useServiceAccountCredentials":true},"kubeScheduler":{"logLevel":2,"image":"gcr.io/google_containers/kube-scheduler:v1.7.4","leaderElection":{"leaderElect":true}},"kubeProxy":{"image":"gcr.io/google_containers/kube-proxy:v1.7.4","cpuRequest":"100m","logLevel":2,"clusterCIDR":"100.96.0.0/11","hostnameOverride":"@aws"},"kubelet":{"kubeconfigPath":"/var/lib/kubelet/kubeconfig","requireKubeconfig":true,"logLevel":2,"podManifestPath":"/etc/kubernetes/manifests","hostnameOverride":"@aws","podInfraContainerImage":"gcr.io/google_containers/pause-amd64:3.0","allowPrivileged":true,"enableDebuggingHandlers":true,"clusterDomain":"cluster.local","clusterDNS":"100.64.0.10","networkPluginName":"kubenet","cloudProvider":"aws","cgroupRoot":"/","nonMasqueradeCIDR":"100.64.0.0/10","networkPluginMTU":9001,"evictionHard":"memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"},"masterKubelet":{"kubeconfigPath":"/var/lib/kubelet/kubeconfig","requireKubeconfig":true,"logLevel":2,"podManifestPath":"/etc/kubernetes/manifests","hostnameOverride":"@aws","podInfraContainerImage":"gcr.io/google_containers/pause-amd64:3.0","allowPrivileged":true,"enableDebuggingHandlers":true,"clusterDomain":"cluster.local","clusterDNS":"100.64.0.10","networkPluginName":"kubenet","cloudProvider":"aws","cgroupRoot":"/","registerSchedulable":false,"nonMasqueradeCIDR":"100.64.0.0/10","networkPluginMTU":9001,"evictionHard":"memory.available\u003c100Mi,nodefs.available\u003c10%,nodefs.inodesFree\u003c5%,imagefs.available\u003c10%,imagefs.inodesFree\u003c5%"},"networking":{"kubenet":{}},"api":{"dns":{}},"authorization":{"alwaysAllow":{}}}},"Role":"Master","Region":"eu-central-1","HostedZoneID":""},"DNSZone":{"Name":"Z3CEXJAAQ67LHN","Lifecycle":"Sync","DNSName":"kubernetes.example.ru","ZoneID":"Z3CEXJAAQ67LHN","Private":null,"PrivateVPC":null}}}
I1011 18:03:37.471189   47617 executor.go:157] Executing task "IAMRolePolicy/additional.masters.kubernetes.example.ru": *awstasks.IAMRolePolicy {"ID":null,"Lifecycle":"Sync","Name":"additional.masters.kubernetes.example.ru","Role":{"ID":"AROAJ7UNLOO6IV3MAAAZA","Lifecycle":"Sync","Name":"masters.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"masters"},"PolicyDocument":{"Name":"","Resource":""}}
I1011 18:03:37.471262   47617 executor.go:157] Executing task "Subnet/eu-central-1b.kubernetes.example.ru": *awstasks.Subnet {"Name":"eu-central-1b.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1b","CIDR":"172.20.64.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1b.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:37.472398   47617 request_logger.go:45] AWS request: iam/GetRolePolicy
I1011 18:03:37.472454   47617 request_logger.go:45] AWS request: iam/GetRolePolicy
I1011 18:03:37.472481   47617 request_logger.go:45] AWS request: ec2/DescribeSubnets
I1011 18:03:37.471584   47617 executor.go:157] Executing task "IAMRolePolicy/additional.nodes.kubernetes.example.ru": *awstasks.IAMRolePolicy {"ID":null,"Lifecycle":"Sync","Name":"additional.nodes.kubernetes.example.ru","Role":{"ID":"AROAI5YUX6GTAMWTAJ67A","Lifecycle":"Sync","Name":"nodes.kubernetes.example.ru","RolePolicyDocument":{"Name":"","Resource":{}},"ExportWithID":"nodes"},"PolicyDocument":{"Name":"","Resource":""}}
I1011 18:03:37.472601   47617 request_logger.go:45] AWS request: iam/GetRolePolicy
I1011 18:03:37.471597   47617 executor.go:157] Executing task "Subnet/eu-central-1a.kubernetes.example.ru": *awstasks.Subnet {"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:37.471588   47617 executor.go:157] Executing task "VPCDHCPOptionsAssociation/kubernetes.example.ru": *awstasks.VPCDHCPOptionsAssociation {"Name":"kubernetes.example.ru","Lifecycle":"Sync","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"DHCPOptions":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"dopt-7bbe9113","DomainName":"eu-central-1.compute.internal","DomainNameServers":"AmazonProvidedDNS"}}
I1011 18:03:37.471628   47617 executor.go:157] Executing task "SecurityGroup/masters.kubernetes.example.ru": *awstasks.SecurityGroup {"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}
I1011 18:03:37.471641   47617 executor.go:157] Executing task "SecurityGroup/nodes.kubernetes.example.ru": *awstasks.SecurityGroup {"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null}
I1011 18:03:37.472139   47617 request_logger.go:45] AWS request: iam/GetRolePolicy
I1011 18:03:37.472186   47617 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I1011 18:03:37.472745   47617 aws_cloud.go:556] Calling DescribeVPC for VPC "vpc-483c0720"
I1011 18:03:37.472791   47617 request_logger.go:45] AWS request: ec2/DescribeSubnets
I1011 18:03:37.473002   47617 request_logger.go:45] AWS request: ec2/DescribeVpcs
I1011 18:03:37.473129   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.473134   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.565815   47617 internetgateway.go:94] found matching InternetGateway "igw-16c2107e"
I1011 18:03:37.605063   47617 subnet.go:81] found matching subnet "subnet-068d4b4b"
I1011 18:03:37.605097   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:37.605106   47617 changes.go:148] comparing maps: kubernetes.io/cluster/kubernetes.example.ru owned owned
I1011 18:03:37.605112   47617 changes.go:148] comparing maps: Name eu-central-1c.kubernetes.example.ru eu-central-1c.kubernetes.example.ru
I1011 18:03:37.639203   47617 subnet.go:81] found matching subnet "subnet-f7a4a88d"
I1011 18:03:37.639241   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:37.639251   47617 changes.go:148] comparing maps: Name eu-central-1b.kubernetes.example.ru eu-central-1b.kubernetes.example.ru
I1011 18:03:37.639258   47617 changes.go:148] comparing maps: kubernetes.io/cluster/kubernetes.example.ru owned owned
I1011 18:03:37.712201   47617 routetable.go:61] found matching RouteTable "rtb-ce6aa2a5"
I1011 18:03:37.712236   47617 subnet.go:81] found matching subnet "subnet-ef844a84"
I1011 18:03:37.712262   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:37.712272   47617 changes.go:148] comparing maps: Name eu-central-1a.kubernetes.example.ru eu-central-1a.kubernetes.example.ru
I1011 18:03:37.712278   47617 changes.go:148] comparing maps: kubernetes.io/cluster/kubernetes.example.ru owned owned
I1011 18:03:37.777364   47617 securitygroup.go:79] found matching SecurityGroup "sg-72c61318"
I1011 18:03:37.777407   47617 changes.go:173] comparing slices: 0 port=22 port=22
I1011 18:03:37.777374   47617 securitygroup.go:79] found matching SecurityGroup "sg-d6c411bc"
I1011 18:03:37.777449   47617 changes.go:173] comparing slices: 0 port=22 port=22
I1011 18:03:37.777459   47617 changes.go:173] comparing slices: 1 port=443 port=443
I1011 18:03:37.777504   47617 changes.go:173] comparing slices: 2 port=4001 port=4001
I1011 18:03:37.777516   47617 changes.go:173] comparing slices: 3 port=4789 port=4789
I1011 18:03:37.777522   47617 changes.go:173] comparing slices: 4 port=179 port=179
I1011 18:03:37.777536   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.777612   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.800247   47617 iam_builder.go:216] Ignoring location "s3://kubernetes-example/kubernetes.example.ru/pki/" because found parent "s3://kubernetes-example/kubernetes.example.ru/"
I1011 18:03:37.800272   47617 iam_builder.go:216] Ignoring location "s3://kubernetes-example/kubernetes.example.ru/secrets/" because found parent "s3://kubernetes-example/kubernetes.example.ru/"
I1011 18:03:37.800282   47617 iam_builder.go:221] Found root location "s3://kubernetes-example/kubernetes.example.ru/"
I1011 18:03:37.875520   47617 securitygroup.go:411] Ignoring security group permission "{\n  FromPort: 1,\n  IpProtocol: \"tcp\",\n  ToPort: 4000,\n  UserIdGroupPairs: [{\n      GroupId: \"sg-72c61318\",\n      UserId: \"426222316478\"\n    }]\n}" (did not match removal rules)
I1011 18:03:37.876432   47617 securitygroup.go:411] Ignoring security group permission "{\n  FromPort: 4003,\n  IpProtocol: \"tcp\",\n  ToPort: 65535,\n  UserIdGroupPairs: [{\n      GroupId: \"sg-72c61318\",\n      UserId: \"426222316478\"\n    }]\n}" (did not match removal rules)
I1011 18:03:37.876475   47617 securitygroup.go:411] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-d6c411bc\",\n      UserId: \"426222316478\"\n    }]\n}" (did not match removal rules)
I1011 18:03:37.876496   47617 securitygroup.go:405] permission matches rule {"Port":22}: {
  FromPort: 22,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0"
    }],
  ToPort: 22
}
I1011 18:03:37.876581   47617 securitygroup.go:411] Ignoring security group permission "{\n  FromPort: 1,\n  IpProtocol: \"udp\",\n  ToPort: 65535,\n  UserIdGroupPairs: [{\n      GroupId: \"sg-72c61318\",\n      UserId: \"426222316478\"\n    }]\n}" (did not match removal rules)
I1011 18:03:37.876604   47617 securitygroup.go:405] permission matches rule {"Port":443}: {
  FromPort: 443,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0"
    }],
  ToPort: 443
}
I1011 18:03:37.877282   47617 securitygroup.go:411] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-72c61318\",\n      UserId: \"426222316478\"\n    }]\n}" (did not match removal rules)
I1011 18:03:37.877316   47617 securitygroup.go:411] Ignoring security group permission "{\n  IpProtocol: \"-1\",\n  UserIdGroupPairs: [{\n      GroupId: \"sg-d6c411bc\",\n      UserId: \"426222316478\"\n    }]\n}" (did not match removal rules)
I1011 18:03:37.877336   47617 securitygroup.go:405] permission matches rule {"Port":22}: {
  FromPort: 22,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0"
    }],
  ToPort: 22
}
I1011 18:03:37.965600   47617 iam_builder.go:216] Ignoring location "s3://kubernetes-example/kubernetes.example.ru/pki/" because found parent "s3://kubernetes-example/kubernetes.example.ru/"
I1011 18:03:37.965629   47617 iam_builder.go:216] Ignoring location "s3://kubernetes-example/kubernetes.example.ru/secrets/" because found parent "s3://kubernetes-example/kubernetes.example.ru/"
I1011 18:03:37.965638   47617 iam_builder.go:221] Found root location "s3://kubernetes-example/kubernetes.example.ru/"
I1011 18:03:37.965756   47617 executor.go:91] Tasks: 48 done / 67 total; 17 can run
I1011 18:03:37.965792   47617 executor.go:157] Executing task "SecurityGroupRule/node-to-master-tcp-4003-65535": *awstasks.SecurityGroupRule {"Name":"node-to-master-tcp-4003-65535","Lifecycle":"Sync","SecurityGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":"tcp","FromPort":4003,"ToPort":65535,"SourceGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I1011 18:03:37.966104   47617 executor.go:157] Executing task "SecurityGroupRule/node-to-master-tcp-1-4000": *awstasks.SecurityGroupRule {"Name":"node-to-master-tcp-1-4000","Lifecycle":"Sync","SecurityGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":"tcp","FromPort":1,"ToPort":4000,"SourceGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I1011 18:03:37.965817   47617 executor.go:157] Executing task "SecurityGroupRule/ssh-external-to-master-0.0.0.0/0": *awstasks.SecurityGroupRule {"Name":"ssh-external-to-master-0.0.0.0/0","Lifecycle":"Sync","SecurityGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":"tcp","FromPort":22,"ToPort":22,"SourceGroup":null,"Egress":null}
I1011 18:03:37.966421   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.966431   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.965934   47617 executor.go:157] Executing task "SecurityGroupRule/node-egress": *awstasks.SecurityGroupRule {"Name":"node-egress","Lifecycle":"Sync","SecurityGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":null,"Egress":true}
I1011 18:03:37.966277   47617 executor.go:157] Executing task "Route/0.0.0.0/0": *awstasks.Route {"Name":"0.0.0.0/0","Lifecycle":"Sync","RouteTable":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"rtb-ce6aa2a5","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}},"Instance":null,"CIDR":"0.0.0.0/0","InternetGateway":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"igw-16c2107e","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"Shared":false},"NatGateway":null}
I1011 18:03:37.966558   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.966027   47617 executor.go:157] Executing task "SecurityGroupRule/all-master-to-master": *awstasks.SecurityGroupRule {"Name":"all-master-to-master","Lifecycle":"Sync","SecurityGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"Egress":null}
I1011 18:03:37.966057   47617 executor.go:157] Executing task "RouteTableAssociation/eu-central-1c.kubernetes.example.ru": *awstasks.RouteTableAssociation {"Name":"eu-central-1c.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"RouteTable":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"rtb-ce6aa2a5","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}},"Subnet":{"Name":"eu-central-1c.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-068d4b4b","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1c","CIDR":"172.20.96.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1c.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}}
I1011 18:03:37.966000   47617 executor.go:157] Executing task "LaunchConfiguration/nodes.kubernetes.example.ru": *awstasks.LaunchConfiguration {"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28","InstanceType":"t2.small","SSHKey":{"Name":"kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a","Lifecycle":"Sync","PublicKey":{"Name":"","Resource":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs1NT4mr0LJcDQ9wu4PbWphFIF9m1u0rXq2u5ZGJZjVSxIolS2/AuLYDhNjNEcJLtPOMKUvGof4A3ywL1WTV5ABGxS6EzQ5Wb2FOLgqqOcoMYdH4YkpPHUMr3PNTJT8/jKcjryxAndkstQpskVQ61DznDKBIRIMSnV7/I6IxqibBWzM1jJ+8ANAmEnRZlXBZlqJzp8RgiRPvXd9PiO0jJKPpOBl7YFW9D7HWKAOZzk82jMSGoT9QGvVka6cvJG1SPqMbKpyk/Qd8owAr1E5E291Q/E+1eyTxbVvvT8xvtPQN5CcskeVMe7ZAmPTkEo4ZngUKnzNhKfPmieZv/vO7bhWn0nly9 [email protected]\n"},"KeyFingerprint":"93:a9:a5:f0:77:55:30:2e:d2:16:bd:c5:10:3b:fc:cc"},"SecurityGroups":[{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null}],"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"AIPAJRVDU7OS5T3PYYYHI"},"RootVolumeSize":128,"RootVolumeType":"gp2","RootVolumeOptimization":null,"SpotPrice":"","ID":null,"Tenancy":null}
I1011 18:03:37.966100   47617 executor.go:157] Executing task "SecurityGroupRule/master-egress": *awstasks.SecurityGroupRule {"Name":"master-egress","Lifecycle":"Sync","SecurityGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":null,"Egress":true}
I1011 18:03:37.966045   47617 executor.go:157] Executing task "LaunchConfiguration/master-eu-central-1a.masters.kubernetes.example.ru": *awstasks.LaunchConfiguration {"Name":"master-eu-central-1a.masters.kubernetes.example.ru","Lifecycle":"Sync","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28","InstanceType":"m3.medium","SSHKey":{"Name":"kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a","Lifecycle":"Sync","PublicKey":{"Name":"","Resource":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs1NT4mr0LJcDQ9wu4PbWphFIF9m1u0rXq2u5ZGJZjVSxIolS2/AuLYDhNjNEcJLtPOMKUvGof4A3ywL1WTV5ABGxS6EzQ5Wb2FOLgqqOcoMYdH4YkpPHUMr3PNTJT8/jKcjryxAndkstQpskVQ61DznDKBIRIMSnV7/I6IxqibBWzM1jJ+8ANAmEnRZlXBZlqJzp8RgiRPvXd9PiO0jJKPpOBl7YFW9D7HWKAOZzk82jMSGoT9QGvVka6cvJG1SPqMbKpyk/Qd8owAr1E5E291Q/E+1eyTxbVvvT8xvtPQN5CcskeVMe7ZAmPTkEo4ZngUKnzNhKfPmieZv/vO7bhWn0nly9 [email protected]\n"},"KeyFingerprint":"93:a9:a5:f0:77:55:30:2e:d2:16:bd:c5:10:3b:fc:cc"},"SecurityGroups":[{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}],"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"AIPAJIL5CNLVETH4PZEA2"},"RootVolumeSize":64,"RootVolumeType":"gp2","RootVolumeOptimization":null,"SpotPrice":"","ID":null,"Tenancy":null}
I1011 18:03:37.967173   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.965799   47617 executor.go:157] Executing task "SecurityGroupRule/all-node-to-node": *awstasks.SecurityGroupRule {"Name":"all-node-to-node","Lifecycle":"Sync","SecurityGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I1011 18:03:37.966183   47617 executor.go:157] Executing task "SecurityGroupRule/all-master-to-node": *awstasks.SecurityGroupRule {"Name":"all-master-to-node","Lifecycle":"Sync","SecurityGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":null,"Protocol":null,"FromPort":null,"ToPort":null,"SourceGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"Egress":null}
I1011 18:03:37.967381   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.967430   47617 request_logger.go:45] AWS request: autoscaling/DescribeLaunchConfigurations
I1011 18:03:37.967438   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.966237   47617 executor.go:157] Executing task "SecurityGroupRule/https-external-to-master-0.0.0.0/0": *awstasks.SecurityGroupRule {"Name":"https-external-to-master-0.0.0.0/0","Lifecycle":"Sync","SecurityGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":"tcp","FromPort":443,"ToPort":443,"SourceGroup":null,"Egress":null}
I1011 18:03:37.967496   47617 request_logger.go:45] AWS request: autoscaling/DescribeLaunchConfigurations
I1011 18:03:37.967595   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.965828   47617 executor.go:157] Executing task "RouteTableAssociation/eu-central-1b.kubernetes.example.ru": *awstasks.RouteTableAssociation {"Name":"eu-central-1b.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"RouteTable":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"rtb-ce6aa2a5","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}},"Subnet":{"Name":"eu-central-1b.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-f7a4a88d","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1b","CIDR":"172.20.64.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1b.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}}
I1011 18:03:37.966224   47617 executor.go:157] Executing task "RouteTableAssociation/eu-central-1a.kubernetes.example.ru": *awstasks.RouteTableAssociation {"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":null,"RouteTable":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"rtb-ce6aa2a5","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}},"Subnet":{"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-ef844a84","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}}
I1011 18:03:37.965940   47617 executor.go:157] Executing task "SecurityGroupRule/ssh-external-to-node-0.0.0.0/0": *awstasks.SecurityGroupRule {"Name":"ssh-external-to-node-0.0.0.0/0","Lifecycle":"Sync","SecurityGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"CIDR":"0.0.0.0/0","Protocol":"tcp","FromPort":22,"ToPort":22,"SourceGroup":null,"Egress":null}
I1011 18:03:37.967912   47617 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I1011 18:03:37.965994   47617 executor.go:157] Executing task "SecurityGroupRule/node-to-master-udp-1-65535": *awstasks.SecurityGroupRule {"Name":"node-to-master-udp-1-65535","Lifecycle":"Sync","SecurityGroup":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null},"CIDR":null,"Protocol":"udp","FromPort":1,"ToPort":65535,"SourceGroup":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null},"Egress":null}
I1011 18:03:37.968020   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.966664   47617 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I1011 18:03:37.966862   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.967154   47617 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I1011 18:03:37.966280   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:37.967984   47617 request_logger.go:45] AWS request: ec2/DescribeRouteTables
I1011 18:03:37.968102   47617 request_logger.go:45] AWS request: ec2/DescribeSecurityGroups
I1011 18:03:38.252952   47617 routetableassociation.go:81] found matching RouteTableAssociation "rtbassoc-0d784a66"
I1011 18:03:38.259217   47617 routetableassociation.go:81] found matching RouteTableAssociation "rtbassoc-c1794baa"
I1011 18:03:38.263003   47617 routetableassociation.go:81] found matching RouteTableAssociation "rtbassoc-6c7e4c07"
I1011 18:03:38.265803   47617 route.go:102] found route matching cidr 0.0.0.0/0
I1011 18:03:38.339990   47617 launchconfiguration.go:109] found existing AutoscalingLaunchConfiguration: "master-eu-central-1a.masters.kubernetes.example.ru-20171011135809"
I1011 18:03:38.340061   47617 aws_cloud.go:587] Calling DescribeImages to resolve name "kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28"
I1011 18:03:38.340198   47617 request_logger.go:45] AWS request: ec2/DescribeImages
I1011 18:03:38.421362   47617 launchconfiguration.go:109] found existing AutoscalingLaunchConfiguration: "nodes.kubernetes.example.ru-20171011135809"
I1011 18:03:38.421433   47617 aws_cloud.go:587] Calling DescribeImages to resolve name "kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28"
I1011 18:03:38.421555   47617 request_logger.go:45] AWS request: ec2/DescribeImages
I1011 18:03:38.435203   47617 aws_cloud.go:632] Resolved image "ami-0072de6f"
I1011 18:03:38.435232   47617 launchconfiguration.go:155] Returning matching ImageId as expected name: "ami-0072de6f" -> "kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28"
I1011 18:03:38.435315   47617 urls.go:85] Using default protokube location: "https://kubeupv2.s3.amazonaws.com/kops/1.7.0/images/protokube.tar.gz"
I1011 18:03:38.435338   47617 context.go:132] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.7.0/images/protokube.tar.gz.sha1
I1011 18:03:38.501126   47617 aws_cloud.go:632] Resolved image "ami-0072de6f"
I1011 18:03:38.501151   47617 launchconfiguration.go:155] Returning matching ImageId as expected name: "ami-0072de6f" -> "kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28"
I1011 18:03:38.501213   47617 context.go:132] Performing HTTP request: GET https://kubeupv2.s3.amazonaws.com/kops/1.7.0/images/protokube.tar.gz.sha1
I1011 18:03:38.600282   47617 apply_cluster.go:803] Found hash "5bd97a02f0793d1906e9f446c548ececf1444737" for "https://kubeupv2.s3.amazonaws.com/kops/1.7.0/images/protokube.tar.gz"
I1011 18:03:38.600806   47617 changes.go:173] comparing slices: 0 *awstasks.SecurityGroup {"Name":null,"Lifecycle":null,"ID":"sg-d6c411bc","Description":null,"VPC":null,"RemoveExtraRules":null,"Shared":null} *awstasks.SecurityGroup {"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}
I1011 18:03:38.759495   47617 apply_cluster.go:803] Found hash "5bd97a02f0793d1906e9f446c548ececf1444737" for "https://kubeupv2.s3.amazonaws.com/kops/1.7.0/images/protokube.tar.gz"
I1011 18:03:38.759859   47617 changes.go:173] comparing slices: 0 *awstasks.SecurityGroup {"Name":null,"Lifecycle":null,"ID":"sg-72c61318","Description":null,"VPC":null,"RemoveExtraRules":null,"Shared":null} *awstasks.SecurityGroup {"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null}
I1011 18:03:38.759933   47617 executor.go:91] Tasks: 65 done / 67 total; 2 can run
I1011 18:03:38.759959   47617 executor.go:157] Executing task "AutoscalingGroup/master-eu-central-1a.masters.kubernetes.example.ru": *awstasks.AutoscalingGroup {"Name":"master-eu-central-1a.masters.kubernetes.example.ru","Lifecycle":"Sync","MinSize":1,"MaxSize":1,"Subnets":[{"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-ef844a84","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}],"Tags":{"k8s.io/role/master":"1"},"LaunchConfiguration":{"Name":"master-eu-central-1a.masters.kubernetes.example.ru","Lifecycle":"Sync","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28","InstanceType":"m3.medium","SSHKey":{"Name":"kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a","Lifecycle":"Sync","PublicKey":{"Name":"","Resource":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs1NT4mr0LJcDQ9wu4PbWphFIF9m1u0rXq2u5ZGJZjVSxIolS2/AuLYDhNjNEcJLtPOMKUvGof4A3ywL1WTV5ABGxS6EzQ5Wb2FOLgqqOcoMYdH4YkpPHUMr3PNTJT8/jKcjryxAndkstQpskVQ61DznDKBIRIMSnV7/I6IxqibBWzM1jJ+8ANAmEnRZlXBZlqJzp8RgiRPvXd9PiO0jJKPpOBl7YFW9D7HWKAOZzk82jMSGoT9QGvVka6cvJG1SPqMbKpyk/Qd8owAr1E5E291Q/E+1eyTxbVvvT8xvtPQN5CcskeVMe7ZAmPTkEo4ZngUKnzNhKfPmieZv/vO7bhWn0nly9 [email protected]\n"},"KeyFingerprint":"93:a9:a5:f0:77:55:30:2e:d2:16:bd:c5:10:3b:fc:cc"},"SecurityGroups":[{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-d6c411bc","Description":"Security group for masters","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22","port=443","port=4001","port=4789","port=179"],"Shared":null}],"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"masters.kubernetes.example.ru","Lifecycle":"Sync","ID":"AIPAJIL5CNLVETH4PZEA2"},"RootVolumeSize":64,"RootVolumeType":"gp2","RootVolumeOptimization":null,"SpotPrice":"","ID":"master-eu-central-1a.masters.kubernetes.example.ru-20171011135809","Tenancy":null}}
I1011 18:03:38.759986   47617 executor.go:157] Executing task "AutoscalingGroup/nodes.kubernetes.example.ru": *awstasks.AutoscalingGroup {"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","MinSize":2,"MaxSize":2,"Subnets":[{"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-ef844a84","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},{"Name":"eu-central-1b.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-f7a4a88d","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1b","CIDR":"172.20.64.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1b.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},{"Name":"eu-central-1c.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-068d4b4b","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1c","CIDR":"172.20.96.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1c.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}],"Tags":{"k8s.io/role/node":"1"},"LaunchConfiguration":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","UserData":{"Name":"","Resource":{}},"ImageID":"kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28","InstanceType":"t2.small","SSHKey":{"Name":"kubernetes.kubernetes.example.ru-76:a1:ab:59:6e:b5:51:44:ca:36:88:21:87:26:7d:5a","Lifecycle":"Sync","PublicKey":{"Name":"","Resource":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs1NT4mr0LJcDQ9wu4PbWphFIF9m1u0rXq2u5ZGJZjVSxIolS2/AuLYDhNjNEcJLtPOMKUvGof4A3ywL1WTV5ABGxS6EzQ5Wb2FOLgqqOcoMYdH4YkpPHUMr3PNTJT8/jKcjryxAndkstQpskVQ61DznDKBIRIMSnV7/I6IxqibBWzM1jJ+8ANAmEnRZlXBZlqJzp8RgiRPvXd9PiO0jJKPpOBl7YFW9D7HWKAOZzk82jMSGoT9QGvVka6cvJG1SPqMbKpyk/Qd8owAr1E5E291Q/E+1eyTxbVvvT8xvtPQN5CcskeVMe7ZAmPTkEo4ZngUKnzNhKfPmieZv/vO7bhWn0nly9 [email protected]\n"},"KeyFingerprint":"93:a9:a5:f0:77:55:30:2e:d2:16:bd:c5:10:3b:fc:cc"},"SecurityGroups":[{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"sg-72c61318","Description":"Security group for nodes","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"RemoveExtraRules":["port=22"],"Shared":null}],"AssociatePublicIP":true,"IAMInstanceProfile":{"Name":"nodes.kubernetes.example.ru","Lifecycle":"Sync","ID":"AIPAJRVDU7OS5T3PYYYHI"},"RootVolumeSize":128,"RootVolumeType":"gp2","RootVolumeOptimization":null,"SpotPrice":"","ID":"nodes.kubernetes.example.ru-20171011135809","Tenancy":null}}
I1011 18:03:38.760311   47617 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I1011 18:03:38.760538   47617 request_logger.go:45] AWS request: autoscaling/DescribeAutoScalingGroups
I1011 18:03:38.894939   47617 changes.go:173] comparing slices: 0 *awstasks.Subnet {"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-ef844a84","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}} *awstasks.Subnet {"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-ef844a84","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:38.894960   47617 changes.go:173] comparing slices: 0 *awstasks.Subnet {"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-ef844a84","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}} *awstasks.Subnet {"Name":"eu-central-1a.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-ef844a84","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1a","CIDR":"172.20.32.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1a.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:38.895124   47617 changes.go:148] comparing maps: k8s.io/role/master 1 1
I1011 18:03:38.895138   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:38.895145   47617 changes.go:148] comparing maps: Name master-eu-central-1a.masters.kubernetes.example.ru master-eu-central-1a.masters.kubernetes.example.ru
I1011 18:03:38.895045   47617 changes.go:173] comparing slices: 1 *awstasks.Subnet {"Name":"eu-central-1b.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-f7a4a88d","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1b","CIDR":"172.20.64.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1b.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}} *awstasks.Subnet {"Name":"eu-central-1b.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-f7a4a88d","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1b","CIDR":"172.20.64.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1b.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:38.895348   47617 changes.go:173] comparing slices: 2 *awstasks.Subnet {"Name":"eu-central-1c.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-068d4b4b","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1c","CIDR":"172.20.96.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1c.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}} *awstasks.Subnet {"Name":"eu-central-1c.kubernetes.example.ru","Lifecycle":"Sync","ID":"subnet-068d4b4b","VPC":{"Name":"kubernetes.example.ru","Lifecycle":"Sync","ID":"vpc-483c0720","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}},"AvailabilityZone":"eu-central-1c","CIDR":"172.20.96.0/19","Shared":false,"Tags":{"KubernetesCluster":"kubernetes.example.ru","Name":"eu-central-1c.kubernetes.example.ru","kubernetes.io/cluster/kubernetes.example.ru":"owned"}}
I1011 18:03:38.895416   47617 changes.go:148] comparing maps: KubernetesCluster kubernetes.example.ru kubernetes.example.ru
I1011 18:03:38.895425   47617 changes.go:148] comparing maps: Name nodes.kubernetes.example.ru nodes.kubernetes.example.ru
I1011 18:03:38.895431   47617 changes.go:148] comparing maps: k8s.io/role/node 1 1
I1011 18:03:38.895449   47617 executor.go:91] Tasks: 67 done / 67 total; 0 can run
I1011 18:03:38.895468   47617 dns.go:152] Pre-creating DNS records
I1011 18:03:38.895567   47617 route53.go:50] AWS request: route53 ListHostedZones
I1011 18:03:39.058537   47617 route53.go:50] AWS request: route53 ListResourceRecordSets
I1011 18:03:39.232971   47617 dns.go:194] Found DNS record api.kubernetes.example.ru. => [203.0.113.123]; won't create
I1011 18:03:39.233000   47617 dns.go:194] Found DNS record api.internal.kubernetes.example.ru. => [203.0.113.123]; won't create
I1011 18:03:39.233008   47617 dns.go:194] Found DNS record etcd-a.internal.kubernetes.example.ru. => [203.0.113.123]; won't create
I1011 18:03:39.233016   47617 dns.go:194] Found DNS record etcd-events-a.internal.kubernetes.example.ru. => [203.0.113.123]; won't create
I1011 18:03:39.233022   47617 context.go:79] deleting temp dir: "/var/folders/wl/rjkst1dd5rz1sk0l4v474n8m0000gn/T/deploy509325126"
I1011 18:03:39.240212   47617 kubectl.go:130] Running command: kubectl config view --output json
I1011 18:03:40.068806   47617 kubectl.go:82] config = "{\n    \"kind\": \"Config\",\n    \"apiVersion\": \"v1\",\n    \"preferences\": {},\n    \"clusters\": [\n        {\n            \"name\": \"kubernetes.example.ru\",\n            \"cluster\": {\n                \"server\": \"https://api.kubernetes.example.ru\",\n                \"certificate-authority-data\": \"REDACTED\"\n            }\n        }\n    ],\n    \"users\": [\n        {\n            \"name\": \"kubernetes.example.ru\",\n            \"user\": {\n                \"client-certificate-data\": \"REDACTED\",\n                \"client-key-data\": \"REDACTED\",\n                \"username\": \"admin\",\n                \"password\": \"CLqiIlMMXo3EqIJedKTzof8zfYJoy4jy\"\n            }\n        },\n        {\n            \"name\": \"kubernetes.example.ru-basic-auth\",\n            \"user\": {\n                \"username\": \"admin\",\n                \"password\": \"CLqiIlMMXo3EqIJedKTzof8zfYJoy4jy\"\n            }\n        }\n    ],\n    \"contexts\": [\n        {\n            \"name\": \"kubernetes.example.ru\",\n            \"context\": {\n                \"cluster\": \"kubernetes.example.ru\",\n                \"user\": \"kubernetes.example.ru\"\n            }\n        }\n    ],\n    \"current-context\": \"kubernetes.example.ru\"\n}"
I1011 18:03:40.091195   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kubecfg/"
I1011 18:03:40.169414   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg/6475652274088147560683131170.crt]
I1011 18:03:40.169446   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg/6475652274088147560683131170.crt"
I1011 18:03:40.245256   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:40.245384   47617 update_cluster.go:247] Exporting kubecfg for cluster
I1011 18:03:40.245422   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/ca/"
I1011 18:03:40.325716   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/ca: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/ca/6475652274709437460616198666.crt]
I1011 18:03:40.325741   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/ca/6475652274709437460616198666.crt"
I1011 18:03:40.404865   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:40.404976   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/ca/"
I1011 18:03:40.484766   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/ca: [s3://kubernetes-example/kubernetes.example.ru/pki/private/ca/6475652274709437460616198666.key]
I1011 18:03:40.484790   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/ca/6475652274709437460616198666.key"
I1011 18:03:40.562387   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:40.562806   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/issued/kubecfg/"
I1011 18:03:40.645590   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg: [s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg/6475652274088147560683131170.crt]
I1011 18:03:40.645617   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/issued/kubecfg/6475652274088147560683131170.crt"
I1011 18:03:40.724345   47617 ca.go:377] Parsing pem block: "CERTIFICATE"
I1011 18:03:40.724478   47617 s3fs.go:210] Listing objects in S3 bucket "kubernetes-example" with prefix "kubernetes.example.ru/pki/private/kubecfg/"
I1011 18:03:40.801166   47617 s3fs.go:236] Listed files in s3://kubernetes-example/kubernetes.example.ru/pki/private/kubecfg: [s3://kubernetes-example/kubernetes.example.ru/pki/private/kubecfg/6475652274088147560683131170.key]
I1011 18:03:40.801193   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/pki/private/kubecfg/6475652274088147560683131170.key"
I1011 18:03:40.878071   47617 ca.go:395] Parsing pem block: "RSA PRIVATE KEY"
I1011 18:03:40.878517   47617 s3fs.go:173] Reading file "s3://kubernetes-example/kubernetes.example.ru/secrets/kube"
I1011 18:03:41.005816   47617 loader.go:357] Config loaded from file /Users/burxanov/.kube/config
I1011 18:03:41.025097   47617 loader.go:357] Config loaded from file /Users/burxanov/.kube/config
Kops has set your kubectl context to kubernetes.example.ru

Cluster changes have been applied to the cloud.


Changes may require instances to restart: kops rolling-update cluster

lastonoga on 11 Oct 2017

@lastonoga In your aws console, what is the state of the autoscaling group? It seems like things are fine on the kops front. Kops creates launch configs and autoscaling groups to spin up instances. I would check to make sure they are scaling up properly.

mikesplain on 11 Oct 2017

@mikesplain How can i do that?

In tutorial it says
All instances created by kops will be built within ASG (Auto Scaling Groups), which means each instance will be automatically monitored and rebuilt by AWS if it suffers any failure.

lastonoga on 11 Oct 2017

mikesplain on 11 Oct 2017

👍2

@mikesplain Finally! Thanks. Problem with account.

lastonoga on 11 Oct 2017

@lastonoga Great!

mikesplain on 11 Oct 2017

@mikesplain Best UI ever.. Only small message somewhere deep. No notification at all even in dashboard.

lastonoga on 11 Oct 2017

👍1

@lastonoga @mikesplain Thank you, you just saved me hours of debugging.

ArneZsng on 17 Jan 2018

@ArneZsng You bet. I'm looking into adding more checks in validation for issues like this.

mikesplain on 17 Jan 2018

@lastonoga Could you provide more details of how to solve your problem? Thanks.

benzwjian on 7 Feb 2018

@benzwjian I created a support ticket to expand accounts limits. But firstly go to history of autoscaling and check that. There should be an error

lastonoga on 7 Feb 2018

I just tried __kops__ today and I just got this same issue. I was using a new AWS account, and I've seen that I have verification issues by viewing EC2 Console under Auto Scaling Events logs. It would be nice if there is a command like kops get ec2-events clusterName --watch :)