Kubevirt: Enhancement: kubevirt on ppc64le

[nitkon]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:
/kind enhancement

What happened:

I am trying to enable Kubevirt on ppc64le arch(nested VM not supported).

What you expected to happen:
Successfully deploy a VM in a K8s cluster via Kubevirt

How to reproduce it (as minimally and precisely as possible):

1. Get a working kubeadm cluster(1 master + 1 worker bare metal) on ppc64le
2. Set env variables as follows:

export DOCKER_PREFIX="index.docker.io/<reponame>"
export KUBECONFIG=/home/<name>/.kube/config
export IMAGE_PULL_POLICY=Always
export DOCKER_TAG=mybuild

3.

docker login -u <repo-name>

1. make && make push / make cluster-build
2. kubectl create -f _out/manifests/release/kubevirt-operator.yaml
3. kubectl apply -f myconfig.yaml

cat myconfig.yaml 
apiVersion: v1
data:
  dev.imagePullPolicy: Always
kind: ConfigMap
metadata: 
  name: kubevirt-config
  namespace: kubevirt

7) kubectl create -f _out/manifests/release/kubevirt-cr.yaml
8) Create a VMI kubectl create -f cluster/examples/vmi-ephemeral.yaml
9)

kubectl get pods
NAME                                READY   STATUS   RESTARTS   AGE
virt-launcher-vmi-ephemeral-cnpvt   0/2     Error    0          19h

kubectl logs virt-launcher-vmi-ephemeral-cnpvt 
Error from server (BadRequest): a container name must be specified for pod virt-launcher-vmi-ephemeral-cnpvt, choose one of: [volumecontainerdisk compute]

Kubectl get events

5m12s       Normal    Pulling            Pod                      pulling image "index.docker.io/niteshkonkar/virt-launcher:mybuild"
5m1s        Normal    Pulled             Pod                      Successfully pulled image "index.docker.io/niteshkonkar/virt-launcher:mybuild"
5m1s        Normal    Created            Pod                      Created container
5m1s        Normal    Started            Pod                      Started container
4m46s       Warning   Unhealthy          Pod                      Readiness probe failed: cat: /var/run/kubevirt-infra/healthy: No such file or directory
5m13s       Normal    SuccessfulCreate   VirtualMachineInstance   Created virtual machine pod virt-launcher-vmi-ephemeral-94hx6
4m44s       Warning   SyncFailed         VirtualMachineInstance   server error. command Launcher.Sync failed: "LibvirtError(Code=8, Domain=44, Message='invalid argument: could not find capabilities for arch=ppc64le domaintype=kvm ')"
0s    Warning   SyncFailed   VirtualMachineInstance   connection is shut down
0s    Normal   Killing   Pod   Killing container with id docker://volumecontainerdisk:Need to kill Pod

Anything else we need to know?:

Environment:

• KubeVirt version (use virtctl version): upstream master
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/ppc64le"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/ppc64le"}

• VM or VMI specifications:
• Cloud provider or hardware configuration:
  Master Node: Ubuntu 1604 VM
  Worker Node: IBM Power 8 System bare metal Ubuntu 1604
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

Linux ubuntu 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:47:54 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

• Install tools:
• Others:

[nitkon]

cc @rmohr @cynepco3hahue

[rmohr]

/cc @fabiand

[jeremyrogers]

Hey all,

We've got a lot of interest in our organization for running VMs on ppc64le. Does anyone have a status update on this?

/jeremy

[cynepco3hahue]

@jeremyrogers I know that @nitkon created some POC PR https://github.com/kubevirt/kubevirt/pull/2145, but I do not sure if he will continue with it :worried:
I will try to find some time the next week to start to finalize and polish the PR.

[dbenoit17]

I can pick this issue up. I'm not completely certain of the existing status of kubevirt on ppc64le, but I'll have some more time at the beginning of next week to really dig into these commits and get a feeling of what sort of enablement work has been done so far.

Thanks!

[cynepco3hahue]

@dbenoit17 It can be wonderful, and feel free to ping me or @rmohr under the slack k8s virtualization group.

[fabiand]

@jeremyrogers @nitkon @dbenoit17 what about syncing on a ml or zoom to sync on your different statuses?

[dbenoit17]

@fabiand Absolutely. Should we sync up on a weekly call, or set aside a dedicated time just for this?

[nitkon]

Hello All, I would like to provide the latest status update on this issue. I have updated my PR and I could deploy a cirros VM on ppc64le via Kubevirt.

Here are the steps I followed:

1. Setup was a 2 node kubeadm cluster. 1 master node VM and 1 bare metal Power system worker node (as nested VM not supported in ppc64le)

root@ubuntu:/home/nitesh# kubectl get nodes 
NAME               STATUS   ROLES    AGE   VERSION
Baremetal   Ready    <none>   10m   v1.14.0
ubuntu             Ready    master   11m   v1.14.0
root@ubuntu:/home/nitesh# kubectl get pods --all-namespaces
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-7wwsb          1/1     Running   0          11m
kube-system   coredns-fb8b8dccf-85wjb          1/1     Running   0          11m
kube-system   etcd-ubuntu                      1/1     Running   0          10m
kube-system   kube-apiserver-ubuntu            1/1     Running   0          10m
kube-system   kube-controller-manager-ubuntu   1/1     Running   0          10m
kube-system   kube-proxy-f2bmq                 1/1     Running   0          11m
kube-system   kube-proxy-zcbst                 1/1     Running   0          10m
kube-system   kube-scheduler-ubuntu            1/1     Running   0          11m
kube-system   weave-net-b92c2                  2/2     Running   0          10m
kube-system   weave-net-dnqpm                  2/2     Running   0          10m

1. kubectl create -f kubevirt.yaml

root@ubuntu:/home/nitesh/yaml_configs# kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-7wwsb            1/1     Running   0          17m
kube-system   coredns-fb8b8dccf-85wjb            1/1     Running   0          17m
kube-system   etcd-ubuntu                        1/1     Running   0          16m
kube-system   kube-apiserver-ubuntu              1/1     Running   0          16m
kube-system   kube-controller-manager-ubuntu     1/1     Running   0          16m
kube-system   kube-proxy-f2bmq                   1/1     Running   0          17m
kube-system   kube-proxy-zcbst                   1/1     Running   0          16m
kube-system   kube-scheduler-ubuntu              1/1     Running   0          17m
kube-system   weave-net-b92c2                    2/2     Running   0          16m
kube-system   weave-net-dnqpm                    2/2     Running   0          16m
kubevirt      virt-api-769f9d9f96-5qcsn          1/1     Running   0          4m33s
kubevirt      virt-api-769f9d9f96-btwnq          1/1     Running   0          4m33s
kubevirt      virt-controller-69c47cc76b-85wzw   1/1     Running   0          4m33s
kubevirt      virt-controller-69c47cc76b-f4cbp   1/1     Running   0          4m33s
kubevirt      virt-handler-cz8dn                 1/1     Running   0          4m33s

1. kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}'
   serviceaccount/default patched

2. kubectl create -f vmi-linux-ppc64le.yaml

root@ubuntu:/home/nitesh/yaml_configs# kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
virt-launcher-vmi-linux-ppc64le-stxw7   2/2     Running   0          111s

root@ubuntu:/home/nitesh/yaml_configs# kubectl get vmi
NAME                AGE     PHASE     IP          NODENAME
vmi-linux-ppc64le   2m42s   Running   10.40.0.8   baremetal123

1. virtctl console vmi-linux-ppc64le

Successfully connected to vmi-linux-ppc64le console. The escape sequence is ^]

login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros login: cirros
Password: 
$ uname -a
Linux cirros 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:20 UTC 2016 ppc64le GNU/Linux

kubevirt.yaml.log
vmi-linux-ppc64le.yaml.log

[fabiand]

/cc @dbenoit17

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[fabiand]

/remove-lifecycle stale

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[fabiand]

/remove-lifecycle rotten

[kubevirt-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

[kubevirt-bot]

@kubevirt-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.