External-storage: unable to provision with iscsi-targetd based on README.md

[gnitzan]

Hi,

I am trying to use the iscsi-provisioner base on the tutorial but it fails with the following, complaining that the file /var/run/secrets/iscsi-provisioner/session-chap-credential.properties not found.

I guess I am missing something?

kubectl describe pvc myclaim
Name:          myclaim
Namespace:     default
StorageClass:  iscsi-targetd-vg-targetd
Status:        Pending
Volume:        
Labels:        
Annotations:   control-plane.alpha.kubernetes.io/leader={"holderIdentity":"de499d5a-e500-11e7-b29d-0242ac110002","leaseDurationSeconds":15,"acquireTime":"2017-12-19T21:09:52Z","renewTime":"2017-12-19T21:09:54Z","lea...
               kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-class":"iscsi-targetd-vg-targetd"},"nam...
               volume.beta.kubernetes.io/storage-class=iscsi-targetd-vg-targetd
               volume.beta.kubernetes.io/storage-provisioner=iscsi-targetd
Capacity:      
Access Modes:  
Events:
  Type     Reason                Age              From                                                                                   Message
  ----     ------                ----             ----                                                                                   -------
  Normal   Provisioning          6s               iscsi-targetd iscsi-provisioner-5db567c895-sgggh de499d5a-e500-11e7-b29d-0242ac110002  External provisioner is provisioning volume for claim "default/myclaim"
  Warning  ProvisioningFailed    6s               iscsi-targetd iscsi-provisioner-5db567c895-sgggh de499d5a-e500-11e7-b29d-0242ac110002  Failed to provision volume with StorageClass "iscsi-targetd-vg-targetd": open /var/run/secrets/iscsi-provisioner/session-chap-credential.properties: no such file or directory
  Normal   ExternalProvisioning  5s (x3 over 7s)  persistentvolume-controller 

If I disable CHAP in storage class I get the following:

kubectl describe pvc myclaim
Name:          myclaim
Namespace:     default
StorageClass:  iscsi-targetd-vg-targetd
Status:        Pending
Volume:        
Labels:        
Annotations:   control-plane.alpha.kubernetes.io/leader={"holderIdentity":"de499d5a-e500-11e7-b29d-0242ac110002","leaseDurationSeconds":15,"acquireTime":"2017-12-19T22:02:45Z","renewTime":"2017-12-19T22:02:47Z","lea...
               kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-class":"iscsi-targetd-vg-targetd"},"nam...
               volume.beta.kubernetes.io/storage-class=iscsi-targetd-vg-targetd
               volume.beta.kubernetes.io/storage-provisioner=iscsi-targetd
Capacity:      
Access Modes:  
Events:
  Type     Reason                Age               From                                                                                   Message
  ----     ------                ----              ----                                                                                   -------
  Normal   ExternalProvisioning  8s (x4 over 15s)  persistentvolume-controller                                                            waiting for a volume to be created, either by external provisioner "iscsi-targetd" or manually created by system administrator
  Normal   Provisioning          0s (x2 over 14s)  iscsi-targetd iscsi-provisioner-5db567c895-sgggh de499d5a-e500-11e7-b29d-0242ac110002  External provisioner is provisioning volume for claim "default/myclaim"
  Warning  ProvisioningFailed    0s (x2 over 14s)  iscsi-targetd iscsi-provisioner-5db567c895-sgggh de499d5a-e500-11e7-b29d-0242ac110002  Failed to provision volume with StorageClass "iscsi-targetd-vg-targetd": {"code":-32603,"message":"bad HTTP Content-Type: text/html"}

Any help would be appreciated.

Best,
Gal

[feresberbeche]

I have this problem too,

[wongma7]

@raffaelespazzoli may be able to help
/area iscsi/targetd

[happyyangyuan]

Same problem. Any solutions?

[cat5]

I feel I have to do a me too in here as well. Got NFS provisioner working, but I need block storage, so trying to get up and running, and have the save issue when trying to provision a PVC:

Name:          iscsi-test
Namespace:     default
StorageClass:  iscsi-targetd-vg-targetd
Status:        Pending
Volume:
Labels:        <none>
Annotations:   control-plane.alpha.kubernetes.io/leader={"holderIdentity":"719f76eb-850c-11e8-bce5-e2071952f55f","leaseDurationSeconds":15,"acquireTime":"2018-07-11T15:30:02Z","renewTime":"2018-07-11T15:30:04Z","lea...
               volume.beta.kubernetes.io/storage-class=iscsi-targetd-vg-targetd
               volume.beta.kubernetes.io/storage-provisioner=iscsi-targetd
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
  Type     Reason                Age                From                                                                                   Message
  ----     ------                ----               ----                                                                                   -------
  Normal   ExternalProvisioning  10s (x3 over 12s)  persistentvolume-controller                                                            waiting for a volume to be created, either by external provisioner "iscsi-targetd" or manually created by system administrator
  Normal   Provisioning          2s (x2 over 12s)   iscsi-targetd iscsi-provisioner-6ccdc4b66c-b9fj2 719f76eb-850c-11e8-bce5-e2071952f55f  External provisioner is provisioning volume for claim "default/iscsi-test"
  Warning  ProvisioningFailed    2s (x2 over 12s)   iscsi-targetd iscsi-provisioner-6ccdc4b66c-b9fj2 719f76eb-850c-11e8-bce5-e2071952f55f  Failed to provision volume with StorageClass "iscsi-targetd-vg-targetd": open /var/run/secrets/iscsi-provisioner/session-chap-credential.properties: no such file or directory

Checking inside the pod, there is no mount for /var/run/secrets/iscsi-p.....

[root@k8s-master new]# kubectl exec -it iscsi-provisioner-6ccdc4b66c-b9fj2 -- bash
[root@iscsi-provisioner-6ccdc4b66c-b9fj2 /]# df -H
Filesystem               Size  Used Avail Use% Mounted on
overlay                   48G  6.8G   41G  15% /
tmpfs                    8.5G     0  8.5G   0% /dev
tmpfs                    8.5G     0  8.5G   0% /sys/fs/cgroup
/dev/mapper/centos-root   48G  6.8G   41G  15% /etc/hosts
shm                       68M     0   68M   0% /dev/shm
tmpfs                    8.5G   13k  8.5G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                    8.5G     0  8.5G   0% /proc/scsi
tmpfs                    8.5G     0  8.5G   0% /sys/firmware
[root@iscsi-provisioner-6ccdc4b66c-b9fj2 /]# ps -faux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        59  0.0  0.0  11788  2924 ?        Ss   15:35   0:00 bash
root        72  0.0  0.0  47456  3412 ?        R+   15:35   0:00  \_ ps -faux
root         1  0.0  0.1  36456 26016 ?        Ssl  13:15   0:01 /iscsi-controller start

Strange error, considering as part of the deployment, ENV vars are defined, and I can see them in the POD as well.

env | grep TARG
TARGETD_ADDRESS=192.168.0.11
TARGETD_USERNAME=test
TARGETD_PASSWORD=use16tocharacter

[lyyao09]

I have this problem too, and solved it by create a file session-chap-credential.properties manually just like this:
node.session.auth.username: YWRtaW4=
node.session.auth.password= Y2lhbw==
node.session.auth.username_in=
node.session.auth.password_in=

then, put this file to /var/run/secrets/iscsi-provisioner/ directory inside iscsi-provisioner pod, or volume it on host path.

I test it works well, and hope to be helpful.

[JaneWangQuest]

@lyyao09 how can you put the file into /var/run/secrets/iscsi-provisioner? Inside this container? I cannot create the missing folder iscsi-provisioner.

[lyyao09]

@JaneWangQuest For test, I just use command docker cp SRC_PATH CONTAINER:DEST_PATH to copy host file into container.
ps: directory /var/run/secrets/iscsi-provisioner/ must already exist before cp file.

[gnitzan]

left iscsi behind, too combersome. Started using Glusterfs with Heketi.

[w1ndy]

I managed to get it working by disabling chap authentication in the storage class configuration:

# whether or not to use chap authentication for discovery operations  
  chapAuthDiscovery: "false"

# whether or not to use chap authentication for session operations  
  chapAuthSession: "false"