Openebs: Use OpenEBS with Rancher 2 and PostgreSQL

[loxy]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT (not sure)

What happened:
I wanted to use openebs in rancher 2.0 to provision a PV for postgresql. The cluster is running fine. I installed openebs though the operator. This works. If I install postgresql (through helm, used a storage class of openebs-standard) the postgres container won't start remaining in status ContainerCreating.

This is the output of corresponding pod kubectl describe po/postgresql-5c7458bf4b-jlqfn -n postgresql-6bxwy8

Name:           postgresql-5c7458bf4b-jlqfn
Namespace:      postgresql-6bxwy8
Node:           vm2/195.201.218.102
Start Time:     Sat, 05 May 2018 12:30:18 +0000
Labels:         app=postgresql
                pod-template-hash=1730146906
Annotations:    <none>
Status:         Pending
IP:
Controlled By:  ReplicaSet/postgresql-5c7458bf4b
Containers:
  postgresql:
    Container ID:
    Image:          postgres:9.6.2
    Image ID:
    Port:           5432/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   exec [sh -c exec pg_isready --host $POD_IP] delay=120s timeout=5s period=10s #success=1 #failure=6
    Readiness:  exec [sh -c exec pg_isready --host $POD_IP] delay=5s timeout=3s period=5s #success=1 #failure=3
    Environment:
      POSTGRES_USER:         postgres
      PGUSER:                postgres
      POSTGRES_DB:
      POSTGRES_INITDB_ARGS:
      PGDATA:                /var/lib/postgresql/data/pgdata
      POSTGRES_PASSWORD:     <set to the key 'postgres-password' in secret 'postgresql'>  Optional: false
      POD_IP:                 (v1:status.podIP)
    Mounts:
      /var/lib/postgresql/data/pgdata from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zwv4t (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  postgresql
    ReadOnly:   false
  default-token-zwv4t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zwv4t
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From          Message
  ----     ------       ----               ----          -------
  Warning  FailedMount  14m (x81 over 2h)  kubelet, vm2  MountVolume.WaitForAttach failed for volume "pvc-11cb12e7-5060-11e8-8d24-9600000a1650" : failed to get any path for iscsi disk, last err seen:
iscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2016-09.com.openebs.jiva:pvc-11cb12e7-5060-11e8-8d24-9600000a1650, portal: 10.43.53.198,3260].
iscsiadm: initiator reported error (12 - iSCSI driver not found. Please make sure it is loaded, and retry the operation)
iscsiadm: Could not log into all portals
Logging in to [iface: default, target: iqn.2016-09.com.openebs.jiva:pvc-11cb12e7-5060-11e8-8d24-9600000a1650, portal: 10.43.53.198,3260] (multiple)
 (exit status 12)
  Warning  FailedMount  4m (x73 over 2h)  kubelet, vm2  Unable to mount volumes for pod "postgresql-5c7458bf4b-jlqfn_postgresql-6bxwy8(121a6ffe-5060-11e8-bd24-9600000a1652)": timeout expired waiting for volumes to attach or mount for pod "postgresql-6bxwy8"/"postgresql-5c7458bf4b-jlqfn". list of unmounted volumes=[data]. list of unattached volumes=[data default-token-zwv4t]

So it seems something went wrong with iscsi. If I ssh into the vm I get:

sudo service iscsid status

iscsid.service - iSCSI initiator daemon (iscsid)
   Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-05-05 16:52:37 CEST; 30min ago
     Docs: man:iscsid(8)
  Process: 29432 ExecStop=/sbin/iscsiadm -k 0 2 (code=exited, status=0/SUCCESS)
  Process: 29755 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)
  Process: 29749 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)
 Main PID: 29758 (iscsid)
    Tasks: 2
   Memory: 1.4M
      CPU: 488ms
   CGroup: /system.slice/iscsid.service
           ├─29757 /sbin/iscsid
           └─29758 /sbin/iscsid

May 05 16:52:37 vm2 systemd[1]: Starting iSCSI initiator daemon (iscsid)...
May 05 16:52:37 vm2 iscsid[29755]: iSCSI logger with pid=29757 started!
May 05 16:52:37 vm2 iscsid[29757]: iSCSI daemon with pid=29758 started!
May 05 16:52:37 vm2 systemd[1]: iscsid.service: Failed to read PID from file /run/iscsid.pid: Invalid argument
May 05 16:52:37 vm2 systemd[1]: Started iSCSI initiator daemon (iscsid).

sudo service iscsi status

open-iscsi.service - Login to default iSCSI targets
   Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2018-05-05 16:53:46 CEST; 29min ago
     Docs: man:iscsiadm(8)
           man:iscsid(8)
  Process: 30674 ExecStop=/lib/open-iscsi/logout-all.sh (code=exited, status=21)
  Process: 30668 ExecStop=/bin/sync (code=exited, status=0/SUCCESS)
  Process: 30643 ExecStop=/lib/open-iscsi/umountiscsi.sh (code=exited, status=0/SUCCESS)
  Process: 30733 ExecStart=/lib/open-iscsi/activate-storage.sh (code=exited, status=0/SUCCESS)
  Process: 30730 ExecStart=/sbin/iscsiadm -m node --loginall=automatic (code=exited, status=21)
  Process: 30723 ExecStartPre=/bin/systemctl --quiet is-active iscsid.service (code=exited, status=0/SUCCESS)
 Main PID: 30733 (code=exited, status=0/SUCCESS)

May 05 16:53:46 vm2 systemd[1]: Starting Login to default iSCSI targets...
May 05 16:53:46 vm2 iscsiadm[30730]: iscsiadm: No records found
May 05 16:53:46 vm2 systemd[1]: Started Login to default iSCSI targets.

I tried to sudo iscsiadm -m discovery -t st -p vm2 (on self):

iscsiadm: cannot make connection to 127.0.1.1: Connection refused
iscsiadm: cannot make connection to 127.0.1.1: Connection refused
iscsiadm: cannot make connection to 127.0.1.1: Connection refused
iscsiadm: cannot make connection to 127.0.1.1: Connection refused
iscsiadm: cannot make connection to 127.0.1.1: Connection refused
iscsiadm: cannot make connection to 127.0.1.1: Connection refused
iscsiadm: connection login retries (reopen_max) 5 exceeded
iscsiadm: Could not perform SendTargets discovery: encountered connection failure

This is my config:

#
# Open-iSCSI default configuration.
# Could be located at /etc/iscsi/iscsid.conf or ~/.iscsid.conf
#
# Note: To set any of these values for a specific node/session run
# the iscsiadm --mode node --op command for the value. See the README
# and man page for iscsiadm for details on the --op command.
#

######################
# iscsid daemon config
######################
# If you want iscsid to start the first time a iscsi tool
# needs to access it, instead of starting it when the init
# scripts run, set the iscsid startup command here. This
# should normally only need to be done by distro package
# maintainers.
#
# Default for Fedora and RHEL. (uncomment to activate).
# iscsid.startup = /etc/rc.d/init.d/iscsid force-start
#
# Default for upstream open-iscsi scripts (uncomment to activate).
iscsid.startup = /usr/sbin/iscsid


#############################
# NIC/HBA and driver settings
#############################
# open-iscsi can create a session and bind it to a NIC/HBA.
# To set this up see the example iface config file.

#*****************
# Startup settings
#*****************

# To request that the iscsi initd scripts startup a session set to "automatic".
# node.startup = automatic
#
# To manually startup the session set to "manual". The default is manual.
node.startup = manual

# For "automatic" startup nodes, setting this to "Yes" will try logins on each
# available iface until one succeeds, and then stop.  The default "No" will try
# logins on all availble ifaces simultaneously.
node.leading_login = No

# *************
# CHAP Settings
# *************

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
#node.session.auth.authmethod = CHAP

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
#node.session.auth.username = username
#node.session.auth.password = password

# To set a CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#node.session.auth.username_in = username_in
#node.session.auth.password_in = password_in

# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
#discovery.sendtargets.auth.authmethod = CHAP

# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
#discovery.sendtargets.auth.username = username
#discovery.sendtargets.auth.password = password

# To set a discovery session CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
#discovery.sendtargets.auth.username_in = username_in
#discovery.sendtargets.auth.password_in = password_in

# ********
# Timeouts
# ********
#
# See the iSCSI REAME's Advanced Configuration section for tips
# on setting timeouts when using multipath or doing root over iSCSI.
#
# To specify the length of time to wait for session re-establishment
# before failing SCSI commands back to the application when running
# the Linux SCSI Layer error handler, edit the line.
# The value is in seconds and the default is 120 seconds.
# Special values:
# - If the value is 0, IO will be failed immediately.
# - If the value is less than 0, IO will remain queued until the session
# is logged back in, or until the user runs the logout command.
node.session.timeo.replacement_timeout = 120

# To specify the time to wait for login to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.login_timeout = 15

# To specify the time to wait for logout to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.logout_timeout = 15

# Time interval to wait for on connection before sending a ping.
node.conn[0].timeo.noop_out_interval = 5

# To specify the time to wait for a Nop-out response before failing
# the connection, edit this line. Failing the connection will
# cause IO to be failed back to the SCSI layer. If using dm-multipath
# this will cause the IO to be failed to the multipath layer.
node.conn[0].timeo.noop_out_timeout = 5

# To specify the time to wait for abort response before
# failing the operation and trying a logical unit reset edit the line.
# The value is in seconds and the default is 15 seconds.
node.session.err_timeo.abort_timeout = 15

# To specify the time to wait for a logical unit response
# before failing the operation and trying session re-establishment
# edit the line.
# The value is in seconds and the default is 30 seconds.
node.session.err_timeo.lu_reset_timeout = 30

# To specify the time to wait for a target response
# before failing the operation and trying session re-establishment
# edit the line.
# The value is in seconds and the default is 30 seconds.
node.session.err_timeo.tgt_reset_timeout = 30


#******
# Retry
#******

# To specify the number of times iscsid should retry a login
# if the login attempt fails due to the node.conn[0].timeo.login_timeout
# expiring modify the following line. Note that if the login fails
# quickly (before node.conn[0].timeo.login_timeout fires) because the network
# layer or the target returns an error, iscsid may retry the login more than
# node.session.initial_login_retry_max times.
#
# This retry count along with node.conn[0].timeo.login_timeout
# determines the maximum amount of time iscsid will try to
# establish the initial login. node.session.initial_login_retry_max is
# multiplied by the node.conn[0].timeo.login_timeout to determine the
# maximum amount.
#
# The default node.session.initial_login_retry_max is 8 and
# node.conn[0].timeo.login_timeout is 15 so we have:
#
# node.conn[0].timeo.login_timeout * node.session.initial_login_retry_max =
#                                                               120 seconds
#
# Valid values are any integer value. This only
# affects the initial login. Setting it to a high value can slow
# down the iscsi service startup. Setting it to a low value can
# cause a session to not get logged into, if there are distuptions
# during startup or if the network is not ready at that time.
node.session.initial_login_retry_max = 8

################################
# session and device queue depth
################################

# To control how many commands the session will queue set
# node.session.cmds_max to an integer between 2 and 2048 that is also
# a power of 2. The default is 128.
node.session.cmds_max = 128

# To control the device's queue depth set node.session.queue_depth
# to a value between 1 and 1024. The default is 32.
node.session.queue_depth = 32

##################################
# MISC SYSTEM PERFORMANCE SETTINGS
##################################

# For software iscsi (iscsi_tcp) and iser (ib_iser) each session
# has a thread used to transmit or queue data to the hardware. For
# cxgb3i you will get a thread per host.
#
# Setting the thread's priority to a lower value can lead to higher throughput
# and lower latencies. The lowest value is -20. Setting the priority to
# a higher value, can lead to reduced IO performance, but if you are seeing
# the iscsi or scsi threads dominate the use of the CPU then you may want
# to set this value higher.
#
# Note: For cxgb3i you must set all sessions to the same value, or the
# behavior is not defined.
#
# The default value is -20. The setting must be between -20 and 20.
node.session.xmit_thread_priority = -20


#***************
# iSCSI settings
#***************

# To enable R2T flow control (i.e., the initiator must wait for an R2T
# command before sending any data), uncomment the following line:
#
#node.session.iscsi.InitialR2T = Yes
#
# To disable R2T flow control (i.e., the initiator has an implied
# initial R2T of "FirstBurstLength" at offset 0), uncomment the following line:
#
# The defaults is No.
node.session.iscsi.InitialR2T = No

#
# To disable immediate data (i.e., the initiator does not send
# unsolicited data with the iSCSI command PDU), uncomment the following line:
#
#node.session.iscsi.ImmediateData = No
#
# To enable immediate data (i.e., the initiator sends unsolicited data
# with the iSCSI command packet), uncomment the following line:
#
# The default is Yes
node.session.iscsi.ImmediateData = Yes

# To specify the maximum number of unsolicited data bytes the initiator
# can send in an iSCSI PDU to a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 262144
node.session.iscsi.FirstBurstLength = 262144

# To specify the maximum SCSI payload that the initiator will negotiate
# with the target for, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the defauls it 16776192
node.session.iscsi.MaxBurstLength = 16776192

# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 262144
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144

# To specify the maximum number of data bytes the initiator will send
# in an iSCSI PDU to the target, edit the following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1).
# Zero is a special case. If set to zero, the initiator will use
# the target's MaxRecvDataSegmentLength for the MaxXmitDataSegmentLength.
# The default is 0.
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0

# To specify the maximum number of data bytes the initiator can receive
# in an iSCSI PDU from a target during a discovery session, edit the
# following line.
#
# The value is the number of bytes in the range of 512 to (2^24-1) and
# the default is 32768
#
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768

# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of enabling the checking, uncomment# one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = CRC32C,None
#node.conn[0].iscsi.DataDigest = CRC32C,None
#
# To allow the targets to control the setting of the digest checking,
# with the initiator requesting a preference of disabling the checking,
# uncomment one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = None,CRC32C
#node.conn[0].iscsi.DataDigest = None,CRC32C
#
# To enable CRC32C digest checking for the header and/or data part of
# iSCSI PDUs, uncomment one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = CRC32C
#node.conn[0].iscsi.DataDigest = CRC32C
#
# To disable digest checking for the header and/or data part of
# iSCSI PDUs, uncomment one or both of the following lines:
#node.conn[0].iscsi.HeaderDigest = None
#node.conn[0].iscsi.DataDigest = None
#
# The default is to never use DataDigests or HeaderDigests.
#

# For multipath configurations, you may want more than one session to be
# created on each iface record.  If node.session.nr_sessions is greater
# than 1, performing a 'login' for that node will ensure that the
# appropriate number of sessions is created.
node.session.nr_sessions = 1

#************
# Workarounds
#************

# Some targets like IET prefer after an initiator has sent a task
# management function like an ABORT TASK or LOGICAL UNIT RESET, that
# it does not respond to PDUs like R2Ts. To enable this behavior uncomment
# the following line (The default behavior is Yes):
node.session.iscsi.FastAbort = Yes

# Some targets like Equalogic prefer that after an initiator has sent
# a task management function like an ABORT TASK or LOGICAL UNIT RESET, that
# it continue to respond to R2Ts. To enable this uncomment this line
# node.session.iscsi.FastAbort = No

I'm not a pro for iscsi and so not sure if iscsi is correct configured. Especially the output

Active: active (exited) ...
May 05 16:53:46 vm2 iscsiadm[30730]: iscsiadm: No records found

makes me wonder. Is that correct?

Any hints?

Environment:

• kubectl get nodes

vm1       Ready     controlplane,etcd,worker   2d        v1.10.1
vm2       Ready     controlplane,etcd,worker   2d        v1.10.1
vm3       Ready     controlplane,etcd,worker   2d        v1.10.1

• kubectl get pods --all-namespaces

NAMESPACE           NAME                                                             READY     STATUS              RESTARTS   AGE
cattle-system       cattle-cluster-agent-5c5fcf6dfc-lnsnm                            1/1       Running             0          2d
cattle-system       cattle-node-agent-c4mps                                          1/1       Running             0          2d
cattle-system       cattle-node-agent-ht72f                                          1/1       Running             0          2d
cattle-system       cattle-node-agent-wpbtx                                          1/1       Running             0          2d
default             maya-apiserver-5fc675bc6-qht4w                                   1/1       Running             0          3h
default             openebs-provisioner-576cbb68d4-4jz99                             1/1       Running             0          3h
ingress-nginx       default-http-backend-564b9b6c5b-tgfwg                            1/1       Running             0          2d
ingress-nginx       nginx-ingress-controller-7gn4t                                   1/1       Running             0          2d
ingress-nginx       nginx-ingress-controller-rbt9g                                   1/1       Running             0          2d
ingress-nginx       nginx-ingress-controller-z57vv                                   1/1       Running             0          2d
kube-system         canal-2x7ts                                                      3/3       Running             0          2d
kube-system         canal-5b8vq                                                      3/3       Running             0          2d
kube-system         canal-svpxx                                                      3/3       Running             0          2d
kube-system         kube-dns-7dfdc4897f-mhslj                                        3/3       Running             0          2d
kube-system         kube-dns-autoscaler-6c4b786f5-tvqvt                              1/1       Running             0          2d
postgresql-6bxwy8   postgresql-5c7458bf4b-jlqfn                                      0/1       ContainerCreating   0          2h
postgresql-6bxwy8   pvc-11cb12e7-5060-11e8-8d24-9600000a1650-ctrl-69bb6467d7-l2zw6   2/2       Running             0          2h
postgresql-6bxwy8   pvc-11cb12e7-5060-11e8-8d24-9600000a1650-rep-c7d9b5d94-bmhtf     1/1       Running             0          2h
postgresql-6bxwy8   pvc-11cb12e7-5060-11e8-8d24-9600000a1650-rep-c7d9b5d94-hfpz7     1/1       Running             0          2h
postgresql-6bxwy8   pvc-11cb12e7-5060-11e8-8d24-9600000a1650-rep-c7d9b5d94-r9d6x     1/1       Running             0          2h

• kubectl get services

NAMESPACE           NAME                                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
default             kubernetes                                          ClusterIP   10.43.0.1       <none>        443/TCP             2d
default             maya-apiserver-service                              ClusterIP   10.43.66.134    <none>        5656/TCP            3h
ingress-nginx       default-http-backend                                ClusterIP   10.43.216.236   <none>        80/TCP              2d
kube-system         kube-dns                                            ClusterIP   10.43.0.10      <none>        53/UDP,53/TCP       2d
postgresql-6bxwy8   postgresql                                          ClusterIP   10.43.191.72    <none>        5432/TCP            2h
postgresql-6bxwy8   pvc-11cb12e7-5060-11e8-8d24-9600000a1650-ctrl-svc   ClusterIP   10.43.53.198    <none>        3260/TCP,9501/TCP   2h

• kubectl get sc

openebs-cassandra    openebs.io/provisioner-iscsi
openebs-es-data-sc   openebs.io/provisioner-iscsi
openebs-jupyter      openebs.io/provisioner-iscsi
openebs-kafka        openebs.io/provisioner-iscsi
openebs-mongodb      openebs.io/provisioner-iscsi
openebs-percona      openebs.io/provisioner-iscsi
openebs-redis        openebs.io/provisioner-iscsi
openebs-standalone   openebs.io/provisioner-iscsi
openebs-standard     openebs.io/provisioner-iscsi
openebs-zk           openebs.io/provisioner-iscsi

• kubectl get pv

NAMESPACE   NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                          STORAGECLASS       REASON    AGE
            pvc-11cb12e7-5060-11e8-8d24-9600000a1650   8Gi        RWO            Delete           Bound     postgresql-6bxwy8/postgresql   openebs-standard             2h

• kubectl get pvc

NAMESPACE           NAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
postgresql-6bxwy8   postgresql   Bound     pvc-11cb12e7-5060-11e8-8d24-9600000a1650   8Gi        RWO            openebs-standard   2h

• Kernel (e.g. uname -a):
  Linux vm1 4.4.0-122-generic #146-Ubuntu SMP Mon Apr 23 15:34:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[kmova]

Hit a similar issue with mongo db. The issues was with kubelet and node having different versions of open-iscsi.

Analysis Steps:

• the iscsiadm command error displays - kernel module not loaded, however -- lsmod from node and kubelet show that libiscsi_tcp module is loaded.
• iscsiadm command to login was failing from within kubelet but passing from node. jiva logs showed that, when logging in from kubelet, the connection was getting closed from initiator:

time="2018-06-22T06:18:58Z" level=info msg="10.42.2.3:3260"
time="2018-06-22T06:18:58Z" level=info msg="Accepting ..."
time="2018-06-22T06:18:58Z" level=info msg="connection is connected from 159.69.41.169:30980...\n"
time="2018-06-22T06:18:58Z" level=info msg="Listening ..."
time="2018-06-22T06:18:58Z" level=warning msg="unexpected connection state: full feature"
time="2018-06-22T06:18:58Z" level=error msg="read tcp 10.42.2.3:3260->159.69.41.169:30980: read: connection reset by peer"

• strace showed that there was difference in what what getting passed to iscsid from kubelet and node.
• apt list --installed | grep open-iscsi showed that the packages were different:
  
  
  
  • kubelet has open-iscsi/stable,now 2.0.874-3~deb9u1 amd64 [installed]
  
  
  • node has open-iscsi/xenial-updates,xenial-updates,now 2.0.873+git0.3b4b4500-14ubuntu3.4 amd64 [installed]
  
  
• Further debugging notes:
  
  
  
  • kubelet and node -- share the /var/run folder.
  
  
  • ps -auxwww : on node and kubelet showed that: iscsid was same in kubelet and node.
  
  
  • killed the iscsid within kubelet and then did a login which succeeded.
  
  

Cause:

• As part of the startup of the node, iscsi was already installed and running
• Kubelet startup - and doesn't start the iscsid as it is already running
• When volume login is initiated, iscsiadm from kubelet - is trying to contact the iscsid (running on the host) to initiate the connection. The version don't match that results in error from iscsid to iscsiadm and prints 12 - module not loaded.

Symptoms:
The following steps can be used to determine this:

  iscsiadm -V
  sudo docker exec kubelet iscsiadm -V

Sample Output:

  root@worker1 ~ # iscsiadm -V
  iscsiadm version 2.0-873
  root@worker1 ~ # sudo docker exec kubelet iscsiadm -V
  iscsiadm version 2.0-874

WorkAround: Remove the iscsi from the node

  service iscsid stop
  sudo apt remove open-iscsi

The above will only remove iscsi from node. But the iscsi inside the kubelet will be running:

iscsiadm -V
ps -auxfwww | grep iscsi
sudo docker exec kubelet iscsiadm -V
sudo docker exec kubelet ps -auxfwww | grep iscsid

Sample Output:

root@worker1 ~ # iscsiadm -V
-bash: /usr/bin/iscsiadm: No such file or directory
root@worker1 ~ # ps -auxfwww | grep iscsi
root       660  0.0  0.0      0     0 ?        S<   16:34   0:00  \_ [iscsi_eh]
root     25460  0.0  0.0  12944  1032 pts/0    S+   17:27   0:00          \_ grep --color=auto iscsi
root@worker1 ~ # sudo docker exec kubelet iscsiadm -V
iscsiadm version 2.0-874
root@worker1 ~ # sudo docker exec kubelet ps -auxfwww | grep iscsid
root     25492  0.0  0.0  12944   924 pts/0    S+   15:27   0:00          \_ grep --color=auto iscsid
root@worker1 ~ #

On reboot of the node, to make sure the iscsi kernel modules are loaded, make sure the following configuration is done on the nodes:

# /etc/modules: kernel modules to load at boot time.
## This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
iscsi_tcp

[kmova]

Related Issue:

• https://github.com/rancher/docs/issues/796
• https://github.com/rancher/rancher/issues/15118

[loxy]

It's more or less related to ranger itself:
https://rancher.com/docs/rancher/v2.x/en/k8s-in-rancher/volumes-and-storage/#iscsi-volumes-with-rancher-launched-kubernetes-clusters

So closing now...