Rook: EdgeFS cluster nodes end up in networking split

Created on 30 May 2019 · 4Comments · Source: rook/rook

Is this a bug report or feature request?

Bug Report

Deviation from expected behavior:

When i kill any pod (mgr, targer) -> the cluster is falling apart
Help only recreate cluster

Expected behavior:

Cluster and all deployments return to green status

How to reproduce it (minimal and precise):

1) deploy k8s cluster
2) deploy operator with edgefs/edgefs-operator:1.1.18
3) deploy cluster with edgefs/edgefs:1.1.31

apiVersion: edgefs.rook.io/v1beta1
kind: Cluster
metadata:
  name: rook-edgefs
  namespace: rook-edgefs
spec:
  edgefsImageName: edgefs/edgefs:1.1.31 
  serviceAccount: rook-edgefs-cluster
  dataDirHostPath: /var/lib/edgefs
  skipHostPrepare: false
  trlogProcessingInterval: 2
  trlogKeepDays: 3
  storage:
    useAllNodes: true
    useAllDevices: true
    config:
      useAllSSD: "true"
  placement:
    all:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: nodepool
              operator: In
              values:
              - edgefs-storage
              - edgefs-gateway
      tolerations:
      - key: edgefs-storage
        operator: Exists
      - key: edgefs-gateway
        operator: Exists
    mgr:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: nodepool
              operator: In
              values:
              - edgefs-gateway
      tolerations:
      - key: edgefs-gateway
        operator: Exists

4) login to ui

create cluster cltest with 1 replicafactor 64 kb
create tenant test
create bucket bk1
create nfs01 cltest/test/bk1
5) edit k8s cluster

  kube-api: 
    extra_args: 
      feature-gates: "PersistentLocalVolumes=true,VolumeScheduling=true,CSINodeInfo=true,CSIDriverRegistry=true,BlockVolume=true,CSIBlockVolume=true"
  kubelet: 
    extra_args: 
      feature-gates: "PersistentLocalVolumes=true,VolumeScheduling=true,CSINodeInfo=true,CSIDriverRegistry=true,BlockVolume=true,CSIBlockVolume=true"

6) lose one pod (from targes on gateway) - it's magic)
7) remove all pods from rook-edgefs
8) cluster broke (it's say that i have one node without gateway node)
9) can't login to UI

Logs (rook-edgefs-mgr)

rook-edgefs-mgr

Running in container 
Populating /opt/nedge/etc from /opt/nedge/etc.target 
Configured /opt/nedge/etc/ccow/ccow.json 
Broker configuration adjusted. 
Waiting for corosync ... 
Waiting for corosync ... 
Waiting for corosync ... 
Waiting for corosync ... 
Starting mgmt in foreground. 
Thu, 30 May 2019 19:00:27 GMT info REST API backends to process: 11 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Auth API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Cluster API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: iSCSI Target API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Msite API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: NFS API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Service API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Swagger spec 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Sysconfig API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: System API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Tenant API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: REST testing API 
Thu, 30 May 2019 19:00:27 GMT info Prometheus exporter listening on 0.0.0.0:8881/metrics 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Job API 
Thu, 30 May 2019 19:00:27 GMT info REST HTTPS server successfully configured. 
Thu, 30 May 2019 19:00:27 GMT info REST API backends to process: 11 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Auth API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Cluster API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: iSCSI Target API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Msite API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: NFS API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Service API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Swagger spec 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Sysconfig API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: System API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Tenant API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: REST testing API 
Thu, 30 May 2019 19:00:27 GMT info Loading API for backend: Job API 
Thu, 30 May 2019 19:00:27 GMT info REST server is up and available at :::4443 
Thu, 30 May 2019 19:00:27 GMT info REST server successfully configured. 
Thu, 30 May 2019 19:00:27 GMT info REST server is up and available at :::8080 
Thu, 30 May 2019 19:00:33 GMT alert ServerID=9FA3178264D9A8088344DE4D8929FE14 IPv6=::ffff:ffff:902:2a0a VdevID=00000000000000000000000000000000 is ONLINE 
Thu, 30 May 2019 19:00:33 GMT alert ServerID=9FA3178264D9A8088344DE4D8929FE14 IPv6=::ffff:ffff:902:2a0a is AVAILABLE 
Thu, 30 May 2019 19:00:51 GMT info GET /nedge-version Headers: { connection: 'close', host: '10.42.2.8:8080' } Params: {} Body: undefined 
Thu, 30 May 2019 19:00:52 GMT info GET /nedge-version Headers: { connection: 'close', host: '10.42.2.8:8080' } Params: {} Body: undefined 
Thu, 30 May 2019 19:01:15 GMT info GET /clusters Headers: { connection: 'close',  host: '10.42.2.8:8080',  authorization: 'Basic YWRtaW46ZWRnZWZz' } Params: {} Body: undefined 
May 30 19:01:32.763 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [151]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:02:32.852 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [184]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:03:15 GMT info GET /system/init Headers: { connection: 'close',  host: '10.42.2.8:8080',  authorization: 'Basic YWRtaW46ZWRnZWZz' } Params: {} Body: undefined 
May 30 19:03:32.956 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [217]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:04:33.060 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [250]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:05:15 GMT error GET /system/service-checkpoint (route: /system/service-checkpoint) failed: ESRCH: Failed to read: Error: ENOENT, open '/opt/nedge/var/run/flexhash.json' 
May 30 19:05:33.164 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [283]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:06:33.265 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [316]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:06:44 GMT info GET /clusters Headers: { connection: 'close',  host: '10.42.2.8:8080',  authorization: 'Basic YWRtaW46ZWRnZWZz' } Params: {} Body: undefined 
May 30 19:07:33.357 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [349]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:08:33.469 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [382]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:08:44 GMT info GET /system/init Headers: { connection: 'close',  host: '10.42.2.8:8080',  authorization: 'Basic YWRtaW46ZWRnZWZz' } Params: {} Body: undefined 
May 30 19:09:33.566 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [415]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:09:33 GMT error GET /clusters (route: /clusters) failed: INIT-ERROR: Error during CCOW tenant initialization: Input/output error 
May 30 19:10:33.662 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [448]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:10:44 GMT error GET /system/service-checkpoint (route: /system/service-checkpoint) failed: ESRCH: Failed to read: Error: ENOENT, open '/opt/nedge/var/run/flexhash.json' 
May 30 19:11:33.762 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [481]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:12:33.855 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [514]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:13:33.947 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [547]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:14:34.047 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [580]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:15:34.136 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [613]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:16:34.244 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [646]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:17:34.340 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [679]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:18:34.445 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [712]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:18:39 GMT info GET /nedge-version Headers: { connection: 'close', host: '10.42.2.8:8080' } Params: {} Body: undefined 
Thu, 30 May 2019 19:18:39 GMT info GET /nedge-version Headers: { connection: 'close', host: '10.42.2.8:8080' } Params: {} Body: undefined 
Thu, 30 May 2019 19:18:42 GMT info GET /clusters Headers: { connection: 'close',  host: '10.42.2.8:8080',  authorization: 'Basic YWRtaW46ZWRnZWZz' } Params: {} Body: undefined 
May 30 19:19:34.537 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [745]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:20:34.633 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [778]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
Thu, 30 May 2019 19:20:42 GMT info GET /system/init Headers: { connection: 'close',  host: '10.42.2.8:8080',  authorization: 'Basic YWRtaW46ZWRnZWZz' } Params: {} Body: undefined 
May 30 19:21:34.726 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [811]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:22:34.826 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [844]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:23:34.918 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [877]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:24:35.023 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [910]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:25:35.131 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [943]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:26:35.231 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [976]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:27:35.324 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [1009]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:28:35.424 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [1042]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
May 30 19:29:35.524 rook-edgefs-mgr-7b8f48c4b4-vkcc6 ccow[1] error   [1075]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480

grpc

Running in container 
Populating /opt/nedge/etc from /opt/nedge/etc.target 
Configured /opt/nedge/etc/ccow/ccow.json 
Broker configuration adjusted. 
Waiting for corosync ... 
Waiting for corosync ... 
Waiting for corosync ... 
Waiting for corosync ... 
Starting grpc-efsproxy in foreground. 
2019-05-30T19:00:28.755Z EFSPROXY in version dev serving on 0.0.0.0:6789 is ready for gRPC clients


> @0.0.0-next start /usr/app 
> NODE_ENV=production node ./build/server 

2019-05-30T19:00:10.696Z : Application env: production 
2019-05-30T19:00:10.698Z : Config: 
2019-05-30T19:00:10.699Z : 
{ 
    "API_ENDPOINT": "http://10.42.2.8:8080", 
    "HTTP_PORT": "3000", 
    "HTTPS_PORT": "3443", 
    "SESSION_MAX_AGE_MINUTES": "60", 
    "LOG_LEVEL_CONSOLE": "info" 
} 
2019-05-30T19:00:10.704Z : [HPM] Proxy created: /  ->  http://10.42.2.8:8080 
2019-05-30T19:00:10.706Z : [HPM] Proxy rewrite rule created: "^/api/" ~> "/" 
2019-05-30T19:00:10.722Z : HTTP server is now running on http://localhost:3000 
2019-05-30T19:00:10.722Z : HTTPs server is now running on https://localhost:3443 
2019-05-30T19:00:24.170Z : ::ffff:10.42.0.0 - GET /api/service 
2019-05-30T19:00:24.176Z : ::ffff:10.42.0.0 - GET /api/system/stats 
2019-05-30T19:00:24.177Z : ::ffff:10.42.0.0 - GET /api/system/service-checkpoint 
2019-05-30T19:00:24.178Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:24.179Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:24.179Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:24.231Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:24.231Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:24.231Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:24.298Z : [PROXY] ::ffff:10.42.0.0 - (user: anon) GET http://10.42.2.8:8080/service 
2019-05-30T19:00:24.301Z : [PROXY] ::ffff:10.42.0.0 - (user: anon) GET http://10.42.2.8:8080/system/stats 
2019-05-30T19:00:24.302Z : [PROXY] ::ffff:10.42.0.0 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint 
2019-05-30T19:00:24.304Z : [PROXY] ::ffff:10.42.0.0 - (user: anon) GET http://10.42.2.8:8080/service - Error: connect ECONNREFUSED 10.42.2.8:8080 
2019-05-30T19:00:24.308Z : [HPM] Error occurred while trying to proxy request /service from 10.253.217.252:31953 to http://10.42.2.8:8080 (ECONNREFUSED) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:00:24.309Z : [PROXY] ::ffff:10.42.0.0 - (user: anon) GET http://10.42.2.8:8080/system/stats - Error: connect ECONNREFUSED 10.42.2.8:8080 
2019-05-30T19:00:24.310Z : [HPM] Error occurred while trying to proxy request /system/stats from 10.253.217.252:31953 to http://10.42.2.8:8080 (ECONNREFUSED) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:00:24.310Z : [PROXY] ::ffff:10.42.0.0 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - Error: connect ECONNREFUSED 10.42.2.8:8080 
2019-05-30T19:00:24.310Z : [HPM] Error occurred while trying to proxy request /system/service-checkpoint from 10.253.217.252:31953 to http://10.42.2.8:8080 (ECONNREFUSED) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:00:50.833Z : ::ffff:10.42.0.0 - GET / 
2019-05-30T19:00:50.835Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:50.886Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:50.947Z : ::ffff:10.42.0.0 - GET / - 304 - 0.113s 
2019-05-30T19:00:51.031Z : ::ffff:10.42.0.0 - GET /main.css?fb61a914c42bd70174c1 
2019-05-30T19:00:51.033Z : ::ffff:10.42.0.0 - GET /main.js?fb61a914c42bd70174c1 
2019-05-30T19:00:51.033Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.034Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.084Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.085Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.138Z : ::ffff:10.42.0.0 - GET /main.css?fb61a914c42bd70174c1 - 304 - 0.107s 
2019-05-30T19:00:51.139Z : ::ffff:10.42.0.0 - GET /main.js?fb61a914c42bd70174c1 - 304 - 0.106s 
2019-05-30T19:00:51.618Z : ::ffff:10.42.0.0 - GET /88b4d1025a38c2940b619645eb339bbe.png 
2019-05-30T19:00:51.619Z : ::ffff:10.42.0.0 - GET /user 
2019-05-30T19:00:51.619Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.619Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.670Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.670Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.699Z : ::ffff:10.42.0.0 - GET /cb0af8bab35b54e3eadea689cb29dc93.woff2 
2019-05-30T19:00:51.700Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.722Z : ::ffff:10.42.0.0 - GET /user - 400 - 0.103s 
2019-05-30T19:00:51.723Z : ::ffff:10.42.0.0 - GET /88b4d1025a38c2940b619645eb339bbe.png - 304 - 0.105s 
2019-05-30T19:00:51.750Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.794Z : ::ffff:10.42.0.0 - GET /version 
2019-05-30T19:00:51.795Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.801Z : ::ffff:10.42.0.0 - GET /cb0af8bab35b54e3eadea689cb29dc93.woff2 - 304 - 0.102s 
2019-05-30T19:00:51.846Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.906Z : ::ffff:127.0.0.1 - GET /api/nedge-version 
2019-05-30T19:00:51.907Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version 
2019-05-30T19:00:51.929Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version - 200 - 0.022s 
2019-05-30T19:00:51.936Z : ::ffff:10.42.0.0 - GET /version - 304 - 0.142s 
2019-05-30T19:00:51.940Z : ::ffff:10.42.0.0 - GET /favicon.png 
2019-05-30T19:00:51.940Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.989Z : ::ffff:10.42.0.0 - GET /version 
2019-05-30T19:00:51.990Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:51.991Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:52.040Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:00:52.043Z : ::ffff:10.42.0.0 - GET /favicon.png - 304 - 0.103s 
2019-05-30T19:00:52.094Z : ::ffff:127.0.0.1 - GET /api/nedge-version 
2019-05-30T19:00:52.095Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version 
2019-05-30T19:00:52.100Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version - 200 - 0.004s 
2019-05-30T19:00:52.102Z : ::ffff:10.42.0.0 - GET /version - 304 - 0.112s 
2019-05-30T19:01:15.219Z : ::ffff:10.42.0.0 - POST /login 
2019-05-30T19:01:15.235Z : ::ffff:10.42.0.0 - POST /login - 200 - 0.016s 
2019-05-30T19:01:15.237Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:01:15.287Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:01:15.341Z : ::ffff:127.0.0.1 - GET /api/clusters 
2019-05-30T19:01:15.343Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters 
2019-05-30T19:01:20.347Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 5s... 
2019-05-30T19:01:25.351Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 10s... 
2019-05-30T19:01:30.356Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 15s... 
2019-05-30T19:01:35.359Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 20s... 
2019-05-30T19:01:40.360Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 25s... 
2019-05-30T19:01:45.366Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 30s... 
2019-05-30T19:01:50.368Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 35s... 
2019-05-30T19:01:55.372Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 40s... 
2019-05-30T19:02:00.376Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 45s... 
2019-05-30T19:02:05.377Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 50s... 
2019-05-30T19:02:10.378Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 55s... 
2019-05-30T19:02:15.382Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 60s... 
2019-05-30T19:02:20.386Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 65s... 
2019-05-30T19:02:25.387Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 70s... 
2019-05-30T19:02:30.391Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 75s... 
2019-05-30T19:02:35.397Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 80s... 
2019-05-30T19:02:40.403Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 85s... 
2019-05-30T19:02:45.408Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 90s... 
2019-05-30T19:02:50.414Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 95s... 
2019-05-30T19:02:55.415Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 100s... 
2019-05-30T19:03:00.421Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 105s... 
2019-05-30T19:03:05.426Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 110s... 
2019-05-30T19:03:10.432Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 115s... 
2019-05-30T19:03:15.242Z : ::ffff:10.42.0.0 - POST /login - [closed by user] - 120.023s 
2019-05-30T19:03:15.346Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - Error: socket hang up 
2019-05-30T19:03:15.348Z : [HPM] Error occurred while trying to proxy request /clusters from 127.0.0.1:3000 to http://10.42.2.8:8080 (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:03:15.353Z : ::ffff:127.0.0.1 - GET /api/system/init 
2019-05-30T19:03:15.355Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init 
2019-05-30T19:03:20.358Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 5s... 
2019-05-30T19:03:25.364Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 10s... 
2019-05-30T19:03:30.366Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 15s... 
2019-05-30T19:03:35.368Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 20s... 
2019-05-30T19:03:40.368Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 25s... 
2019-05-30T19:03:45.374Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 30s... 
2019-05-30T19:03:50.377Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 35s... 
2019-05-30T19:03:55.380Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 40s... 
2019-05-30T19:04:00.384Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 45s... 
2019-05-30T19:04:05.386Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 50s... 
2019-05-30T19:04:10.392Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 55s... 
2019-05-30T19:04:15.396Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 60s... 
2019-05-30T19:04:20.402Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 65s... 
2019-05-30T19:04:25.408Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 70s... 
2019-05-30T19:04:30.409Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 75s... 
2019-05-30T19:04:35.415Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 80s... 
2019-05-30T19:04:40.421Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 85s... 
2019-05-30T19:04:45.422Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 90s... 
2019-05-30T19:04:50.423Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 95s... 
2019-05-30T19:04:55.429Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 100s... 
2019-05-30T19:05:00.432Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 105s... 
2019-05-30T19:05:05.436Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 110s... 
2019-05-30T19:05:10.445Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 115s... 
2019-05-30T19:05:15.358Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - Error: socket hang up 
2019-05-30T19:05:15.360Z : [HPM] Error occurred while trying to proxy request /system/init from 127.0.0.1:3000 to http://10.42.2.8:8080 (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:05:15.362Z : Cluster not initialized: Proxy error: Error: socket hang up 
2019-05-30T19:05:15.365Z : ::ffff:127.0.0.1 - GET /api/system/service-checkpoint 
2019-05-30T19:05:15.367Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint 
2019-05-30T19:05:15.375Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - 500 - 0.008s 
2019-05-30T19:06:44.031Z : ::ffff:10.42.0.0 - POST /login 
2019-05-30T19:06:44.034Z : ::ffff:10.42.0.0 - POST /login - 200 - 0.003s 
2019-05-30T19:06:44.036Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:06:44.087Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:06:44.141Z : ::ffff:127.0.0.1 - GET /api/clusters 
2019-05-30T19:06:44.143Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters 
2019-05-30T19:06:49.146Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 5s... 
2019-05-30T19:06:54.152Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 10s... 
2019-05-30T19:06:59.158Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 15s... 
2019-05-30T19:07:04.163Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 20s... 
2019-05-30T19:07:09.166Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 25s... 
2019-05-30T19:07:14.172Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 30s... 
2019-05-30T19:07:19.176Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 35s... 
2019-05-30T19:07:24.182Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 40s... 
2019-05-30T19:07:29.184Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 45s... 
2019-05-30T19:07:34.189Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 50s... 
2019-05-30T19:07:39.195Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 55s... 
2019-05-30T19:07:44.198Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 60s... 
2019-05-30T19:07:49.204Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 65s... 
2019-05-30T19:07:54.206Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 70s... 
2019-05-30T19:07:59.212Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 75s... 
2019-05-30T19:08:04.215Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 80s... 
2019-05-30T19:08:09.220Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 85s... 
2019-05-30T19:08:14.224Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 90s... 
2019-05-30T19:08:19.226Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 95s... 
2019-05-30T19:08:24.232Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 100s... 
2019-05-30T19:08:29.237Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 105s... 
2019-05-30T19:08:34.241Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 110s... 
2019-05-30T19:08:39.246Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 115s... 
2019-05-30T19:08:44.037Z : ::ffff:10.42.0.0 - POST /login - [closed by user] - 120.007s 
2019-05-30T19:08:44.146Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - Error: socket hang up 
2019-05-30T19:08:44.149Z : [HPM] Error occurred while trying to proxy request /clusters from 127.0.0.1:3000 to http://10.42.2.8:8080 (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:08:44.154Z : ::ffff:127.0.0.1 - GET /api/system/init 
2019-05-30T19:08:44.155Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init 
2019-05-30T19:08:49.158Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 5s... 
2019-05-30T19:08:54.158Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 10s... 
2019-05-30T19:08:59.164Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 15s... 
2019-05-30T19:09:04.170Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 20s... 
2019-05-30T19:09:09.176Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 25s... 
2019-05-30T19:09:14.181Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 30s... 
2019-05-30T19:09:19.187Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 35s... 
2019-05-30T19:09:24.192Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 40s... 
2019-05-30T19:09:29.196Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 45s... 
2019-05-30T19:09:34.200Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 50s... 
2019-05-30T19:09:39.206Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 55s... 
2019-05-30T19:09:44.212Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 60s... 
2019-05-30T19:09:49.217Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 65s... 
2019-05-30T19:09:54.221Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 70s... 
2019-05-30T19:09:59.226Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 75s... 
2019-05-30T19:10:04.232Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 80s... 
2019-05-30T19:10:09.238Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 85s... 
2019-05-30T19:10:14.244Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 90s... 
2019-05-30T19:10:19.250Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 95s... 
2019-05-30T19:10:24.255Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 100s... 
2019-05-30T19:10:29.261Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 105s... 
2019-05-30T19:10:34.267Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 110s... 
2019-05-30T19:10:39.272Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 115s... 
2019-05-30T19:10:44.158Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - Error: socket hang up 
2019-05-30T19:10:44.160Z : [HPM] Error occurred while trying to proxy request /system/init from 127.0.0.1:3000 to http://10.42.2.8:8080 (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:10:44.162Z : Cluster not initialized: Proxy error: Error: socket hang up 
2019-05-30T19:10:44.166Z : ::ffff:127.0.0.1 - GET /api/system/service-checkpoint 
2019-05-30T19:10:44.168Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint 
2019-05-30T19:10:44.174Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - 500 - 0.006s 
2019-05-30T19:18:37.808Z : ::ffff:10.42.0.0 - GET / 
2019-05-30T19:18:37.810Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:37.861Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:37.914Z : ::ffff:10.42.0.0 - GET / - 304 - 0.106s 
2019-05-30T19:18:38.048Z : ::ffff:10.42.0.0 - GET /main.css?fb61a914c42bd70174c1 
2019-05-30T19:18:38.048Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.050Z : ::ffff:10.42.0.0 - GET /main.js?fb61a914c42bd70174c1 
2019-05-30T19:18:38.051Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.099Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.101Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.150Z : ::ffff:10.42.0.0 - GET /main.css?fb61a914c42bd70174c1 - 304 - 0.102s 
2019-05-30T19:18:38.153Z : ::ffff:10.42.0.0 - GET /main.js?fb61a914c42bd70174c1 - 304 - 0.103s 
2019-05-30T19:18:38.707Z : ::ffff:10.42.0.0 - GET /user 
2019-05-30T19:18:38.707Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.715Z : ::ffff:10.42.0.0 - GET /88b4d1025a38c2940b619645eb339bbe.png 
2019-05-30T19:18:38.716Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.758Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.767Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.810Z : ::ffff:10.42.0.0 - GET /user - 400 - 0.103s 
2019-05-30T19:18:38.819Z : ::ffff:10.42.0.0 - GET /88b4d1025a38c2940b619645eb339bbe.png - 304 - 0.104s 
2019-05-30T19:18:38.819Z : ::ffff:10.42.0.0 - GET /cb0af8bab35b54e3eadea689cb29dc93.woff2 
2019-05-30T19:18:38.820Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.870Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.907Z : ::ffff:10.42.0.0 - GET /version 
2019-05-30T19:18:38.908Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:38.921Z : ::ffff:10.42.0.0 - GET /cb0af8bab35b54e3eadea689cb29dc93.woff2 - 304 - 0.102s 
2019-05-30T19:18:38.957Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:39.012Z : ::ffff:127.0.0.1 - GET /api/nedge-version 
2019-05-30T19:18:39.014Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version 
2019-05-30T19:18:39.018Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version - 200 - 0.005s 
2019-05-30T19:18:39.021Z : ::ffff:10.42.0.0 - GET /version - 304 - 0.114s 
2019-05-30T19:18:39.060Z : ::ffff:10.42.0.0 - GET /version 
2019-05-30T19:18:39.061Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:39.112Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:39.164Z : ::ffff:127.0.0.1 - GET /api/nedge-version 
2019-05-30T19:18:39.165Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version 
2019-05-30T19:18:39.168Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/nedge-version - 200 - 0.003s 
2019-05-30T19:18:39.172Z : ::ffff:10.42.0.0 - GET /version - 304 - 0.112s 
2019-05-30T19:18:42.603Z : ::ffff:10.42.0.0 - POST /login 
2019-05-30T19:18:42.604Z : ::ffff:10.42.0.0 - POST /login - 200 - 0.001s 
2019-05-30T19:18:42.605Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:42.654Z : [session-file-store] will retry, error on last attempt: Error: ENOENT: no such file or directory, open '.sessions/cQjGKIMxZCDuJbNDjwAGfKTeAw1jBUyF.json' 
2019-05-30T19:18:42.707Z : ::ffff:127.0.0.1 - GET /api/clusters 
2019-05-30T19:18:42.708Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters 
2019-05-30T19:18:44.880Z : ::ffff:10.42.0.0 - POST /login - [closed by user] - 2.277s 
2019-05-30T19:18:47.711Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 5s... 
2019-05-30T19:18:52.716Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 10s... 
2019-05-30T19:18:57.721Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 15s... 
2019-05-30T19:19:02.726Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 20s... 
2019-05-30T19:19:07.732Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 25s... 
2019-05-30T19:19:12.736Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 30s... 
2019-05-30T19:19:17.740Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 35s... 
2019-05-30T19:19:22.744Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 40s... 
2019-05-30T19:19:27.746Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 45s... 
2019-05-30T19:19:32.749Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 50s... 
2019-05-30T19:19:37.753Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 55s... 
2019-05-30T19:19:42.756Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 60s... 
2019-05-30T19:19:47.761Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 65s... 
2019-05-30T19:19:52.766Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 70s... 
2019-05-30T19:19:57.771Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 75s... 
2019-05-30T19:20:02.777Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 80s... 
2019-05-30T19:20:07.783Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 85s... 
2019-05-30T19:20:12.786Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 90s... 
2019-05-30T19:20:17.791Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 95s... 
2019-05-30T19:20:22.793Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 100s... 
2019-05-30T19:20:27.797Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 105s... 
2019-05-30T19:20:32.800Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 110s... 
2019-05-30T19:20:37.804Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - wait for 115s... 
2019-05-30T19:20:42.714Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/clusters - Error: socket hang up 
2019-05-30T19:20:42.716Z : [HPM] Error occurred while trying to proxy request /clusters from 127.0.0.1:3000 to http://10.42.2.8:8080 (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:20:42.720Z : ::ffff:127.0.0.1 - GET /api/system/init 
2019-05-30T19:20:42.721Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init 
2019-05-30T19:20:47.724Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 5s... 
2019-05-30T19:20:52.730Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 10s... 
2019-05-30T19:20:57.736Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 15s... 
2019-05-30T19:21:02.740Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 20s... 
2019-05-30T19:21:07.745Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 25s... 
2019-05-30T19:21:12.750Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 30s... 
2019-05-30T19:21:17.754Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 35s... 
2019-05-30T19:21:22.759Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 40s... 
2019-05-30T19:21:27.764Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 45s... 
2019-05-30T19:21:32.768Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 50s... 
2019-05-30T19:21:37.772Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 55s... 
2019-05-30T19:21:42.776Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 60s... 
2019-05-30T19:21:47.781Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 65s... 
2019-05-30T19:21:52.786Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 70s... 
2019-05-30T19:21:57.795Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 75s... 
2019-05-30T19:22:02.801Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 80s... 
2019-05-30T19:22:07.806Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 85s... 
2019-05-30T19:22:12.808Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 90s... 
2019-05-30T19:22:17.812Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 95s... 
2019-05-30T19:22:22.815Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 100s... 
2019-05-30T19:22:27.816Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 105s... 
2019-05-30T19:22:32.819Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 110s... 
2019-05-30T19:22:37.824Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - wait for 115s... 
2019-05-30T19:22:42.725Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/init - Error: socket hang up 
2019-05-30T19:22:42.727Z : [HPM] Error occurred while trying to proxy request /system/init from 127.0.0.1:3000 to http://10.42.2.8:8080 (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors) 
2019-05-30T19:22:42.729Z : Cluster not initialized: Proxy error: Error: socket hang up 
2019-05-30T19:22:42.732Z : ::ffff:127.0.0.1 - GET /api/system/service-checkpoint 
2019-05-30T19:22:42.733Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint 
2019-05-30T19:22:47.736Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 5s... 
2019-05-30T19:22:52.742Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 10s... 
2019-05-30T19:22:57.745Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 15s... 
2019-05-30T19:23:02.750Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 20s... 
2019-05-30T19:23:07.755Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 25s... 
2019-05-30T19:23:12.760Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 30s... 
2019-05-30T19:23:17.766Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 35s... 
2019-05-30T19:23:22.770Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 40s... 
2019-05-30T19:23:27.776Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 45s... 
2019-05-30T19:23:32.777Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 50s... 
2019-05-30T19:23:37.782Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 55s... 
2019-05-30T19:23:42.788Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 60s... 
2019-05-30T19:23:47.792Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 65s... 
2019-05-30T19:23:52.796Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 70s... 
2019-05-30T19:23:57.802Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 75s... 
2019-05-30T19:24:02.805Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 80s... 
2019-05-30T19:24:07.809Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 85s... 
2019-05-30T19:24:12.815Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 90s... 
2019-05-30T19:24:17.816Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 95s... 
2019-05-30T19:24:22.819Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 100s... 
2019-05-30T19:24:27.821Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 105s... 
2019-05-30T19:24:32.827Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 110s... 
2019-05-30T19:24:37.829Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - wait for 115s... 
2019-05-30T19:24:42.737Z : [PROXY] ::ffff:127.0.0.1 - (user: anon) GET http://10.42.2.8:8080/system/service-checkpoint - Error: socket hang up 
2019-05-30T19:24:42.739Z : [HPM] Error occurred while trying to proxy request /system/service-checkpoint from 127.0.0.1:3000 to http://10.42.2.8:8080 (ECONNRESET) (https://nodejs.org/api/errors.html#errors_common_system_errors)

Logs rook-edgefs-nfs-nfs01

nfs pod restart loop

Running in container 
Populating /opt/nedge/etc from /opt/nedge/etc.target 
Configured /opt/nedge/etc/ccow/ccow.json 
Broker configuration adjusted. 
Starting ganesha in foreground. 
May 30 19:34:21.762 rook-edgefs-nfs-nfs01-cdffc54bd-dt5xb ccow[1] error   [22]  server-list.c:378  : SERVER LIST GET request timeout after 240 attempts seqid:1.480 
2019-05-30T19:34:21.769Z Invalid gRPC service nfs01

Logs rook-edgefs-target on edgefs-storage-1 node

daemon

Running in container 
Configured /opt/nedge/etc/ccow/ccow.json 
Configured /opt/nedge/etc/ccow/ccowd.json 
Server configuration adjusted. 
Waiting for corosync ... 
Waiting for corosync ... 
Waiting for corosync ... 
Waiting for corosync ... 
Waiting for corosync ... 
Reloading corosync.conf... 
Done 
Starting ccowd in foreground. 
May 30 19:00:44.413 rook-edgefs-target-0 ccowd[12790] notice  [12790]         util.c:654  : SERVERID: 951A46B4D029376A585DC6532EF8511E suffix= 
May 30 19:00:44.433 rook-edgefs-target-0 ccowd[12790] notice  [13354]  reptrans-rd.c:11573 : Local data store scsi-0QEMU_QEMU_HARDDISK_3dfa58e9-2d76-4d4b-b kdev=/dev/sdd initialized 
May 30 19:00:44.433 rook-edgefs-target-0 ccowd[12790] notice  [13354]  reptrans-rd.c:11573 : Local data store scsi-0QEMU_QEMU_HARDDISK_d3941ba0-6100-4f9b-a kdev=/dev/sdb initialized 
May 30 19:00:44.433 rook-edgefs-target-0 ccowd[12790] notice  [13354]  reptrans-rd.c:11573 : Local data store scsi-0QEMU_QEMU_HARDDISK_ae52db1b-2689-4802-a kdev=/dev/sde initialized 
May 30 19:00:44.433 rook-edgefs-target-0 ccowd[12790] notice  [13354]  reptrans-rd.c:11573 : Local data store scsi-0QEMU_QEMU_HARDDISK_c6ba7a04-4233-4d61-9 kdev=/dev/sdc initialized 
May 30 19:00:44.477 rook-edgefs-target-0 ccowd[12790] notice  [13478]  reptrans-rd.c:3775 : Dev(scsi-0QEMU_QEMU_HARDDISK_3dfa58e9-2d76-4d4b-b) plevel 0 opening main env using part /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_3dfa58e9-2d76-4d4b-b-part1 mapsize=104508776960 
May 30 19:00:44.485 rook-edgefs-target-0 ccowd[12790] notice  [13528]  reptrans-rd.c:3775 : Dev(scsi-0QEMU_QEMU_HARDDISK_d3941ba0-6100-4f9b-a) plevel 0 opening main env using part /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_d3941ba0-6100-4f9b-a-part1 mapsize=104508776960 
May 30 19:00:44.501 rook-edgefs-target-0 ccowd[12790] notice  [13478]  reptrans-rd.c:10538 : Dev(scsi-0QEMU_QEMU_HARDDISK_3dfa58e9-2d76-4d4b-b): journal 1:0 (32 maxentries 8388736 maxchunk) initialized on /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_3dfa58e9-2d76-4d4b-b-part2 
May 30 19:00:44.501 rook-edgefs-target-0 ccowd[12790] notice  [13528]  reptrans-rd.c:10538 : Dev(scsi-0QEMU_QEMU_HARDDISK_d3941ba0-6100-4f9b-a): journal 1:0 (32 maxentries 8388736 maxchunk) initialized on /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_d3941ba0-6100-4f9b-a-part2 
May 30 19:00:44.525 rook-edgefs-target-0 ccowd[12790] notice  [13594]  reptrans-rd.c:3775 : Dev(scsi-0QEMU_QEMU_HARDDISK_ae52db1b-2689-4802-a) plevel 0 opening main env using part /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_ae52db1b-2689-4802-a-part1 mapsize=104508776960 
May 30 19:00:44.529 rook-edgefs-target-0 ccowd[12790] notice  [13645]  reptrans-rd.c:3775 : Dev(scsi-0QEMU_QEMU_HARDDISK_c6ba7a04-4233-4d61-9) plevel 0 opening main env using part /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_c6ba7a04-4233-4d61-9-part1 mapsize=104508776960 
May 30 19:00:44.549 rook-edgefs-target-0 ccowd[12790] notice  [13594]  reptrans-rd.c:10538 : Dev(scsi-0QEMU_QEMU_HARDDISK_ae52db1b-2689-4802-a): journal 1:0 (32 maxentries 8388736 maxchunk) initialized on /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_ae52db1b-2689-4802-a-part2 
May 30 19:00:44.549 rook-edgefs-target-0 ccowd[12790] notice  [13645]  reptrans-rd.c:10538 : Dev(scsi-0QEMU_QEMU_HARDDISK_c6ba7a04-4233-4d61-9): journal 1:0 (32 maxentries 8388736 maxchunk) initialized on /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_c6ba7a04-4233-4d61-9-part2 
May 30 19:00:44.753 rook-edgefs-target-0 ccowd[12790] notice  [13354]     reptrans.c:6725 : Dev(/dev/sdd) loaded TS=1808231690 
May 30 19:00:44.753 rook-edgefs-target-0 ccowd[12790] notice  [13354]     reptrans.c:6725 : Dev(/dev/sdb) loaded TS=1808240612 
May 30 19:00:44.753 rook-edgefs-target-0 ccowd[12790] notice  [13354]     reptrans.c:6725 : Dev(/dev/sde) loaded TS=1808257555 
May 30 19:00:44.753 rook-edgefs-target-0 ccowd[12790] notice  [13354]     reptrans.c:6725 : Dev(/dev/sdc) loaded TS=1808289452 
May 30 19:00:44.757 rook-edgefs-target-0 ccowd[12790] notice  [13354]     clengine.c:1341 : Cluster Engine join cluster: <corosync> node:<951A46B4D029376A585DC6532EF8511E> ip:<::ffff:ffff:603:2a0a> 4over6:1 
May 30 19:00:44.757 rook-edgefs-target-0 ccowd[12790] notice  [13354]     clengine.c:315  : hostcount changed from 0 to 1 
May 30 19:00:44.765 rook-edgefs-target-0 ccowd[12790] notice  [13354]     clengine.c:334  : ringid changed from 0 to 8 
May 30 19:00:47.773 rook-edgefs-target-0 ccowd[12790] notice  [13354]     flexhash.c:2953 : flexhash_devs_perrow corrected ret: 4, mean: 4 
May 30 19:00:47.773 rook-edgefs-target-0 ccowd[12790] notice  [13354]     flexhash.c:2965 : flexhash_devs_perrow res: 4, delta: 0, rows: 8,  newdevs: 0, lvdevcount: 4 
May 30 19:00:47.773 rook-edgefs-target-0 ccowd[12790] notice  [13354]    rebalance.c:1416 : Flexhash rebalance completed numdevs: 4 devices_per_row: 4 numrows: 8 rows_per_dev: 8 
May 30 19:00:47.773 rook-edgefs-target-0 ccowd[12790] notice  [13354]     flexhash.c:1425 : Dumping the file to disk 
May 30 19:00:47.773 rook-edgefs-target-0 ccowd[12790] notice  [13354]     flexhash.c:1425 : Dumping the file to disk 
May 30 19:00:47.773 rook-edgefs-target-0 ccowd[12790] notice  [13354]     clengine.c:603  : Sending out a topo learn to 1 nr_nodes 
May 30 19:00:47.777 rook-edgefs-target-0 ccowd[12790] notice  [13354]     clengine.c:1029 : Received new FlexHash update genid=1559242847774286 
May 30 19:01:08.769 rook-edgefs-target-0 ccowd[12790] notice  [13354]     clengine.c:445  : Cluster is formed with 1 nodes

corosync

Running in container 
Configured /opt/nedge/etc/ccow/ccow.json 
Configured /opt/nedge/etc/ccow/ccowd.json 
Server configuration adjusted. 

Configured /opt/nedge/etc/corosync/corosync.conf to use IPv4 and explicit node list, netmtu=1450 
Configured /opt/nedge/etc/ccow/ccow.json 
Configured /opt/nedge/etc/ccow/ccowd.json, transport=rtrd 
RTRD configuration for NodeType: target {Devices:[{Name:scsi-0QEMU_QEMU_HARDDISK_3dfa58e9-2d76-4d4b-b Device:/dev/sdd Psize:16384 MdcacheReserved:0 HddReadahead:0 VerifyChid:1 Journal: Metadata: Bcache:0 BcacheWritearound:0 PlevelOverride:0 Sync:1} {Name:scsi-0QEMU_QEMU_HARDDISK_d3941ba0-6100-4f9b-a Device:/dev/sdb Psize:16384 MdcacheReserved:0 HddReadahead:0 VerifyChid:1 Journal: Metadata: Bcache:0 BcacheWritearound:0 PlevelOverride:0 Sync:1} {Name:scsi-0QEMU_QEMU_HARDDISK_ae52db1b-2689-4802-a Device:/dev/sde Psize:16384 MdcacheReserved:0 HddReadahead:0 VerifyChid:1 Journal: Metadata: Bcache:0 BcacheWritearound:0 PlevelOverride:0 Sync:1} {Name:scsi-0QEMU_QEMU_HARDDISK_c6ba7a04-4233-4d61-9 Device:/dev/sdc Psize:16384 MdcacheReserved:0 HddReadahead:0 VerifyChid:1 Journal: Metadata: Bcache:0 BcacheWritearound:0 PlevelOverride:0 Sync:1}]}Configured master daemon /opt/nedge/etc/ccow/rt-rd.json 
Configured /opt/nedge/etc/ccow/auditd.ini 

Configuration applied successfully. 

Configuration applied. 
Starting corosync in foreground. 
    ring0_addr: rook-edgefs-target-0.rook-edgefs-target.rook-edgefs.svc.cluster.local. 
    ring0_addr: rook-edgefs-target-1.rook-edgefs-target.rook-edgefs.svc.cluster.local. 
May 30 19:00:43 notice  [MAIN  ] Corosync Cluster Engine 2.99.3.91-87ee starting up 
May 30 19:00:43 info    [MAIN  ] Corosync built-in features: pie relro bindnow 
May 30 19:00:43 notice  [TOTEM ] Initializing transport (Kronosnet). 
May 30 19:00:43 info    [TOTEM ] totemknet initialized 
May 30 19:00:43 notice  [SERV  ] Service engine loaded: corosync configuration map access [0] 
May 30 19:00:43 info    [QB    ] server name: cmap 
May 30 19:00:43 notice  [SERV  ] Service engine loaded: corosync configuration service [1] 
May 30 19:00:43 info    [QB    ] server name: cfg 
May 30 19:00:43 notice  [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2] 
May 30 19:00:43 info    [QB    ] server name: cpg 
May 30 19:00:43 notice  [SERV  ] Service engine loaded: corosync profile loading service [4] 
May 30 19:00:43 notice  [QUORUM] Using quorum provider corosync_votequorum 
May 30 19:00:43 notice  [QUORUM] This node is within the primary component and will provide service. 
May 30 19:00:43 notice  [QUORUM] Members[0]: 
May 30 19:00:43 notice  [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5] 
May 30 19:00:43 info    [QB    ] server name: votequorum 
May 30 19:00:43 notice  [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3] 
May 30 19:00:43 info    [QB    ] server name: quorum 
May 30 19:00:43 info    [KNET  ] host: host: 29703 (passive) best link: 0 (pri: 1) 
May 30 19:00:43 warning [KNET  ] host: host: 29703 has no active links 
May 30 19:00:43 info    [KNET  ] host: host: 29703 (passive) best link: 0 (pri: 1) 
May 30 19:00:43 warning [KNET  ] host: host: 29703 has no active links 
May 30 19:00:43 info    [KNET  ] host: host: 29703 (passive) best link: 0 (pri: 1) 
May 30 19:00:43 warning [KNET  ] host: host: 29703 has no active links 
May 30 19:00:43 notice  [TOTEM ] A new membership (1022558633:8) was formed. Members joined: 1022558633 
May 30 19:00:43 warning [CPG   ] downlist left_list: 0 received 
May 30 19:00:43 notice  [QUORUM] Members[1]: 1022558633 
May 30 19:00:43 notice  [MAIN  ] Completed service synchronization, ready to provide service. 
May 30 19:00:44 notice  [CFG   ] Config reload requested by node 1022558633 
May 30 19:00:44 info    [TOTEM ] Configuring link 0 
May 30 19:00:44 info    [TOTEM ] Configured link number 0: local addr: 10.42.3.6, port=5405

auditd

Running in container
Configured /opt/nedge/etc/ccow/ccow.json
Configured /opt/nedge/etc/ccow/ccowd.json
Server configuration adjusted.
Waiting for corosync ...
Waiting for corosync ...
Waiting for corosync ...
Waiting for corosync ...
Waiting for corosync ...
Starting auditd in foreground.

Logs rook-edgefs-target on edgefs-gateway-1 node

daemon

Running in container
Configured /opt/nedge/etc/ccow/ccow.json
Configured /opt/nedge/etc/ccow/ccowd.json
Server configuration adjusted.
Waiting for corosync ...
Waiting for corosync ...
Waiting for corosync ...
Reloading corosync.conf...
Done
Starting ccowd in foreground.
May 30 19:00:27.667 rook-edgefs-target-1 ccowd[24937] notice [24937] util.c:654 : SERVERID: 9FA3178264D9A8088344DE4D8929FE14 suffix=
May 30 19:00:27.679 rook-edgefs-target-1 ccowd[24937] notice [25404] clengine.c:1341 : Cluster Engine join cluster: <corosync> node:<9FA3178264D9A8088344DE4D8929FE14> ip:<::ffff:ffff:902:2a0a> 4over6:1
May 30 19:00:27.695 rook-edgefs-target-1 ccowd[24937] notice [25404] clengine.c:315 : hostcount changed from 0 to 1
May 30 19:00:27.699 rook-edgefs-target-1 ccowd[24937] notice [25404] clengine.c:334 : ringid changed from 0 to 8

corosync

Running in container
Configured /opt/nedge/etc/ccow/ccow.json
Configured /opt/nedge/etc/ccow/ccowd.json
Server configuration adjusted.
DNS lookup retry 0 failed for rook-edgefs-target-1.rook-edgefs-target.rook-edgefs. waiting for next DNS lookup in 10 seconds
Configured /opt/nedge/etc/corosync/corosync.conf to use IPv4 and explicit node list, netmtu=1450
Configured /opt/nedge/etc/ccow/ccow.json
Configured /opt/nedge/etc/ccow/ccowd.json, transport=rtrd
RTRD configuration for NodeType: gateway {Devices:[]}Configured master daemon /opt/nedge/etc/ccow/rt-rd.json
Configured /opt/nedge/etc/ccow/auditd.ini
Configuration applied successfully.
Configuration applied.
Starting corosync in foreground.
ring0_addr: rook-edgefs-target-0.rook-edgefs-target.rook-edgefs.svc.cluster.local.
ring0_addr: rook-edgefs-target-1.rook-edgefs-target.rook-edgefs
May 30 19:00:23 notice [MAIN ] Corosync Cluster Engine 2.99.3.91-87ee starting up
May 30 19:00:23 info [MAIN ] Corosync built-in features: pie relro bindnow
May 30 19:00:23 notice [TOTEM ] Initializing transport (Kronosnet).
May 30 19:00:24 info [TOTEM ] totemknet initialized
May 30 19:00:24 notice [SERV ] Service engine loaded: corosync configuration map access [0]
May 30 19:00:24 info [QB ] server name: cmap
May 30 19:00:24 notice [SERV ] Service engine loaded: corosync configuration service [1]
May 30 19:00:24 info [QB ] server name: cfg
May 30 19:00:24 notice [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
May 30 19:00:24 info [QB ] server name: cpg
May 30 19:00:24 notice [SERV ] Service engine loaded: corosync profile loading service [4]
May 30 19:00:24 notice [QUORUM] Using quorum provider corosync_votequorum
May 30 19:00:24 notice [QUORUM] This node is within the primary component and will provide service.
May 30 19:00:24 notice [QUORUM] Members[0]:
May 30 19:00:24 notice [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
May 30 19:00:24 info [QB ] server name: votequorum
May 30 19:00:24 notice [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
May 30 19:00:24 info [QB ] server name: quorum
May 30 19:00:24 info [KNET ] host: host: 425 (passive) best link: 0 (pri: 1)
May 30 19:00:24 warning [KNET ] host: host: 425 has no active links
May 30 19:00:24 info [KNET ] host: host: 425 (passive) best link: 0 (pri: 1)
May 30 19:00:24 warning [KNET ] host: host: 425 has no active links
May 30 19:00:24 info [KNET ] host: host: 425 (passive) best link: 0 (pri: 1)
May 30 19:00:24 warning [KNET ] host: host: 425 has no active links
May 30 19:00:24 notice [TOTEM ] A new membership (2045867015:8) was formed. Members joined: 2045867015
May 30 19:00:24 warning [CPG ] downlist left_list: 0 received
May 30 19:00:24 notice [QUORUM] Members[1]: 2045867015
May 30 19:00:24 notice [MAIN ] Completed service synchronization, ready to provide service.
May 30 19:00:27 notice [CFG ] Config reload requested by node 2045867015
May 30 19:00:27 info [TOTEM ] Configuring link 0
May 30 19:00:27 info [TOTEM ] Configured link number 0: local addr: 10.42.2.9, port=5405

auditd

Running in container
Configured /opt/nedge/etc/ccow/ccow.json
Configured /opt/nedge/etc/ccow/ccowd.json
Server configuration adjusted.
Waiting for corosync ...
Waiting for corosync ...
Starting auditd in foreground.

Environment:

OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Kernel (e.g. uname -a):

Linux edgefs-client-1 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Cloud provider or hardware configuration:
on openstack
edgefs-brain-1 2x4
edgefs-gateway-1 4x8
edgefs-client-1 4x8
edgefs-storage-1 4x8, 4 disk X 100Gb
Rook version (use rook version inside of a Rook Pod):
edgefs/edgefs-operator:1.1.18
edgefs/edgefs:1.1.31
Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):

RKE (rancher 2.2.3)

EdgeFS bug

Source

Negashev

All 4 comments

Thank you for the comprehensive bug report! An impressive collection of the info!
I found this line in corosync.log:

May 30 19:00:43 notice  [QUORUM] Members[1]: 1022558633

That means that corosync coordination network that we use within a local cluster segment isn't seeing more than one member, i.e. in a split.

This is a known issue and we are working on it.
The workaround is to login to a target node(s) toolbox and execute the following command:

corosync-cfgtool -R

Monitor corosync container log, and make sure it has all members that you expecting to see, including the gateways.

dyusupov on 31 May 2019

👍1

@dyusupov Cool corosync-cfgtool -R helped me!

Negashev on 31 May 2019

The fix for it is on its way - https://github.com/Nexenta/edgefs/issues/279

dyusupov on 19 Jul 2019

The issue was fixed in https://github.com/Nexenta/edgefs/releases/tag/v1.2.23 . Reopen if still happens.

dyusupov on 20 Jul 2019

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Rook not creating PVC and errors

alialshamali · 3Comments

LoadBalancer IPs rather than ServiceIPs

ryholt · 4Comments

Rook does not deploy on k8s 1.5 anymore.

kokhang · 4Comments

minikube : monitor data filesystem reached concerning levels of available storage space

ksingh7 · 4Comments

Rook CSI plugin

kokhang · 3Comments