Parity-ethereum: Parity does not start at Kubernetes with Azure File Share storage

[TimBobkov]

I'm running:

• Which Parity version?: 1.8.4 (Docker image)
• Which operating system?: Linux node at Azure Kubernetes (AKS)
• How installed?: via kubectl
• Are you fully synchronized?: no
• Did you try to restart the node?: yes

I use PoA config converted to Kubernetes style without members.
When I use as volume Azure Disk all start and work properly. But When I switch to Azure File Share - parity goes down with error: "Client service error: Client(Database("IO error: directory: Invalid argument"))". I have check access to folder by root user wich is used in container - user has full access to file share. Parity also create missed folders and files during startup.
Full startup log:

Loading config file from /parity/config/authority.toml
2017-12-14 17:07:00 UTC Starting Parity/v1.8.4-beta-c74c8c1-20171211/x86_64-linux-gnu/rustc1.22.1
2017-12-14 17:07:00 UTC Keys path /root/.local/share/io.parity.ethereum/keys/parity-poa-playground
2017-12-14 17:07:00 UTC DB path /root/.local/share/io.parity.ethereum/chains/parity-poa-playground/db/d24f9ddc2dd71e78
2017-12-14 17:07:00 UTC Path to dapps /root/.local/share/io.parity.ethereum/dapps
2017-12-14 17:07:00 UTC State DB configuration: fast
2017-12-14 17:07:00 UTC Operating mode: active
2017-12-14 17:07:02 UTC Configured for parity-poa-playground using AuthorityRound engine
Client service error: Client(Database("IO error: directory: Invalid argument"))

Steps to reproduce

1. Deploy Azure AKS service with Kubernetes version 1.8.1
2. Create Storage account
3. Create file share "Authority2"
4. Create secret for accessing storage account (use this docs: https://docs.microsoft.com/en-us/azure/aks/azure-files)
5. Change paths in authority.toml
   5.1. chain = "/root/.local/share/io.parity.ethereum/config/chain.json"
   5.2. password = ["/root/.local/share/io.parity.ethereum/authority.pwd"]
6. Copy config files to file share:
   6.1. /config to /config
   6.2. /keys to /keys
   6.3. /authorities/authority2.json to /keys/authority.json
   6.4. /authorities/authority2.pwd to /authority.pwd
   6.5. /node2.network.key to /network/key
7. Deploy Parity authority:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: authority2
namespace: blockchain-test
spec:
replicas: 1
template:
metadata:
labels:
app: authority2
spec:
containers:
- name: authority2
image: parity/parity:v1.8.4
command: ["/parity/parity"]
args: ["--config", "/root/.local/share/io.parity.ethereum/config/authority.toml", "--gasprice", "0", "--gas-floor-target", "0", "--engine-signer", "0x002e28950558fbede1a9675cb113f0bd20912019"]
ports:
- containerPort: 8545
- containerPort: 30303
volumeMounts:
- name: authority2
mountPath: /root/.local/share/io.parity.ethereum
readOnly: false
restartPolicy: Always
volumes:
- name: authority2
azureFile:
secretName: azure-secret
shareName: authority2
readOnly: false

Expected behavior
Normal start of Parity with PoA network

Actual
Error:

Loading config file from /parity/config/authority.toml
2017-12-14 17:07:00 UTC Starting Parity/v1.8.4-beta-c74c8c1-20171211/x86_64-linux-gnu/rustc1.22.1
2017-12-14 17:07:00 UTC Keys path /root/.local/share/io.parity.ethereum/keys/parity-poa-playground
2017-12-14 17:07:00 UTC DB path /root/.local/share/io.parity.ethereum/chains/parity-poa-playground/db/d24f9ddc2dd71e78
2017-12-14 17:07:00 UTC Path to dapps /root/.local/share/io.parity.ethereum/dapps
2017-12-14 17:07:00 UTC State DB configuration: fast
2017-12-14 17:07:00 UTC Operating mode: active
2017-12-14 17:07:02 UTC Configured for parity-poa-playground using AuthorityRound engine
Client service error: Client(Database("IO error: directory: Invalid argument"))

[5chdn]

Loading config file from /parity/config/authority.toml

Please share your config :)

[TimBobkov]

For now I use this config:

[parity]
chain = "/root/.local/share/io.parity.ethereum/config/chain.json"

[rpc]
interface = "0.0.0.0"
cors = "all"
hosts = ["all"]
apis = ["web3", "eth", "net", "parity", "traces", "rpc", "personal", "parity_accounts", "signer", "parity_set"]

[network]
bootnodes = [
"enode://147573f46fe9f5cc38fbe070089a31390baec5dd2827c8f2ef168833e4d0254fbee3969a02c5b9910ea5d5b23d86a6ed5eabcda17cc12007b7d9178b6c697aa5@10.0.166.1:30303",
"enode://1412ee9b9e23700e4a67a8fe3d8d02e10376b6e1cb748eaaf8aa60d4652b27872a8e1ad65bb31046438a5d3c1b71b00ec3ce0b4b42ac71464b28026a3d0b53af@10.0.166.2:30303",
"enode://9076c143a487aa163437a86f7d009f257f405c50bb2316800b9c9cc40e5a38fef5b414a47636ec38fdabc8a1872b563effa8574a7f8f85dc6bde465c368f1bf5@10.0.166.3:30303"
]

[account]
password = ["/root/.local/share/io.parity.ethereum/authority.pwd"]

[mining]
reseal_on_txs = "none"

[ddorgan]

Is it possible to run this with strace -fF and check at the end where you see "IO error: directory: Invalid argument" in the logfile. It should have a filename with an open() in front of that somewhere near the end of the logs.

What's the filename?

[TimBobkov]

The log file attached.
log.LOG

The last opened files:
open("/tmp/io.parity.ethereum/chains/parity-poa-playground/db/d24f9ddc2dd71e78/overlayrecent/db/003192.log", O_RDWR|O_CREAT|O_TRUNC, 0644) = 16
openat(AT_FDCWD, "/tmp/io.parity.ethereum/chains/parity-poa-playground/db/d24f9ddc2dd71e78/overlayrecent/db", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 17

P.S. I have change config little bit for running test parity from different location. All other configs are the same.

[folsen]

Closing as stale.

[TimBobkov]

Hmmm... @folsen you close issue, but it still present... I check it right now with Parity 1.11.1 from Docker Hub.

So maybe there is some solution for this issue?

[folsen]

@TimBobkov I don't think anyone here is an expert at running nodes in K8s on Azure, I know other people have done it before so it's not impossible. Do you have any suggested configuration changes to any particular part of our stack?

[TimBobkov]

@folsen It's not necessary to use K8s for reproduce this issue. You can just use Azure File Share or any SMB shares. Btw with NFS share everything works fine. Does Parity use symbolic link anywhere?

[Tbaut]

Maybe @ddorgan or @fevo1971 will be able to have another look :)

[TimBobkov]

How can I contact them?

[Tbaut]

I pinged them so they will have a look at the issue and let you know if they have questions or thoughts ;)

[ddorgan]

We have run parity on k8s. This is a known working config:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: default
  name: pv-default-100g-disk01
  annotations:
    volume.beta.kubernetes.io/storage-class: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
  name: parity-service
  namespace: default
spec:
  selector:
    app: parity
  ports:
    - name: eth-net
      port: 30303
      protocol: TCP
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: parity-config
  namespace: default
data:
  parity.toml: |
    [parity]
    mode = "dark"
    base_path = "/data"
    [footprint]
    db_compaction = "hdd"
    pruning_memory = 128
    tracing = "off"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: parity
  namespace: default
  labels:
    app: parity
spec:
  replicas: 1
  selector:
    matchLabels:
      app: parity
  template:
    metadata:
      labels:
        app: parity
        name: parity
    spec:
      terminationGracePeriodSeconds: 20
      containers:
        - image: parity/parity:v1.9.4
          name: parity
          imagePullPolicy: IfNotPresent
          args: ["--config=/config/parity.toml"]
          volumeMounts:
            - name: parity-config
              mountPath: /config
            - name: pv 
              mountPath: /data
      volumes:
        - name: parity-config
          configMap:
            name: parity-config
        - name: pv
          persistentVolumeClaim:
            claimName: pv-default-100g-disk01

[TimBobkov]

@ddorgan does your PV config use File Share or Blob storage?
And can you explain what does "mode", "pruning_memory" and "tracing" configs do?

[ddorgan]

@TimBobkov please try with blob storage. For the other question can you please open another issue or ask on gitter.

[TimBobkov]

@ddorgan but I was asking about File Storage... I know that blob storage working fine, but I need File Storage.

[TimBobkov]

As you can see at last screen the authority.toml loading successfully. And when I run the same config file with storing data at NFS storage - everything work fine. But when I change storage to SMB - the error above appears.
The difference in locations is corresponding to changing configuration during time.

[fevo1971]

@TimBobkov running parity on a network-file-system (at least on CIFS) is not possible
at the moment, so this is not related to azure / kubernetes, it's related
to the underlying file-system.

best solution is to use kubernetes persistent volumes like in our example
posted above or describes here: https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv

maybe(!) by tweaking the mount-options for your container.
as described here: http://unethicalblogger.com/2017/12/01/aks-storage-research.html?utm_source=feedburner&utm_medium=twitter&utm_campaign=Feed%3A%20UnethicalBlogger%20(unethical%20blogger)#azure-file

[TimBobkov]

Thanks for answer @fevo1971.
I already try PV with mount options, but I don't know which one I should add :( do you have any suggestions?

[onpaws]

@ddorgan thanks for the .yaml config, that's helpful.

@fevo1971 are you aware of any Parity PRs or open issues for supporting CIFS shares?
I'm curious what the technical reason is (e.g. does Parity data depend on symlinks/etc)

[onpaws]

I have a "Parity on k8n" security question and would appreciate any insight.

So I deployed the Parity k8n manifests @ddorgan shared above -- they're great, thanks!
The logs on the Parity pod show Parity syncing to the blockchain - excellent.

I see the k8n docs indicate services of type ClusterIP "makes the service only reachable from within the cluster."
But because I see the blockchain syncing in the logs I think they mean something like a traditional NAT setup - traffic _originating_ inside my Pod can get out, but originating _outside_ the cluster won't route to the Service/Pod.

OK, so far so good.

Enter my app, which I've Dockerized, and depends on the Parity JSON-RPC. I know I shouldn't JSON-RPC to the outside world. At home, I know NAT should prevent direct connections to my machines behind the router. I believe, but am not 100% sure, it should be safe to adjust my service to this...what do you think?

apiVersion: v1
kind: Service
metadata:
  name: parity-service
  namespace: default
spec:
  selector:
    app: parity
  ports:
    - name: eth-net
      port: 30303
      protocol: TCP
    - name: json-rpc-http
      port: 8545
      protocol: TCP
    - name: json-rpc-ws
      port: 8546
      protocol: TCP

[Tbaut]

@onpaws this is a different issue right? Please open a different issue.

[onpaws]

Yes I suppose so. Sorry for hijacking, will open a new issue.

[Tbaut]

I'll close as stale.