Docs: windows: Linux builds don't work on windows filesystems (via docker/WSL)

[ssmun]

I tried to test CockroachDB and followed instructions on this link: https://www.cockroachlabs.com/docs/stable/start-a-local-cluster-in-docker.html#os-windows

ERROR: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument

Log file:

I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] file created at: 2018/05/22 20:22:08
I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] running on machine: roach1
I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] binary: CockroachDB CCL v2.0.2 (x86_64-unknown-linux-gnu, built 2018/05/21 14:55:20, go1.10)
I180522 20:22:08.011632 1 util/log/clog.go:1134 [config] arguments: [/cockroach/cockroach start --insecure]
I180522 20:22:08.011632 1 util/log/clog.go:1134 line format: [IWEF]yymmdd hh:mm:ss.uuuuuu goid file:line msg utf8=✓
W180522 20:22:08.011625 1 cli/start.go:909 RUNNING IN INSECURE MODE!

• Your cluster is open for any client that can access .
• Any user, even root, can log in without providing a password.
• Any user, connecting as root, can read or write any data in your cluster.
• There is no network encryption nor authentication, and thus no confidentiality.

Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v2.0/secure-a-cluster.html
I180522 20:22:08.046850 1 server/config.go:330 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
W180522 20:22:08.066892 1 cli/start.go:812 Using the default setting for --cache (128 MiB).
A significantly larger value is usually needed for good performance.
If you have a dedicated server a reasonable setting is --cache=.25 (495 MiB).
I180522 20:22:08.067224 1 server/config.go:330 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
W180522 20:22:08.067282 1 cli/start.go:825 Using the default setting for --max-sql-memory (128 MiB).
A significantly larger value is usually needed in production.
If you have a dedicated server a reasonable setting is --max-sql-memory=.25 (495 MiB).
I180522 20:22:08.067330 1 cli/start.go:923 CockroachDB CCL v2.0.2 (x86_64-unknown-linux-gnu, built 2018/05/21 14:55:20, go1.10)
I180522 20:22:08.296553 1 server/config.go:330 available memory from cgroups (8.0 EiB) exceeds system memory 1.9 GiB, using system memory
I180522 20:22:08.296800 1 server/config.go:430 system total memory: 1.9 GiB
I180522 20:22:08.297016 1 server/config.go:432 server configuration:
max offset 500000000
cache size 128 MiB
SQL memory pool size 128 MiB
scan interval 10m0s
scan max idle time 200ms
event log enabled true
I180522 20:22:08.297153 1 cli/start.go:789 using local environment variables: COCKROACH_CHANNEL=official-docker
I180522 20:22:08.297233 1 cli/start.go:796 process identity: uid 0 euid 0 gid 0 egid 0
I180522 20:22:08.297314 1 cli/start.go:461 starting cockroach node
I180522 20:22:08.317992 21 storage/engine/rocksdb.go:552 opening rocksdb instance at "/cockroach/cockroach-data/cockroach-temp451412792"
E180522 20:22:08.454632 1 cli/error.go:109 failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument
Error: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument

[quartex]

Hi,
the same for me on Windows 8 with Docker 18.03.01

docker run -d --name=roach1 --hostname=roach1 --net=roachnet -p 26257:26257 -p 8080:8080 -v "D:dockercockroachdata1:/cockroach/cockroach-data" cockroachdb/cockroach:v2.0.2 start --insecure

report me

• ERROR: failed to start server: could not create temp storage: could not open rocksdb instance: IO error: While fsync: a directory: Invalid argument
  *
  Failed running "start"

[jseldess]

Thanks for the report, @ssmun and @quartex.

@BramGruneir, @bdarnell, @a-robinson, based on the provided logs, do you have ideas about what's going on?

[bdarnell]

Our linux builds don't currently work very well on windows filesystems (this affects both the windows linux subsystem and docker). We recommend using the native windows binaries on windows. Or if you want to run linux on windows for some reason, create and mount a linux filesystem instead of having them write directly to your windows filesystem.

[quartex]

Hi,
I had the same problem with Mysql with Direct_io settings that use fsync.
Using a docker-compose with all 3 nodes settings inside it and using VOLUMES instead directory remapping for storage all works on Windows 10 with last version of Docker!

I'm using a docker.compose.yaml like this

`
version: '3'

services:

roach1:
volumes:
#- D:dockercockroachdata1:/cockroach/cockroach-data
- data-volume1:/cockroach/cockroach-data
image: cockroachdb/cockroach:v2.0.2
command: start --insecure
ports:
- 26257:26257
- 8080:8080
networks:
- roachnet
roach2:
volumes:
#- D:dockercockroachdata2:/cockroach/cockroach-data
- data-volume2:/cockroach/cockroach-data
image: cockroachdb/cockroach:v2.0.2
depends_on:
- roach1
command: start --insecure --join=roach1
networks:
- roachnet
roach3:
volumes:
#- D:dockercockroachdata3:/cockroach/cockroach-data
- data-volume3:/cockroach/cockroach-data
image: cockroachdb/cockroach:v2.0.2
depends_on:
- roach1
command: start --insecure --join=roach1
networks:
- roachnet
networks:
roachnet:
volumes:
data-volume1:
data-volume2:
data-volume3:
`
Thank you

[BramGruneir]

@quartex Wonderful!

We'll try that out and update our docs accordingly.

[jazoom]

I have been using a CockroachDB cluster in Swarm Mode in Docker for Windows for a few months now and it has worked well (was using volumes). I'd never tried direct path mapping to Windows file system. What's actually the difference that makes one work and not the other? It seems docker volumes are pretty much the same thing except docker keeps the data in a special directory on the host that you don't need to know the path of.

Now I'm moving everything over to Kubernetes in Docker for Windows and it seems hostPath isn't working for CockroachDB. I haven't found an equivalent of the docker volumes here that work around this issue. What can we do in Kubernetes?

[jazoom]

Circling back to answer my own question in case it helps someone else in future.

This is how I got around this issue in Kubernetes in Docker for Windows.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: cockroachdb
spec:
  replicas: 1
  template:
    spec:
      volumes:
      - name: data
        $patch: delete
      - name: data
        emptyDir: {}
      - name: dump
        hostPath:
          path: /E/kubernetes-volumes/cockroachdb-dump
          type: DirectoryOrCreate
      initContainers:
      - name: restore-data
        image: busybox:1.29.2
        command: ['sh', '-c', 'if [ -e /cockroach/dump/data/ ]; then echo "restoring data..." && cp -rf /cockroach/dump/data/. /cockroach/cockroach-data/ && echo "data restored"; fi']
        volumeMounts:
        - name: data
          mountPath: /cockroach/cockroach-data
        - name: dump
          mountPath: /cockroach/dump
      containers:
      - name: cockroachdb
        volumeMounts:
        - name: dump
          mountPath: /cockroach/dump
        command:
        - "/bin/bash"
        - "-ecx"
        - "exec /cockroach/cockroach start --insecure --logtostderr --cache 25% --max-sql-memory 25% --http-host 0.0.0.0"
        lifecycle:
          preStop:
            exec:
              command:  ['sh', '-c', 'echo "dumping data..." && cp -r /cockroach/cockroach-data/ /cockroach/dump/data-temp/ && rm -rf /cockroach/dump/data/ && mv /cockroach/dump/data-temp/ /cockroach/dump/data/ && echo "data dumped"']

The above is actually the file I use to patch the production cockroachdb.yaml with Kustomize (which is where $patch: delete comes from, which isn't valid Kubernetes config YAML). It essentially shows the changes that need to be made to a normal cockroachdb config.

It's a little hacky, but it seems to work well. After the container has received the signal to terminate it saves CockroachDB's data to a hostPath. Before the pod starts next time an initContainer runs and restores that data to CockroachDB's data directory. I used an initContainer instead of a postStart hook because postStart runs in parallel to the container's entry command.

This gets around the issue because CockroachDB doesn't ever have to do its operations on a hostPath, yet the data is still persisted to a hostPath between pod restarts.

[BramGruneir]

@rmloveland This needs to be tested and our docs need to be updated, we're leading users astray.

[jazoom]

Also I should add that my solution above does have a problem, so there still isn't a good solution to this. The problem is that the preStop hook doesn't seem to run when shutting down Docker or Windows, which means I need to manually delete the CockroachDB pod if I want the current state persisted to disk. It's pretty annoying.

[jseldess]

@lhirata, @rolandcrosby, this problem has come up several time, so we may want to prioritize a docs update for 2.2.

[dlouwers]

Is there a proposed workaround like a certain modification to the cockroachdb-statefulset-secure.yaml file to make this work on WIndows?

[xambas84]

sorry for the bump. but are there any plans to fix this?

[jseldess]

Thanks for the bump, @xambas84. I'm looking into this now. I think this issue encompasses two separate issues. I'll update again soon.

[jseldess]

@jazoom, @dlouwers, @xambas84, I've split the Kubernetes-specific problem into a new issue: https://github.com/cockroachdb/docs/issues/4333. We're planning to update our docs for the Docker problem for our April release. We'll aim to address the Kubernetes issue as well, if time allows.

[jseldess]

For this issue:

• [ ] Remove docker and kubernetes from the install-cockroachdb-windows page and replace with suggestion that you can run a linux vm.
• [ ] Figure out and document more complete instructions (exploring @quartex's docker-compose suggestion first).

[jseldess]

Adding @rolandcrosby for PM support.

[louishust]

I encounter the same problem with vagrant where the crdb is in the /vagrant shared directory.

[jseldess]

I should've passed this to you long ago, @taroface and @johnrk. The crux seems to be that running Cockroach via Docker or Kubernetes on Windows doesn't work because our linux builds don't currently work very well on windows filesystems.

I've split off a separate issue to focus on Kubernetes: https://github.com/cockroachdb/docs/issues/4333. I'm not sure that was the right approach. Feel free to collapse into one again.

@bdarnell said:

Our linux builds don't currently work very well on windows filesystems (this affects both the windows linux subsystem and docker). We recommend using the native windows binaries on windows. Or if you want to run linux on windows for some reason, create and mount a linux filesystem instead of having them write directly to your windows filesystem

@bdarnell, I'm trying to understand the implications:

• Does this mean that if you want to run in either Kubernetes, Docker, or in a Linux VM on Windows, you need to install a linux filesystem (which?) and make sure volumes map to that?
• A user above suggested that running in docker was working when they weren't mapping volumes directly to the host. Does that suggest another option (just for testing purposes)?

[bdarnell]

you need to install a linux filesystem (which?)

Maybe that would work, but it seems complicated, and I doubt it's going to be a great experience.

A user above suggested that running in docker was working when they weren't mapping volumes directly to the host. Does that suggest another option (just for testing purposes)?

Yes, for testing purposes (i.e. all your data goes away on restart) this would work.

We should redo some tests here - it's possible that things have gotten better (WSL 2 made a lot of improvements in compatibility but I'm not sure if it fixes the filesystem-level concerns). But assuming the old comments here are still accurate, I think we should probably tell people not to use docker or WSL on windows, and either use the native windows binaries or a full-fledged linux VM.