Cockroach: docs: can't use cockroachdb on Windows Docker

[JKhawaja]

Error:
log: exiting because of error: log: cannot create log: symlink cockroach-data/logs/cockroach.roach1. root.2016-12-29T07_16_39Z.000001.INFO.log cockroach-data/logs/cockroach.INFO: protocol error

This is a container runtime error (causing the container to Exit within seconds) that occurs after starting a container via: $ docker run --name=roach1 --hostname=roach1 --net=roachnet -p 26257:26257 -p 8080:8080 -v "${PWD} /cockroach-data/roach1:/cockroach/cockroach-data" cockroachdb/cockroach:beta-20161215 start --insecure via Docker ToolBox on Windows.

[knz]

You are running into http://unix.stackexchange.com/questions/197642/getting-protocol-error-when-trying-to-create-a-symlink-in-docker

This is because Windows does not support symlinks in this way. This problem not specific to CockroachDB.

I guess we may need to tweak our docs to account for Windows-related limitations in Docker.

[knz]

This is a dup of https://github.com/cockroachdb/docs/issues/907

[bdarnell]

This isn't quite the same as cockroachdb/docs#907. That issue is about discouraging people from using docker when they have other options (unless they have strong preexisting commitments to docker). Docker is our sole recommendation for windows. It looks like now that the docker-machine VM is no longer used, the docker-on-windows environment is somewhat non-standard. We need to either tolerate windows filesystems mounted into docker (would it be enough to ignore errors when creating these log symlinks, or will rocksdb also have problems on such a filesystem?) or come up with alternate recommendations for windows users (even if that means running a full-fledged linux VM).

[a-robinson]

Ugh, yeah that's rough. You have to wonder whether Docker will go back to making it easy to run Linux container images in VMs on Windows in addition to running Windows container images. Judging by posts on their forums, they mostly got rid of the UX around that but it seems as though they could bring it back easily if they wanted.

[knz]

Someone tells me that we may have success using / advertising the new "linux mode" in Windows 10. This gets you a native unix shell where (perhaps) linux docker can run natively.

[JKhawaja]

I think the direction this should go is "windows currently not supported" as Bash on Ubuntu on Windows is a feature for Windows 10 users only. The cockroach binary actually runs in the Linux sub-system environment (on Windows 10) but when trying to start a node via cockroach start --background you get a Unix datagram socket error listen unixgram /tmp/sdnotify*/notify.sock: setsockopt: invalid argument. I think it is trying to send data through a listening socket?

The other direction is to ignore the symlink errors and try to get the docker image to run on Windows. RocksDB apparently has a Windows port (done my Microsoft): RocksDB Windows Port ... but I personally know nothing about RocksdB.

This blogpost gives a better rough idea on the purpose of Ubuntu on Windows: Ubuntu on Windows if you still want to pursue that direction. I think trying to get Docker to work on Ubuntu on Windows might be more of a hassle than trying to get the cockroach binary to work (if anything is going to work).

Overall, the direction I would suggest is def telling Windows users to use a Linux VM (for now).

Edit: uh yeah, according to this video: Linux Command Line on windows the Linux sub-system is basically a kernel driver that is translating Linux system calls to Windows system calls (you are not getting the Linux kernel or any virtualization) + a (reduced) Ubuntu userspace. So, it's cool but not helpful here.

[knz]

Thanks for your comments.
The listen error is a bug I think we could look into this - #12755.

[bdarnell]

The unix datagram socket is only used with start --background. If you start the process in the foreground then it will at least get further in the initialization process.

[JKhawaja]

@bdarnell Foreground process started with cockroach start produces an open file descriptors limit error (2048 for my machine). Trying to alter via production settings: CockroachDB Production Settings does not actually work for me. But, there seems to be something constantly overriding the limit as this blog post suggests: Raising the Maximum Number of File Descriptors under the Finding That One Last Thing You Still Need to Do section.

I created an issue for it here: Microsoft/BashOnWindows#1576 ... but it may just be me doing something wrong too. ¯\_(ツ)_/¯

[bdarnell]

Hmm. 2048 file descriptors should be plenty for a brand-new cluster. We recommend more than that because you'll need more as your data and traffic grow, but it should take a while to reach that level. There must be something else going on here.

[JKhawaja]

Sorry, should have pasted the full error in previous comment.

Error: cockroach server exited with error: failed to create engines: hard open file descriptor limit of 2048 is under the minimum required 2256
please see https://www.cockroachlabs.com/docs/recommended-production-settings.html for more details
Failed running "start"

[bdarnell]

Ah, so that's our error and not actually running out of file descriptors. Maybe we should just tweak the constants to require 1500 file descriptors per store instead of 2000. The default values we had encountered in practice were either 512 FDs or 4096+, so it didn't matter too much where exactly we set the limit, but if there's a common platform where the default limit is 2048 then we might as well aim to stay under that limit.

[JKhawaja]

@bdarnell

From: https://github.com/Microsoft/BashOnWindows/issues/1576#issuecomment-271417029

Most of the limits are not currently supported, but we are tracking the work.

And it seems like it might take Microsoft some time getting around to supporting limits. So, do you think that this should be opened as a separate issue for trying to get cockroach start to work on Bash on Windows?

[bdarnell]

I don't think supporting the windows linux subsystem should be a priority; it's likely to have more rough edges than docker or a full VM. I'd be happy to be proven wrong, though.

[BramGruneir]

cc: @jseldess

So I spent some time getting cockroach to work in BashOnWindows. There are a few steps that are required. I'm going to describe each step here:

1) Install the linux subsystem: https://msdn.microsoft.com/en-us/commandline/wsl/install_guide
2) Update the versions of CLIBCXX and CXXABI

sudo apt-get install libstdc++6
sudo add-apt-repository ppa:ubuntu-toolchain-r/test 
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

3) Copy the latest binary and run it.

However, that won't work right now as pointed out earlier in this thread due to the system's hard open file descriptor limit. This limit cannot be incrased:
https://wpdev.uservoice.com/forums/266908-command-prompt-console-bash-on-ubuntu-on-windo/suggestions/17310124-add-ability-to-change-max-number-of-open-files-for

I created a custom binary with a lower FD limit and it worked perfectly. I'm going create a PR to lower the minimum number of open FD to 2000 instead of 2256 and I think this would be a viable (but experimental) option for running cockroach on windows without docker.

A few other notes:

• Similar to docker, windows pro 10 is required.
• It is not possible to build cockroach, or even call go get right now. So this is not an option for cockroach development, only for running local nodes.
• It's not even possible to build Golang from source either.
• I think keeping this as an experimental way to run cockroach in windows should be fine, but we should not spend too much time supporting it as there may be some hidden dragons.

In the end, we are going to need a native windows executable instead of relying on these virtualizations. Docker, VMs (HyperV or VirtualBox) and even the Bash on Windows subsystem is not enough.

[tamird]

FWIW, you should not need to run step 2 for any binary produced after
https://github.com/cockroachdb/cockroach/pull/13102.

On Mon, Feb 6, 2017 at 4:56 PM, Bram Gruneir [e-mail]
wrote:

cc: @jseldess https://github.com/jseldess

So I spent some time getting cockroach to work in BashOnWindows. There are
a few steps that are required. I'm going to describe each step here:

1. Install the linux subsystem: https://msdn.microsoft.com/en-
   us/commandline/wsl/install_guide
   https://msdn.microsoft.com/en-us/commandline/wsl/install_guide
2. Update the versions of CLIBCXX and CXXABI

sudo apt-get install libstdc++6
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade

1. Copy the latest binary and run it.

However, that won't work right now as pointed out earlier in this thread
due to the system's hard open file descriptor limit. This limit cannot be
incrased:
https://wpdev.uservoice.com/forums/266908-command-prompt-
console-bash-on-ubuntu-on-windo/suggestions/17310124-
add-ability-to-change-max-number-of-open-files-for

I created a custom binary with a lower FD limit and it worked perfectly.
I'm going create a PR to lower the minimum number of open FD to 2000
instead of 2256 and I think this would be a viable (but experimental)
option for running cockroach on windows without docker.

A few other notes:

• Similar to docker, windows pro 10 is required.
• It is not possible to build cockroach, or even call go get right
  now. So this is not an option for cockroach development, only for running
  local nodes.
• It's not even possible to build Golang from source either.
• I think keeping this as an experimental way to run cockroach in
  windows should be fine, but we should not spend too much time supporting it
  as there may be some hidden dragons.

In the end, we are going to need a native windows executable instead of
relying on these virtualizations. Docker, VMs (HyperV or VirtualBox) and
even the Bash on Windows subsystem is not enough.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cockroachdb/cockroach/issues/12593#issuecomment-277826824,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABdsPILzYf5e8lZlA6OmJ9r7iMeEeVogks5rZ5cDgaJpZM4LXZvt
.

[bdarnell]

@tamird's last comment refers to TYPE=release binaries; the default build will still require the other libraries to be installed/updated.

[petermattis]

@BramGruneir This has been fixed, right?

[BramGruneir]

Yep, closing.