Che: BC break in Docker Che entrypoint.sh

[davidwindell]

A recent change to entrypoint.sh https://github.com/eclipse/che/commit/4813a1a712abb1e0bb949b1c2dcdc7a7b206fd63#diff-1da4013f60e1b92b5613cf9680423a88 seems to have caused our environments to start up without the users database or custom stacks.

Are there upgrade docs for this or do we need to set CHE_HOME in an env var?

[benoitf]

@riuvshin ?

[davidwindell]

For context, here is the current service definition

che-user:
  environment:
    - CHE_DOCKER_IP_EXTERNAL=user.cheserverdomain.com
    - CHE_DOCKER_SERVER__EVALUATION__STRATEGY=docker-local
    - CHE_PORT=80
    - CHE_WORKSPACE_VOLUME=/var/run/docker.sock:/var/run/docker.sock;/root/data/eclipse/users/user/ssh:/root/.ssh
    - VIRTUAL_HOST=user.cheserverdomain.com
  expose:
    - '80'
  image: 'eclipse/che-server:nightly'
  restart: always
  volumes:
    - '/var/run/docker.sock:/var/run/docker.sock'
    - '/root/data/eclipse/users/user:/data'
    - '/root/data/eclipse/templates:/data/templates'
    - '/root/data/eclipse/stacks/stacks.json:/tmp/stacks.json'

[ghost]

@davidwindell using nightly images in a real env can be risky. In your case one nightly setup has become broken for another nightly. And at this point I don't know how to reproduce, since these are not tagged images.

Sure, @riuvshin and myself will take a look at yoru composefile and changes in the commit.

[riuvshin]

hi @davidwindell I have doubts that this commit https://github.com/eclipse/che/commit/4813a1a712abb1e0bb949b1c2dcdc7a7b206fd63#diff-1da4013f60e1b92b5613cf9680423a88 broke your che it just allows to use properties files to configure CHE. As @eivantsov said you should use only released tag versions in real env, nightly could be broken time to time due to active development.
So could you please double check that your data folder is still there with projects etc. Also please send us a logs.

[riuvshin]

@benoitf @eivantsov here is last commit that affects docker-compose file: https://github.com/eclipse/che/commit/a13068e1403ad3eeb618f726df2000d84942a7ac

[riuvshin]

@davidwindell could you please also start your workspace, execute docker inspect <WS_CONTAINER_ID> and share output with us

[davidwindell]

Hi chaps, no need for a lecture about nightly ;) I'm raising these issues to help QA!

Ok, so as suspected, the change to the entrypoint.sh script is indeed a BC break. Before the change, if there was no config file in /conf, the script would create a fresh one in /data/conf and set CHE_LOCAL_CONF_DIR to /data/conf. This change gets rid of all the logic that copies the built in conf file to /data and also means that if there is no /conf/che.properties file then the mounts never get set properly etc.

I think this change needs reverting and some more thought put into how this logic should work.

[riuvshin]

@davidwindell

Before the change, if there was no config file in /conf, the script would create a fresh one in /data/conf and set CHE_LOCAL_CONF_DIR to /data/conf

yes but this functionality was never working, and we agreed to remove this part because we configure CHE with ENV variables. So no need to manage default che.properties this way.
And again ANY custom mounted OR embedded che.properties weren't worked before this commit, it was broken and this commit actually fixes using custom properties.

And more we still have default embedded https://github.com/eclipse/che/blob/master/assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/codenvy/che.properties
in WAR.

The default configuration of CHE is defined in che.env and overrides everything in https://github.com/eclipse/che/blob/master/assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/codenvy/che.properties
this is a proper way to configure CHE, but we still allow users to mount custom che.properties

If you have problems with mounting custom che.properties let us know.

[davidwindell]

Ok, so in the default assembly che.properties, it has che.workspace.storage=${che.home}/workspaces

But this won't work for us because we don't set CHE_HOME and entrypoint.sh sets it to some alternative path.

[riuvshin]

I don't see how this change affects CHE_HOME it was not set there even before changes.

[davidwindell]

It affects CHE_HOME because now, as there is no /conf/che.properties the sed calls on che.properties never get called which means CHE_HOME is required as the sed commands don't override the built in ${che.home}/workspaces with /data/workspaces like they used to.

That's where the issue lies I think.

[riuvshin]

before that commit any property file weren't used at all. and everything is configured here:
https://github.com/eclipse/che/blob/master/dockerfiles/init/modules/che/templates/che.env.erb#L26-L33
even if it were worked before this change, ENV var priority is higher then props so https://github.com/eclipse/che/blob/master/dockerfiles/init/modules/che/templates/che.env.erb#L26-L33 those things were used.
So I still doubt that this change affected something. We have to check your storage is it still consistent.

[davidwindell]

@riuvshin all data is correctly mounted from our host into the container at /data. This works fine with our setup above on 5.10.0. In 5.11.0 (latest tag), all workspaces are missing.

Here are the key parts from inspect.

[
    {
    ....
        "HostConfig": {
            "Binds": [
                "/data/eclipse/users/david:/data:rw",
                "/data/eclipse/stacks/stacks.json:/tmp/stacks.json:rw",
                "/data/eclipse/templates:/data/templates:rw",
                "/var/run/docker.sock:/var/run/docker.sock:rw"
            ]
    },
        "Config": {
            "Hostname": "che-beta-1",
            "Env": [
                "CHE_DOCKER_IP_EXTERNAL=beta.chedomain.com",
                "CHE_WORKSPACE_VOLUME=/var/run/docker.sock:/var/run/docker.sock;/data/eclipse/users/david/ssh:/root/.ssh",
                "CHE_WORKSPACE_AUTO__SNAPSHOT=false",
                "CHE_DOCKER_SERVER__EVALUATION__STRATEGY=docker-local",
                "VIRTUAL_HOST=beta.chedomain.com",
                "CHE_PORT=80",
                "CHE_IN_CONTAINER=true"
            ],
            "Image": "eclipse/che-server:latest"
        }        
    }
]

Note that we don't mount a conf file at all. Before the change, the entrypoint would create a /data/conf/che.properties file but this no longer happens. Deleting this file makes no difference.

Help! ;)

[ghost]

Hm... I have started a 5.10 Che, then used the same local path to mount into /data for 5.11. I was able to see my workspace.

Can you confirm that in /data/conf/storage you have got the same che.mv.db that is mounted locally ie in /data/eclipse/users/david/data/db/che.mv.db?

[davidwindell]

It's not in /data/conf/ (never was), but yes, it's there:

bash-4.3# ls -la /data/storage/db/
total 452
drwxr-xr-x    2 root     root          4096 May  6 01:05 .
drwxr-xr-x    3 root     root          4096 May  6 01:05 ..
-rw-r--r--    1 root     root        454656 May 25 05:43 che.mv.db

[ghost]

Yes, I mean storage path. Is it identical to /data/eclipse/users/david/data/db/che.mv.db?

[davidwindell]

Yep. That's on the host and mounted in exactly the same way. How do I dump Che's environment as I'm convinced it's related to that?

[ghost]

Dump Che's environment? Can you elaborate? You mean exact envs that ws master uses?

[davidwindell]

So we can see exactly what merged configuration is being used and then compare that from 5.10 to 5.11.

[ghost]

I'd exec into a container and run env there.

[davidwindell]

Both containers are the same, the issue here lies with changes to how the environment is handled by Che internally so we need a way to dump the che.env or che.properties that is actually being used.

[ghost]

5.10.0:
2017-05-25 07:00:50,131[main] [INFO ] [o.a.c.s.VersionLoggerListener 115] - Command line argument: -Dche.local.conf.dir=/data/conf

5.11.0:

2017-05-25 07:10:31,459[main] [INFO ] [o.a.c.s.VersionLoggerListener 115] - Command line argument: -Dche.local.conf.dir=/conf/

[ghost]

@davidwindell me and @riuvshin have tested upgrade from 5.10 to 5.11 using the same local folder mounted int /data. It worked for both of us on different machines.

Can you provide the following information?

1. Do you have a custom assembly?
2. How exactly do you start Che?
3. Provide output from the following command executed in a local storage folder mounted into data: ls -ld $(find .)
4. Content of both che.env (the one in the root of a local storage folder and the one in /instance/config)
5. Your docker version and OS

[ghost]

che.env should be in the root of a local dir that is mounted into /data

[davidwindell]

That's empty for me:

bash-4.3# ls -la /data
total 36
drwxr-xr-x    9 root     root          4096 May 25 06:13 .
drwxr-xr-x   40 root     root          4096 May 25 11:34 ..
drwxr-xr-x    2 root     root          4096 May 24 11:42 conf
drwxr-xr-x    3 root     root          4096 May 25 11:34 lib
drwx------    2 root     root          4096 May  8 13:36 ssh
drwxr-xr-x    2 root     root          4096 May  9 14:16 stacks
drwxr-xr-x    3 root     root          4096 May  6 01:05 storage
drwxr-xr-x    2 root     root          4096 May 10 11:53 templates
drwxr-xr-x   12 root     root          4096 May 25 04:45 workspaces

[riuvshin]

@davidwindell can you please share full command line how you start eclipse che

[davidwindell]

1. Do you have a custom assembly?

No

1. How exactly do you start Che?

With Docker Cloud which just uses the compose definition I linked to above. In the case of this issue, using the eclipse/che-server:latest image.

1. Provide output from the following command executed in a local storage folder mounted into data: ls -ld $(find .)

# ls -ld $(find . -not -path "./workspaces/*")
drwxr-xr-x  9 root root     4096 May 25 06:13 .
drwxr-xr-x  2 root root     4096 May 24 11:42 ./conf
-rw-r--r--  1 root root    14185 May 24 11:42 ./conf/che.properties
drwxr-xr-x  3 root root     4096 May 25 11:34 ./lib
drwxr-xr-x  4 root root     4096 May 25 11:34 ./lib/linux_amd64
drwxr-xr-x  2 root root     4096 May 25 11:34 ./lib/linux_amd64/exec
-rw-r--r--  1 root root  2765979 May 25 11:34 ./lib/linux_amd64/exec/exec-agent-linux_amd64.tar.gz
drwxr-xr-x  2 root root     4096 May 25 11:34 ./lib/linux_amd64/terminal
-rw-r--r--  1 root root  2698443 May 25 11:34 ./lib/linux_amd64/terminal/websocket-terminal-linux_amd64.tar.gz
-rw-r--r--  1 root root 45145097 May 25 11:34 ./lib/ws-agent.tar.gz
drwxr-xr-x  2 root root     4096 May  9 14:16 ./stacks
-rw-r--r--  1 root root     3955 May 24 11:42 ./stacks/stacks.json
drwxr-xr-x  3 root root     4096 May  6 01:05 ./storage
drwxr-xr-x  2 root root     4096 May  6 01:05 ./storage/db
-rw-r--r--  1 root root   593920 May 25 08:25 ./storage/db/che.mv.db
drwxr-xr-x  2 root root     4096 May  5 20:44 ./templates
drwxr-xr-x 12 root root     4096 May 25 04:45 ./workspaces

1. Content of both che.env (the one in the root of a local storage folder and the one in /instance/config)

@eivantsov Where exactly do I find the che.env, the file doesn't exist according to find / -name "che.env"

1. Your docker version and OS

Docker version 1.11.2-cs5, build d364ea1 on Ubuntu 14.04.4 LTS

[ghost]

@davidwindell so you are not using CLI. This is the problem.

[riuvshin]

@davidwindell yeah... you should use CHE CLI to run CHE, not che-server directly.
as a workaround you could add mount of your custom che.properties with correct paths to /conf

I will take a look how we can manage this as it was in 5.10
But again CHE should be started using CLI, because there is a lot of logic inside which could do some migrations and etc. I would say you could run che-server directly at your own risk.

[davidwindell]

@riuvshin @eivantsov @TylerJewell

Sorry but, what? This is officially documented use of Che https://www.eclipse.org/che/docs/setup/docker/index.html. We use Docker Cloud and will never use the CLI.

The change I linked to above has broken the che-server image, please revert that or at least put out a BC break announcement.

[ghost]

@davidwindell yes, we will fix that. Although being documented, this is a non standard way of running Che. @riuvshin will take care of it.

[riuvshin]

@davidwindell yeah sorry about that, we will fix that.

[davidwindell]

Thanks, I appreciate that, lots of people are using orchestrators.

[riuvshin]

@davidwindell here is a PR to fix that https://github.com/eclipse/che/pull/5202 we will include it in next bugfix release.

[riuvshin]

@davidwindell I've just released bugfix 5.11.1, so can you please try this out? just want to make sure it is OK on your side.

[davidwindell]

@riuvshin checking

[davidwindell]

@riuvshin I'm getting cp: can't create '/conf/che.properties': No such file or directory

Looks like you need to mkdir?

[riuvshin]

@davidwindell yeah, my bad, I thought /conf is always in the image but that folder appear with mount using CLI. I've added mkdir and repushed 5.11.1 tag. can you please do docker pull eclipse/che-server:5.11.1 and try it again

[davidwindell]

For existing workspaces, i'm getting "Project exists in workspace but not in filesystem". I don't have time to investigate right now but there must be a change in where data paths are obtained from/expected. Before, workspaces were expected in /data/workspaces

[davidwindell]

I think this might be related to this https://github.com/eclipse/che/pull/4931

[ghost]

@davidwindell I have tested it this way:

Started Che 5.10 and created a workspace with a project:

docker run -ti -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock -v ~/Documents/510:/data -e "CHE_IP=172.20.0.1" -e "CHE_PORT=8080" eclipse/che-server:5.10.0

Stopped Che and started 5.11:

docker run -ti -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock -v ~/Documents/510:/data -e "CHE_IP=172.20.0.1" -e "CHE_PORT=8080" eclipse/che-server:5.11.1

My project was there.

[davidwindell]

@eivantsov #4931 is an issue exclusive to orchestrators. I'll have to investigate why that patch no longer works since these changes

[JonathanSpooner]

@riuvshin I'm getting an error starting 5.11.1 after an upgrade from 5.10.0 which may have to do with the changes you've just made. Here's the relevant output from the CLI:

INFO: (che start): Services booting...
INFO: (che start): Server logs at "docker logs -f che"
ERROR: (che start): Timeout waiting for server. Run "docker logs che" to inspect.
jonathan@workstation ~ $ docker logs che
Using embedded assembly...
mkdir: can't create directory '/conf': File exists

The last two lines repeat several times. Here's the output of the info command:

WARNING: No swap limit support
INFO: (che cli): 5.11.1 - using docker 17.05.0-ce / native
CLI:
TTY: true
Daemon: /var/run/docker.sock
Image: eclipse/che:5.11.1
Version: 5.11.1
Command: info
Parameters: info
Mounts:
/data: /home/jonathan/Projects
/data/instance: not set
/data/backup: not set
/repo: not set
/assembly: not set
/sync: not set
/unison: not set
/chedir: not set
System:
Docker: native
Proxy: not set
Internal:
CHE_VERSION: 5.11.1
CHE_HOST: 192.168.0.199
CHE_INSTANCE: /home/jonathan/Projects/instance
CHE_CONFIG: /home/jonathan/Projects
CHE_BACKUP: /home/jonathan/Projects/backup
CHE_REGISTRY: /version
CHE_DEBUG: false
IP Detection: 192.168.0.199
Initialized: true
Image Registry:
IMAGE_INIT=eclipse/che-init:5.11.1
IMAGE_CHE=eclipse/che-server:5.11.1
IMAGE_COMPOSE=docker/compose:1.8.1
IMAGE_TRAEFIK=traefik:v1.3.0-rc1
BOOTSTRAP_IMAGE_ALPINE=alpine:3.4
BOOTSTRAP_IMAGE_CHEIP=eclipse/che-ip:5.11.1
UTILITY_IMAGE_CHEACTION=eclipse/che-action:5.11.1
UTILITY_IMAGE_CHEDIR=eclipse/che-dir:5.11.1
UTILITY_IMAGE_CHETEST=eclipse/che-test:5.11.1
UTILITY_IMAGE_CHEMOUNT=eclipse/che-mount:5.11.1
che.env:

[davidwindell]

Probably needs a -p

[riuvshin]

hm strange if folder is there file should be there too... ok I will add check for /conf dir and perfrom mkdir -p /conf only if it does not exist

[riuvshin]

@davidwindell I've updated 5.11.1 tag, can you please pull and try it.

[JonathanSpooner]

@riuvshin That fixed the issue that I was seeing. Thanks and good luck to you both ;)

[davidwindell]

@riuvshin this is working for us now. Can you also merge https://github.com/eclipse/che/pull/4931 as we have to hack into the entrypoint file every time we launch it until that's been merged.

p.s. you don't need to check if /conf exists, -p will stop mkdir throwing errors.

[riuvshin]

@davidwindell yeah I know what -p does, check is more obvious for ones who does not know. I will merge
https://github.com/eclipse/che/pull/4931 now to master. If you want I can also push this to 5.11.1 tag

[davidwindell]

@riuvshin :) yes please!

[riuvshin]

@davidwindell done

[davidwindell]

Thanks @riuvshin for sorting this out. I promise to give you a break until Tuesday ;)

[riuvshin]

@davidwindell sorry for this silly bugs. You can ping any time you need and we will try to help you asap.
Hava a nice weekends!