Electron-builder: docker: poorly formatted environment:variable in travis (TRAVIS_COMMIT_MESSAGE)

Created on 7 Jan 2018 · 12Comments · Source: electron-userland/electron-builder

Version: 19.49.2

Target: Windows and Linux using Docker.

While building for windows and linux using docker on trusty environment in travis, i am getting this error:

docker: poorly formatted environment: variable 'Refactored Settings system by moving it from dexie to JSON. Results in less complexity and efficiency as listeners are removed.' has white spaces.

I have used the travis config as provided in the docs sample project. The builds were running fine with the same config before. Between the last successful build and the current build, the .travis.yml file has not been touched at all.

The mac builds are working perfectly, so I assume the error is due to some config issue.

docker

Source

quantumkv

Most helpful comment

@xsokaris I found another way here https://github.com/moby/moby/issues/12997#issuecomment-307665540

Here's my full script:

  npm run build:prod
  ENVS=`env | grep -iE '^(DEBUG|NODE_|ELECTRON_|YARN_|NPM_|CI|CIRCLE|TRAVIS|APPVEYOR_|CSC_|_TOKEN|_KEY|AWS_|STRIP|BUILD_)([A-Z]|_)*=' | sed -n '/^[^\t]/s/=.*//p' | sed '/^$/d' | sed 's/^/-e /g' | tr '\n' ' '`
  if [ "$TRAVIS_OS_NAME" == "linux" ]; then
    docker run $ENVS --rm \
      -v ${PWD}:/project \
      -v ~/.cache/electron:/root/.cache/electron \
      -v ~/.cache/electron-builder:/root/.cache/electron-builder \
      electronuserland/builder:wine \
      /bin/bash -c "npm run release:linux && npm run release:win"
  else
    npm run release:mac
  fi

Istead of passing vars with --env-file, it creates a list of -e VAR1 -e VAR2, and docker will be able to find the value of each itself.
I also fixed the regex so it only takes vars starting by, and not only containing "DEBUG", "NODE_" etc.

Eventually you'll have to add your environment variable pattern if it is not already matched by the regular expression.

Clover-Lindo on 16 Sep 2018

👍8 ❤3 🎉1

All 12 comments

Please show result of env command (except TOKEN and other passwords).

develar on 19 Jan 2018

The output given by env is as follows:

MANPATH=/home/travis/.nvm/versions/node/v8.9.1/share/man:/home/travis/.kiex/elixirs/elixir-1.4.5/man:/home/travis/.rvm/rubies/ruby-2.4.1/share/man:/usr/local/man:/usr/local/cmake-3.9.2/man:/usr/local/clang-5.0.0/share/man:/usr/local/share/man:/usr/share/man:/home/travis/.rvm/man
TRAVIS_FILTERED=redirect_io
rvm_bin_path=/home/travis/.rvm/bin
GEM_HOME=/home/travis/.rvm/gems/ruby-2.4.1
TRAVIS_STACK_JOB_BOARD_REGISTER=/.job-board-register.yml
TRAVIS_STACK_LANGUAGES=__garnet__ c c++ clojure cplusplus cpp default go groovy java node_js php pure_java python ruby scala
IRBRC=/home/travis/.rvm/rubies/ruby-2.4.1/.irbrc
TRAVIS_COMMIT=7a962098eb2918aa30f5c65ea5920c20cc8b03cd
TRAVIS_OS_NAME=linux
CASHER_DIR=/home/travis/.casher
TRAVIS_UID=2000
OLDPWD=/home/travis/build
MY_RUBY_HOME=/home/travis/.rvm/rubies/ruby-2.4.1
MIX_ARCHIVES=/home/travis/.kiex/mix/elixir-1.4.5
NVM_DIR=/home/travis/.nvm
USER=travis
TRAVIS_LANGUAGE=generic
rvm_path=/home/travis/.rvm
TRAVIS=true
TRAVIS_REPO_SLUG=quantumkv/nighthawk
TRAVIS_COMMIT_MESSAGE=Added Queue Manager [skip ci]
TRAVIS_PULL_REQUEST=false
PERLBREW_ROOT=/home/travis/perl5/perlbrew
TRAVIS_STACK_TIMESTAMP=2017-12-05 19:33:09 UTC
rvm_prefix=/home/travis
PATH=/home/travis/.rvm/gems/ruby-2.4.1/bin:/home/travis/.rvm/gems/ruby-2.4.1@global/bin:/home/travis/.rvm/rubies/ruby-2.4.1/bin:/home/travis/.rvm/bin:/home/travis/bin:/home/travis/.local/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.9.1/bin:/home/travis/.kiex/elixirs/elixir-1.4.5/bin:/home/travis/.kiex/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.7.4.linux.amd64/bin:/usr/local/phantomjs/bin:/usr/local/phantomjs:/usr/local/neo4j-3.2.7/bin:/usr/local/maven-3.5.2/bin:/usr/local/cmake-3.9.2/bin:/usr/local/clang-5.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin
MAIL=/var/mail/travis
TRAVIS_PRE_CHEF_BOOTSTRAP_TIME=2017-12-05T19:32:55
PERLBREW_HOME=/home/travis/.perlbrew
TRAVIS_SUDO=true
TRAVIS_TAG=
TRAVIS_ALLOW_FAILURE=false
TRAVIS_JOB_NUMBER=25.2
TRAVIS_EVENT_TYPE=api
HOME=/home/travis
GOROOT=/home/travis/.gimme/versions/go1.7.4.linux.amd64
CI=true
TRAVIS_BUILD_ID=330710674
LOGNAME=travis
TRAVIS_STACK_FEATURES=basic cassandra chromium couchdb disabled-ipv6 docker docker-compose elasticsearch firefox go-toolchain google-chrome jdk memcached mongodb mysql neo4j nodejs_interpreter perl_interpreter perlbrew phantomjs postgresql python_interpreter rabbitmq redis riak ruby_interpreter sqlite xserver
TRAVIS_PULL_REQUEST_SLUG=
GEM_PATH=/home/travis/.rvm/gems/ruby-2.4.1:/home/travis/.rvm/gems/ruby-2.4.1@global
TRAVIS_SECURE_ENV_VARS=true
NVM_BIN=/home/travis/.nvm/versions/node/v8.9.1/bin
GOPATH=/home/travis/gopath
TRAVIS_STACK_NODE_ATTRIBUTES=/.node-attributes.yml
TRAVIS_STACK_NAME=garnet
TRAVIS_BRANCH=master
TRAVIS_COMMIT_RANGE=fee88ef21692c0cdfa04bb1918edeecbecad1233...7a962098eb2918aa30f5c65ea5920c20cc8b03cd
TRAVIS_PULL_REQUEST_BRANCH=
TRAVIS_JOB_ID=330710676
TRAVIS_BUILD_DIR=/home/travis/build/quantumkv/nighthawk
TRAVIS_BUILD_NUMBER=25

The failure seems to be at random. The next commit worked perfectly. Seems like TRAVIS_COMMIT_MESSAGE is causing the problems.

quantumkv on 19 Jan 2018

@quantumkv How did you determine that TRAVIS_COMMIT_MESSAGE was the cause? I think I have a similar issue with that variable causing OSX builds to fail in one of my repos.

stevenzeck on 9 Mar 2018

The error I got referred to my git message string. After some googling i found out that docker has a well documented issue with newlines when any such text is passed to --env-file

The error you are getting with TRAVIS_COMMIT_MESSAGE might be due to newlines in you git commit message. Keeping your newlines as LF instead of CRLF might solve the issue.

quantumkv on 10 Mar 2018

GitHub PR merges are pre-filled and formatted that way. I noticed when going through your old Travis builds that were failing that your commit messages were structured the same way.

We're just blanking out that variable in Travis now until (or if) we figure out what the actual problem is. But I was just curious if this was something similar and if you found a root cause.

stevenzeck on 10 Mar 2018

👍2

~~Filtering out multiline env vars with grep -v '\r' seems to work for me, e.g.:~~

Edit: it actually fails on releases. See my other workaround below

docker run --rm \
          --env-file <(env | grep -v '\r' | grep -iE 'DEBUG|NODE_|ELECTRON_|YARN_|NPM_|CI|CIRCLE|TRAVIS|APPVEYOR_|CSC_|_TOKEN|_KEY|AWS_|STRIP|BUILD_') \
          ...

(excluding TRAVIS_COMMIT_MESSAGE used to work before too, but it stopped recently, possibly because of another variable...)

sebn on 12 Sep 2018

❤3

(EDIT: See my cleaner suggestion two messages below, this one only removes all variable)

\r wasn't enough on my side, I had to exclude \n too

grep -vE '\r|\n'

Here's the full line:

--env-file <(env | grep -vE '\r|\n' | grep -iE 'DEBUG|NODE_|ELECTRON_|YARN_|NPM_|CI|CIRCLE|TRAVIS|APPVEYOR_|CSC_|_TOKEN|_KEY|AWS_|STRIP|BUILD_') \

I'm not sure if that's a good workaround though, but that did the trick for me.

Clover-Lindo on 15 Sep 2018

👍6

@Clover-Lindo not for me ... i can't make this working ... after your solution I have :

• building block mapblockMapFile=dist/CookieTouch Setup 0.0.1-beta.10.exe.blockmap
• uploadingfile=CookieTouch Setup 0.0.1-beta.10.exe.blockmapprovider=S3
• uploadingfile=CookieTouch Setup 0.0.1-beta.10.exeprovider=S3
  ⨯ NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
github.com/develar/app-builder/pkg/publisher.getBucketRegion

and this is making sense since all white spaces are removed i think

xsokaris on 16 Sep 2018

@xsokaris I found another way here https://github.com/moby/moby/issues/12997#issuecomment-307665540

Here's my full script:

  npm run build:prod
  ENVS=`env | grep -iE '^(DEBUG|NODE_|ELECTRON_|YARN_|NPM_|CI|CIRCLE|TRAVIS|APPVEYOR_|CSC_|_TOKEN|_KEY|AWS_|STRIP|BUILD_)([A-Z]|_)*=' | sed -n '/^[^\t]/s/=.*//p' | sed '/^$/d' | sed 's/^/-e /g' | tr '\n' ' '`
  if [ "$TRAVIS_OS_NAME" == "linux" ]; then
    docker run $ENVS --rm \
      -v ${PWD}:/project \
      -v ~/.cache/electron:/root/.cache/electron \
      -v ~/.cache/electron-builder:/root/.cache/electron-builder \
      electronuserland/builder:wine \
      /bin/bash -c "npm run release:linux && npm run release:win"
  else
    npm run release:mac
  fi

Eventually you'll have to add your environment variable pattern if it is not already matched by the regular expression.

Clover-Lindo on 16 Sep 2018

👍8 ❤3 🎉1

@Clover-Lindo this somehow leaves out my tokens (GH_TOKEN in this case).

johannesjo on 26 Sep 2018

this somehow leaves out my tokens (GH_TOKEN in this case).

I believe matching only vars starting with _TOKEN won't work :)
I ended up with the following, which seems to work for now, including releases:

docker run --rm \
          $(env | \
            grep -Eo '^[^\s=]*(DEBUG|NODE_|ELECTRON_|YARN_|NPM_|CI|CIRCLE|TRAVIS|APPVEYOR_|CSC_|_TOKEN|_KEY|AWS_|STRIP|BUILD_)[^\s=]*' | \
            sed '/^$/d;s/^/-e /' | \
            paste -sd ' ' \
          ) \
          -v ${PWD}:/project \
          -v ~/.cache/electron:/root/.cache/electron \
          -v ~/.cache/electron-builder:/root/.cache/electron-builder \
          ... \ # your docker base image
          ... # your build command

Looks like the env can sometimes contain weird stuff like variables in variables (?), at least on my distro, hence the [^\s=] exclusion in grep. But maybe it would be cleaner to just allow alphanums, I don't know.
Also sed can take multiple ;-separated scripts at once. And I suppose paste is best to merge multiple lines in one.

sebn on 27 Sep 2018

👍2

@Clover-Lindo Thanks for your line, it was the only solution that worked for me! Here's my full command

        docker run --rm \
         --env-file <(env | grep -vE '\r|\n' | grep -iE 'DEBUG|NODE_|ELECTRON_|YARN_|NPM_|CI|CIRCLE|TRAVIS_TAG|TRAVIS|TRAVIS_REPO_|TRAVIS_BUILD_|TRAVIS_BRANCH|TRAVIS_PULL_REQUEST_|APPVEYOR_|CSC_|GH_|GITHUB_|BT_|AWS_|STRIP|BUILD_') \
          --env ELECTRON_CACHE="/root/.cache/electron" \
          --env ELECTRON_BUILDER_CACHE="/root/.cache/electron-builder" \
          -v ${PWD}:/project \
          -v ~/.cache/electron:/root/.cache/electron \
          -v ~/.cache/electron-builder:/root/.cache/electron-builder \
          electronuserland/builder:wine-02.17 \
          /bin/bash -c "yarn --link-duplicates --pure-lockfile && yarn release --linux --win"