Deno: Can't build master

[jamiebuilds]

Running:

RUST_BACKTRACE=1 cargo build -vv

Yields:

Unable to load AWS_CREDENTIAL_FILE ()

Full stack trace:

--- stderr
Failed to fetch file gs://chromium-clang-format/025ca7c75f37ef4a40f3a67d81ddd11d7d0cdb9b for /Users/jamiebuilds/Projects/deno/core/libdeno/buildtools/mac/clang-format. [Err: /Users/jamiebuilds/Projects/deno/third_party/depot_tools/external_bin/gsutil/gsutil_4.28/gsutil/third_party/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
  warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
]
thread 'main' panicked at 'assertion failed: status.success()', core/build.rs:94:9
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/vsts/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /Users/vsts/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:214
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::begin_panic
   8: build_script_build::gn::Build::run
             at core/build.rs:94
   9: build_script_build::main
             at core/build.rs:9
  10: std::rt::lang_start::{{closure}}
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libstd/rt.rs:64
  11: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:49
  12: std::panicking::try::do_call
             at src/libstd/panicking.rs:296
  13: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  14: std::panicking::try
             at src/libstd/panicking.rs:275
  15: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  16: std::rt::lang_start_internal
             at src/libstd/rt.rs:48
  17: std::rt::lang_start
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libstd/rt.rs:64
  18: build_script_build::main

I did already have depot_tools cloned on my machine and setup in my PATH but I removed it to see if there was some unusual conflict there

[chrmoritz]

Could you try to unset AWS_CREDENTIAL_FILE in case you have this environment variable set.

[jamiebuilds]

@chrmoritz Just tried that out and it failed in the same way

[ry]

@jamiebuilds Try unset AWS_ACCESS_KEY_ID and unset AWS_SECRET_ACCESS_KEY - this has to do with our sscache setup... It might actually be a bug in sscache (I think they shouldn't error)

[jamiebuilds]

@ry Just tried that too, still failing with

0> Failed to fetch file gs://chromium-clang-format/025ca7c75f37ef4a40f3a67d81ddd11d7d0cdb9b for /Users/jamiebuilds/Projects/deno/core/libdeno/buildtools/mac/clang-format, skipping. [Err: /Users/jamiebuilds/Projects/deno/third_party/depot_tools/external_bin/gsutil/gsutil_4.28/gsutil/third_party/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
  warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
]

[ry]

@jamiebuilds Can you examine your env and let us know which env vars are set? (Please be careful not to reveal any secret keys!) Are there any related to AWS or Google Cloud?

[jamiebuilds]

I don't keep any keys in my env right now. But I do have my local copy of depot_tools in my PATH, I tried removing that though and it failed with the same error as above.

USER=jamiebuilds
PATH=/Users/jamiebuilds/<<<REDACTED>>>/bin:/Users/jamiebuilds/.npm-global/bin:/Users/jamiebuilds/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Library/Apple/bin:/Users/jamiebuilds/.npm-global/bin:/Users/jamiebuilds/.cargo/bin:/Users/jamiebuilds/Projects/depot_tools
LOGNAME=jamiebuilds
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.lwLRa4TvXm/Listeners
HOME=/Users/jamiebuilds
SHELL=/bin/zsh
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
TMPDIR=/var/folders/nb/d402mpcs1dl_dj7tv7p_r1qw0000gn/T/
XPC_SERVICE_NAME=0
XPC_FLAGS=0x0
GREP_COLOR=37;45
LESS_TERMCAP_mb=
TERM_PROGRAM=vscode
LESS_TERMCAP_md=
LESS_TERMCAP_me=
TERM=xterm-256color
BAY_HOME=/Users/jamiebuilds/<<<REDACTED>>>
TERM_PROGRAM_VERSION=1.39.2
LESS_TERMCAP_ue=
LS_COLORS=di=34:ln=35:so=32:pi=33:ex=31:bd=36;01:cd=33;01:su=31;40;07:sg=36;40;07:tw=32;40;07:ow=33;40;07:
GREP_COLORS=mt=37;45
KUBECONFIG=/Users/jamiebuilds/<<<REDACTED>>>
VIRTUAL_ENV_DISABLE_PROMPT=12
PAGER=less
LSCOLORS=exfxcxdxbxGxDxabagacad
LESS_TERMCAP_us=
PWD=/Users/jamiebuilds/Projects/deno
EDITOR=vim
LANG=en_US.UTF-8
SHLVL=1
LESS=-F -g -i -M -R -S -w -X -z-4
VISUAL=nano
LESS_TERMCAP_so=
BROWSER=open
COLORTERM=truecolor
LESS_TERMCAP_se=
APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL=true
OLDPWD=/Users/jamiebuilds/Projects/deno
PROMPT_EOL_MARK=
_=/usr/bin/printenv

I also tried grepping around declare -f and couldn't find anything relevant.

[ry]

@jamiebuilds Would you mind trying to edit your path so it doesn't have that depot_tools and see if that is the problem? (We provide our own copy of depot_tools in third_party/depot_tools)

[jamiebuilds]

@ry Yeah, I did that when I saw it was there and got the same result.

[ry]

@jamiebuilds Oops sorry I missed that. So to give you some context, "cargo build" is running core/build.rs which is failing on line 94, trying to run "tools/setup.py"
https://github.com/denoland/deno/blob/1f52c66ced9bed0cae6bff065dfa7563cbfaee29/core/build.rs#L85-L94
tool/setup.py fetches some binary dependencies from Google Cloud Storage - it seems to be failing trying to get clang_format. It might be helpful if you just ran "tools/setup.py" by itself and see if there is a better error message.

One annoying thing is that clang_format isn't even required to build deno - it's just for people submitting changes and running tools/format.py ... but that's tangental.

@chrmoritz any ideas?

[jamiebuilds]

Output of ./tools/setup.py

❯ ./tools/setup.py                                                                                          ✘ 130 
binary download
/Users/jamiebuilds/Projects/deno/third_party/depot_tools/cipd install gn/gn/mac-amd64 git_revision:152c5144ceed9592c20f0c8fd55769646077569b -root /Users/jamiebuilds/Projects/deno/core/libdeno/buildtools/mac
Package gn/gn/mac-amd64 is up-to-date.
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/jamiebuilds/Projects/deno/third_party/depot_tools/download_from_google_storage.py --platform=darwin --no_auth --bucket=chromium-clang-format --sha1_file /Users/jamiebuilds/Projects/deno/core/libdeno/buildtools/mac/clang-format.sha1
0> Failed to fetch file gs://chromium-clang-format/025ca7c75f37ef4a40f3a67d81ddd11d7d0cdb9b for /Users/jamiebuilds/Projects/deno/core/libdeno/buildtools/mac/clang-format, skipping. [Err: /Users/jamiebuilds/Projects/deno/third_party/depot_tools/external_bin/gsutil/gsutil_4.28/gsutil/third_party/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
  warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
]
Failed to fetch file gs://chromium-clang-format/025ca7c75f37ef4a40f3a67d81ddd11d7d0cdb9b for /Users/jamiebuilds/Projects/deno/core/libdeno/buildtools/mac/clang-format. [Err: /Users/jamiebuilds/Projects/deno/third_party/depot_tools/external_bin/gsutil/gsutil_4.28/gsutil/third_party/boto/boto/pyami/config.py:69: UserWarning: Unable to load AWS_CREDENTIAL_FILE ()
  warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
]
Downloading 1 files took 2.054418 second(s)

[chrmoritz]

Oh wait, I can now reproduce the same error locally too (with a clean checkout) and I'm sure everything was fine a few days ago.

Given that the erroring code lives in ./third_party/depot_tools/external_bin/gsutil/gsutil_4.28/gsutil/third_party/boto/boto/pyami/config.py, which isn't checked in code (from external_bin), but something which is dynamically downloaded by depot_tools, this looks like some recent macOS specific breaking caused by a upstream change by Google.

[jamiebuilds]

I am on macOS Catalina if that is relevant.

[chrmoritz]

The way to reproduce this is to remove ./third_party/depot_tools/external_bin on macOS and let depot_tools recreate it.

```
rm -rf third_party/depot_tools/external_bin
./tools/setup.py

[chrmoritz]

Error throwing code looks like this

            if "AWS_CREDENTIAL_FILE" in os.environ:
                full_path = expanduser(os.environ['AWS_CREDENTIAL_FILE'])
                try:
                    self.load_credential_file(full_path)
                except IOError:
                    warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)

But I definitely don't have AWS_CREDENTIAL_FILE set either, so some previously running python code has to set it to an invalid value.

[chrmoritz]

It is getting set here to an empty string, which still satisfy the condition that it's set.

~But commenting this out just results in failing to download it at a later point without a meaningful error message :(~

[chrmoritz]

@jamiebuilds: As a workaround you could just comment out third_party.download_clang_format() and third_party.download_sccache() because they are not strictly needed for building deno:

https://github.com/denoland/deno/blob/6c5a981fd2afad21af73a1345c4e30fb6b30b09a/tools/setup.py#L28-L33

[chrmoritz]

@ry Upgrading depot_tools to 5abb9b7c5d0c0a5f81f4a0da741680df9b9d128c (the one used in v8 7.9) fixes this issue for me. Since this doesn't happen on CI this might be a Catalina specific issue fixed within a newer depot_tools version.

[chrmoritz]

We're running into https://bugs.chromium.org/p/chromium/issues/detail?id=1012612 here, fixed in https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/1854010 == https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/6a1d77869d77b61189247ce8b47ac9d86684a6c2.

[chrmoritz]

@jamiebuilds This should be fixed now with #3175. Could you retry with the latest master?

[dnalborczyk]

@chrmoritz I found this issue after opening a new issue: https://github.com/denoland/deno/issues/3440

the problem still seems to exist on macos catalina.