Sdk: "No such file or directory" running `fetch dart`

[DanTup]

I'm setting up the SDK code on my corp MacBook following the instructions on the wiki and got this error:

Syncing projects: 100% (84/84), done.                                     
Traceback (most recent call last):
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 3105, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 3091, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/Users/dantup/Dev/Google/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 2783, in CMDsync
    ret = client.RunOnDeps('update', args)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 1691, in RunOnDeps
    self.RunHooksRecursively(self._options, pm)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 1155, in RunHooksRecursively
    hook.run(self.root.root_dir)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 229, in run
    cmd, cwd=cwd, always=self._verbose)
  File "/Users/dantup/Dev/Google/depot_tools/gclient_utils.py", line 314, in CheckCallAndFilterAndHeader
    return CheckCallAndFilter(args, **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/gclient_utils.py", line 525, in CheckCallAndFilter
    **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/subprocess2.py", line 262, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory.
Check that /Users/dantup/Dev/Google/dart-sdk or download_from_google_storage exist and have execution permission.
Traceback (most recent call last):
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 306, in <module>
    sys.exit(main())
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 301, in main
    return run(options, spec, root)
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 295, in run
    return checkout.init()
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 137, in init
    self.run_gclient(*sync_cmd)
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 82, in run_gclient
    return self.run(cmd_prefix + cmd, **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 71, in run
    subprocess.check_call(cmd, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python', '/Users/dantup/Dev/Google/depot_tools/gclient.py', 'sync')' returned non-zero exit status 1

Here's what I did:

• cd ~/Dev/Google
• git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
• mkdir dart-sdk
• cd dart-sdk
• fetch dart

(note I already had ~/Dev/Google/depot_tools in PATH from when I copied paths from my other machine).

The folder mention exists and download_from_google_storage exists and is executable (I ran it at the end of the session included below to show that).

At the end of this, I do have an sdk folder inside ~/Dev/Google/dart-sdk.

Here's the full terminal session

dantup-macbookpro:Google dantup$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
Cloning into 'depot_tools'...
remote: Sending approximately 31.08 MiB ...
remote: Total 25677 (delta 16532), reused 25677 (delta 16532)
Receiving objects: 100% (25677/25677), 31.08 MiB | 8.37 MiB/s, done.
Resolving deltas: 100% (16532/16532), done.
dantup-macbookpro:Google dantup$ echo $PATH
/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:~/Dev/dart-sdk/latest/bin:~/Dev/Google/flutter/bin:~/Library/Android/sdk/tools:~/Library/Android/sdk/tools/bin:~/Dev/Google/depot_tools
dantup-macbookpro:Google dantup$ mkdir dart-sdk
dantup-macbookpro:Google dantup$ cd dart-sdk
dantup-macbookpro:dart-sdk dantup$ fetch dart
Running: /usr/bin/python /Users/dantup/Dev/Google/depot_tools/gclient.py root
Running: /usr/bin/python /Users/dantup/Dev/Google/depot_tools/gclient.py config --spec 'solutions = [
  {
    "url": "https://dart.googlesource.com/sdk.git",
    "managed": False,
    "name": "sdk",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
'
Running: /usr/bin/python /Users/dantup/Dev/Google/depot_tools/gclient.py sync

________ running 'git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://dart.googlesource.com/sdk.git /Users/dantup/Dev/Google/dart-sdk/_gclient_sdk_TWTvRU' in '/Users/dantup/Dev/Google/dart-sdk'
Cloning into '/Users/dantup/Dev/Google/dart-sdk/_gclient_sdk_TWTvRU'...
remote: Sending approximately 567.37 MiB ...        
remote: Counting objects: 881, done        
remote: Finding sources: 100% (71/71)           
Receiving objects:  75% (531330/706236), 462.86 MiB | 8.42 MiB/s   
[0:01:00] Still working on:
[0:01:00]   sdk
Receiving objects:  92% (649738/706236), 548.70 MiB | 8.42 MiB/s   
[0:01:10] Still working on:
[0:01:10]   sdk
remote: Total 706236 (delta 577868), reused 706220 (delta 577868)        
Receiving objects: 100% (706236/706236), 567.31 MiB | 8.18 MiB/s, done.
Resolving deltas:  29% (168985/577868)   
[0:01:20] Still working on:
[0:01:20]   sdk
Resolving deltas:  62% (358817/577868)   
[0:01:30] Still working on:
[0:01:30]   sdk
Resolving deltas:  74% (427851/577868)   
[0:01:40] Still working on:
[0:01:40]   sdk
Resolving deltas:  78% (450738/577868)   
[0:01:50] Still working on:
[0:01:50]   sdk
Resolving deltas:  81% (469885/577868)   
[0:02:00] Still working on:
[0:02:00]   sdk
Resolving deltas:  85% (492055/577868)   
[0:02:10] Still working on:
[0:02:10]   sdk
Resolving deltas:  88% (512000/577868)   
[0:02:20] Still working on:
[0:02:20]   sdk
Resolving deltas: 100% (577868/577868), done.

[0:02:30] Still working on:
[0:02:30]   sdk
Syncing projects:   0% ( 0/ 2) 
[0:02:35] Still working on:
[0:02:35]   sdk
Syncing projects: 100% (84/84), done.                                     
Traceback (most recent call last):
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 3105, in 
    sys.exit(main(sys.argv[1:]))
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 3091, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/Users/dantup/Dev/Google/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 2783, in CMDsync
    ret = client.RunOnDeps('update', args)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 1691, in RunOnDeps
    self.RunHooksRecursively(self._options, pm)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 1155, in RunHooksRecursively
    hook.run(self.root.root_dir)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 229, in run
    cmd, cwd=cwd, always=self._verbose)
  File "/Users/dantup/Dev/Google/depot_tools/gclient_utils.py", line 314, in CheckCallAndFilterAndHeader
    return CheckCallAndFilter(args, **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/gclient_utils.py", line 525, in CheckCallAndFilter
    **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/subprocess2.py", line 262, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory.
Check that /Users/dantup/Dev/Google/dart-sdk or download_from_google_storage exist and have execution permission.
Traceback (most recent call last):
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 306, in 
    sys.exit(main())
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 301, in main
    return run(options, spec, root)
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 295, in run
    return checkout.init()
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 137, in init
    self.run_gclient(*sync_cmd)
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 82, in run_gclient
    return self.run(cmd_prefix + cmd, **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/fetch.py", line 71, in run
    subprocess.check_call(cmd, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/bin/python', '/Users/dantup/Dev/Google/depot_tools/gclient.py', 'sync')' returned non-zero exit status 1
dantup-macbookpro:dart-sdk dantup$ download_from_google_storage
Usage: download_from_google_storage.py [options] target
Target must be:
  (default) a sha1 sum ([A-Za-z0-9]{40}).
  (-s or --sha1_file) a .sha1 file, containing a sha1 sum on the first line.
  (-d or --directory) A directory to scan for .sha1 files.

download_from_google_storage.py: error: Missing target.
dantup-macbookpro:dart-sdk dantup$ 

[DanTup]

If I try to run gclient sync it fails with the same error:

dantup-macbookpro:dart-sdk dantup$ gclient sync
Bootstrapping cipd client for mac-amd64 from https://chrome-infra-packages.appspot.com/client?platform=mac-amd64&version=git_revision:4d19637ec2c3d1efd8c6a1b05285118b786919e2...
Syncing projects: 100% (84/84), done.                                     
Traceback (most recent call last):
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 3105, in <module>
    sys.exit(main(sys.argv[1:]))
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 3091, in main
    return dispatcher.execute(OptionParser(), argv)
  File "/Users/dantup/Dev/Google/depot_tools/subcommand.py", line 252, in execute
    return command(parser, args[1:])
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 2783, in CMDsync
    ret = client.RunOnDeps('update', args)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 1691, in RunOnDeps
    self.RunHooksRecursively(self._options, pm)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 1155, in RunHooksRecursively
    hook.run(self.root.root_dir)
  File "/Users/dantup/Dev/Google/depot_tools/gclient.py", line 229, in run
    cmd, cwd=cwd, always=self._verbose)
  File "/Users/dantup/Dev/Google/depot_tools/gclient_utils.py", line 314, in CheckCallAndFilterAndHeader
    return CheckCallAndFilter(args, **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/gclient_utils.py", line 525, in CheckCallAndFilter
    **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/subprocess2.py", line 262, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory.
Check that /Users/dantup/Dev/Google/dart-sdk or download_from_google_storage exist and have execution permission.
dantup-macbookpro:dart-sdk dantup$ 

[DanTup]

I wondered if maybe depot_tools should come first in my PATH, but in these docs it says "Add depot_tools to the end of your PATH" (emphasis on end theirs).

[bwilkerson]

@whesse Who would best be able to answer this question?

[DanTup]

It may be an issue with depot_tools or something else, but I couldn't find any answers online (lots of people have had similar issues with gclient and the answer vary from them being out of date (I only just cloned mine) to "just ignore it and build anyway").

FWIW, I had similar issues (same error in fact) when I previously did this and I think I just ignored them because I was only making fairly minor modifications and the analyzer code seemed to work; however I did have issues with things like the pre-commit checks when using Gerrit that may have been related so I was hoping to get things set up "properly" on my corp laptop.

[DanTup]

I edited the python script to output what it was trying to run and seems that it's dying on this:

['download_from_google_storage', '--no_auth', '--no_resume', '--bucket', 'dart-dependencies', '--recursive', '--directory', 'sdk/third_party/d8']

Running that manually gives this:

dantup-macbookpro:dart-sdk dantup$ download_from_google_storage --no_auth --no_resume --bucket dart-dependencies --recursive --directory sdk/third_party/d8
Traceback (most recent call last):
  File "/Users/dantup/Dev/Google/depot_tools/download_from_google_storage.py", line 604, in <module>
    sys.exit(main(sys.argv))
  File "/Users/dantup/Dev/Google/depot_tools/download_from_google_storage.py", line 600, in main
    options.extract)
  File "/Users/dantup/Dev/Google/depot_tools/download_from_google_storage.py", line 401, in download_from_google_storage
    gsutil.check_call('version')
  File "/Users/dantup/Dev/Google/depot_tools/download_from_google_storage.py", line 108, in check_call
    timeout=self.timeout)
  File "/Users/dantup/Dev/Google/depot_tools/subprocess2.py", line 458, in communicate
    proc = Popen(args, **kwargs)
  File "/Users/dantup/Dev/Google/depot_tools/subprocess2.py", line 262, in __init__
    % (str(e), kwargs.get('cwd'), args[0]))
OSError: Execution failed with error: [Errno 2] No such file or directory.
Check that None or vpython exist and have execution permission.

So I got that to print what it's trying to run and I get:

['vpython', '/Users/dantup/Dev/Google/depot_tools/gsutil.py', '--force-version', '4.28', 'version']

Running that manually results in:

dantup-macbookpro:dart-sdk dantup$ vpython /Users/dantup/Dev/Google/depot_tools/gsutil.py --force-version 4.28 version
Traceback (most recent call last):
  File "/Users/dantup/Dev/Google/depot_tools/gsutil.py", line 182, in <module>
    sys.exit(main())
  File "/Users/dantup/Dev/Google/depot_tools/gsutil.py", line 179, in main
    clean=args.clean)
  File "/Users/dantup/Dev/Google/depot_tools/gsutil.py", line 152, in run_gsutil
    return subprocess.call(cmd, shell=IS_WINDOWS)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Make that print results in this:

['vpython', '-vpython-spec', '/Users/dantup/Dev/Google/depot_tools/gsutil.vpython', '--', '/Users/dantup/Dev/Google/depot_tools/external_bin/gsutil/gsutil_4.28/gsutil/gsutil', '-o', 'GSUtil:software_update_check_period=0', 'version']

Running that manually seems to run with no errors:

dantup-macbookpro:dart-sdk dantup$ vpython -vpython-spec /Users/dantup/Dev/Google/depot_tools/gsutil.vpython -- /Users/dantup/Dev/Google/depot_tools/external_bin/gsutil/gsutil_4.28/gsutil/gsutil -o GSUtil:software_update_check_period=0 version
gsutil version: 4.28
dantup-macbookpro:dart-sdk dantup$ 

I'm not sure why that command seems to work for me but fails in the script... Here's the offending code:

  # Run "gsutil" through "vpython". We need to do this because on GCE instances,
  # expectations are made about Python having access to "google-compute-engine"
  # and "boto" packages that are not met with non-system Python (e.g., bundles).
  cmd = [
      'vpython',
      '-vpython-spec', os.path.join(THIS_DIR, 'gsutil.vpython'),
      '--',
      gsutil_bin
  ] + disable_update + args
  return subprocess.call(cmd, shell=IS_WINDOWS)

I'll try and do a bit more digging...

[DanTup]

So, I fixed it! It turned out to be fairly simple - I'd used tildes ~ in my PATH, and they resolved fine in the terminal but not inside Python, which is why it culdn't find vpython. Doh.

[poioit]

DanTup, thanks for this thread and it helps me a lot. I changed ~ in my PATH. it can be resolved.

[saber47]

Replace ~ with absolute path also works for me as well. Thanks DanTup!