Dvc: unexpected error for aws training machine

[marksein07]

Bug Report

dvc pull and reproduce raise a unexpected error on aws gpu training machine

I use gitlab on aws g3xlarge and try to dvc pull data from gdrive

$ dvc version 1.10.0
$ git version 2.17.1

$ dvc repro
ERROR: unexpected error - expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/main.py", line 89, in main
    cmd = args.func(args)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/command/base.py", line 40, in __init__
    self.repo = Repo(uninitialized=self.UNINITIALIZED)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 177, in __init__
    self._ignore()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 256, in _ignore
    self.scm.ignore_list(flist)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in ignore_list
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in <listcomp>
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 195, in ignore
    entry, gitignore = self._get_gitignore(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 181, in _get_gitignore
    if not path_isin(os.path.realpath(gitignore), self.root_dir):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 156, in path_isin
    parent = os.path.join(normalize_path(parent), "")
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 154, in normalize_path
    return os.path.normpath(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/posixpath.py", line 340, in normpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/bin/dvc", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/main.py", line 119, in main
    dvc_info = get_dvc_info()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/info.py", line 37, in get_dvc_info
    repo = Repo()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 177, in __init__
    self._ignore()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 256, in _ignore
    self.scm.ignore_list(flist)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in ignore_list
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in <listcomp>
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 195, in ignore
    entry, gitignore = self._get_gitignore(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 181, in _get_gitignore
    if not path_isin(os.path.realpath(gitignore), self.root_dir):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 156, in path_isin
    parent = os.path.join(normalize_path(parent), "")
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 154, in normalize_path
    return os.path.normpath(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/posixpath.py", line 340, in normpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

the error raised before any verbose output

[skshetry]

@marksein07, could you please provide complete dvc version output? It has more information than dvc --version. Thanks.

[marksein07]

In fact There is the same Error

$ dvc version
ERROR: unexpected error - expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/main.py", line 90, in main
    ret = cmd.run()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/command/version.py", line 12, in run
    dvc_info = get_dvc_info()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/info.py", line 37, in get_dvc_info
    repo = Repo()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 177, in __init__
    self._ignore()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 256, in _ignore
    self.scm.ignore_list(flist)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in ignore_list
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in <listcomp>
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 195, in ignore
    entry, gitignore = self._get_gitignore(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 181, in _get_gitignore
    if not path_isin(os.path.realpath(gitignore), self.root_dir):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 156, in path_isin
    parent = os.path.join(normalize_path(parent), "")
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 154, in normalize_path
    return os.path.normpath(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/posixpath.py", line 340, in normpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/bin/dvc", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/main.py", line 119, in main
    dvc_info = get_dvc_info()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/info.py", line 37, in get_dvc_info
    repo = Repo()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 177, in __init__
    self._ignore()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/repo/__init__.py", line 256, in _ignore
    self.scm.ignore_list(flist)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in ignore_list
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/base.py", line 81, in <listcomp>
    return [self.ignore(path) for path in p_list]
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 195, in ignore
    entry, gitignore = self._get_gitignore(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/scm/git.py", line 181, in _get_gitignore
    if not path_isin(os.path.realpath(gitignore), self.root_dir):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 156, in path_isin
    parent = os.path.join(normalize_path(parent), "")
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/dvc/utils/fs.py", line 154, in normalize_path
    return os.path.normpath(path)
  File "/home/ubuntu/anaconda3/lib/python3.7/posixpath.py", line 340, in normpath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

[skshetry]

@marksein07, can you please upgrade to the latest version and see if it works?

[marksein07]

I tried 1.11.2, and It doesn't work.

[skshetry]

@marksein07, can you provide more information? Does this happen when running inside a dvc/git repo?
And, what's there location are? Are they same?

It would be easier to debug if you could provide a script.

[marksein07]

In fact, I can operate dvc perfectly inside private gitlab in local pc with ubuntu 18.04 and conda.

However, after I do the same operation on aws cloud machine, the every dvc command is just crashed except for --help work well.

There is even no any other verbose I could provide, and I also confused.

Is there any Environment variable the dvc default to use ?

[efiop]

@marksein07 Anything about the path where you run dvc in on your aws machine? What is the absolute path to it? E.g. could you show $ pwd output?

[marksein07]

ubuntu@ip-10-0-1-126:~/git/vnpc-dialog-service/sa$ pwd 
/home/ubuntu/git/vnpc-dialog-service/sa

I tried to run dvc init in other folder in git :
and the Error message is

ERROR: unexpected error - expected str, bytes or os.PathLike object, not NoneType

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[efiop]

@marksein07 Any symlinks in that path?

[marksein07]

@efiop No

[skshetry]

@marksein07, could you please try 1.11.6 version? We have made some changes there, so maybe that'll work? Although the chances are slim. Thanks.

[efiop]

@marksein07 Hard to tell what is going on here. Please try @skshetry 's suggestion and if it still fails, and you are familiar with pdb and keen on debugging a bit, try running dvc version --pdb and seeing what went wrong. Without us being able to reproduce it, it will take longer to try to figure out what went wrong. This does make us think about including additional debugging info into our error logs (e.g. args in the stack) to be able to quicker tell what broke. :slightly_frowning_face: