dvc: dvc_walk() and related things are broken by design

[Suor]

A few things:

• when we collect stages in a branch dvcignore is not used at all, i.e. we go into .dvc and other ignored dirs
• dvc_walk() and walk_files() always walk the working dir, but might accept dvcignore built from a branch. The results will make no sense.

We may approach this from two sides:

• wrap dvcignore into a tree, i.e. dvctree = TreeWithIgnores(some_tree); dvctree.walk(...).
• check that we only use dvcignore built from working dir if we os.walk().

[efiop]

when we collect stages in a branch dvcignore is not used at all, i.e. we go into .dvc and other ignored dirs

It is used https://github.com/iterative/dvc/blob/master/dvc/repo/__init__.py#L401 . Could you clarify?

[Suor]

Not used in GitTree.walk() https://github.com/iterative/dvc/blob/ee70822bd434adb36c41e20c63939af69bf0bc64/dvc/scm/git/tree.py#L139-L152

[efiop]

@Suor Good catch! CC @pared

[ghost]

@Suor, what do you mean by a tree?

[efiop]

@mroutis trees that we use. E.g. dvc/dvc/scm/git/tree.py

[ghost]

thanks, @efiop

[pared]

Confirmed, dvcignore has not been properly working for branches since day 1.
https://github.com/iterative/dvc/pull/1820