dvc ignore "!" syntax different behavior from git

[dorp92]

From:
https://dvc.org/doc/command-reference/check-ignore

echo "file*\n\!file2" >> .dvcignore
touch file1 file2 other
dvc check-ignore file*
file1
file2

Why file2 is ignored?
Correct me if I'm worng but in git file2 shouldn't be ignored.

I am encountering this issue as a result of trying to ignore all files except certain extensions.
This works with git but not in dvc

*
!*.pth
!*.npy
!*/
.idea/

There is anther way to do it in dvc?

[jorgeorpinel]

It's a good question but I tried the same in .gitignore and git check-ignore has the same result: file2 is ignored.

Ref: https://git-scm.com/docs/gitignore

[dorp92]

Yes you are right about the output of check-ignore but still I think there is a bug.
I created .gitignore and .dvcignore files containing:

file*
!file2

Then

$ dvc add file2
Adding...                                                                                                     
ERROR: Path 'file2' is ignored by
.dvcignore:1:file*
.dvcignore:2:!file2

but git add file2 works fine.

[jorgeorpinel]

Yes, in that case DVC behaves different to Git by design (I think cc @efiop). It gives the error do that the user knows what's going and can address the situation if needed.

What is the problem you are having, or what are you trying to achieve?

[skshetry]

Looks like a bug to me. We had a discussion on #4103 which was fixed by @karajan1001.

It might be a mistake on DVC's part, and I wonder If it could be fixed by something similar to following (@karajan1001, please correct me if I am wrong):

diff --git a/dvc/ignore.py b/dvc/ignore.py
index 301e6669..49621fdd 100644
--- a/dvc/ignore.py
+++ b/dvc/ignore.py
@@ -129,7 +129,10 @@ class DvcIgnorePatterns(DvcIgnore):
                     result.append(pattern.file_info)
             else:
                 if regex.match(path):
-                    result.append(pattern.file_info)
+                    if ignore[1] is False:  # similarly for `is_dir`
+                        result = []
+                    else:
+                        result.append(pattern.file_info)
         return result

[dorp92]

Yes, in that case DVC behaves different to Git by design (I think cc @efiop). It gives the error do that the user knows what's going and can address the situation if needed.

What is the problem you are having, or what are you trying to achieve?

But it shouldn't give an error, Git just added the file.
With !file2 in gitignore file2 isn't ignored, But in dvc it is ignored.

My final goal is to ignore everything except of certain extensions

[karajan1001]

echo "file*\n\!file2" >> .dvcignore
touch file1 file2 other
dvc check-ignore file*
file1
file2

is not a bug, the Git does the same thing.

$ dvc add file2
Adding...                                                                                                     
ERROR: Path 'file2' is ignored by
.dvcignore:1:file*
.dvcignore:2:!file2

This is a bug.

[karajan1001]

@skshetry Seems that because of this

Here I add two prints into DvcIgnorePatterns.ignore and DvcIgnorePatterns.ignore_details. We can see that checking if file2 is ignored runs into DvcIgnorePatterns.ignore_details which is actually designed for dvc check-ignore it has a different behaviour fromDvcIgnorePatterns.ignore which is used for other funcs.

DvcIgnorePatterns.ignore return true for file matches ignore pattern ( starts without !) but DvcIgnorePatterns.ignore return true for file matches any pattern ( includes ones starts with ! )

Maybe we should rename DvcIgnorePatterns.ignore_details to something else or add some comments to prevent misusage.

[karajan1001]

The problem is here.
https://github.com/iterative/dvc/blob/a7d665157cd838a9406e30c249065111d93a9e4a/dvc/output/base.py#L533

[skshetry]

@karajan1001, thanks. So, we just have to change it to use is_ignored() instead in that file.

For other internal functions, for example, _ignored_details, a few comments should be enough. I just reached there by debugging.

I think check_ignore also should have a few docstring, that's enough I guess.

[karajan1001]

It is OK in version 1.4.0.