Dvc: .dvcignore is broken on negation when blacklisting all

[skshetry]

Bug Report

Please provide information about your setup

I am not sure how far this extends to, but a lot of trials for .dvcignore failed when I blacklisted all and tried to whitelist some:

*
!scripts

*
!/scripts

/*
!/scripts/

/*
!/scripts/*

/*
!/scripts/**

/*
!scripts/

What worked:

/*
!/scripts

Why do I feel suddenly the spelling of scripts is wrong? :smile:

[efiop]

CC @pared is this expected?

[pared]

Considering that we have been designing .dvcignore to comply with .gitignore, I will refer to original:
It seems to me that we have some disrepancies in behaviour between git and dvc. I prepared a script to illustrate that:

#!/bin/bash

rm -rf repo
mkdir repo

pushd repo

git init --quiet
dvc init --quiet

git commit -am "ble"

mkdir data
echo 1 >> data/1
echo 2 >> data/2
echo 3 >> data/3
echo not_ignored >> data/file
mkdir data/dir
echo not_ignored >> data/dir/file

echo 'dvci*' >> .gitignore 

echo unrelated >> dvci0

echo '*' > dvci1

echo '*' > dvci2
echo '!/dir' >> dvci2
echo '!/file' >> dvci2

echo '*' > dvci3
echo 'dir' >> dvci3
echo 'file' >> dvci3

echo '/*' > dvci4
echo '!/dir/' >> dvci4
echo '!/file/' >> dvci4

echo '/*' > dvci5
echo '!/dir/*' >> dvci5

echo '/*' > dvci6
echo '!/dir/**' >> dvci6

echo '/*' > dvci7
echo '!dir/' >> dvci7
echo '!file/' >> dvci7

echo '/*' > dvci8
echo '!/dir' >> dvci8
echo '!/file' >> dvci8


for i in {0..8}
do
    echo '### START ###'
    echo 'ignore content:'
    cat dvci$i
    echo '----------------'
    cp dvci$i data/.dvcignore
    echo 'dvc results:' 
    python -c "from dvc.repo import Repo;print(list(Repo().tree.walk_files('data')))"


    # git test
    echo ''
    echo 'git results:'
    cp dvci$i data/.gitignore
    res=()
    for value in "data/1" "data/2" "data/3" "data/dir/file" "data/file"
    do
        if [[ ! $(git check-ignore $value) ]]; then
            res+="'$value', "
        fi
    done
    echo "[${res[@]}]"

    echo '### STOP ###'
    echo ''
    echo ''
done

And the two differences occur in following situations:

ignore content:
/*
!/dir/
!/file/
----------------
dvc results:
[]

git results:
['data/dir/file', ]

and

ignore content:
/*
!dir/
!file/
----------------
dvc results:
[]

git results:
['data/dir/file', ]

I also compared the results from before latest dvc optimization, https://github.com/iterative/dvc/pull/3967 results were the same as for master.

So, in conclusion, I think its a bug.

[karajan1001]

I compared the result to #4120 and there is no difference.

[skshetry]

@karajan1001, I think this issue is with the upstream library, as this issue looks similar to: https://github.com/cpburnz/python-path-specification/issues/19, but I haven't looked closely to verify.

[karajan1001]

@karajan1001, I think this issue is with the upstream library, as this issue looks similar to: cpburnz/python-path-specification#19, but I haven't looked closely to verify.

@skshetry I agreed.

I had looked into the PathSpec code and the following result is weird to me. It is very different from the logic I learned from the PathSpec code.

ignore content:
/*
!dir/
!file/

---

git results:
['data/dir/file', ]