Dvc: Cannot dvc add an already tracked file that lies in a folder

[lefos99]

Bug Report

dvc version: 1.8.0

----Use case problem----
So I have folder that looks like that:

.
├── Annotations.json -> /media/newhdd/data/ki67_er_pr/data/dvc_cache/44/7b62afed38955cb2a2f4ca35c5133c
├── Annotations.json.dvc
└── annotator
    ├── ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_0.json -> /media/newhdd/data/ki67_er_pr/data/dvc_cache/12/6285610f25480393ca3e153378352b
    ├── ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_0.json.dvc
    ├── ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_1.json -> /media/newhdd/data/ki67_er_pr/data/dvc_cache/6b/303f04558c946332e3c746781ac940
    └── ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_1.json.dvc

1 directory, 6 files

When I run dvc add annotator/ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_0.json -> I get
ERROR: output 'annotator/ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_0.json' is already specified in stage: 'annotator/ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_0.json.dvc'. :exclamation:

But when I do cd annotator && dvc add ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_0.json -> I get correctly :white_check_mark:

Stage is cached, skipping                                                                                                                
  0% Add|                                                                                                       |0/0 [00:02,     ?file/s]

And the process is skipped.

If I cat ki67_20x__perez_2019_04_12__7__1___ Annotation_0_hotspot_0.json.dvc I get:

outs:
- md5: 126285610f25480393ca3e153378352b
  path: ki67_20x__perez_2019_04_12__7__1___Annotation_0_hotspot_0.json

[efiop]

Hi @lefos99 !

Could you show the contents of annotator/Annotation_0_hotspot_0.json.dvc, please?

[lefos99]

Hi @efiop

Which file do you mean? I don't have such a file in this directory.

Update:
Sorry now I saw the confusion, I fixed my initial comment. (Sorry I had to hide some sensitive information)

[efiop]

@lefos99 Thanks! Makes sense now! Ok, I'm able to reproduce with:

#!/bin/bash

set -e
set -x

rm -rf myrepo
mkdir myrepo
cd myrepo 
git init
dvc init
DATA=/tmp/$(uuidgen)
echo foo > $DATA
mkdir data
ln -s $DATA data/foo
dvc add data/foo
dvc add data/foo

looks like our dag check broke for symlinks that are in a repo subdir :scream: Bumping up the priority...

[skshetry]

@efiop, I see it working as expected.

The first dvc add data/foo adds to the cache and links from the cache (by default, here with the copy replacing the symlink).
Also, when the path is symlink, dvc creates stage file in the working directory as foo.dvc instead of data/foo.dvc.

The second dvc add data/foo tries to create the .dvc file in data/foo.dvc as data/foo is no longer a symlink, and fails as there's already a different stage.

We could definitely fix this by making contains_symlink_up_to smarter (here :arrow_down:):
https://github.com/iterative/dvc/blob/f0a729df4d6ae259f3b40be32ea2f9f6be0aa3db/dvc/utils/fs.py#L71

But, this whole different link strategy in symlink/non-symlink case will for sure bite us somewhere else. More research might be needed to fix this issue to fix repeated dvc add calls (if this even is an issue).

[pmrowla]

Should note that the original issue (#1648) which contains_symlink_up_to was supposed to address appears to be broken again (for new git related reason) in current master:

...
2020-10-23 17:29:45,759 ERROR: /Users/pmrowla/git/scratch/test-4654/dvc_example/my_repo/data/data_file
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pmrowla/git/dvc/dvc/command/add.py", line 17, in run
    self.repo.add(
  File "/Users/pmrowla/git/dvc/dvc/repo/__init__.py", line 51, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/pmrowla/git/dvc/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/Users/pmrowla/git/dvc/dvc/repo/add.py", line 90, in add
    stage.save()
  File "/Users/pmrowla/git/dvc/dvc/stage/__init__.py", line 386, in save
    self.save_outs(allow_missing=allow_missing)
  File "/Users/pmrowla/git/dvc/dvc/stage/__init__.py", line 398, in save_outs
    out.save()
  File "/Users/pmrowla/git/dvc/dvc/output/base.py", line 263, in save
    self.ignore()
  File "/Users/pmrowla/git/dvc/dvc/output/base.py", line 243, in ignore
    self.repo.scm.ignore(self.fspath)
  File "/Users/pmrowla/git/dvc/dvc/scm/git.py", line 195, in ignore
    entry, gitignore = self._get_gitignore(path)
  File "/Users/pmrowla/git/dvc/dvc/scm/git.py", line 181, in _get_gitignore
    raise FileNotInRepoError(path)
dvc.scm.base.FileNotInRepoError: /Users/pmrowla/git/scratch/test-4654/dvc_example/my_repo/data/data_file
------------------------------------------------------------