dvc status regression after 0.30 release

[mflis]

Please provide information about your setup
Platform: Arch linux
Method of installation: pip
Python version: 3.6.8

After upgrade to 0.3.0 all my files show changed checksum at dvc status. As a sanity check I chose one file created via dvc add(by one of earlier versions of dvc), checked outs with md5sum and it matches checksum in file.

Example dvc file:

md5: fc2ea87b490d2f382601c489822a1e96
outs:
- cache: true
  md5: 068b1464866e460e58ce216cd82dcf9b
  metric: false
  path: plwiki-latest-pages-articles.xml

Series of outputs with different versions.

 pip install dvc==0.28

dvc --version
0.28.0

dvc status plwiki-latest-pages-articles.xml.dvc
Pipeline is up to date. Nothing to reproduce.

pip install dvc==0.30

dvc --version
0.29.0+220b4b.mod

dvc status plwiki-latest-pages-articles.xml.dvc
plwiki-latest-pages-articles.xml.dvc:
    changed checksum



md5-9520a4b723bb82622895575a0deeb0b8



pip install git+git://github.com/iterative/dvc

dvc --version
0.30.0+9175c4
dvc status plwiki-latest-pages-articles.xml.dvc
plwiki-latest-pages-articles.xml.dvc:
    changed checksum

[mflis]

git bisect pointed me at this commit: https://github.com/iterative/dvc/commit/38536903e101b7449e65fc668c4b19519e61869c

[shcheklein]

@mflis I can confirm the regression. Taking a look now.

[shcheklein]

@mflis thank you for reporting this! I've created a PR to fix. We'll release a new version asap!

[mflis]

@shcheklein thanks for super fast reaction. I've rebuilt newest version from master (0.30.1+97c099) and it fixes problem with file I've attached in this issue.

However not all files were fixed. Here's another problematic file:

md5: 54e861d73bd6788ac59b39b6cac00a4f
outs:
- cache: true
  md5: 1e7dd6fac926ffeb6689b4aa1ec690af.dir
  metric: false
  path: vocabulary
wdir: .

It was created yesterday, so it was most likely 0.30.0 version, also via dvc add

Other examples in my repo with changed checksum are my fault, because i've manually edited dvc files. I wanted to cleanup loads of dvc files in my main directory, so I moved them to separate directory. Then, to fix path issues, I've added wdir: .. Now paths are pointing to right places, but I have changed checksum. Is there a way I can force recalculation of these checksums?

BTW It would be nice to have reference describing format of dvc file. It's mostly self explanatory, I had to browse source code to learn that md5 between deps and outs refers to checksum of dvc file content itself.

[efiop]

@mflis If you are sure you want to simply re-calculate checksums and save whatever you have, you could use dvc commit -f to commit everything or dvc commit -f stage.dvc to commit specific stage file. It will make sure you have files in cache and will re-calc all needed checksums with files that you have without running the command. Please let us know if that helped :slightly_smiling_face: Also, 0.30.1 is officially released, so you don't have to use upstream version anymore :slightly_smiling_face:

[mflis]

@efiop dvc commit -f helped with changed checksum. Thanks

[shcheklein]

@mflis sorry for this mess, the problem was that files that 0.30 were creating (actually, the way it was calculating md5 when wdir is empty or default '.') was not compatible with neither 0.29, nor 0.30.1. That's why in 0.30.1 you saw that files created in 0.30 are marked as changed. Again, sorry for this premature release, we should have tested this way more rigorous before releasing 0.30.