Dvc: Computing md5 for a large directory. This is only done once. Runs every time.

[andrethrill]

$ dvc --version
0.18.15

$ uname -v
#24~16.04.1-Ubuntu SMP Wed Aug 29 12:54:36 UTC 2018

I'm working on a repository together with 3 other colleagues. We work on the same machine with remote ssh sessions. We use DVC with shared cache. Each one of us works on his/her own repository copy located in distinct dirs. DVC cache is located outside the repository dirs but on the same disk partition as each of our repository dirs.

Before, when we ran, for instance, dvc status each one of us would get the following message for all the bigger files/dirs being tracked by DVC:

Computing md5 for a large directory path/to/dir1. This is only done once.
[##############################] 100%  path/to/dir1
Computing md5 for a large directory  path/to/dir2. This is only done once.
[##############################] 100%   path/to/dir2
(...)

This would run only once, coherently with the messages printed by dvc.

I've noticed that, I think every time dvc was updated to a newer version, everything would be calculated again, but just once! I assume this would due to some details on dvc calculates hashes change slightly from version to version.

Recently, one of my colleagues started to have the md5 recalculated every time. I think it coincided with a dvc update but honestly, I am not sure. But, it was only him. (we always update dvc to newer versions at the same time)
Yesterday, another colleague started to have the same behaviour. This is a pain because the md5 calculations take around 5/10 min and break our workflow.

So, currently, two of us have no problems at all. The other two always have the md5's recalculated.

What we tried so far:

• creating a fresh python virtual environment (both 3.6 and 3.7)
• creating a fresh copy of the git repository, and running dvc status twice.

• restarting the machine (yes, everyone tries this at some point right? 🤓 )

  The behaviour continues.

Unfortunately, I am not sure how to create a reproducible example of this.

Any suggestions would be highly appreciated.

Thanks,
André

[efiop]

Hi @andrethrill !

Sounds like you are affected by the recent state file size limit, which results in such behavior 🙁Could you please downgrade to 0.18.11 and give it a try? The introduced limit is a temporary measure and it is going to be lifted in 0.19 very soon. When introducing the limit I thought that 10K entries should be plenty but looks like I was definitely wrong. The patch with the fix is already merged, I will try to get everything ready and release ASAP(hopefully until tomorrow). So sorry for the inconvenience.

Thanks,
Ruslan

[andrethrill]

Hi @efiop,

Indeed! Tried it with 0.18.11 and the symptoms disappeared!

Initially I excluded that limitation as a cause because only some of us were experiencing the problem. Although I still don’t understand why is it like that, I’m happy we found the cause. Looking forward for v0.19 ;)

Thank you very much for your support.

[efiop]

Hi @andrethrill !

Sorry it takes longer than expected. I think the change is going to be released until monday for sure.
Thank you for your patience!

-Ruslan

[andrethrill]

No worries @efiop, 0.18.11 has been doing its job for now.

Thanks for keeping me update!

[efiop]

Sorry for the delay. The patch for this has been merged and we're getting ready to release 0.19 until the end of the week.

Thanks,
Ruslan

[efiop]

Closing this issue since the patch is merged. Preparing to release 0.19.

[efiop]

Hi @andrethrill !

0.19 is out. Please feel free to upgrade.

Thanks,
Ruslan

[andrethrill]

Thanks for the news @efiop! Will update!