Dvc: 0.90.0 requires write permission on cache dir

[florianspecker]

Since 0.90.0, dvc fetch, dvc checkout and dvc pull all fail with:

$ dvc pull
ERROR: unexpected error - [Errno 30] Read-only file system: '/Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd'

/Volumes/dvc is a read-only NFS mount. Up and including 0.89.0, this setup worked like a charm.

$ more .dvc/config
[core]
analytics = false
remote = datasetmaster
['remote "datasetmaster"']
url = s3:///eac-plv-dataset-iphone5-4k-okr_q4_2019
profile = datasets

$ more .dvc/config.local
[cache]
dir = /Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/

Background on our setup:
DVC pushes images to S3. Our dataset engineers have the .dvc/config as pasted above. The same config is used for our ML training runs that are being executed on AWS.
Our ML developers need the same datasets, and so do the ML trainings that are executed onprem (we have our own GPUs; AWS is only used for peaks). Both these use cases have .dvc/config.local as pasted above (in addition to the same .dvc/config as everybody else). It points to a NFS share, where we sync the content of our S3 bucket. It is read-only to make sure it stays consistent.

Environment (this is my machine - the same issue occurred on a colleagues machine running Ubuntu):
DVC version: 0.90.0
Python version: 3.7.7
Platform: Darwin-19.3.0-x86_64-i386-64bit
Binary: False
Package: brew
Supported remotes: azure, gdrive, gs, http, https, s3, ssh, oss
Cache: reflink - supported, hardlink - supported, symlink - supported

[efiop]

Hi @florianspecker !

Thanks for reporting this issue! Could you show us verbose log with $ dvc pull -v, please?

I'm pretty sure it is coming from https://github.com/iterative/dvc/blob/0.90.2/dvc/remote/local.py#L457 , but just want to double check.

[florianspecker]

Hi @efiop thanks a lot for looking into it!

$ dvc pull -v
2020-03-19 14:35:24,577 DEBUG: PRAGMA user_version;
2020-03-19 14:35:24,577 DEBUG: fetched: [(3,)]
2020-03-19 14:35:24,577 DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
2020-03-19 14:35:24,578 DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
2020-03-19 14:35:24,578 DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
2020-03-19 14:35:24,578 DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
2020-03-19 14:35:24,578 DEBUG: PRAGMA user_version = 3;
2020-03-19 14:35:25,372 DEBUG: Preparing to download data from 's3://<our-S3-bucket>/eac-plv-dataset-iphone5-4k-okr_q4_2019'
2020-03-19 14:35:25,372 DEBUG: Preparing to collect status from s3://<our-S3-bucket>/eac-plv-dataset-iphone5-4k-okr_q4_2019
2020-03-19 14:35:25,373 DEBUG: Collecting information from local cache...
2020-03-19 14:35:25,375 DEBUG: Path '../../../../../Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd' inode '403712'
2020-03-19 14:35:25,376 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=?
2020-03-19 14:35:25,376 DEBUG: fetched: [('1580748406000000000', '10042082', '2cb5d8b77eb1ac4f56dfbce8aa2f4dfd', '1584624915719617024')]
2020-03-19 14:35:25,377 DEBUG: UPDATE state SET timestamp = ? WHERE inode = ?
2020-03-19 14:35:25,377 DEBUG: cache '../../../../../Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd' expected '2cb5d8b77eb1ac4f56dfbce8aa2f4dfd' actual '2cb5d8b77eb1ac4f56dfbce8aa2f4dfd'
2020-03-19 14:35:25,379 DEBUG: SELECT count from state_info WHERE rowid=?
2020-03-19 14:35:25,379 DEBUG: fetched: [(506,)]
2020-03-19 14:35:25,380 DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?
2020-03-19 14:35:25,381 ERROR: unexpected error - [Errno 30] Read-only file system: '/Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/command/data_sync.py", line 31, in run
    recursive=self.args.recursive,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/__init__.py", line 28, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/pull.py", line 28, in pull
    recursive=recursive,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/fetch.py", line 52, in _fetch
    used, jobs, remote=remote, show_checksums=show_checksums
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/data_cloud.py", line 82, in pull
    cache, jobs=jobs, remote=remote, show_checksums=show_checksums
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 382, in pull
    download=True,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 346, in _process
    download=download,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 255, in status
    local_exists = self.cache_exists(md5s, jobs=jobs, name=self.cache_dir)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 214, in cache_exists
    + ("cache in " + name if name else "local cache"),
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 216, in <listcomp>
    if not self.changed_cache_file(checksum)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/base.py", line 759, in changed_cache_file
    self.protect(cache_info)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 448, in protect
    os.chmod(path, mode)
OSError: [Errno 30] Read-only file system: '/Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd'
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

$ dvc fetch -v
2020-03-19 14:35:50,571 DEBUG: PRAGMA user_version;
2020-03-19 14:35:50,572 DEBUG: fetched: [(3,)]
2020-03-19 14:35:50,572 DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
2020-03-19 14:35:50,572 DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
2020-03-19 14:35:50,572 DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
2020-03-19 14:35:50,572 DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
2020-03-19 14:35:50,573 DEBUG: PRAGMA user_version = 3;
2020-03-19 14:35:51,361 DEBUG: Preparing to download data from 's3://<our-S3-bucket>/eac-plv-dataset-iphone5-4k-okr_q4_2019'
2020-03-19 14:35:51,361 DEBUG: Preparing to collect status from s3://<our-S3-bucket>/eac-plv-dataset-iphone5-4k-okr_q4_2019
2020-03-19 14:35:51,361 DEBUG: Collecting information from local cache...
2020-03-19 14:35:51,363 DEBUG: Path '../../../../../Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd' inode '403712'
2020-03-19 14:35:51,363 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=?
2020-03-19 14:35:51,364 DEBUG: fetched: [('1580748406000000000', '10042082', '2cb5d8b77eb1ac4f56dfbce8aa2f4dfd', '1584624925377250048')]
2020-03-19 14:35:51,364 DEBUG: UPDATE state SET timestamp = ? WHERE inode = ?
2020-03-19 14:35:51,365 DEBUG: cache '../../../../../Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd' expected '2cb5d8b77eb1ac4f56dfbce8aa2f4dfd' actual '2cb5d8b77eb1ac4f56dfbce8aa2f4dfd'
2020-03-19 14:35:51,366 DEBUG: SELECT count from state_info WHERE rowid=?
2020-03-19 14:35:51,366 DEBUG: fetched: [(506,)]
2020-03-19 14:35:51,367 DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?
2020-03-19 14:35:51,368 ERROR: unexpected error - [Errno 30] Read-only file system: '/Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/command/data_sync.py", line 72, in run
    recursive=self.args.recursive,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/__init__.py", line 28, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/__init__.py", line 508, in fetch
    return self._fetch(*args, **kwargs)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/fetch.py", line 52, in _fetch
    used, jobs, remote=remote, show_checksums=show_checksums
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/data_cloud.py", line 82, in pull
    cache, jobs=jobs, remote=remote, show_checksums=show_checksums
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 382, in pull
    download=True,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 346, in _process
    download=download,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 255, in status
    local_exists = self.cache_exists(md5s, jobs=jobs, name=self.cache_dir)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 214, in cache_exists
    + ("cache in " + name if name else "local cache"),
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 216, in <listcomp>
    if not self.changed_cache_file(checksum)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/base.py", line 759, in changed_cache_file
    self.protect(cache_info)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 448, in protect
    os.chmod(path, mode)
OSError: [Errno 30] Read-only file system: '/Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/2c/b5d8b77eb1ac4f56dfbce8aa2f4dfd'
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

$ dvc checkout -v
2020-03-19 14:35:56,999 DEBUG: PRAGMA user_version;
2020-03-19 14:35:56,999 DEBUG: fetched: [(3,)]
2020-03-19 14:35:56,999 DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
2020-03-19 14:35:57,000 DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
2020-03-19 14:35:57,000 DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
2020-03-19 14:35:57,000 DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
2020-03-19 14:35:57,000 DEBUG: PRAGMA user_version = 3;
2020-03-19 14:35:57,697 DEBUG: SELECT * FROM link_state
2020-03-19 14:35:57,717 DEBUG: checking if 'short_range/pitch_4/IMG_2543.png'('{'md5': '97f8c0bdc301b6c7711b9d03a0f4da81'}') has changed.
2020-03-19 14:35:57,718 DEBUG: 'short_range/pitch_4/IMG_2543.png' doesn't exist.
2020-03-19 14:35:57,721 DEBUG: Path '../../../../../Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/97/f8c0bdc301b6c7711b9d03a0f4da81' inode '403738'
2020-03-19 14:35:57,721 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=?
2020-03-19 14:35:57,722 DEBUG: fetched: []
2020-03-19 14:35:58,743 DEBUG: Path '../../../../../Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/97/f8c0bdc301b6c7711b9d03a0f4da81' inode '403738'
2020-03-19 14:35:58,744 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=?
2020-03-19 14:35:58,744 DEBUG: fetched: []
2020-03-19 14:35:58,745 DEBUG: INSERT INTO state(inode, mtime, size, md5, timestamp) VALUES (?, ?, ?, ?, ?)
2020-03-19 14:35:58,745 DEBUG: cache '../../../../../Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/97/f8c0bdc301b6c7711b9d03a0f4da81' expected '97f8c0bdc301b6c7711b9d03a0f4da81' actual '97f8c0bdc301b6c7711b9d03a0f4da81'
2020-03-19 14:35:58,749 DEBUG: SELECT count from state_info WHERE rowid=?
2020-03-19 14:35:58,749 DEBUG: fetched: [(506,)]
2020-03-19 14:35:58,749 DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?
2020-03-19 14:35:58,750 ERROR: unexpected error - [Errno 30] Read-only file system: '/Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/97/f8c0bdc301b6c7711b9d03a0f4da81'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/command/checkout.py", line 71, in run
    recursive=self.args.recursive,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/__init__.py", line 28, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/__init__.py", line 504, in checkout
    return self._checkout(*args, **kwargs)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/repo/checkout.py", line 83, in _checkout
    filter_info=filter_info,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/funcy/decorators.py", line 39, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/stage.py", line 161, in rwlocked
    return call()
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/funcy/decorators.py", line 60, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/stage.py", line 993, in checkout
    filter_info=filter_info,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/output/base.py", line 301, in checkout
    filter_info=filter_info,
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/base.py", line 980, in checkout
    checksum, path_info=path_info, filter_info=filter_info
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/base.py", line 798, in changed_cache
    return self.changed_cache_file(checksum)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/base.py", line 759, in changed_cache_file
    self.protect(cache_info)
  File "/usr/local/Cellar/dvc/0.90.0/libexec/lib/python3.7/site-packages/dvc/remote/local.py", line 448, in protect
    os.chmod(path, mode)
OSError: [Errno 30] Read-only file system: '/Volumes/dvc/eac-plv-dataset-iphone5-4k-okr_q4_2019/97/f8c0bdc301b6c7711b9d03a0f4da81'
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[efiop]

@florianspecker Yep, indeed the issue raises where expected. Thanks for clarifying! Will try to prepare a patch for it today along with a new release.

[florianspecker]

@efiop great, thanks a lot for the quick response!

[efiop]

Hi @florianspecker ! 0.91.1 is out with the fix. Please upgrade and give it a try.

[florianspecker]

Hi @efiop thank you very much! I can confirm that the issue is gone. Thanks a lot for the timely fix!