dvc pull not fetching all data (cache file not found)

Created on 8 Jan 2019  路  28Comments  路  Source: iterative/dvc

Please provide information about your setup

dvc --version
0.23.2

uname -a
Linux arachne-postgres 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

On server-1
data pushed to remote is in cache and in s3

$ ls -l .dvc/cache/9c/
total 63360
-rw-rw-r-- 1 ubuntu ubuntu   783548 Jan  4 20:53 01b58cb0faab4ee28a9228552ffd8d
-rw-rw-r-- 1 ubuntu ubuntu  3719779 Jan  4 20:53 2861308e6110dc7f4850cbe331e63a
-rw-rw-r-- 2 ubuntu ubuntu    14722 Jan  4 20:51 2fa17d3b0c9486c5af435329f62151
-rw-rw-r-- 1 ubuntu ubuntu   849013 Jan  4 20:52 416b598605a8fcd6fc04c2edab4edc
-rw-rw-r-- 2 ubuntu ubuntu    22852 Jan  4 20:52 55bb368600627e7e14ad7648d8f26b
-rw-rw-r-- 2 ubuntu ubuntu    39899 Jan  4 20:52 5ffd6a38f12b14c6a5e6aafacf133c
-rw-rw-r-- 1 ubuntu ubuntu   614053 Jan  4 20:51 711b4315305a21dabfa44e74740ff7
-rw-rw-r-- 2 ubuntu ubuntu    23825 Jan  4 20:52 765b898179e0c88af48a638dfe6586
-rw-rw-r-- 1 ubuntu ubuntu   148555 Jan  7 07:36 7b5eca364544bf04f63c390dde7f6e.dir
-rw-rw-r-- 1 ubuntu ubuntu   865287 Jan  4 20:53 7f3444015a68ee039d84986d5f9a98
-rw-rw-r-- 2 ubuntu ubuntu    18086 Jan  4 20:52 8bc9b783a8e0568e71d350e4a1fc37
-rw-rw-r-- 2 ubuntu ubuntu     9823 Jan  4 20:52 95e0f6ac3455b66b69974c3936eba7
-rw-rw-r-- 1 ubuntu ubuntu   673028 Jan  4 20:52 997353cce6f0997dc97e311893f3fb
-rw-rw-r-- 1 ubuntu ubuntu 54562699 Jan  4 22:43 9d7b83b536edc6666c76d16e9bfc6b
-rw-rw-r-- 1 ubuntu ubuntu   436909 Jan  4 20:52 a485af08514391eaf58f7a607a1aaa
-rw-rw-r-- 1 ubuntu ubuntu   365795 Jan  4 20:51 a76a77aa1343aba087211e42f6d2b7
-rw-rw-r-- 1 ubuntu ubuntu  1232517 Jan  4 20:53 ae610c9a0246025b573888e84d766e
-rw-rw-r-- 1 ubuntu ubuntu   364246 Jan  4 20:51 bec0f4fc35ecf1377e62c3859362c4
-rw-rw-r-- 1 ubuntu ubuntu    59057 Nov  9 02:47 cf83393276d56191a88c3d54ef6a5d
-rw-rw-r-- 2 ubuntu ubuntu    33169 Jan  4 20:52 e00d88888f810720aee5b46c3f0772

aws  --endpoint=https://ceph.acc.ohsu.edu s3 ls s3://bmeg/dvc/9c/
2019-01-08 04:21:07     783548 01b58cb0faab4ee28a9228552ffd8d
2019-01-08 04:20:50    3719779 2861308e6110dc7f4850cbe331e63a
2019-01-08 04:01:19      14722 2fa17d3b0c9486c5af435329f62151
2019-01-08 04:20:47     849013 416b598605a8fcd6fc04c2edab4edc
2019-01-08 04:02:18      22852 55bb368600627e7e14ad7648d8f26b
2018-12-18 21:41:50      62535 5b6377d120103a5a8e841a7b94ff4c
2019-01-08 04:02:14      39899 5ffd6a38f12b14c6a5e6aafacf133c
2019-01-08 04:19:52     614053 711b4315305a21dabfa44e74740ff7
2019-01-08 04:01:46      23825 765b898179e0c88af48a638dfe6586
2019-01-08 04:01:17     148555 7b5eca364544bf04f63c390dde7f6e.dir
2019-01-08 04:20:18     865287 7f3444015a68ee039d84986d5f9a98
2019-01-08 04:01:38      18086 8bc9b783a8e0568e71d350e4a1fc37
2019-01-08 04:01:44       9823 95e0f6ac3455b66b69974c3936eba7
2019-01-08 04:19:13     673028 997353cce6f0997dc97e311893f3fb
2019-01-04 22:41:27   54562699 9d7b83b536edc6666c76d16e9bfc6b
2019-01-08 04:20:23     436909 a485af08514391eaf58f7a607a1aaa
2019-01-08 04:19:44     365795 a76a77aa1343aba087211e42f6d2b7
2019-01-08 04:20:40    1232517 ae610c9a0246025b573888e84d766e
2018-12-18 21:41:50     799324 b818767f237d4c9647c1208ce8c28b
2019-01-08 04:20:53     364246 bec0f4fc35ecf1377e62c3859362c4
2019-01-07 07:07:32      59057 cf83393276d56191a88c3d54ef6a5d
2019-01-08 04:02:17      33169 e00d88888f810720aee5b46c3f0772

On server-2
dvc never loads all files

aws  --endpoint=https://ceph.acc.ohsu.edu s3 ls s3://bmeg/dvc/9c/

2019-01-07 20:21:07     783548 01b58cb0faab4ee28a9228552ffd8d
2019-01-07 20:20:50    3719779 2861308e6110dc7f4850cbe331e63a
2019-01-07 20:01:19      14722 2fa17d3b0c9486c5af435329f62151
2019-01-07 20:20:47     849013 416b598605a8fcd6fc04c2edab4edc
2019-01-07 20:02:18      22852 55bb368600627e7e14ad7648d8f26b
2018-12-18 13:41:50      62535 5b6377d120103a5a8e841a7b94ff4c
2019-01-07 20:02:14      39899 5ffd6a38f12b14c6a5e6aafacf133c
2019-01-07 20:19:52     614053 711b4315305a21dabfa44e74740ff7
2019-01-07 20:01:46      23825 765b898179e0c88af48a638dfe6586
2019-01-07 20:01:17     148555 7b5eca364544bf04f63c390dde7f6e.dir
2019-01-07 20:20:18     865287 7f3444015a68ee039d84986d5f9a98
2019-01-07 20:01:38      18086 8bc9b783a8e0568e71d350e4a1fc37
2019-01-07 20:01:44       9823 95e0f6ac3455b66b69974c3936eba7
2019-01-07 20:19:13     673028 997353cce6f0997dc97e311893f3fb
2019-01-04 14:41:27   54562699 9d7b83b536edc6666c76d16e9bfc6b
2019-01-07 20:20:23     436909 a485af08514391eaf58f7a607a1aaa
2019-01-07 20:19:44     365795 a76a77aa1343aba087211e42f6d2b7
2019-01-07 20:20:40    1232517 ae610c9a0246025b573888e84d766e
2018-12-18 13:41:50     799324 b818767f237d4c9647c1208ce8c28b
2019-01-07 20:20:53     364246 bec0f4fc35ecf1377e62c3859362c4
2019-01-06 23:07:32      59057 cf83393276d56191a88c3d54ef6a5d
2019-01-07 20:02:17      33169 e00d88888f810720aee5b46c3f0772

ls -l .dvc/cache/9c/
total 908
-rw-rw-r-- 1 ubuntu ubuntu  62535 Nov  9 16:09 5b6377d120103a5a8e841a7b94ff4c
-rw-rw-r-- 1 ubuntu ubuntu 799324 Nov 11 16:54 b818767f237d4c9647c1208ce8c28b
-rw-rw-r-- 1 ubuntu ubuntu  59057 Nov  9 16:22 cf83393276d56191a88c3d54ef6a5d

dvc fetch runs without errors (although it constantly recalcs md5)
dvc pull always returns the following.

Warning: Cache '9c7b5eca364544bf04f63c390dde7f6e.dir' not found. File '{'path': '/mnt/bmeg/bmeg-etl/source/ccle/vcfs', 'scheme': 'local'}' won't be created.

both servers are at the same git branch / commit

bug
Source
picture bwalsh

All 28 comments

Also,

dvc status -c -v
shows the file

Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'

dvc pull -v source/ccle/vcfs.dvc Debug: PRAGMA user_version; Debug: fetched: [(3,)] Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL) Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER) Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL) Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info) Debug: PRAGMA user_version = 3; Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None' Preparing to download data from 's3://bmeg/dvc' Preparing to collect status from s3://bmeg/dvc Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None' [##############################] 100% Collecting information [##############################] 100% Analysing status. Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None' Preparing to download data from 's3://bmeg/dvc' Preparing to collect status from s3://bmeg/dvc Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None' [##############################] 100% Collecting information [##############################] 100% Analysing status. Debug: SELECT count from state_info WHERE rowid=1 Debug: fetched: [(419,)] Debug: UPDATE state_info SET count = 419 WHERE rowid = 1 Debug: PRAGMA user_version; Debug: fetched: [(3,)] Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL) Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER) Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL) Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info) Debug: PRAGMA user_version = 3; Debug: SELECT * FROM link_state Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None' Warning: Cache '9c7b5eca364544bf04f63c390dde7f6e.dir' not found. File '{'path': '/mnt/bmeg/bmeg-etl/source/ccle/vcfs', 'scheme': 'local'}' won't be created. Debug: SELECT count from state_info WHERE rowid=1 Debug: fetched: [(419,)] Debug: UPDATE state_info SET count = 419 WHERE rowid = 1 Debug: Analytics is enabled. Debug: Trying to spawn '['/usr/bin/python', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmphGXZPr']' Debug: Spawned '['/usr/bin/python', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmphGXZPr']' arachne-postgres bmeg-etl (vep-merge)

bwalsh picture bwalsh on 8 Jan 2019

Hi @bwalsh !

The fact that there are fewer cache files might be normal because it might contain cache files that are not related to the vcfs directory. Still, the fact that 9c7b5eca364544bf04f63c390dde7f6e.dir is missing is strange. Have you seen "Corrupted cache file" warnings by chance?

efiop picture efiop on 8 Jan 2019

Re."Corrupted cache file", no, haven't seen them.

My latest attempt was to remove the state file and the dir.

$ mv .dvc/state .dvc/state-backup
dvc pull -v source/ccle/vcfs.dvc
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 3;
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(0,)]
Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
Debug: PRAGMA user_version;
Debug: fetched: [(3,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 3;
Debug: SELECT * FROM link_state
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
Warning: Cache '9c7b5eca364544bf04f63c390dde7f6e.dir' not found. File '{'path': '/mnt/bmeg/bmeg-etl/source/ccle/vcfs', 'scheme': 'local'}' won't be created.
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(0,)]
Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
Debug: Analytics is enabled.
Debug: Trying to spawn '['/usr/bin/python', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmpv6vnAi']'
Debug: Spawned '['/usr/bin/python', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmpv6vnAi']'
arachne-postgres bmeg-etl (vep-merge)

$ rm  -r .dvc/cache/9c
arachne-postgres bmeg-etl (vep-merge)
dvc pull -v source/ccle/vcfs.dvc
Debug: PRAGMA user_version;
Debug: fetched: [(3,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 3;
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(0,)]
Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
Debug: PRAGMA user_version;
Debug: fetched: [(3,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 3;
Debug: SELECT * FROM link_state
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/9c/7b5eca364544bf04f63c390dde7f6e.dir', md5 '9c7b5eca364544bf04f63c390dde7f6e.dir', actual 'None'
Warning: Cache '9c7b5eca364544bf04f63c390dde7f6e.dir' not found. File '{'path': '/mnt/bmeg/bmeg-etl/source/ccle/vcfs', 'scheme': 'local'}' won't be created.
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(0,)]
Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
Debug: Analytics is enabled.
Debug: Trying to spawn '['/usr/bin/python', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmp3789Xc']'
Debug: Spawned '['/usr/bin/python', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmp3789Xc']'
bwalsh picture bwalsh on 8 Jan 2019

And does dvc status -c print anything? I don't think you've posted a log for it. Could you please show dvc status -c output on both server1 and server2?

efiop picture efiop on 8 Jan 2019

@efiop Ruslan, thanks for the quick response. I have the status -c logs for server1 & server2. They are large ~ 260K. Is there a good way to email them to you?

bwalsh picture bwalsh on 8 Jan 2019

@bwalsh Sure, my email is in my github profile. Thanks! :)

efiop picture efiop on 8 Jan 2019

For the record: I've received the logs and looking into them. :)

efiop picture efiop on 8 Jan 2019

@bwalsh Could you please roll back to 0.22.0 and see if that works? We've done some heavy refactoring in 0.23.2, so I really suspect it to be the cause.

efiop picture efiop on 8 Jan 2019

@efiop Ruslan, thank you. I think (haven't processed all the files yet) that may have solved the issue. I'll update the issue tomorrow.

My process to downgrade:

pip uninstall dvc
pip install dvc==0.22.0
rm .dvc/state
dvc [pull|push]
bwalsh picture bwalsh on 9 Jan 2019
👍1

Thanks for trying that out! So we have a regression then. Looking into it.

efiop picture efiop on 9 Jan 2019

Hi @bwalsh !

Could you please try installing the experimental version, see if it works and share the logs? To install it:

pip uninstall -y dvc
pip install git+https://github.com/efiop/dvc
efiop picture efiop on 9 Jan 2019

On server-2 - I stabilized dvc state on v 0.22.0 then I installed the experimental version

$ pip freeze | grep dvc
dvc==0.22.0
$ pip uninstall -y dvc
Uninstalling dvc-0.22.0:
  Successfully uninstalled dvc-0.22.0
$ pip install git+https://github.com/efiop/dvc
 ...
$ dvc pull
Warning: State file version '2' is too old. Reformatting to the current version '3'.
Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
[###########################   ] 90% Collecting informationComputing md5 for a large file .dvc/cache/c3/90251bd07b4220eb4bf824a5240b08. This is only done once.
   ...  many lines of md5 recalc
   ... successful 

$ dvc pull
Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
  .... many more lines of md5 recalc!!!
bwalsh picture bwalsh on 10 Jan 2019

Subsequently on server 2

$ pip uninstall -y dvc
$ pip install dvc==0.22.0
$ dvc pull
  ... many lines of md5 calc
$ dvc pull
   ... many lines of "Data 'xxxx' didn't change."
bwalsh picture bwalsh on 10 Jan 2019

@bwalsh Ah, I totally forgot to say that you should try it on a clean repo(without cache) and see if the directory is created now with the experimental version. Could you try that out and show logs, please?

efiop picture efiop on 10 Jan 2019

Clarification: we are mainly interested in sever-2.

efiop picture efiop on 10 Jan 2019

@bwalsh So sorry for the delay. We've released 0.24.0 with a few related bugs fixed. Could you please give it a try?

efiop picture efiop on 24 Jan 2019

@bwalsh Do you still experience that issue? We have 0.27.1 released, consider trying it out :slightly_smiling_face:

efiop picture efiop on 16 Feb 2019

Thank you, we will try it out next week.

bwalsh picture bwalsh on 22 Feb 2019
👍1

@efiop Ruslan - after updating to 0.29.0 when I run a dvc push the first time, it runs through all the files 

(1/7279): [##############################] 100% 
...
(3891/7279): [##############################] 100% outputs/ccle/drug_response.Program.Vertex.json.gz
...

I figured this might be expected after an update.

However, running dvc push a second time, causes dvc to run through all 7K+ files again? Is this expected?

bwalsh picture bwalsh on 2 Mar 2019

Still having problems:

server1:

$ cat outputs.ccle.dvc
cmd: python3 transform/ccle/samples.py
deps:
- md5: b4cf38f5366ef8b58e7db826f872e0d4
  path: source/ccle/DepMap-2018q4-celllines.csv
md5: 815a241a1e4c564640317097555f53e7
outs:
- cache: true
  md5: 9499f47f8cc33c79cf48c1799776905e
  path: outputs/ccle/Aliquot.Vertex.json.gz
- cache: true
  md5: 484ba288601db239b17b9e3363f48cba
  path: outputs/ccle/AliquotFor.Edge.json.gz
- cache: true
  md5: 9909dd770481cea740e1db0983cfbf80
  path: outputs/ccle/Sample.Vertex.json.gz
- cache: true
  md5: 54597ce80e241a4678266cdd44374027
  path: outputs/ccle/SampleFor.Edge.json.gz
- cache: true
  md5: 7a05f11efc9fce44fdda461d121d9c00
  path: outputs/ccle/Case.Vertex.json.gz
- cache: true
  md5: 499ebf693e66144ea82ed7ea8615c461
  path: outputs/ccle/InProject.Edge.json.gz
- cache: true
  md5: 88a7ccd29b1efd70c5274ba7bc0d0288
  path: outputs/ccle/Project.Vertex.json.gz
- cache: true
  md5: 87520203d93cbbd8e6f00171567f9749
  path: outputs/ccle/InProgram.Edge.json.gz
- cache: true
  md5: 69ab8b00c4bb56805506569ec19da523
  path: outputs/ccle/Program.Vertex.json.gz
- cache: true
  md5: bcb95cf6dfdb04448f15af60072c5d80
  path: outputs/ccle/Phenotype.Vertex.json.gz
- cache: true
  md5: 88c6a6095ecc42a2974e5181568bd401
  path: outputs/ccle/PhenotypeOf.Edge.json.gz

# spot check one file

# exists in cache
ls -l .dvc/cache/94/99f47f8cc33c79cf48c1799776905e
-rw-rw-r-- 2 ubuntu ubuntu 15794 Feb  5 19:23 .dvc/cache/94/99f47f8cc33c79cf48c1799776905e

# exists in object store
aws  --endpoint=https://ceph.acc.ohsu.edu s3 ls  s3://bmeg/dvc/94/99f47f8cc33c79cf48c1799776905e
2019-03-01 23:11:57      15794 99f47f8cc33c79cf48c1799776905e

on server2:

dvc pull outputs.ccle.dvc
+-------------------------------------------+
|                                           |
|     Update available 0.29.0 -> 0.30.0     |
|       Run pip install dvc --upgrade       |
|                                           |
+-------------------------------------------+

Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
Warning: Cache '9499f47f8cc33c79cf48c1799776905e' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/Aliquot.Vertex.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '484ba288601db239b17b9e3363f48cba' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/AliquotFor.Edge.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '9909dd770481cea740e1db0983cfbf80' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/Sample.Vertex.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '54597ce80e241a4678266cdd44374027' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/SampleFor.Edge.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '7a05f11efc9fce44fdda461d121d9c00' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/Case.Vertex.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '499ebf693e66144ea82ed7ea8615c461' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/InProject.Edge.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '88a7ccd29b1efd70c5274ba7bc0d0288' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/Project.Vertex.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '87520203d93cbbd8e6f00171567f9749' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/InProgram.Edge.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '69ab8b00c4bb56805506569ec19da523' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/Program.Vertex.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache 'bcb95cf6dfdb04448f15af60072c5d80' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/Phenotype.Vertex.json.gz', u'scheme': u'local'}' won't be created.
Warning: Cache '88c6a6095ecc42a2974e5181568bd401' not found. File '{u'path': u'/mnt/bmeg/bmeg-etl/outputs/ccle/PhenotypeOf.Edge.json.gz', u'scheme': u'local'}' won't be created.
Everything is up to date.

# didn't create cache ?
$ ls .dvc/cache/94/99f47f8cc33c79cf48c1799776905e
ls: cannot access '.dvc/cache/94/99f47f8cc33c79cf48c1799776905e': No such file or directory

# no problem seeing file in s3
$ aws  --endpoint=https://ceph.acc.ohsu.edu s3 ls 
 s3://bmeg/dvc/94/99f47f8cc33c79cf48c1799776905e
2019-03-01 15:11:57      15794 99f47f8cc33c79cf48c1799776905e

both servers have the same git log

git log -1 | head -1
commit 7b8a84af1165894794ca442bcd4055b308db3848

both servers have the same .dvc/config & aws credentials

md5sum .dvc/config
ee101b2b0b63f5624b3ba3d1e366953b  .dvc/config

md5sum ~/.aws/credentials
ee51dd10c7cdb3ca527581dbd09ecef6  /home/ubuntu/.aws/credentials

Tried deleting .dvc/cache and .dvc/state - same results

Just seems like fetch not attempting to download data ?

dvc fetch  outputs.ccle.dvc --verbose
+-------------------------------------------+
|                                           |
|     Update available 0.29.0 -> 0.30.0     |
|       Run pip install dvc --upgrade       |
|                                           |
+-------------------------------------------+

Debug: PRAGMA user_version;
Debug: fetched: [(3,)]
Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
Debug: PRAGMA user_version = 3;
Preparing to download data from 's3://bmeg/dvc'
Preparing to collect status from s3://bmeg/dvc
[###########################   ] 90% Collecting informationDebug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/94/99f47f8cc33c79cf48c1799776905e', md5 '9499f47f8cc33c79cf48c1799776905e', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/88/a7ccd29b1efd70c5274ba7bc0d0288', md5 '88a7ccd29b1efd70c5274ba7bc0d0288', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/bc/b95cf6dfdb04448f15af60072c5d80', md5 'bcb95cf6dfdb04448f15af60072c5d80', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/54/597ce80e241a4678266cdd44374027', md5 '54597ce80e241a4678266cdd44374027', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/69/ab8b00c4bb56805506569ec19da523', md5 '69ab8b00c4bb56805506569ec19da523', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/88/c6a6095ecc42a2974e5181568bd401', md5 '88c6a6095ecc42a2974e5181568bd401', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/49/9ebf693e66144ea82ed7ea8615c461', md5 '499ebf693e66144ea82ed7ea8615c461', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/7a/05f11efc9fce44fdda461d121d9c00', md5 '7a05f11efc9fce44fdda461d121d9c00', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/99/09dd770481cea740e1db0983cfbf80', md5 '9909dd770481cea740e1db0983cfbf80', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/48/4ba288601db239b17b9e3363f48cba', md5 '484ba288601db239b17b9e3363f48cba', actual 'None'
Debug: File '/mnt/bmeg/bmeg-etl/.dvc/cache/87/520203d93cbbd8e6f00171567f9749', md5 '87520203d93cbbd8e6f00171567f9749', actual 'None'
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
Debug: SELECT count from state_info WHERE rowid=1
Debug: fetched: [(191,)]
Debug: UPDATE state_info SET count = 191 WHERE rowid = 1
Everything is up to date.
Debug: Analytics is disabled.

Files in the cache that are fetched

pwd ; ls -l
/mnt/bmeg/bmeg-etl/.dvc/cache
total 108
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 00
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 01
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 02
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 03
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 04
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 05
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 06
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 07
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 18:00 08
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 09
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 18:00 0a
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 0b
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 18:00 0c
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 0d
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 0e
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 0f
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 18:00 10
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 18:01 11
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 18:00 12
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 18:00 13
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 14
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 15
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 16
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 17
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 18
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 19
drwxrwxr-x 2 ubuntu ubuntu 4096 Mar  1 17:59 1a

find .  -type f -print
./19/5546b01347f02255977ef992710bf4
./19/8711e4b92c46cfd8e9418dd91041cc
./19/466c730334df723d3465aa99fed6ac
./0c/effb04d6cd879f2e0604c8c8207185
./0c/197fc28ff4e93b1938079071f5c18e
./0c/52a2c9b9748fd7a68d9038af1d86b4
./0c/7e9ee215d6d46a9ccc47be1bdf6bc1
./1a/1c04d62e4c696b345e8e0d09051352
./1a/625fc1fb5f8be0e164d49737bea64f
./14/91201f10f6099b04a1d192b26d7efd
./14/05ae23d8ccaa0c0f7aa3fad5b4e86e
./14/fcb3f6329aa6cccfa556152a64c4d8
./14/1cddf5d9c9785a09a66cb9c4920e8a
./14/dfcbc3049e33f31db97f7a1286034b
./14/3912a953b484686e0b330545e546d5
./0b/13cbd0734793ec0734a13939f408cc
./0b/c6440470bf79b6daba51853df63e04
./0b/a4233585b69eb076339cd2f994dd8d
./0b/552be7e27cf653050f57f3b39cc7c3
./0b/58980fae3db6a598d5fbc482e04a7c
./0b/e46ddc5cdf02c5f42a2233df874914
./0b/0acc0bed862108b54c8cfde039bddc
./0b/c0ea78289fecfb0122ce7a33b749ad
./15/525092ad0e0598b95b874c9660bf6c
./15/0a0cf635ee42341540a5a5f3b3f3fd
./15/8de78286aa90054560ae70ec0b7d17
./15/b9763875d50a057f7c3ce478d469e2
./16/02328d5d8aa2ddfa2074524827edf1
./16/000da0d61e881bc25de4a4e78be2da
./16/4569e28d5aa41203a30f8687bd61c4
./16/55bc8f3787b8b77243931c0892ed5a
./16/f8ac2f9f6d945316502e6c66d8a95c
./16/5c1a80e7968588be64a2f81bc51e7c
./16/c972bd55baa0f006f85c7499e85251
./16/c62c7d30b060bb174986a31a44c6d9
./16/22e3a74a16f6b1dd9c60671992b96a
./01/7b6b252b4d4da36b94210bde952039
./01/343bf23cd40defbd353d4243a6749c
./01/54c36fb6c048ec147a9fcdc43313c3
./01/114a2dd13b1f12cdaf401eec5c910b
./02/815e23a4c15d05d50462793e895d16
./02/0ecb8e355c0999f19c0dbd2ae50756
./02/a738b5fd58032943596a5d46f55051
./02/247a72e4e96f7427abbb8f338e567b
./02/be5b4d241e530028625e1eeabad756
./02/7d999a8ca2deaafb3f41b7215a7219
./02/7878674cd8183b87ebbd3bee9b87d3
./02/dbe1de6d7f32430c042bf1bb9a7cc9
./02/f8994795acff36c199f2735f3ec810
./02/547b3d5149533e3a1bc950a50badaf
./02/cb171b0c6214ba416e827da0fcbb0c
./18/4099262aa9bd2419eaeaefcae9b1f1
./18/672158e507e204dbfbbe8240859fb9
./18/e2cb70d818250b30623f5e213304e8
./07/773541ca834a31fb1658c3bca4474b
./07/b930da26e4a06dcf8c9a0faff57be1
./07/9b6c5a8ceeceb97ce65e7f701889f9
./07/25593fe43ac634d2875e06e020d07e
./07/287e866f93c1861355172bf79846ed
./07/b26fea58ef2b4c2ed509af34af1175
./0f/5fb6608f02254a1e5a764a3756a5de
./0f/082a803ca112c65cdfe3604324dc1f
./0f/d43c43a0aca790b98b5455283446e7
./0f/db66b514d8a5ac4f0a1072fc09ce96
./0f/78328745bf44ce8c8a33f23025cfa6
./03/1fdb57b933d717887c8bf9882cfc04
./03/b28697e0f406d477813138bb91348a
./03/40b418506f46f8632793f29470f02a
./03/a73284fcd5bffa1b519cd8dd1d7aa7
./03/f6866d73217c6ce3e877021da20c31
./03/56e3359965e80ddffd9704f610e50e
./03/1f818afa2b3e17826b263ff92eb221
./03/e2a3ae3f5d481af9db23c3dad11ff4
./06/32c731ba653b17f102bc42e34c1ae3
./06/e7fe6763115edf5774928b04687207
./06/09230f16c756363558e005b0d195d8
./06/d386be1323174a10a62fea3355a978
./06/c681edccd6d70c3c8eb7384b084c01
./05/4f238677bd3e3fd49954922114f8b3
./05/8633b758f634edef065ba20ef6a16c
./05/fbe738e292209c83cff1359913f1ec
./05/542db109ae94423d2ae1d0734facc7
./0e/5dedb2528e5d471b033c652cd569e9
./0e/594dad9b459b0fd175573f155ae635
./0e/b9066e87cf11ed186e4fa98220a8d0
./0e/a648a9ac361bef7dd870562bf29736
./0e/79c51c46895489b8270ece93fdef7f
./09/961c5e0e22dea2588011c54d33f2c8
./09/8ddab84239c8a187a2ec26d90414f9
./09/eee53fed61f0a448cbb872457167f1
./09/ba31d0fb5e8dd98c585e271d0b8e79
./09/a80210472a1a4d25fbc3601562fd03
./09/14f0b44fb7fa5b533909535193bc73
./09/e3c8d467f6ed3d97e491ebd28ac025
./09/641e30887dfe586c9c0ff9c9c151ad
./09/0a8ec1471a7cb3b2dbaee60d082d72
./09/279f27a37d2d749553cbebc6aaa17d.dir
./08/ec6eb40ad76b48210aa0e939ae7aa1
./08/ee08009c543d7a6ab4c4e34c54649e
./08/0c831c6626e90d0ec6384b6d62e485
./08/b3a86717995c50fbb32194726e922c
./11/9df30d020f3acb2b41ef046b1e218f
./11/7ba2556dca3904ea8201311386d9a0
./11/e9da270928da83829c8a058f6773f1
./11/b51f694020c30b30e168a87a26ee19
./11/9ac65734c7075caf375c273fe5412c
./10/d4aeec639c0af51d566d710f18c45c
./10/a14a8b317a34784e5b3e62c3fa387a
./10/12330624f64dd52697069e469f71c2
./10/90a663e7fa3967a3caaface7d32aea
./10/f62d38c4de444d285f57781caad743
./12/7c8e230cdfed7a934bb4a6fcd12438
./12/41ab5231a101a08fcf38cfb5f4c7d7
./12/76ff6586e210949efd835145b966a2
./12/dca0a2d42b2632a5ed681180197656
./12/6a47b3fe34fee0f0a601367c765779
./12/66bc70d039def454e00a90ed8e92d1
./12/3a9fff8d6ac034373a9c934c5c8c1c
./0d/9dbb25dde6e857c5e1bbe549889881
./0d/b5d44fa54f3df2ff404c53e68fcf85
./0d/2155078977b9397ebfc541983aa03c
./0d/63cb93b52ac38bf105c0c92abc9c13
./0d/2fbac550c3f8df986df76b2f289b01
./0d/25b271f196a66926ebf77461f1e7f8
./0d/503feee2b0bd091b4bfb54a0de0403
./17/9c3ada7fb7464df4fdff09545a596e
./17/d488c5a5a7a6aa84b6b80f92293d94
./13/56a888a32b8e693091a325334c90e5
./13/5e2139fd4dc6153950861c1154ca61
./13/a4dd8a6c2c37686c763b487f5b89f9
./13/bfe0bc6efe7932eab7b31b0711c464
./13/115283108d66db5cd5674dfc473968
./13/c9e93014c062a2c959a5d69eb2727a
./13/11040b11cfaa27056159481e261f70
./04/312f49a9710ed396e583a997a64ca2
./04/c034ca69c33c2901a007d26656bac6
./04/6e0d65655adaf153fcba204453df42
./04/12550fa46867945f748dda7970abe8
./00/ae7b052f208fc16af326a2027d5e9d
./00/0878611a40e38c21ce0b4b22cb80a6
./0a/2592d9e0d65b5ddeceb8b05d53c6a3
./0a/8fca7e09689f0f20ef696a4e650796
./0a/9772b910b3b312881a8d1670f7f4e8
./0a/389e553410f31a7769e4d3e3d5a508
./0a/2548960e959769992cd82f3c4145e1

It looks suspiciously like it is only fetching the first page of files?

Could it be related to list_objects_v2 support in ceph http://docs.ceph.com/docs/mimic/radosgw/s3/python/#listing-a-bucket-s-content

https://github.com/iterative/dvc/blob/dc5a676306d94d46c65fbd4d7b04242a5a64d528/dvc/remote/s3.py#L147

bwalsh picture bwalsh on 2 Mar 2019

Wrote a quick test to confirm:

    s3 = boto3.client('s3',
                      endpoint_url=credentials['endpoint_url'],
                      aws_access_key_id=credentials['access_key'],
                      aws_secret_access_key=credentials['secret_key'])

    response = s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix)
    for item in response['Contents']:
        print(item['Key'])

# retrieves 1000 items (1 page)

./list.py  | awk -F"/" '{print $1 "/" $2}' | sort | uniq
dvc/00
dvc/01
dvc/02
dvc/03
dvc/04
dvc/05
dvc/06
dvc/07
dvc/08
dvc/09
dvc/0a
dvc/0b
dvc/0c
dvc/0d
dvc/0e
dvc/0f
dvc/10
dvc/11
dvc/12
dvc/13
dvc/14
dvc/15
dvc/16
dvc/17
dvc/18
dvc/19
dvc/1a

# same as the results of fetch

hypothesis: only fetching first page

https://github.com/iterative/dvc/blob/9528ad6a1dbe205644431bfb0e02b1e2ae8449bb/dvc/remote/s3.py#L221

Confirmed 

# always None
print(response.get("NextContinuationToken", None))
>>> None
print(response.keys())
>>> dict_keys(['ResponseMetadata', 'IsTruncated', 'Contents', 'Name', 'Prefix', 'MaxKeys', 'EncodingType'])

Could we use list_objects instead ?

    paginator = s3.get_paginator('list_objects')
    for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix):
        for item in page['Contents']:
            print(item['Key'])

# returns all the items in the bucket
bwalsh picture bwalsh on 2 Mar 2019
❤1

@bwalsh Wow, amazing investigation! Is there a way to detect that it is ceph that we are connecting to and not a regular aws s3? If there is, we could use list_objects for ceph specifically and list_objects_v2(which the recommmended one) for everything else. What do you think?

efiop picture efiop on 2 Mar 2019

Looks like the ceph team have an open issue on it. https://tracker.ceph.com/issues/23908 - but no updates. :-(

Poking around, I'm not seeing any api in boto3 that returns anything like "level of s3 compatibility".

This will work, but has the expense of an extra api call.

    s3 = boto3.client('s3',
                      endpoint_url=credentials['endpoint_url'],
                      aws_access_key_id=credentials['access_key'],
                      aws_secret_access_key=credentials['secret_key'])

    list_objects_api = 'list_objects_v2'
    probe_s3_support = s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix)
    if 'NextContinuationToken' not in probe_s3_support:
        list_objects_api = 'list_objects'

    paginator = s3.get_paginator(list_objects_api)
    for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix):
        for item in page['Contents']:
            print(item['Key'])
bwalsh picture bwalsh on 2 Mar 2019

@bwalsh Nice! I can confirm that NextContinuationToken is not present at all in the response keys when working with a regular aws s3, so we could definitely use that workaround (will need to definitely add a comment about it though ). Would you like to submit a patch for it? 馃檪

efiop picture efiop on 3 Mar 2019

@bwalsh Ah, wait. That is actually the problem, we'll get two calls(list_objects_v2 and then list_objects) for single-page responses from regular aws s3, since there will be no NextContinuationToken at all. Sorry, missed it yesterday :slightly_smiling_face: That is not ideal, for sure, but we could live with it, especially since it only affects single-page responses, which are quite fast anyway. That said, I still don't quite like the idea of fixing it on our side, since it is ceph's fault for not supporting an API standard properly. Maybe there is some way to detect that it is ceph that we are connecting to after all, so we could only use that workaround for it and not affect other scenarios? Also, could you please ping ceph about this issue, maybe they'll prioritize it or something? Or at least they would be notified that there is a user that needs that feature implemented.

efiop picture efiop on 3 Mar 2019

Sorry for the delay, I can take a stab at this.Is there a mechanism or pattern I can follow to add a parameter ( or ENV variable )

bwalsh picture bwalsh on 7 Mar 2019

@bwalsh Something like dvc remote modify mys3 use_list_objects true? That would be even better. Take a look at AWS_ENDPOINT_URL here https://github.com/iterative/dvc/blob/master/dvc/config.py#L195 and https://github.com/iterative/dvc/blob/master/dvc/remote/s3.py#L56 and add that new config option. It is pretty much all you'll need. Please feel free to ping us if you need any help. Thanks a lot for looking into it!

efiop picture efiop on 8 Mar 2019

@efiop Created a PR. Unclear how to extend test work. Will experiment with this on our system today.

bwalsh picture bwalsh on 11 Mar 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ghost picture ghost  路  3Comments
mfrata picture mfrata  路  3Comments
dnabanita7 picture dnabanita7  路  3Comments
TezRomacH picture TezRomacH  路  3Comments
analystanand picture analystanand  路  3Comments