Dvc: "dvc pull" fails to find files in a pushed directories subfolder on Windows

[sremm]

Setup info
Ubuntu:

• DVC version: 0.41.3
• Python version: 3.6.7
• Platform: Linux-4.15.0-51-generic-x86_64-with-debian-buster-sid
• Installed with pip

Windows:

• DVC version: 0.41.3
• Python version: 3.6.7
• Platform: Windows-10-10.0.16299-SP0
• Installed with pip

After pushing a folder from Ubuntu, which included some model files and one subfolder with some variable files, I tried to pull it from Windows. It managed to get all the files in the initial folder, but ran into an error with the files in the one subfolder.

dvc pull very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-others.dvc
c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\git\repo\base.py:128: UserWarning: The use of environment variables in paths is deprecated
for security reasons and may be removed in the future!!
  "\nfor security reasons and may be removed in the future!!")
Preparing to download data from 'F:\localpath\dvc-q99hd-cache'
Preparing to collect status from F:\localpath\dvc-q99hd-cache
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
[####                          ] 16% very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-others
[#########                     ] 33% very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-others
[###############               ] 50% very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-others
[###################           ] 66% very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-others
ERROR: failed to pull data from the cloud - [Errno 2] No such file or directory: 'F:\\localpath\\localpath\\localpath\\localpath\\localpath\\localpath\\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-others\\variables\\variables.data-00000-of-00001'

After a brief converation with Ruslan over email, he suspect the following:

I suspect it is most likey the filename that is to blame. It is either not handled by windows at all, or somehow butchered or mishandled by dvc, so I need to know more about that filename.

[sremm]

As Ruslan also pointed out a minimal working example would be useful. I would have to come back to providing as soon as I have time again.

[Suor]

@sremm can you please post the full folder name or even better the full output of dvc pull -v command?

Also, if any folder or file names contain \ then it works on Unix-like systems and breaks on Windows when you checkout/pull. This is a known issue - #2059.

[efiop]

@Suor No \ or /, I've asked about that in the mailing list.

[Suor]

@efiop ok, good, I will check whether semicolons are the culprit then - this might be the case if we don't quote filenames properly somewhere.

[sremm]

@Suor Here is the full verbose output, with all special characters preserved in the path, there is also a % which is probably a bit irregular, but that has not been a problem for any other files.
(tf-keras) F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder>dvc pull "very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc.dvc" -v c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\git\repo\base.py:128: UserWarning: The use of environment variables in paths is deprecated for security reasons and may be removed in the future!! "\nfor security reasons and may be removed in the future!!") DEBUG: Trying to spawn '['c:\\users\\srg\\appdata\\local\\continuum\\anaconda3\\envs\\tf-keras\\python.exe', '-m', 'dvc', 'daemon', '-q', 'updater']' DEBUG: Spawned '['c:\\users\\srg\\appdata\\local\\continuum\\anaconda3\\envs\\tf-keras\\python.exe', '-m', 'dvc', 'daemon', '-q', 'updater']' DEBUG: PRAGMA user_version; DEBUG: fetched: [(3,)] DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL) DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER) DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL) DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info) DEBUG: PRAGMA user_version = 3; DEBUG: Path F:\localpath\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir inode 2866823111190838224 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2866823111190838224 DEBUG: fetched: [('1560335121671053312', '481', '3b2b5789fe406ca3a4f3a370f2c75278', '1560766590630647296')] DEBUG: UPDATE state SET timestamp = "1560766726938328064" WHERE inode = 2866823111190838224 DEBUG: cache '..\..\..\..\..\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir' expected '3b2b5789fe406ca3a4f3a370f2c75278.dir' actual '3b2b5789fe406ca3a4f3a370f2c75278' DEBUG: Path F:\localpath\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir inode 2866823111190838224 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2866823111190838224 DEBUG: fetched: [('1560335121671053312', '481', '3b2b5789fe406ca3a4f3a370f2c75278', '1560766726938328064')] DEBUG: UPDATE state SET timestamp = "1560766726942330112" WHERE inode = 2866823111190838224 DEBUG: cache '..\..\..\..\..\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir' expected '3b2b5789fe406ca3a4f3a370f2c75278.dir' actual '3b2b5789fe406ca3a4f3a370f2c75278' DEBUG: Downloading 'F:\localpath\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir' to 'C:\Users\srg\AppData\Local\Temp\tmp1dngmi_v' Preparing to download data from 'F:\localpath\dvc-q99hd-cache' Preparing to collect status from F:\localpath\dvc-q99hd-cache [######### ] 30% Collecting information DEBUG: Path F:\localpath\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir inode 2866823111190838224 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2866823111190838224 DEBUG: fetched: [('1560335121671053312', '481', '3b2b5789fe406ca3a4f3a370f2c75278', '1560766726942330112')] DEBUG: UPDATE state SET timestamp = "1560766726961319424" WHERE inode = 2866823111190838224 DEBUG: cache '..\..\..\..\..\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir' expected '3b2b5789fe406ca3a4f3a370f2c75278.dir' actual '3b2b5789fe406ca3a4f3a370f2c75278' DEBUG: Path F:\localpath\dvc-local-cache\a5\39af0a0b8a4a8f7e371e735d32012e inode 2949476153769092464 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2949476153769092464 DEBUG: fetched: [('1560335747630845184', '1269', 'a539af0a0b8a4a8f7e371e735d32012e', '1560766590666496512')] DEBUG: UPDATE state SET timestamp = "1560766726966317056" WHERE inode = 2949476153769092464 DEBUG: cache '..\..\..\..\..\dvc-local-cache\a5\39af0a0b8a4a8f7e371e735d32012e' expected 'a539af0a0b8a4a8f7e371e735d32012e' actual 'a539af0a0b8a4a8f7e371e735d32012e' DEBUG: Path F:\localpath\dvc-local-cache\b2\7ca6ff20930db88048b0fd7dbf0784 inode 1874976821782326377 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=1874976821782326377 DEBUG: fetched: [('1560335747938848512', '212413661', 'b27ca6ff20930db88048b0fd7dbf0784', '1560766590676697088')] DEBUG: UPDATE state SET timestamp = "1560766726975330560" WHERE inode = 1874976821782326377 DEBUG: cache '..\..\..\..\..\dvc-local-cache\b2\7ca6ff20930db88048b0fd7dbf0784' expected 'b27ca6ff20930db88048b0fd7dbf0784' actual 'b27ca6ff20930db88048b0fd7dbf0784' DEBUG: Path F:\localpath\dvc-local-cache\a3\252ceaee23ab5de79654464c28b354 inode 2949476153774917709 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2949476153774917709 DEBUG: fetched: [('1560335747631844096', '598381', 'a3252ceaee23ab5de79654464c28b354', '1560766590684696832')] DEBUG: UPDATE state SET timestamp = "1560766726983242240" WHERE inode = 2949476153774917709 DEBUG: cache '..\..\..\..\..\dvc-local-cache\a3\252ceaee23ab5de79654464c28b354' expected 'a3252ceaee23ab5de79654464c28b354' actual 'a3252ceaee23ab5de79654464c28b354' DEBUG: Path F:\localpath\dvc-local-cache\fc\299cbdf94305d1565cce18001eafe7 inode 2784169151568729663 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2784169151568729663 DEBUG: fetched: [('1560335747628843264', '1075', 'fc299cbdf94305d1565cce18001eafe7', '1560766590697049856')] DEBUG: UPDATE state SET timestamp = "1560766726994292224" WHERE inode = 2784169151568729663 DEBUG: cache '..\..\..\..\..\dvc-local-cache\fc\299cbdf94305d1565cce18001eafe7' expected 'fc299cbdf94305d1565cce18001eafe7' actual 'fc299cbdf94305d1565cce18001eafe7' DEBUG: Path F:\localpath\dvc-local-cache\9f\298981c4d59131e5562be8b2016a32 inode 2784169151569894712 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2784169151569894712 DEBUG: fetched: [('1560335747972844032', '212311928', '9f298981c4d59131e5562be8b2016a32', '1560766590651392000')] DEBUG: UPDATE state SET timestamp = "1560766726999293696" WHERE inode = 2784169151569894712 DEBUG: cache '..\..\..\..\..\dvc-local-cache\9f\298981c4d59131e5562be8b2016a32' expected '9f298981c4d59131e5562be8b2016a32' actual '9f298981c4d59131e5562be8b2016a32' DEBUG: Path F:\localpath\dvc-local-cache\2c\e1264d2e1e033272c660b0ee95e996 inode 2866823110791226417 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2866823110791226417 DEBUG: fetched: [('1560335747633842176', '7804', '2ce1264d2e1e033272c660b0ee95e996', '1560766590659496192')] DEBUG: UPDATE state SET timestamp = "1560766727008297984" WHERE inode = 2866823110791226417 DEBUG: cache '..\..\..\..\..\dvc-local-cache\2c\e1264d2e1e033272c660b0ee95e996' expected '2ce1264d2e1e033272c660b0ee95e996' actual '2ce1264d2e1e033272c660b0ee95e996' [##############################] 100% Collecting information [##############################] 100% Analysing status. DEBUG: SELECT count from state_info WHERE rowid=1 DEBUG: fetched: [(83,)] DEBUG: UPDATE state_info SET count = 83 WHERE rowid = 1 DEBUG: PRAGMA user_version; DEBUG: fetched: [(3,)] DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL) DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER) DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL) DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info) DEBUG: PRAGMA user_version = 3; DEBUG: SELECT * FROM link_state DEBUG: checking if 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc'('ordereddict([('md5', '3b2b5789fe406ca3a4f3a370f2c75278.dir')])') has changed. DEBUG: 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc' doesn't exist. DEBUG: Path F:\localpath\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir inode 2866823111190838224 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2866823111190838224 DEBUG: fetched: [('1560335121671053312', '481', '3b2b5789fe406ca3a4f3a370f2c75278', '1560766726961319424')] DEBUG: UPDATE state SET timestamp = "1560766729647678208" WHERE inode = 2866823111190838224 DEBUG: cache '..\..\..\..\..\dvc-local-cache\3b\2b5789fe406ca3a4f3a370f2c75278.dir' expected '3b2b5789fe406ca3a4f3a370f2c75278.dir' actual '3b2b5789fe406ca3a4f3a370f2c75278' DEBUG: Path F:\localpath\dvc-local-cache\a5\39af0a0b8a4a8f7e371e735d32012e inode 2949476153769092464 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2949476153769092464 DEBUG: fetched: [('1560335747630845184', '1269', 'a539af0a0b8a4a8f7e371e735d32012e', '1560766726966317056')] DEBUG: UPDATE state SET timestamp = "1560766729655697408" WHERE inode = 2949476153769092464 DEBUG: cache '..\..\..\..\..\dvc-local-cache\a5\39af0a0b8a4a8f7e371e735d32012e' expected 'a539af0a0b8a4a8f7e371e735d32012e' actual 'a539af0a0b8a4a8f7e371e735d32012e' DEBUG: Path F:\localpath\dvc-local-cache\b2\7ca6ff20930db88048b0fd7dbf0784 inode 1874976821782326377 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=1874976821782326377 DEBUG: fetched: [('1560335747938848512', '212413661', 'b27ca6ff20930db88048b0fd7dbf0784', '1560766726975330560')] DEBUG: UPDATE state SET timestamp = "1560766729658679296" WHERE inode = 1874976821782326377 DEBUG: cache '..\..\..\..\..\dvc-local-cache\b2\7ca6ff20930db88048b0fd7dbf0784' expected 'b27ca6ff20930db88048b0fd7dbf0784' actual 'b27ca6ff20930db88048b0fd7dbf0784' DEBUG: Path F:\localpath\dvc-local-cache\a3\252ceaee23ab5de79654464c28b354 inode 2949476153774917709 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2949476153774917709 DEBUG: fetched: [('1560335747631844096', '598381', 'a3252ceaee23ab5de79654464c28b354', '1560766726983242240')] DEBUG: UPDATE state SET timestamp = "1560766729664680192" WHERE inode = 2949476153774917709 DEBUG: cache '..\..\..\..\..\dvc-local-cache\a3\252ceaee23ab5de79654464c28b354' expected 'a3252ceaee23ab5de79654464c28b354' actual 'a3252ceaee23ab5de79654464c28b354' DEBUG: Path F:\localpath\dvc-local-cache\fc\299cbdf94305d1565cce18001eafe7 inode 2784169151568729663 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2784169151568729663 DEBUG: fetched: [('1560335747628843264', '1075', 'fc299cbdf94305d1565cce18001eafe7', '1560766726994292224')] DEBUG: UPDATE state SET timestamp = "1560766729667679232" WHERE inode = 2784169151568729663 DEBUG: cache '..\..\..\..\..\dvc-local-cache\fc\299cbdf94305d1565cce18001eafe7' expected 'fc299cbdf94305d1565cce18001eafe7' actual 'fc299cbdf94305d1565cce18001eafe7' DEBUG: Path F:\localpath\dvc-local-cache\9f\298981c4d59131e5562be8b2016a32 inode 2784169151569894712 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2784169151569894712 DEBUG: fetched: [('1560335747972844032', '212311928', '9f298981c4d59131e5562be8b2016a32', '1560766726999293696')] DEBUG: UPDATE state SET timestamp = "1560766729674684672" WHERE inode = 2784169151569894712 DEBUG: cache '..\..\..\..\..\dvc-local-cache\9f\298981c4d59131e5562be8b2016a32' expected '9f298981c4d59131e5562be8b2016a32' actual '9f298981c4d59131e5562be8b2016a32' DEBUG: Path F:\localpath\dvc-local-cache\2c\e1264d2e1e033272c660b0ee95e996 inode 2866823110791226417 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2866823110791226417 DEBUG: fetched: [('1560335747633842176', '7804', '2ce1264d2e1e033272c660b0ee95e996', '1560766727008297984')] DEBUG: UPDATE state SET timestamp = "1560766729677687296" WHERE inode = 2866823110791226417 DEBUG: cache '..\..\..\..\..\dvc-local-cache\2c\e1264d2e1e033272c660b0ee95e996' expected '2ce1264d2e1e033272c660b0ee95e996' actual '2ce1264d2e1e033272c660b0ee95e996' DEBUG: Checking out 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc' with cache '3b2b5789fe406ca3a4f3a370f2c75278.dir'. DEBUG: Linking directory 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc'. DEBUG: checking if 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\dataset_info.json'('{'md5': 'a539af0a0b8a4a8f7e371e735d32012e'}') has changed. DEBUG: 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\dataset_info.json' doesn't exist. DEBUG: Cache type 'reflink' is not supported: reflink is not supported DEBUG: Created 'copy': F:\localpath\dvc-local-cache\a5\39af0a0b8a4a8f7e371e735d32012e -> F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\dataset_info.json DEBUG: Path F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\dataset_info.json inode 1709669208767743758 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=1709669208767743758 DEBUG: fetched: [] DEBUG: INSERT INTO state(inode, mtime, size, md5, timestamp) VALUES (1709669208767743758, "1560766729704231680", "1269", "a539af0a0b8a4a8f7e371e735d32012e", "1560766729708261120") DEBUG: checking if 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\frozen.pb'('{'md5': 'b27ca6ff20930db88048b0fd7dbf0784'}') has changed. DEBUG: 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\frozen.pb' doesn't exist. DEBUG: Created 'copy': F:\localpath\dvc-local-cache\b2\7ca6ff20930db88048b0fd7dbf0784 -> F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\frozen.pb DEBUG: Path F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\frozen.pb inode 2949476153786568199 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2949476153786568199 DEBUG: fetched: [] DEBUG: INSERT INTO state(inode, mtime, size, md5, timestamp) VALUES (2949476153786568199, "1560766731287842560", "212413661", "b27ca6ff20930db88048b0fd7dbf0784", "1560766731291842816") DEBUG: checking if 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\saved_model.pb'('{'md5': 'a3252ceaee23ab5de79654464c28b354'}') has changed. DEBUG: 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\saved_model.pb' doesn't exist. DEBUG: Created 'copy': F:\localpath\dvc-local-cache\a3\252ceaee23ab5de79654464c28b354 -> F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\saved_model.pb DEBUG: Path F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\saved_model.pb inode 2949476153787733248 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2949476153787733248 DEBUG: fetched: [] DEBUG: INSERT INTO state(inode, mtime, size, md5, timestamp) VALUES (2949476153787733248, "1560766731319860736", "598381", "a3252ceaee23ab5de79654464c28b354", "1560766731325842176") DEBUG: checking if 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\train_config.json'('{'md5': 'fc299cbdf94305d1565cce18001eafe7'}') has changed. DEBUG: 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\train_config.json' doesn't exist. DEBUG: Created 'copy': F:\localpath\dvc-local-cache\fc\299cbdf94305d1565cce18001eafe7 -> F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\train_config.json DEBUG: Path F:\localpath\DataworkerRepo\local%20pathwitperc\path-with-dash\folder\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\train_config.json inode 2618862266068995094 DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=2618862266068995094 DEBUG: fetched: [] DEBUG: INSERT INTO state(inode, mtime, size, md5, timestamp) VALUES (2618862266068995094, "1560766731350848768", "1075", "fc299cbdf94305d1565cce18001eafe7", "1560766731354842368") DEBUG: checking if 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\variables\variables.data-00000-of-00001'('{'md5': '9f298981c4d59131e5562be8b2016a32'}') has changed. DEBUG: 'very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\variables\variables.data-00000-of-00001' doesn't exist. DEBUG: SELECT count from state_info WHERE rowid=1 DEBUG: fetched: [(83,)] DEBUG: UPDATE state_info SET count = 87 WHERE rowid = 1 ERROR: failed to pull data from the cloud - [Errno 2] No such file or directory: 'F:\\localpath\\DataworkerRepo\\local%20pathwitperc\\tf-pose-estimation\\models\\graph\\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\\variables\\variables.data-00000-of-00001' ------------------------------------------------------------ Traceback (most recent call last): File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\command\data_sync.py", line 46, in do_run recursive=self.args.recursive, File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\repo\pull.py", line 27, in pull target=target, with_deps=with_deps, force=force, recursive=recursive File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\repo\checkout.py", line 54, in checkout stage.checkout(force=force, progress_callback=progress_callback) File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\stage.py", line 851, in checkout force=force, tag=self.tag, progress_callback=progress_callback File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\output\base.py", line 288, in checkout progress_callback=progress_callback, File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\remote\base.py", line 688, in checkout return self._checkout(path_info, checksum, force, progress_callback) File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\remote\base.py", line 698, in _checkout path_info, checksum, force, progress_callback=progress_callback File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\remote\base.py", line 630, in _checkout_dir self.link(entry_cache_info, entry_info) File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\remote\local\slow_link_detection.py", line 59, in call return decorated(remote_local, *args, **kwargs) File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\remote\local\slow_link_detection.py", line 26, in __call__ result = self.method(*args, **kwargs) File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\site-packages\dvc\remote\local\__init__.py", line 148, in link self.CACHE_TYPE_MAP[self.cache_types[0]](cache, path) File "c:\users\srg\appdata\local\continuum\anaconda3\envs\tf-keras\lib\shutil.py", line 121, in copyfile with open(dst, 'wb') as fdst: FileNotFoundError: [Errno 2] No such file or directory: 'F:\\localpath\\DataworkerRepo\\local%20pathwitperc\\tf-pose-estimation\\models\\graph\\very-long_folder-name_with;semicolons;and;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;fillerXxxYyyZzz;_and-some-dash.dvc\\variables\\variables.data-00000-of-00001' ------------------------------------------------------------ Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!

[sremm]

@Suor I remembered that there is a maximum path length limitation in windows https://docs.microsoft.com/en-us/windows/desktop/FileIO/naming-a-file#maximum-path-length-limitation

And since the full path was more than 260 characters I tested with the same files but shorter folder name. As it turns out there was no issues pulling with the new full path less than 260 characters in length.

And having semicolons in a shorter name was not an issue either. So I could either rename the issue if there would be reason to find a workaround for long path names in windows and leave it open, or close the issue.

[Suor]

@sremm thanks for getting back and thank you very much for the MAX_PATH clue. Looks like this is only API limit not filesystem one and could be bypassed by using "\\?\" path prefix on Windows.

@efiop should we look into using that prefix? This can significantly complicate our code I think.

[efiop]

@Suor If using "\\?\" is all we need, wouldn't it be pretty easy to fix it in PathInfo.fspath?

[Suor]

@efiop not all our ops use PathInfo. Any path longer than 260 chars will need to be prefixed, e.g. dvc-files, gitignore, anything you pass to system functions.

[efiop]

@Suor Touche. Also, judging by googling around, it seems like that limitation should be handled by python itself and by windows (e.g. here is a guide how to enable long paths), so I don't think we should be handling this, especially with that amount of hustle. @sremm Could you please try enabling long paths on your machine and seeing if that helps? We would at least add it our docs, so we could point people to it.

[sremm]

@efiop I enabled long paths following the guide you sent and then the pull command worked, all the files were pulled without error

[efiop]

We will be sure to add it to our user guide: https://github.com/iterative/dvc.org/issues/395 . Thanks a lot for the feedback @sremm ! :)