Dvc: Problems with dvc push

[LarryThermo]

Bug Report

Please provide information about your setup

Output of dvc version:

> dvc version
DVC version: 1.1.10
Python version: 3.7.5
Platform: Windows-10-10.0.17763-SP0
Binary: True
Package: exe
Supported remotes: azure, gdrive, gs, hdfs, http, https, s3, ssh, oss
Cache: reflink - not supported, hardlink - supported, symlink - not supported
Filesystem type (cache directory): ('NTFS', 'C:\\')
Repo: dvc, git
Filesystem type (workspace): ('NTFS', 'C:\\')

Additional Information (if any):
I am having problems with dvc push no longer working after I upgraded to the latest dvc.

My steps in a Windows Powershell window (if it appears a little bit disjointed its because I copied and pasted the output verbatim):

dvc init

You can now commit the changes to git.

+---------------------------------------------------------------------+
| |
| DVC has enabled anonymous aggregate usage analytics. |
| Read the analytics documentation (and how to opt-out) here: |
| https://dvc.org/doc/user-guide/analytics |
| |
+---------------------------------------------------------------------+

What's next?

• Check out the documentation: https://dvc.org/doc
• Get help and share ideas: https://dvc.org/chat
• Star us on GitHub: https://github.com/iterative/dvc
  > dvc remote add -d myremote s3://analysis-training-sets/masu-autofocus-models2
  Setting 'myremote' as a default remote.
  > dvc add "model-all-00038-0.70911.pt"
  100% Add|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:02, 2.99s/file]

To track the changes with git, run:

    git add model-all-00038-0.70911.ptj.dvc

dvc push
ERROR: unexpected error - unable to open database file

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

[efiop]

Hi @LarryThermo . Could you please show full dvc push -v log?

[LarryThermo]

Thanks for the response Ruslan.

MasuAutoFocusTraining.Simple.ManufacturingVariability.Beads> dvc push -v log
2020-07-20 09:12:12,240 DEBUG: Trying to spawn '['C:\Program Files (x86)\Data Version Control\dvc.exe', 'daemon', '-q', 'updater']'
2020-07-20 09:12:18,739 DEBUG: Spawned '['C:\Program Files (x86)\Data Version Control\dvc.exe', 'daemon', '-q', 'updater']'
2020-07-20 09:12:18,748 DEBUG: fetched: [(3,)]
2020-07-20 09:12:18,753 DEBUG: Checking if stage 'log' is in 'dvc.yaml'
2020-07-20 09:12:18,771 DEBUG: Assuming 'log' to be a stage inside 'dvc.yaml'
2020-07-20 09:12:18,772 DEBUG: fetched: [(6,)]

2020-07-20 09:12:18,789 ERROR: failed to push data to the cloud - 'log' does not exist as an output or a stage name in 'dvc.yaml': 'dvc.yaml' does not exist.

Traceback (most recent call last):
File "dvc\repo__init__.py", line 318, in collect_granular
File "dvc\repo__init__.py", line 255, in collect
File "dvc\repo__init__.py", line 215, in get_stages
File "dvc\repo__init__.py", line 207, in get_stage
File "dvc\dvcfile.py", line 210, in stages
File "dvc\dvcfile.py", line 92, in _load
dvc.stage.exceptions.StageFileDoesNotExistError: 'dvc.yaml' does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "dvc\commanddata_sync.py", line 59, in run
File "dvc\repo__init__.py", line 36, in wrapper
File "dvc\repo\push.py", line 32, in push
File "dvc\repo__init__.py", line 377, in used_cache
File "dvc\repo__init__.py", line 373, in
File "dvc\repo__init__.py", line 325, in collect_granular

dvc.exceptions.NoOutputOrStageError: 'log' does not exist as an output or a stage name in 'dvc.yaml'

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

MasuAutoFocusTraining.Simple.ManufacturingVariability.Beads> dvc push -v
2020-07-20 09:13:39,123 DEBUG: Trying to spawn '['C:\Program Files (x86)\Data Version Control\dvc.exe', 'daemon', '-q', 'updater']'
2020-07-20 09:13:45,665 DEBUG: Spawned '['C:\Program Files (x86)\Data Version Control\dvc.exe', 'daemon', '-q', 'updater']'
2020-07-20 09:13:45,671 DEBUG: fetched: [(3,)]
2020-07-20 09:13:45,941 DEBUG: fetched: [(6,)]

2020-07-20 09:13:45,953 ERROR: unexpected error - unable to open database file

Traceback (most recent call last):
File "dvc\main.py", line 53, in main
File "dvc\commanddata_sync.py", line 59, in run
File "dvc\repo__init__.py", line 36, in wrapper
File "dvc\repo\push.py", line 35, in push
File "dvcdata_cloud.py", line 66, in push
File "dvc\remote\base.py", line 78, in wrapper
File "dvc\remote\index.py", line 87, in __enter__
File "dvc\remote\index.py", line 153, in load
File "dvc\state.py", line 484, in _connect_sqlite

sqlite3.OperationalError: unable to open database file

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

From: Ruslan Kuprieiev [mailto:[email protected]]
Sent: Friday, July 17, 2020 9:45 PM
To: iterative/dvc [e-mail]
Cc: Rystrom, Larry [e-mail]; Mention [e-mail]
Subject: Re: [iterative/dvc] Problems with dvc push (#4234)

CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to [email protected].

Hi @LarryThermohttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=8RO59mJXM9K4E8utvZpf9QRI_ZZkP47sKP0Irq1v3O8&s=K0tzKPRXcIvKDCy6ip7jePX9j3U2mCgGtIRUAJIUsi8&e= . Could you please show full dvc push -v log?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D660424482&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=8RO59mJXM9K4E8utvZpf9QRI_ZZkP47sKP0Irq1v3O8&s=AkFi6HDS05yhxqtHdyEMC4s8jHl6hGIdamZUEQXRm1A&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HHQV6EFK2IQMASZLK3R4ESDRANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=8RO59mJXM9K4E8utvZpf9QRI_ZZkP47sKP0Irq1v3O8&s=Krc9ZttJGoMCZYqj7-q8LWdRLtSldSvZ92GSFXxp3lQ&e=.

[efiop]

@LarryThermo Thanks. Looks very strange, could you please show the contents of model-all-00038-0.70911.ptj.dvc ?

[LarryThermo]

Hi Ruslan,

I had to regenerate it due to wanting to put it under version control in the interim. Complete sequence of regeneration commands:

dvc add "model-all-00038-0.70911.ptj"
Adding...
ERROR: output 'model-all-00038-0.70911.ptj' is already tracked by SCM (e.g. Git).
You can remove it from Git, then add to DVC.
To stop tracking from Git:
git rm -r --cached 'model-all-00038-0.70911.ptj'
git commit -m "stop tracking model-all-00038-0.70911.ptj"

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

git rm -r --cached 'model-all-00038-0.70911.ptj'
rm 'model-all-00038-0.70911.ptj'
git commit -m "stop tracking model-all-00038-0.70911.ptj"
[Simple.ManufacturingVariability.Beads 9a2bc32] stop tracking model-all-00038-0.70911.ptj
1 file changed, 0 insertions(+), 0 deletions(-)
delete mode 100644 model-all-00038-0.70911.ptj
dvc add "model-all-00038-0.70911.ptj"
100% Add|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:02, 2.88s/file]

To track the changes with git, run:

    git add model-all-00038-0.70911.ptj.dvc .gitignore

dvc push
ERROR: unexpected error - unable to open database file

Contents of model-all-00038-0.70911.ptj.dvc:

outs:

• md5: da3d1f8746a142ba52706dab6c6fed53
  path: model-all-00038-0.70911.ptj

From: Ruslan Kuprieiev [mailto:[email protected]]
Sent: Monday, July 20, 2020 9:27 AM
To: iterative/dvc [e-mail]
Cc: Rystrom, Larry [e-mail]; Mention [e-mail]
Subject: Re: [iterative/dvc] Problems with dvc push (#4234)

CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to [email protected].

@LarryThermohttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=7PLItGQH1dOW24iNE935NzE8Ws7fR4SvUR2iRzElpyA&s=CvKQpVl3y5cxTNLMfNaooUgGDXax4G1HyfxdRwzples&e= Thanks. Looks very strange, could you please show the contents of model-all-00038-0.70911.ptj.dvc ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D661154788&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=7PLItGQH1dOW24iNE935NzE8Ws7fR4SvUR2iRzElpyA&s=_wGn2TYI0ZUsAXP3lEBF9eZY_9XRcNwG_GBNF_dnZUE&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HAYM5UPTP4DNBW73UTR4RV4XANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=7PLItGQH1dOW24iNE935NzE8Ws7fR4SvUR2iRzElpyA&s=17z6C_n-GV1iUaP7rsdR5mImEw8DAqoYeu2TtYvekRY&e=.

[efiop]

@LarryThermo Thanks!

@pmrowla Looks like something is up with the sqlite index db (though the very first log is different and confusing), maybe you have some ideas about it?

[pmrowla]

@LarryThermo can you try deleting the .dvc/tmp/index folder from your repo, and then re-run dvc push -v?

[LarryThermo]

dvc push -v
2020-07-21 09:26:45,576 DEBUG: Trying to spawn '['C:\Program Files (x86)\Data Version Control\dvc.exe', 'daemon', '-q', 'updater']'
2020-07-21 09:26:52,107 DEBUG: Spawned '['C:\Program Files (x86)\Data Version Control\dvc.exe', 'daemon', '-q', 'updater']'
2020-07-21 09:26:52,114 DEBUG: fetched: [(3,)]
2020-07-21 09:26:52,382 DEBUG: fetched: [(6,)]

2020-07-21 09:26:52,391 ERROR: unexpected error - unable to open database file

Traceback (most recent call last):
File "dvc\main.py", line 53, in main
File "dvc\commanddata_sync.py", line 59, in run
File "dvc\repo__init__.py", line 36, in wrapper
File "dvc\repo\push.py", line 35, in push
File "dvcdata_cloud.py", line 66, in push
File "dvc\remote\base.py", line 78, in wrapper
File "dvc\remote\index.py", line 87, in __enter__
File "dvc\remote\index.py", line 153, in load
File "dvc\state.py", line 484, in _connect_sqlite

sqlite3.OperationalError: unable to open database file

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

From: Peter Rowlands (변기호) [mailto:[email protected]]
Sent: Monday, July 20, 2020 5:47 PM
To: iterative/dvc [e-mail]
Cc: Rystrom, Larry [e-mail]; Mention [e-mail]
Subject: Re: [iterative/dvc] Problems with dvc push (#4234)

CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to [email protected].

@LarryThermohttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=G849SKpEPXS7kSOsdNrj8kHDt0HoSbOLb8zNcUG8WAU&e= can you try deleting the .dvc/tmp/index folder from your repo, and then re-run dvc push -v?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D661501707&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=XDP_21UwbanwBtQDv7vWa05wzZABjE1HMYUFbMvsp9A&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HECYF2MC45VS2JLMJDR4TQR5ANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=o3TnaZkqf31DZrYcVPp3cJnQNYCx6p2ZgYz9fbTEAgo&e=.

[LarryThermo]

Actually I might have self-diagnosed the problem. I think the problem has do with a long directory or path name.

The full path to the dvc repository is:
C:\Projects-Git\MasuAutoFocusTraining.Hybrid\MasuAutoFocusTraining.Simple\MasuAutoFocusTraining.Simple.ManufacturingVariability\MasuAutoFocusTraining.Simple.ManufacturingVariability.Beads

If I shorten it to:
C:\Projects-Git\MasuAutoFocusTraining.Hybrid\MasuAutoFocusTraining.Simple\MasuAutoFocusTraining.Simple.ManufacturingVariabilitya

then the push succeeds.

Best,

Larry

From: Peter Rowlands (변기호) [mailto:[email protected]]
Sent: Monday, July 20, 2020 5:47 PM
To: iterative/dvc [e-mail]
Cc: Rystrom, Larry [e-mail]; Mention [e-mail]
Subject: Re: [iterative/dvc] Problems with dvc push (#4234)

CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to [email protected].

@LarryThermohttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_LarryThermo&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=G849SKpEPXS7kSOsdNrj8kHDt0HoSbOLb8zNcUG8WAU&e= can you try deleting the .dvc/tmp/index folder from your repo, and then re-run dvc push -v?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D661501707&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=XDP_21UwbanwBtQDv7vWa05wzZABjE1HMYUFbMvsp9A&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HECYF2MC45VS2JLMJDR4TQR5ANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=mO4VrG4Ij6stdYoKU97iDMGIYfRF9tZpgeFq8bF8OCY&s=o3TnaZkqf31DZrYcVPp3cJnQNYCx6p2ZgYz9fbTEAgo&e=.

[pmrowla]

I don't think there's much we can do about the path length limitation on Windows, but we could at least check for this and give a better error message if we fail to open the index DB.

[LarryThermo]

Yes, it really has nothing to do with dvc and improving the error message is about what you can do.

Please note that I am really enjoying using dvc and thank you.

From: Peter Rowlands (변기호) [mailto:[email protected]]
Sent: Thursday, July 23, 2020 8:15 AM
To: iterative/dvc [e-mail]
Cc: Rystrom, Larry [e-mail]; Mention [e-mail]
Subject: Re: [iterative/dvc] Problems with dvc push (#4234)

CAUTION: This email originated from outside of Thermo Fisher Scientific. If you believe it to be suspicious, report using the Report Phish button in Outlook or send to [email protected].

I don't think there's much we can do about the path length limitation on Windows, but we could at least check for this and give a better error message if we fail to open the index DB.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_iterative_dvc_issues_4234-23issuecomment-2D663065050&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=__9ePWo8DZTHM_AgliBR8tlk_1DVuTImEPDEIbBBZs0&s=pkQjuZ99GVvBcM6eDjdkqJ3MggxINHqzdpebk2UW7TQ&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AESR4HDZCZRBK4QMUNRLGILR5BHV5ANCNFSM4O7BOULQ&d=DwMCaQ&c=q6k2DsTcEGCcCb_WtVSz6hhIl8hvYssy7sH8ZwfbbKU&r=HYF8_GLywj_CyctTtBeHSUbdP3vNJRlUPrSvd7uyd9o&m=__9ePWo8DZTHM_AgliBR8tlk_1DVuTImEPDEIbBBZs0&s=Z6Mmj_INo8iFpc9PE6WPad5lkLV_2RmU9gLsxSNbL9A&e=.

[efiop]

Hm, btw, we use an absolute path when opening the db, so we probably could've avoided this issue if we were to use a relative path instead. This is something that we are constantly talking about in many places in dvc, and is prone to hitting MAX_PATH on other platforms as well. So the best solution will be if we make that switch, but that won't happen in the nearest future :(

As to the error(or warning), a meaningful one will require some sysconf magic (MAX_PATH might not be 260, it might be more or might even be ignored by the system) and will bring us questionable results. Shame sqlite doesn't raise a meaningful exception for this :slightly_frowning_face: So considering this, I think we should just close this issue for now, since there is an easy workaround. If we get more reports - we'll reopen and will probably try to use relpaths at least when dealing with sqlite, but that will likely make the issue arise in a different place :slightly_frowning_face:

If anyone else is running into this issue, please leave a comment so we are aware. Thank you.