Azure-devops-cli-extension: az artifacts universal download fails with Access denied and file not found

[wpschaub]

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name
az artifacts universal download Extension Name: azure-devops. Version: 0.12.0.

Errors:

First error when using verbose logging:

An error occurred while extracting the Universal Packages tooling: [WinError 5] Access is denied: 'C:\\Users\\WS19312\\.azure\\azuredevops\\cli\\tools\\artifacttool\\a445ec82-2166-4964-946e-d868ead0c4e3' -> 'C:\\Users\\WS19312\\.azure\\azuredevops\\cli\\tools\\artifacttool\\ArtifactTool_win10-x64_0.2.104'

Error trace reported by tool:

[WinError 2] The system cannot find the file specified
Traceback (most recent call last):
pip-install-76yvurxi\knack\knack\cli.py, ln 206, in invoke
cli\core\commands\__init__.py, ln 575, in execute
cli\core\commands\__init__.py, ln 633, in _run_jobs_serially
cli\core\commands\__init__.py, ln 624, in _run_job
azext_devops\dev\common\exception_handler.py, ln 26, in azure_devops_exception_handler
    reraise(*sys.exc_info())
Temp\pip-install-76yvurxi\six\six.py, ln 693, in reraise
cli\core\commands\__init__.py, ln 603, in _run_job
...
Azure\CLI2\lib\subprocess.py, ln 709, in __init__
    restore_signals, start_new_session)
Azure\CLI2\lib\subprocess.py, ln 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• az login
• az extension add --name azure-DevOps
• az artifacts universal download --organization {} --feed {} --name {} --version {} --path {} --debug

Expected Behavior

Environment Summary

Windows-10-10.0.16299-SP0
Python 3.6.6
Shell: powershell.exe

azure-cli 2.0.68 *

Extensions:
azure-devops 0.12.0

Additional Context

[shubham90]

@wpschaub Do you have write permission to "C:\Users\WS19312"? Can you try running the command in an administrators prompt to eliminate the possibility of permissions issue?

[wpschaub]

I have write permissions to the folder and have tried running it in an administrators prompt.

From: Shubham Bhawsinka [e-mail]
Sent: September 17, 2019 5:02 PM
To: Azure/azure-devops-cli-extension [e-mail]
Cc: Willy-Peter Schaub [e-mail]; Mention [e-mail]
Subject: Re: [Azure/azure-devops-cli-extension] az artifacts universal download fails with Access denied and file not found (#776)

@wpschaubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwpschaub&data=02%7C01%7C%7Cffebfd8919fc4312931708d73bcb650d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637043617061840676&sdata=nY06xhAbUtWt5LB2RKtK0JvDhGxIthsS04zsh2mnNtk%3D&reserved=0 Do you have write permission to "C:\Users\WS19312"? Can you try running the command in an administrators prompt to eliminate the possibility of permissions issue?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-devops-cli-extension%2Fissues%2F776%3Femail_source%3Dnotifications%26email_token%3DAC44ZK2VL77LAHUJMRJKDBDQKFVWTA5CNFSM4ITKIJ42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD66IJ7Q%23issuecomment-532448510&data=02%7C01%7C%7Cffebfd8919fc4312931708d73bcb650d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637043617061850687&sdata=7Lj1MqnJUG%2FHU6IhDWebehZuJ1zyVcfdnEhQUN84BKw%3D&reserved=0, or mute the threadhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAC44ZKYM2TVMMKJTA3XKJ6TQKFVWTANCNFSM4ITKIJ4Q&data=02%7C01%7C%7Cffebfd8919fc4312931708d73bcb650d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637043617061850687&sdata=Bhh4nEnUomW0l1OZUSGyc3%2FPC4sb1yo8WB6ccF2EJMM%3D&reserved=0.

[shubham90]

That's interesting, I was unable to repro the issue on my box and on an isolated VM. I checked the code and this is doing a os.rename(path1, path2). Since, you have write permissions to the folder, it's interesting that it is failing in admin mode. I will keep looking on why this could potentially fail.

[mmenorgi]

Same error here, personal folder and even in Admin Mode i get the same error. The temp folder in the .azure subdir gets created and then error. On the 4th retry it worked, who knows why....

[elbatk]

@shubham90 did you come up with anything here? If we can't get a solid repro from us or the customer(s) that reported, we may have to close this for now.

[shubham90]

@elbatk I looked at this for sometime and it looks like an issue with retrying os.rename(path1, path2). I don't understand the root cause of the issue because os.rename is not controlled by the top level client. We can add this bug to the internal backlog to add retries or accept any contributions from the community.

[elbatk]

@shubham90 thanks for the update, I say we leave it up to the community for now and close this. If we see more occurrences we can take a bigger step and get it on our backlog.

[marklindell]

I have the same problem.

[kippesp]

I also see this on Windows. If I create the UUID as a directory in the .../artifacttool location, this directory (or any directory) gets quickly deleted when running the "az artifacts..." command. Once I see about 85% of the tool getting downloaded, the UUID directory will reappear for a second then quickly get removed with the "An error occurred while extracting..." error message.

Is it possible my corporation's McAfee protection software is permitting Python from taking the next step?

I'll try this on WSL next.

[kippesp]

I've got it working by obtaining the binary ArtifactTool_win10-x64_0.2.128 (via a less secure Windows system) and placing it into the correct directory manually. Another person in my org used the --debug data to identify the .zip and manually download the zip and do the same thing.

I suspect @marklindell would have success by running the az command again with the --debug flag, capture the output, find the line with "ArtifactTool", grab that URL, download with curl (or maybe a browser), unzip the contents into USER/.azure/azuredevops/cli/tools/artifacttool/ArtifactTool_win10-x64_0.2.128 (or whatever the zip file root name is).

No idea why I have to do this, but I think there is behind the scenes security software involved.

[marklindell]

I would agree with the assessment that it is related to my corporate McAfee software. It is locked down and does not allow me to disable or ignore any folders. It is draconian and prevents me from may of my daily development task. I will try another approach. :-(

[shubham90]

I've reopened this issue to see if we can add a retry while downloading the file. If the antivirus is deleting the file then there is not much we can do. You will have to whitelist the path in the antivirus setting.

If the antivirus is locking the file while we attempt to rename it, then retry should solve this issue. I will make the fix and update once it's released.

[shreyansh-zazz]

Any update on this one? I am having the same issue.

I also have corporate Mcafee Antivirus installed, it may be stopping this but have no idea. What should I do?

[PhilipSkinner]

I second this, same issue, McAfee installed. Other developers machines with the same software installed do not have this issue. Doesn't work for me no matter how many times it tries to install.

[davidlopez9210]

I have McAfee installed too and we are having the same problem. Any knows how to add the component to a whitelist in McAfee?

[PhilipSkinner]

You shouldn't have to. It's probably ms windows file locking. The installed needs to be able to deal with files being locked by waiting until the lock is released.

[atbagga]

The fix was released in 0.17.0 for retry logic.

@shubham90 Do you want to keep the issue open or should we close it now?

[shreyansh-zazz]

The fix was released in 0.17.0 for retry logic.

But the problem is still persisting.

[shubham90]

The fix was to add retries to the file handling and adding additional logging to get more information out of this error. If you are using azure devops extension 0.17.0 and still seeing the issue then please paste the detailed logs and we can investigate this further.

@atbagga Let's keep this open for now and we will check the new logs to determine if this is the same issue or not.

Adding @satbai as well.

[PhilipSkinner]

I'll ask all our devs to upgrade and feedback - we only had 1 out of 9 people install and use it successfully.

[thommiano]

I'm having the same problem on Mac. I connected to my subscription:

az login

Logged in via browser and confirmed that my subscriptions appeared connected properly. Installed the extension:

az extension add --name azure-DevOps

Attempted to download my artifact (a Python package)

az artifacts universal download \
    --organization ORGANIZATION \
    --feed FEED \
    --name NAME \
    --version VERSION \
    --path . 

Then got this:

Can't find the package 'PACKAGE' with version 'VERSION' in feed 'FEED'.

When I know the PACKAGE, VERSION, and FEED are correct and exist.

I have McAfee installed as well.

[thommiano]

Running below got me past this issue.

az devops configure --defaults organization=https://dev.azure.com/ORGANIZATION project=PROJECT-NAME

However, when it goes to run the download I end up with the error:

Can't find the package PACKAGE with version VERSION in feed FEED.

Others appear to have a similar issue: https://github.com/Azure/azure-cli-extensions/issues/554

I'm creating the artifact with below using TwineAuthenticate.

- task: TwineAuthenticate@1
  displayName: 'Twine Authenticate'
  artifactFeed: FEED
- script: twine upload -r FEED --verbose --config-file $(PYPIRC_PATH) dist/*
  displayName: "Upload package as artifact"

Using the above I'm able to download the artifact in a build context. The following works

- task: DownloadPackage@1
  displayName: 'Download NAME python package'
  inputs:
    packageType: pypi
    feed: FEED
    view: VIEW
    definition: NAME
    downloadPath: '$(System.DefaultWorkingDirectory)'  # default: "$(System.ArtifactsDirectory)"
    version: VERSION

But I can't download it locally using universal download.

[seqdan]

It seems like two issues are mixed up here:

• The original issue is about An error occurred while extracting the Universal Packages tooling: [WinError 5] Access is denied:. The az cli tries to install the Azure Devops "artifacttool" and fails copying it.
• Can't find the package PACKAGE with version VERSION in feed FEED. looks like an issue finding a feed, but the "artifacttool" seems to work in this case.

I found this post because I ran into the original issue: "artifacttool" could not be installed. I had tried 2 times before reading this. One time again while reading it, and as it seemed to be a "flaky" issue, once afterwards.

It worked afterwards, and since then without any issues. "artifacttool" is now installed. I did not change anything in the antivirus settings or security software. I only run the same command several times.

[samuelkoppes]

@thommiano Please make sure your configuration is set up correctly per the Azure Artifacts documentation. Also verify that the package and version you are looking for are available in the target feed.

@seqdan Please let us know if you see this issue again.