Azure-pipelines-tasks: ExtractFiles should allow overwriting destination

[TFleury]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
bug

Enter Task Name: ExtractFiles

Environment

• Server : Azure Pipelines

• Agent : Private : Windows Server 2019

Issue Description

Task fails if "Clean destination folder before extracting" is not checked and destination folder is not empty.

Task logs

2019-10-19T12:13:02.9307087Z ##[section]Starting: Extract files 
2019-10-19T12:13:02.9446365Z ==============================================================================
2019-10-19T12:13:02.9446437Z Task         : Extract files
2019-10-19T12:13:02.9446466Z Description  : Extract a variety of archive and compression files such as .7z, .rar, .tar.gz, and .zip
2019-10-19T12:13:02.9446510Z Version      : 1.151.1
2019-10-19T12:13:02.9446551Z Author       : Microsoft Corporation
2019-10-19T12:13:02.9446577Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/extract-files
2019-10-19T12:13:02.9446604Z ==============================================================================
2019-10-19T12:13:03.2123419Z No path specified for search pattern: *.zip defaulting to: C:\azagent\A1\_work\r1\a
2019-10-19T12:13:03.2124863Z Searching for: *.zip under directory: C:\azagent\A1\_work\r1\a
2019-10-19T12:13:03.2200742Z Found: 1 files to extract:
2019-10-19T12:13:03.2200895Z C:\azagent\A1\_work\r1\a\Staging - CI\AssetToDeply\AssetToDeply.zip
2019-10-19T12:13:03.2207021Z Extracting file: C:\azagent\A1\_work\r1\a\Staging - CI\AssetToDeply\AssetToDeply.zip
2019-10-19T12:13:03.2219725Z [command]C:\azagent\A1\_work\_tasks\ExtractFiles_5e1e3830-fbfb-11e5-aab1-090c92bc4988\1.151.1\7zip\7z.exe x -oC:\Apps\AssetDest\dev C:\azagent\A1\_work\r1\a\Staging - CI\AssetToDeply\AssetToDeply.zip
2019-10-19T12:13:03.2435313Z 
2019-10-19T12:13:03.2435539Z 
2019-10-19T12:13:03.2435783Z ERROR: Can't allocate required memory!
2019-10-19T12:13:03.2435822Z 7-Zip [64] 16.00 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-10
2019-10-19T12:13:03.2435867Z 
2019-10-19T12:13:03.2435912Z Scanning the drive for archives:
2019-10-19T12:13:03.2435961Z 1 file, 35814372 bytes (35 MiB)
2019-10-19T12:13:03.2435994Z 
2019-10-19T12:13:03.2436042Z Extracting archive: C:\azagent\A1\_work\r1\a\Staging - CI\AssetToDeply\AssetToDeply.zip
2019-10-19T12:13:03.2436104Z --
2019-10-19T12:13:03.2436184Z Path = C:\azagent\A1\_work\r1\a\Staging - CI\AssetToDeply\AssetToDeply.zip
2019-10-19T12:13:03.2436216Z Type = zip
2019-10-19T12:13:03.2436292Z Physical Size = 35814372
2019-10-19T12:13:03.2436347Z 
2019-10-19T12:13:03.2436377Z 
2019-10-19T12:13:03.2436424Z Would you like to replace the existing file:
2019-10-19T12:13:03.2436475Z   Path:     C:\Apps\AssetDest\dev\Api.Abstractions.dll
2019-10-19T12:13:03.2436519Z   Size:     10240 bytes (10 KiB)
2019-10-19T12:13:03.2436548Z   Modified: 2019-10-19 10:16:40
2019-10-19T12:13:03.2436639Z with the file from archive:
2019-10-19T12:13:03.2436730Z   Path:     Api.Abstractions.dll
2019-10-19T12:13:03.2436761Z   Size:     10240 bytes (10 KiB)
2019-10-19T12:13:03.2436841Z   Modified: 2019-10-19 10:16:40
2019-10-19T12:13:03.2436915Z ? (Y)es / (N)o / (A)lways / (S)kip all / A(u)to rename all / (Q)uit? 
2019-10-19T12:13:03.2436981Z Archives with Errors: 1
2019-10-19T12:13:03.2437000Z 
2019-10-19T12:13:03.2438414Z code: 2

Troubleshooting

Need to add "-aoa" switch (Overwrite All existing files without prompt.) to 7zip command.

Related issues

• ExtractFiles failing in Release pipeline with ERROR: Can't allocate required memory! #11135
• ExtractFiles to support override files on the target path #5522

[jrybacek]

This may be a stupid question, but based on what I can see it looks like this change never made it into production. Is it still stuck in review? Once the code appears here on Github how long does it take until Azure DevOps starts leveraging the updated ExtractFiles task?

[JohnFF]

I agree with jrbacek - we need this feature, and it's not present.

[jrybacek]

I agree with jrbacek - we need this feature, and it's not present.

I've been begging, I've opened tickets. Its crickets, Microsoft, please. Help!

[arkein]

Just faced this issue, I think it's a bug rather than an enhancement.
Consider this: I have explicitly stated in task parameters that I do not want to clear the destination folder - which implies that I'm OK with any possible file name conflict during the extraction.

[stephenmichaelf]

I will get this PR merged this week. Thanks!

[murugan-mp]

I will get this PR merged this week. Thanks!

Any update on this

[stephenmichaelf]

I was on call last week but I have time blocked on it today, I am going to open a new PR and will link to it here!

[rajdeol]

Has the PR been merged ? I am waiting for this

[jrybacek]

I'm excited to have this fixed as well. We are waiting on migrating to Azure DevOps until this is fixed. Stephen, I'm sure you are quite busy, but we appreciate your help. Once its committed, do you know how long it will take to end up in Azure DevOps services?

[murugan-mp]

@stephenmichaelf We're waiting for your response and PR.

[scheffler]

Hit this today. Any update on timing?

[jrybacek]

As far as I can tell, Microsoft is the only one that can actually accept the pull request. I could be wrong here, but we are still waiting on Microsoft. Please keep adding comments to the thread, it appears to be the only way it will get noticed. Microsoft ignores our complains through our enterprise contract. I wonder why we even pay it.

[JohnFF]

I had to use the standard copy files as a work around. @stephenmichaelf we need this merged asap!

[JohnFF]

Just seen 167 open pull requests. WTH!?

[Miles-Davies-HORIBA]

@ShreyasRmsft please review this. I have had to use pwsh to work around this issue which should be a simple fix for you guys :-)

[ShreyasRmsft]

@Miles-Davies-HORIBA sadly i'm not the owner for this task and the practice is generally for code owners or the team that owns the task to approve the PR.

Also FYI a couple of tests are failing https://github.com/microsoft/azure-pipelines-tasks/pull/11594/checks?check_run_id=673857387

I'll see what I can do and drop a mail to @stephenmichaelf and a few others who might be able to help here.

[stephenmichaelf]

PR in progress here, almost done with testing then will merge: https://github.com/microsoft/azure-pipelines-tasks/pull/12902

[murugan-mp]

@stephenmichaelf Do you have any good news here?

[jrybacek]

We are excitedly waiting for this feature, so we can finally put this issue to bed. Is there something we can do with our corporate Microsoft contacts to help prioritize this work?

[murugan-mp]

PR in progress here, almost done with testing then will merge: #12902

Any update?

[DavidLHannah]

I echo the need for an overwrite option. Or perhaps just an _arguments_ option where we can pass any 7zip argument.
In the mean time, easy enough to make due in PowerShell.

[jrybacek]

I echo the need for an overwrite option. Or perhaps just an _arguments_ option where we can pass any 7zip argument.
In the mean time, easy enough to make due in PowerShell.

Hi David, would once you figure out the PowerShell script, would you mind posting it back here? I know what 7zip arguments I need, but I'm not super familiar with what environment variable names reference the zip file and output directory.

[DavidLHannah]

@jrybacek I just used the microsoft.powershell.archive module in a PowerShell task. Pretty simple.

[AlexA-1976]

Please add this capability!

[jrybacek]

@stephenmichaelf when this PR is accepted, how long will it take to get into the Azure DevOps cloud product?

[murugan-mp]

@stephenmichaelf when this PR is accepted?

[anatolybolshakov]

Hi @murugan-mp @jrybacek @AlexA-1976 could you please check it now? We have rolled out related changes for ExtractFiles task - so it should be available now

[anatolybolshakov]

@TFleury @murugan-mp @jrybacek @AlexA-1976 I'm closing it at the moment - please let us know if there are still any issues