Azure-pipelines-tasks: Publish Pipeline Artifact task fails when path to publish is a file

[tvald]

Agent Version and Platform

Version: 2.140.2
OS: Hosted Ubuntu 1604

Task Type and Version

==============================================================================
Task         : Publish Pipeline Artifact
Description  : Publish Pipeline Artifact
Version      : 0.139.0
Author       : Microsoft Corporation
Help         : Publish a local directory or file as a named artifact for the current pipeline.
==============================================================================

What's not working?

The task fails when Path to publish points to a file rather than a directory.

Task Logs

When publishing a file: (fails)

2018-10-18T17:37:09.0819573Z Uploading pipeline artifact from /home/vsts/work/1/s/dist.zip for build #15
2018-10-18T17:37:11.6205372Z ##[error]Microsoft.VisualStudio.Services.BlobStore.WebApi.Exceptions.InvalidLocalDirectoryException: The source directory provided does not exist, or is not a directory.
   at Microsoft.VisualStudio.Services.BlobStore.WebApi.BuildDropManager.<PublishAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Agent.Plugins.PipelineArtifact.PipelineArtifactServer.<UploadAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Agent.Plugins.PipelineArtifact.PublishPipelineArtifactTask.<ProcessCommandInternalAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Agent.Plugins.PipelineArtifact.PipelineArtifactTaskPluginBase.<RunAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Agent.PluginHost.Program.Main(String[] args)

When publishing a directory: (succeeds)

2018-10-18T17:37:12.1639551Z Uploading pipeline artifact from /home/vsts/work/1/s for build #15
2018-10-18T17:37:14.0002031Z Information, 9318 files to be processsed in 94 groups.
<...>
2018-10-18T17:38:02.8487800Z Information, Upload is done!
2018-10-18T17:38:02.8506948Z 
2018-10-18T17:38:04.8750126Z Associated artifact 1 with build 15
2018-10-18T17:38:04.8752235Z Uploading pipeline artifact finished.

[igor-sadchenko]

I have the same issue

[anishkny]

Yes this was surprising to me too. Please allow publishing just files too.

https://dev.azure.com/anishkarandikar/custom-pipelines-task/_build/results?buildId=45

[federicorinaldi]

I don't think you should recommend switching to Pipeline Artifacts (here) until you have the same level of support

[ArieHein]

why are you trying to do that in the first place ?

idea is that if you want certain files form the build directory you use a copy task and in the content section you put relative path / specific files / exclude files to the artifact dir (a folder under the _work path of the agent) and the next step would be publish to an artifact name from that 'a' folder.

I would agree that this requires a check that the user didnt place a file in the 'path' which is probably missing which gives an error.

[federicorinaldi]

According to the help Path to publish is "The folder or file path to publish"

And for your suggestion of using the copy task, the docs also recommends against this:

In build artifacts, it was common to first copy files to $(Build.ArtifactStagingDirectory) and then use the Publish Build Artifacts task to publish that directory. With Pipeline Artifacts, we recommend pointing the Publish Pipeline Artifacts tasks directly to the paths to be published. This saves your build the time of creating a copy of the files you wish to publish

I've already created a zip file with the files I need, so I only need to publish that, is true that I can publish the entire dir with only that file but this behaviour is not what you would expect from the help and also from the previous Publish Build Artifacts (that this task intends to replace).

[tvald]

this behaviour is not what you would expect from the help and also from the previous Publish Build Artifacts (that this task intends to replace)

Precisely. The best practices recommended in documentation are broken, and have been broken for several months without the slightest acknowledgement from MS. This does not instill confidence in the service. (We gave up trying to get DevOps configured, will maybe try again next year after the service is more mature.)

Flagging top recent contributors to maybe boost visibility:
@DavidStaheli @Ajay-MS @madhurig @prawalagarwal @bishal-pdMSFT

[ArieHein]

@federicorinaldi perhaps that is poorly worded or miscommunication of the writer, for me a file path is not path to file. A file path is \server\share not a \server\share\file.ext so maybe an assumption of the writer vs English terms.

While I agree that this causes a longer build time as youre copying files from one folder on the agent to another folder on the same agent BUT, as you probably know, compiling code leaves extra unnecessary files that were brought over by the "Get Sources" step and the Build process itself, so this allows you to more then tell the Copy step what to copy its what NOT to copy, depending on how you set your Build Output parameter. Ifyou set a special folder for the output then yes, you do NOT need the Copy step, but ive seen way more times in my work as DevOps and ALM consultant that the current method a company uses, creates folders with mass amounts of 'garbage' that has no reason to be promoted as an artifact so that Copy step is essential for the beginning to make them start working correctly and then in the second step we fix the build procedure so that extra copy step is not needed.

That said, because the copy and publish steps, both use Robocopy, the copy of the files is very fast, especially if you tell it what to copy and what NOT to copy.

If youre comfortable with the output of the Build and used a zip file to archive, simply state in the Archive step to place the zip under a 'Output' folder, which will make the zip the ONLY file in it
and then the Publish step will take it from that folder.

@tvald While documentation is lacking at times, you and I have the power to fix it and offer fixes via opening a pull request. I've done so myself couple of time in the past, with ease.

However I completely disagree with your statement concerning the maturity of Azure Devops.
As someone who worked with TFS when it was Visual source safe, seeing the transition and helping numerous organizations convert to it and use it, its a Delightful experience and the service is very mature, very intuitive and allows degree of freedom to do whatever you want with it, if you write your own PowerShell scripts, your own extension and your own applications to interact with the RESTAPI it offeres.

[anishkny]

+1 to @ArieHein

I have been delighted by Azure DevOps as I try to use it in my workflows and build a custom task. Even the best of services has some gotchas and this was one. Perhaps a better error message if you try to publish a file instead of a folder - or better yet just allow publishing a file 😀

[mitchdenny]

@ArieHein and @tvald thanks for your feedback on the Pipeline Artifacts feature. Clearly we need to address this issue. The docs say one thing and the code does the other.

We want Pipeline Artifacts to be a drop in replacement for the existing artifact store so it makes sense to support handling file paths. I'll create a bug on our team to go and address this.

[mitchdenny]

OK I've logged a bug for this. In the meantime if you want to specify the directory go right ahead. If you have files in the directory along with the file you want to upload you can use the .artifactignore file to constrain what you publish:

https://docs.microsoft.com/en-us/azure/devops/artifacts/reference/artifactignore?view=vsts

[GongT]

Try to upload a node project as artifact. After half an hour, it still uploading files from node_modules... 🤣

Why it download as a zip file, but upload as files...

[mitchdenny]

@GongT are you using the Publish Pipeline Artifact task or the Publish Build Artifacts task. The later will be quite slow for a large node_modules folder. The former should be relatively quick. Can you paste your logs - perhaps in a fresh issue.

[GongT]

I'm using Publish Build Artifacts one. I see many HTTP-thing and know why it slow, now I'm creating a zip by myself.

Publish Pipeline Artifact cannot download again from other Builds. (Why? So what is the goal of Pipeline Artifact?)

[mitchdenny]

@GongT the Download Pipeline Artifact task does support downloading artifacts from other pipelines. The trick is that you need to provide the _Pipeline ID_ (or _Build ID_). The Pipeline ID is not the definition ID, its the actual build instance that you want to download from. So you'll need to find that programmatically.

We are working on some enhancements to this task which will allow you to do things like copy from the latest artifacts from a pipeline which will make this easier in some cases.

If you are need to move artifacts between pipelines, you might find that Universal Packages is a better fit since you publish to a feed and then explicitly version the packages.

https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/universal-packages?view=vsts&tabs=yaml

[GongT]

Really thanks for your reply!!! I'm trying to use artifacts now. (hope I know it a few days ago)

[ArieHein]

What ever step youre using, do not include the node modules subfolder. That's a bad habbit for any development that is based on packages. The packages should be restored as part of the build process.

[elbatk]

Hi all, I believe this issue has been fixed since the last activity so I'm going to close. Please let me know if you're still having issues!

[aromanrod]

just replace in variables: '**/Dockerfile' by '$(Build.SourcesDirectory)/Dockerfile' that's works for me.