Azure-pipelines-tasks: Upload failed with Error: write EPIPE

[bhaveshmaniya]

Environment

Software: Azure Pipelines
OS: Windows (Azure Web App)

Issue Description

• The FTP Upload task in Azure release pipeline throws Error: write EPIPE.

• Basically in our case, the FTP upload pipeline copies the files from the build artifacts to the azure web app. Everything was working as expected from last few months and suddenly we are getting below error from last 2 days.

• Refer below screen capture:

Error logs

[error]upload failed: /site/wwwroot/Data/xyz.yml due to error: Error: write EPIPE

[yacaovsnc]

Thanks for reporting this issue. Which agent are you using?

[yacaovsnc]

Also, could you please run your build with "Syste.Debug" set to "true", and upload the relevant logs? Does it fail immediately after job starts? Does it take a while? If the job repeatedly fails, do they fail around the same time spot?

[bhaveshmaniya]

Thanks for reporting this issue. Which agent are you using?

Agent: Hosted Agent (Pool: Hosted VS2017)

[groja]

I am experiencing the same problem from an Azure DevOps instance deploying to an App Service (I found this ticket by googling my error!).

We are using the "Hosted" pool, and the failure seems to consistently occur after a few files have succeeded.

The full error line is
2019-03-11T17:03:37.8815983Z FTP upload failed: "upload failed: [filename removed] due to error: Error: write EPIPE". FTP log: "[connection] < '226 Transfer complete.\r\n',[parser] < '226 Transfer complete.\r\n',[parser] Response: code=226, buffer='Transfer complete.',[connection] > 'PWD'".

The only thing that seems to have changed since our pipeline last succeeded is that the following is now repeatedly logged at the start of the FTP task, where it didn't used to be:

2019-03-11T17:03:32.8476235Z (node:2936) Warning: Use Cipheriv for counter mode of aes-256-ctr

Some sort of incompatibility or point of failure introduced by changing available cipher sets maybe?

[yacaovsnc]

@groja I am still trying to understand the problem, I couldn't reproduce this myself. I am trying to understand why some files were successfully uploaded if this is a compat issue.

[yacaovsnc]

Could you please describe your deployment process? Do you stop the app service before you start ftp upload? Are you deploying to prod or stage slot? Is the file being ftp'ed a new file or is it overwriting existing file?

[groja]

For what it's worth I've just re-run in debug mode and it hasn't shown anything more enlightening - all that I've got is more lines showing what appear to be successful FTP commands.

I've gone and looked at the App Service end, and it looks like whichever file it "fails" on gets left as a 0 byte file, so clearly there is some sort of problem with the connection cutting out. I did try to find the FTP logs at that end, but apparently that's not supported for end users.

In response to your questions:

• No we don't stop the app normally. However, I've just re-tested having manually turned off the service (given it's a staging one so I can do that), and it failed the same way.
• Staging slot, I'm dry-running a production release.
• FTP is overwriting, don't think there are any completely new files in this deployment compared to previous.

However, I have tested a different change that does seem to have helped - I've changed to the ftp:// endpoint rather than ftps://, and that seems to have fixed it. That does strongly indicate some sort of weird network library incompatibility, but I couldn't tell you what! Obviously we don't want to use non-secure FTP if we can possibly avoid it, but that at least gives me an interim workaround.

@bhaveshmaniya it might be worth trying that as a temporary workaround.

[steveellis]

I can confirm for us too that ftps deployment no longer works using the ms ftp pipeline task. Changing the endpoint to ftp fixes it which of course is not ideal.

The warning message Use Cipheriv for counter mode of aes-256-ctr also appears on app service deployments but they aren't failing. Other people have reported this:

https://github.com/Microsoft/azure-pipelines-tasks/issues/9783

This is the nodejs issue for the warning: https://github.com/nodejs/node/issues/16746.

[yacaovsnc]

@steveellis @groja thanks for the workaround and confirmation. We are considering our options.

[ugaboy]

I just came across this issue today, but with a completely different pipeline task. Azure DevOps Server on-prem running a Bash, shell or command task that attempts to use anything that fetches data from the Internet fails with the Node.js error concerning cipher counter mode. Using a Bash inline script task that simply calls wget or curl logs this:

(node:10993) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:10993) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:10993) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:10993) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:10993) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:10993) Warning: Use Cipheriv for counter mode of aes-256-ctr
(node:10993) Warning: Use Cipheriv for counter mode of aes-256-ctr

And will sit there until the job is manually killed. We are behind a corporate proxy and I can successfully execute wget or curl while in an interactive terminal session on the build agent. I am unable to run any of our builds that use a linux agent version 2.149.0 (i haven't tried on a Windows one). It would be a horrible headache to band-aid the builds to circumvent this. :(

[bhaveshmaniya]

I've updated my FTP upload task to use the ftp but it's not working. I am getting below error:

Note: I've tried the same credentials using FileZilla and I am able to connect. Does anyone suggest what could be the issue?

[yacaovsnc]

The v2 FTP task has been release and it will auto retry for failed connections. Please open new bugs against v2 if you continue experiencing issues. I also verified ftps works with v2 task.

[kjng]

I can reproduce this on v1 of the FTP task starting 1/21/2020. FTPS no longer works, but FTP does.

[dcartamantiglia]

I’ve just encountered the same issue.
Solved by updating the task version of the FTP upload from 1.* to 2.*

[tommmato]

We currently have the same issue in our release pipeline that worked for months. Started out of nowhere about 2 weeks ago. Switching from task version 1 to 2 did not solve it. Strange fact: we deploy to four different development/staging slots of our app service - 1 slot still works, 3 not. I have not tried the production slot yet.

_2020-02-06T19:30:45.6541513Z ##[error]upload failed: site/wwwroot/example/assets/css/bootstrap-theme.css due to error: Error: write EPIPE_

[reunefe]

Ever since January 28th I'm having the same issue.
The deploys went fine for almost a year. Then out of nowhere it started failing.
Switching to v2 or FTP instead of FTPS does not fix things.

The directories are created as expected, but once starting to write the first file the entire thing fails...
2020-02-12T07:20:09.1139904Z FTP upload failed: "upload failed: site/wwwroot/xx/xx/xxxx.txt due to error: Error: write EPIPE". FTP log: "[connection] < '226 Transfer complete.\r\n',[parser] < '226 Transfer complete.\r\n',[parser] Response: code=226, buffer='Transfer complete.',[connection] > 'PWD'".

[tommmato]

I finally found a solution that solved it for me. I have always specified the remote directory in the format "site/wwwwroot/example". Now I put a "/" in front, so it looks like "/site/wwwwroot/example". The errors are gone.

As I said, it had worked for months without the leading slash. MS must have changed something here.

[reunefe]

Thanks, that in combination to switching to v2 did the trick!