Azure-pipelines-tasks: IIS Web App Deploy Task No Longer Available

[dylanjaye]

Question, Bug, or Feature?
Type: bug

Enter Task Name: IIS Web App Deploy
ID: 1b467810-6725-4b6d-accd-886174c09bba
Version: 0.156.11

Environment

• Server - Azure Pipelines or TFS on-premises?

  • Azure Pipelines
    Account name: poolcorporation
    Project name: Software
    Release pipeline: All releases using IIS Web App Deploy

#

• Agent - Hosted or Private:

  • Private agent
    OS: Microsoft Windows Server 2016 Datacenter
    Agent Version: 2.169.1

Issue Description

Around 3:00 PM Central time, all of our release pipelines using the IIS Web App Deploy task started failing with the following error:

Failed to download task 'IISWebAppDeploymentOnMachineGroup'. Error No task definition found matching ID 1b467810-6725-4b6d-accd-886174c09bba and version 0.156.11. You must register the task definition before uploading the package.

The task icon has disappeared from ADO as seen in the following screenshot:

This behavior is consistent across all releases that use this task.

I found the same issue being reported here:
https://developercommunity.visualstudio.com/content/problem/1059329/failed-to-download-task-iiswebappdeploymentonmachi.html

Task Logs

tasklog_4.log

Error logs

Failed to download task 'IISWebAppDeploymentOnMachineGroup'. Error No task definition found matching ID 1b467810-6725-4b6d-accd-886174c09bba and version 0.156.11. You must register the task definition before uploading the package.

[vblain]

They were working at 12:29 PM PST today. All release pipelines are down for IIS on our end also. Also reported at https://developercommunity.visualstudio.com/content/problem/1059329/failed-to-download-task-iiswebappdeploymentonmachi.html?inRegister=true

[zalos]

Same. I just found this thread.

Failed to download task 'IISWebAppDeploymentOnMachineGroup'. Error No task definition found matching ID and version 0.156.11. You must register the task definition before uploading the package.

Will just stay posted for this to hopefully resolve itself. We did not change anything, this just started to block all our deployments. Tried a few things but no go thus far. Saw some info about the task.loc.json files, but they are inside the specific version folders. and this one does not exist:

[mc1brew]

Tango down here as well.

[alexpacer]

Same happening on my side. While all tasks remains the same. I'm see this error for all IIS related deployment

No task definition found matching ID 1b467810-6725-4b6d-accd-886174c09bba and version 0.156.11. You must register the task definition before uploading the package.

[sharadbharadia]

Facing exact same issue. It was working till yesterday and it started failing now.

I am using yml file for deployment using 'IISWebAppDeploymentOnMachineGroup@0' task.

[stephenmichaelf]

Sorry for the issue here!

Could someone try to hit:

https://.visualstudio.com/_apis/distributedtask/tasks/1b467810-6725-4b6d-accd-886174c09bba/0.156.11

Replacing with your Organization and let me know if it downloads the task zip?

Also, if anyone can email me (stfrance at microsoft dot com) with your Organization name that would be very helpful.

[shaynemarriage]

I get the error message below from: https://.visualstudio.com/_apis/distributedtask/tasks/1b467810-6725-4b6d-accd-886174c09bba/0.156.11
{
$id: "1",
innerException: null,
message: "No task definition found matching ID 1b467810-6725-4b6d-accd-886174c09bba and version 0.156.11. You must register the task definition before uploading the package.",
typeName: "Microsoft.TeamFoundation.DistributedTask.WebApi.TaskDefinitionNotFoundException, Microsoft.TeamFoundation.DistributedTask.WebApi",
typeKey: "TaskDefinitionNotFoundException",
errorCode: 0,
eventId: 3000
}

[stephenmichaelf]

Can you all try running your releases again? We pushed out a change.

[stephenmichaelf]

Or hit the same endpoint as before but this time make the ending 0.156.12 instead of 11.

[ilushka85]

Does not work. Same error is produced and still points to .11 in azure dev ops.

[dylanjaye]

Just re-ran one of our releases, and can confirm it does seem to be working now:

EDIT: Nevermind, the initialize task succeeded, but the release failed when trying to run the task:

[ajayush]

Still failing for us, however the error changed to:
File not found: 'C:azagent\A2_work_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.12\deployiiswebapp.js'

[ldk107]

We are also receiving the same error: File not found with deployiiswebapp.js

[hkurma]

Sorry for the issue here!

Could someone try to hit:

https://.visualstudio.com/_apis/distributedtask/tasks/1b467810-6725-4b6d-accd-886174c09bba/0.156.11

Replacing with your Organization and let me know if it downloads the task zip?

Also, if anyone can email me (stfrance at microsoft dot com) with your Organization name that would be very helpful.

I tried this and was able to download the zip. However when I tried to run the task I get the below error.

Still failing for us, however the error changed to:
File not found: 'C:azagent\A2_work_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.12\deployiiswebapp.js'

[stephenmichaelf]

Thanks! That is a second unrelated issue, unfortunately. We found that issue and are rolling out a fix now. Should be resolved in less than 30 minutes.

[stephenmichaelf]

Could you please try again? The version is now 0.156.13.

[Amir818]

Thanks @stephenmichaelf. We're having the same issue. Got around it by manually unizipping task.zip, but looking forward to the fix. Thanks.

[ldk107]

ran a new release and still receiving the error file not found tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.12\deployiiswebapp.js. The version is still .12

[Amir818]

ran a new release and still receiving the error file not found tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.156.12\deployiiswebapp.js. The version is still .12

@ldk107, try unizipping the task.zip manually inside that folder. Worked for us. I just ran a new release as well, and it's still .12. Not sure how the version upgrade is triggered.

[vblain]

Back to working

[dylanjaye]

Pipelines appear to be stable again

[goodtocode]

Resolved here as well. Thanks Azure DevOps team!

[stephenmichaelf]

Thanks everyone! Sorry again. I am writing up an RCA and will post it here soon that describes what went wrong. I appreciate your patience.

[zalos]

yep all working! Thanks for the work on this. Did someone not properly QA before their automated CI ran out that bit of code? Shame, shame 🤣

[stephenmichaelf]

This issue has been fixed with the new task version, 0.156.13.

We realize this was extremely inconvenient and caused blocks for all of you and we apologize for that.

We ended up running into two problems here:

1. Yesterday we were deploying a new sprint payload to some of our scale units, and customers reported an (unrelated) regression so we rolled back the deployment. We hit an issue with the rollback such that the rollback was not clean, leaving data in an inconsistent state, which caused the task to fail. The DB was pointing to 0.156.11, but due to partial servicing 0.156.12 was installed, causing the errors.
2. In order to fix the task from issue 1, we had to run a hotfix to update the task. This runs a special build in our CI process that builds 1 specific task and creates the zip and PowerShell files to install it. We had recently updated this process and merged to master. This change, however, didn't get into our current releases branch (missed it by 1 day). So the package that was generated for the hotfix was mostly correct but not fully correct. This caused the second issue where 0.156.12 was not working. The second fix was to fix this CI process.

Follow up items:

1. Fix the servicing process so that even if it's canceled, things are not left in an inconsistent state.
2. For this one we have to be more careful with the day things are committed and when the release branch is cut off. We missed it by a couple of hours but thought it had gone in.

Sorry again for the inconvenience this caused and we will make sure to follow up to prevent this type of issue in the future.