[email protected] leading to broken Function App when when trying to publish

[vbui1]

Using the [email protected] Python Function Apps can't be published.
Following command is used to publish the Python Function App:
func azure functionapp publish FUNCTION_APP_NAME --python --build remote

The output prints:
Writing the artifacts to a squashfs file Failed to retrieve remote build status, please visit https://FUNCTION_APP_NAME.scm.azurewebsites.net/api/deployments Response status code does not indicate success: 500 (Internal Server Error).

The Function App is afterwards left in a broken state. Accessing the URL returns "The Service is unavailable." and the Function App can't be deleted due to error message:
{"Code":"InternalServerError","Message":null,"Target":null,"Details":[{"Message":null},{"Code":"InternalServerError"},{"ErrorEntity":{"Code":"InternalServerError","Message":null}}],"Innererror":null}

or

Cannot modify this site because another operation is in progress.

PowerShell Function Apps can be published without any problem.

When downgrading the azure-functions-core-tools to version 2.7.2508, publishing works flawlessly.

[anthonychu]

@Hazhzeng @anirudhgarg FYI

[Hazhzeng]

Hi @vbui1, thanks for raising this issue to us.

Is this a transient issue?
Please share us with your function app name. If you feel uncomfortable sharing the function app name publicly, please follow the guideline in here to share it with us privately. Thanks.

[vbui1]

Hello @Hazhzeng ,

since our Function App are in a broken state, we cannot run one of our functions to get the id. Is there any other way to share the Function Apps privately?

We were able to reproduce the issue since last Thursday, 25th June, accross multiple Function Apps.

[dan-leanix]

Just to chime in here version 3.0.2630 seems to be also affected. Our company currently has several support tickets open with Linux-based Functions (Python) in a broken state after trying to publish the Function.

[balag0]

@vbui1 I added an alternate method to share the function app name privately at the bottom of the same page. https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately

@dan-leanix Could you also share the name of one of the impacted apps using those steps.

[dan-leanix]

@balag0 11:56 PM UTC f081c418-20dd-4ccf-ac24-066288a41f7a West Europe

Otherwise I already pointed the support engineers to this issue. Might be that they internally reach out to you or the team.

[vbui1]

@balag0

07:34 AM UTC b686bf9a-ae41-4d40-85ba-fa1e2df1029d West Europe

I hope it worked. The site returned The service is unavailable.

[dan-leanix]

@balag0

We were successful with version 3.0.2534 to deploy our Function in North Europe.

West Europe seems to have other issues now, like Function Runtime not available and service unavailable.

[balag0]

@vbui1 Thanks for sharing the guid. I was able to locate the app name and looks like it is failing due to duplicate appsettings (may be there is an appsetting with the same name as a keyvault reference name?)

Since the app is deleted now I couldn't verify the appsettings but if you are seeing the issue with another app let me know.

[balag0]

@dan-leanix I think this app also has the same config issue as above. (duplicate app setting name and key vault reference name) This site is also deleted now so can't confirm if this is the issue, but please let me know if you have a different app we could look into. Thanks

It would be great if you or @vbui1 could confirm if your apps are good after resolving the appsetting issue.

[dan-leanix]

@balag0 As I mentioned it Azure Support was working on it and after we found out that the recent Core Tools version is the case they proceed with cleaning up.

The issue is really reproducible. Spin up a Python Azure Function App, have some Key Vault secret references and then try to publish the Function with Core Tools version 3.0.2630.

What I can say for sure that we did not have app settings and Key Vault secret references with the same name.

[balag0]

I tried this today and couldn't repro it. Looks like there is some specific config that seems to trigger it so I will keep trying. It would be easier/faster to investigate this if there is an app in this broken state.

Since both the apps mentioned in this thread have been deleted , for any one else facing similar issue please share the app name using the instructions at https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately . Thanks

[vbui1]

Hello @balag0 ,

Here is another broken Function App:

10:09 AM UTC e3313585-1b1a-4ac1-b5cb-f76638ce8151 West Europe.

We were able to reproduce the bug again using [email protected].
We tried to publish with following command:
func azure functionapp publish FUNCTIONAPPNAME --python --build remote

[balag0]

Thank you. This is helpful and I will share an update once I have something.

[kummerer94]

I can confirm this. We're seeing the same issue and it is breaking every Function App on a consumption plan we have deployed. I also have a support ticket open for this. Key Vault references and managed identity turned on seem to break the app.

[balag0]

Thanks everyone. We were able to identify the issue. As discussed on this thread the impact is to function apps using Key vault references after deploying with the latest core tools.

We have started deploying a fix which should complete in the next 12 hours. There is no need to update the core tools version you are using. Currently West Europe has been updated.

Next steps
If your app is in West Europe:

0) The app should already be fine. If not please continue with Step 1
1) Restart the app. This is to make sure your app is running on the latest container with the fix applied.
2) Check if the app is working now. If not please redeploy the app in case the previous deployment had failed in an inconsistent state. There is no need to update the core tools version you are already using.
3) If you are still seeing issues, please update the thread with the app name and I will investigate (https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately )

If your app is not in west europe:
Please share the app name and I will prioritize those apps. In any case we expect the rollout to complete in the next 12 hours across all regions. I will provide an update when this is complete.

[dan-leanix]

@balag0 FYI. In the support tickets I have mentioned above, we had the issue also in North Europe with the latest core tools version.

But as we downgraded to 3.0.2534 we haven't had that again.

[kummerer94]

@balag0 Thanks a lot! I've just noticed that our serverless Function Apps are back and I am able to work with them again.

[balag0]

The fix has been deployed to all regions now. Please let me know if you are still seeing any issues. Thanks everyone for your patience.