Kudu: Can no longer push to remote git repo in Azure App Service Editor (preview)

[jk-ncsc]

The issue is that I'm no longer able to "push" to a remote Git repo from Azure App Service Editor or Kudu console.

I started this by outlining _my_ specific error, then realised this happens even on a new and empty Web App.... so I've tried to make the repro below as generic as I can:

Repro from Blank

• Create a new github repo (e.g. https://github.com/jk-ncsc/testing-ASE)
• Create a new Web App in Azure (Free Tier - F1, B1 also tested) (e.g. "testing-ase[.]azurewebsites[.]net")
• Open "App Service Editor (preview)" in WebApp options
• Select "Git" section and add a git repo (sync works fine)
• Make a change (e.g. add a file)
• Select "Git" section and commit change with a message (success)
• Select "Git" dropdown and select "push" (hangs a while then errors, blank message in UI)
• Developer tools error message is "There was an issue running a git operation. Please use the console to check the state of your repository." (same on latest Chrome & Edge)
• Open console, check "git status" and then do a"git push" (fails)
• In this example, it just hangs, with no output
• I had the following explicit output on my main projects (both api apps give same errors)

error: cannot create standard input pipe for git credential-manager get: Function not implemented
error: cannot create standard input pipe for "$HOME/bin/cred.sh" get: Function not implemented
error: cannot create standard output pipe for D:\home\data\monaco\home\empty.cmd: Function not implemented
error: cannot create standard input pipe for bash: Function not implemented
fatal: could not read Password for 'https://<USERNAME>@bitbucket.org/<NAME>/<PROJECT>.git': Function not implemented

I've been using the ASE for months successfully. Last successful push was 17th Dec I think!

[davidebbo]

Please note that App Service Editor is not related to Kudu. But we can still discuss here as a placeholder (App Service MSDN forum would be the more correct place to discuss).

[jk-ncsc]

Apologies... top left (logo) of ASE links to http://go.microsoft.com/fwlink/?LinkId=816520 which is in this project

[davidebbo]

I see, we used this wiki to host that Editor related page. Anyway, no worries, let's discuss here.

First guess is that it may somehow be related to the upgrade to Windows 2016, but we'll need to dig deeper.

BTW, I'm also finding that when making changes, it's no longer displaying the diff in App Service Editor. Did you by a chance notice the same thing around the same time? It could be related.

[jk-ncsc]

Thanks! I'm still seeing the diff (as a side by side), but no longer seeing any visual cues / highlighting on changes. If it's any consolation about this topic being here, I'm getting the same errors in Kudu console as ASE. Given the timings of when this last worked, it could be related to wider upgrade. Was just surprised to not hear more chatter on the topic.

[davidebbo]

Yes, the highlighting is what I meant. It looks like it got broken as well. :(

[victorliafook]

Hi there, any news on this issue? Im having the same thing. I just cant push, if I use the command line it just hangs forever as if doing something, and if I try using the dropdown, it is disabled. I can create a new branch, add changes, commit, but never push.

[jk-ncsc]

Still there for me too. Very frustrating. The error message bug is fixed (I can see them now), but difference highlighting and general git push/pull is broken.

[davidebbo]

@jk-ncsc are you seeing this with a public or private repo? I'm surprised that you see git pull being broken if the repo is public (as in your initial steps).

The issue appears related to doing a push on an https git endpoint. If you go to Kudu process explorer, you can see that it tries to launch git-credential-manager, and maybe gets stuck there because that tries to pop up UI.

One workaround is to create a Personal access token (https://github.com/settings/tokens) and then embed that in your git remote. e.g. your remote would look like https://youruser:[email protected]/jk-ncsc/testing-ASE.git

[victorliafook]

@davidebbo it is exactly this problem (at least for me). Ive noticed that and tried killing the process (in kudu), then it promptly shows me the modal where I am supposed to put my bitbucket creds, I do it and still it doesn't work. Then the underlying op is always timed out.

For some reason the process for inputting the git creds is not working properly.

[davidebbo]

@victorliafook Indeed. Though I'm curious: how can that work at all if you have two factor auth enabled? Or are you using a personal access token as in my previous response?

[victorliafook]

@davidebbo actually none of them, Im not using Bitbucket's two step auth, I was just trying to set the remote to version my existing bot app code in the simple way.

[davidebbo]

@victorliafook well, embedding the creds in the remote should work, even if it's not ideal.

[victorliafook]

Yes, it should, I am going to give it a try. As long as no one else is contributing, it should be ok for now. Thanks for your attention, @davidebbo .

[jk-ncsc]

@davidebbo - both public & private git repos. My main issue came from using a private bitbucket repo and entering creds via the ASE popup each time I did a pull (personal pref). I tried the public github one before posting, noted the same behaviour, and used that for the example here.

It was working most of last year, until the update in Nov/Dec 17.

I'd rather not store a token in the AppService for git, but I'll give it a try to check. Thanks for the response.

[Bloom-web]

I'm having the same or a similar problem. After not working on a site for a while and coming back to it, I've also found that both App Service Editor and Kudu can't push to Github anymore. I was getting those silent errors that would just hang the environment or give a generic error.

Since it was happening immediately I suspected that it had to do with the Windows Credentials Manager messing something up. After trying all commands to reset, remove, and change the credentials manager ended up not helping anything, I tried changing some of the config files manually. After then changing the username from the email address to the github username, I started finally getting some error messages through kudu:

D:\home\site\wwwroot>git push github
git credential-manager get: /mingw32/libexec/git-core/git: Permission denied
git credential-cache get: /mingw32/libexec/git-core/git: Permission denied
3900 [main] bash 9796 fork: child -1 - CreateProcessW failed for 'D:\Program Files (x86)\Git\usr\bin\bash.exe', errno 13
bash: fork: Permission denied
error: failed to execute prompt script (exit code 254)
fatal: could not read Password for 'https://@github.com': No error

I think in my absence, someone did something on Azure to break everything about credentials to allow Kudu/ASE/Monaco authenticate with git and has not realized for months that it's been broken. I'm guessing this is because not many people use Azure this way—but I do, so it would be really nice if this functionality that I based my workflow on, was fixed.

[davidebbo]

Indeed, the root problem here is that Monaco (App Service Editor) is not an area that is receiving much investment, and the goal is roughly to "keep it mostly working as long as it's not expensive". Primary goal is to allow editing files with an editor that is superior to what's in Kudu. But the git support is sadly in neglect.

So yes, clearly something related to the credential manager broke, and right now no one knows why. One candidate is that around January, we moved from Server 2012 to Server 2016, which maybe affected that at some level.

If someone feels like trying to figure it out, you can find all the Monaco sources under D:\Program Files (x86)\SiteExtensions\Monaco from Kudu Console. There is even a way you can run a modified version of it using a private site extension. It's a Node app that's running on an ancient version of Node (0.10.29).

Another thing worth mentioning is that we're about to move from 32 bit to 64 bit git on the VM, to deal with random git issues that have been plaguing us (see https://github.com/projectkudu/kudu/issues/2714 and https://github.com/git-for-windows/git/wiki/32-bit-issues). Not sure how that will affect Monaco at this point.

[davidebbo]

Looking at the process tree after attempting a push is quite telling:

So I think what's happening is that whatever logic Monaco is using to register with the credential manager (so it can put u its custom UI) is no longer working. As a result, it's probably falling back to prompting for creds on the git command line, which hangs because there is no interactive console to type things in.

So the challenge is to figure out why the credential manager custom UI got busted.

[asifip]

Hi everyone. I have more than a dozen of live webapps and I am facing the very same problem just when I enabled GIT on them. git push just doesn't work no matter how many different cmd lines I used. Since I am losing my sleep I had to reach out. Please do your best @jk-ncsc ,@davidebbo , et al. Meanwhile I would too think hard. Now that the UI is known to cause it. Tx

[davidebbo]

@asifip I think this may be a new issue, related to the transition to 64 bit git. If you go to the App Service Editor console (not Kudu Console) and just type git and enter, what happens?

[asifip]

Hi @davidebbo I ran cmd 'git' and it output a guidance content as seen in https://git-scm.com/docs/git.
Other than that I was able to perform push to 'Visual Studio Team Services GIT Repo' instead of GITHUB. During that step I received a UI prompt asking VSTS credentials. Is there a similar possibility for UI interface to do GITHUB? Please let me know if saving password in a file in webapp scm could work with GITHUB authentication during push requests.

[suwatch]

The cred hook is done thru credential.helper. App Service Editor adds this config in webapp's D:\home\data\monaco\home\.gitconfig.

\ [master]> git config --global --list
...
credential.helper=!"$HOME/bin/cred.sh"

The $HOME points to App Service Editor's home which is d:\Program Files (x86)\SiteExtensions\Monaco\1.0.0-20180530\server\home. The issue seems to be the git-credential-manager does not take credential.helper as precedent over its built in auths for GitHub\Bitbucket.. etc. If I remove either git-credential-manager.exe or GitHub.Authentication.exe from d:\Program Files (x86)\Git\mingw32\libexec\git-core, then credential.helper will kick it. We are checking with dev owning to see what is the cleaner way and why it was working before.

[asifip]

Microsoft's acquiring GitHub. :+1:
How does this affect them users? Incredibly positively I presume. Anyhoo! expecting a great deal out of this.

[whoisj]

@suwatch why is this being done?

credential.helper=!"$HOME/bin/cred.sh"

The Git Credential Manager for Windows (aka GCM) is bin placed into the mingw64/libexec/git-core/ folder of the Git for Windows (64-bit) layout. This means that credential-manager is a verb that git.exe can use.

Better to use credential.helper = manager, and let Git linkup the parts itself. OR you can bypass the git-credential subsystem by using git credential-manager as your command.

The issue seems to be the git-credential-manager does not take credential.helper as precedent over its built in auths for GitHub\Bitbucket.. etc.

Um... the GCM doesn't even come into play in the decision of what to use. Git decides what to use and what not to use, the GCM is but a utility in this situation. If you're seeing that other helpers are being used instead of the one you're hoping for, it is quite likely that however Git is being invoked is explicitly instructing Git to do so.

[suwatch]

This is App Service Editor where git operations are provided via Web Application. I added more info you asked on https://github.com/Microsoft/Git-Credential-Manager-for-Windows/issues/649.

To address issue at hand, there are two workarounds mentioned in https://github.com/Microsoft/Git-Credential-Manager-for-Windows/issues/649 and they both worked great for me. So please try it and let us know.

As for overriding the modal dialog, sure. Use environment variable GCM_MODAL_PROMPT=0 or git config credential.modalPrompt false.

Note: For environment solution, do add AppSettings to Web App instead.

[davidebbo]

Thanks @suwatch, this provides a very simple workaround for this problem: go to your Web App's App Settings and add GCM_MODAL_PROMPT=0. And everything should start working again :)

Others, can you verify?

[jk-ncsc]

@davidebbo Very pleased to confirm that the suggested workaround does work for me (Add Application Setting GCM_MODAL_PROMPT of 0 to Web App, and refresh ASE).

ASE is one of the most underrated features of the Azure Portal for me - so thank you David and to everyone else involved for the workaround and keeping this thread alive.

[davidebbo]

Going to fix this so it's set by default in App Service Editor. Worth doing a one liner for this :)

[davidebbo]

Good news: the fix has now been deployed, so it should all work with you needing to do anything special. Please do verify.

[LaloCo]

@davidebbo you saved me with that GCM_MODAL_PROMPT=0, that is indeed how I manage to solve this. I don't understand exactly what happened, literally was working 5 hours ago and the suddenly stopped, but this solved it, thanks!

[whoisj]

@davidebbo you saved me with that GCM_MODAL_PROMPT=0, that is indeed how I manage to solve this. I don't understand exactly what happened, literally was working 5 hours ago and the suddenly stopped, but this solved it, thanks!

There are a number of factors at play. Some of them expire after a few hours (site logon) which an impact the GCM ability to non-interactively authenticate.