Xamarin-android: [Bug] application deploy on Android 11 emulator fails with Mono.AndroidTools.AdbException: secure_mkdirs failed: Permission denied

[pmahend1]

Description

Last week upgraded Visual Studio 2019(Professional) which shipped Android 11 SDK updates
I changed the target SDK in both manifest and project application properties.
When I try to deploy the app I get the error. This error doesnt show up for Android Pie and Q emulators

Steps to Reproduce

1. Update Visual Studio 2019 to 16.7
2. Update all android SDKs
3. Create an x86 emulator with Android 11(R)
4. Build and Run Android app selecting Android 11 emulator

Expected Behavior

App should deploy and run successfully.

Actual Behavior

Severity    Code    Description Project File    Line    Suppression State
Error       ADB1000:  Deployment failed
Xamarin.AndroidTools.AndroidDeploymentException: FailedToSynchronizeFastDevAssemblies ---> Mono.AndroidTools.AdbException: secure_mkdirs failed: Permission denied
   at Mono.AndroidTools.Util.AggregateAsyncResult.CheckError(CancellationToken token)
   at Mono.AndroidTools.Adb.AdbSyncClient.EndPushSyncItems(IAsyncResult result)
   at Mono.AndroidTools.Internal.AdbClientTaskExtensions.InnerInvoke(TaskCompletionSource`1 tcs, Func`2 endMethod, IAsyncResult l)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Xamarin.AndroidTools.AndroidDeploySession.<InstallAssemblies>d__121.MoveNext()
   --- End of inner exception stack trace ---
   at Xamarin.AndroidTools.AndroidDeploySession.<InstallAssemblies>d__121.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Xamarin.AndroidTools.AndroidDeploySession.<FastDevAsync>d__118.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Xamarin.AndroidTools.AndroidDeploySession.<RunAsync>d__110.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Xamarin.AndroidTools.AndroidDeploySession.<RunLoggedAsync>d__108.MoveNext()           0   

Basic Information

• Version with issue: Xamarin Forms 4.8.0.XXXX
• Last known good version: Not sure
• IDE: Visual Studio 2019 Professional
• Platform Target Frameworks:
  
  
  
  • iOS: N/A
  
  
  • Android: Android 11
  
  
  • UWP: N/A
  
  
• Android Support Library Version:
• Nuget Packages: Xamarin Forms 4.8.0.XXXX Xamarin.Android.Support 28.0.0.3
• Affected Devices:

Screenshots

Reproduction Link

Workaround

Turn Use Fast Deployment Off
Reference : https://forums.xamarin.com/discussion/11383/deployment-failed-due-to-an-error-in-fastdev-assembly-synchronization

[pmahend1]

Update

I noticed this exception goes away if we turn Use Fast Deployment off

So this can be thought as a workaround

[pmahend1]

And I could find related Visual Studio bug as raised in community as well

https://developercommunity.visualstudio.com/content/problem/1075071/xamarinandroid-deployment-often-fails.html

[dellis1972]

@pmahend1 can you provide a full diagnostic deployment log? It would be interesting to see where its trying to fast deploy the assemblies to. I suspect we no longer have write permissions to the directory we try to use for fast deployment.

The easiest way to get use a deployment log is to turn fast dev back on, un install the app from the device/emulator then do the following from the Visual Studio command prompt.

msbuild <your android project.csproj> /t:Install /bl

This will provide a msbuild.binlog file which we can then investigate.

You can also capture this from the IDE, but you'll need to make sure that the log output is set to diagnostic and that you capture the deployment, not the build log as the build log won't have the data we need.

[pmahend1]

@dellis1972 logs have been shared with you over email.

[dellis1972]

@pmahend1 thanks for the logs.

Ok so the odd thing is this directory it is trying to create /storage/emulated/0/Android/data/com.companyname.app5/files/.__override__ is a perfectly valid directory.
It looks like Android R has changed the external storage system/capabilities https://developer.android.com/preview/privacy/storage. Looks like apps with no longer be able to use the external storage directory.

There are some commands to try , this for example will enable management of external storage.

adb shell appops set --uid PACKAGE_NAME MANAGE_EXTERNAL_STORAGE allow

You could also try adding preserveLegacyExternalStorage to your manifest. see if that helps. Although you probably don't want to do that for release apps.

We are working on a new Fast Deployment system which will be compatible with Android R, but that is not slated for release until 16.9 (16.8 is the one currently in preview).

[Mertsch]

Caution: After you update your app to target Android 11 (API level 30),
the system ignores the requestLegacyExternalStorage attribute when your app is running on Android 11 devices,
so your app must be ready to support scoped storage and to migrate app data for users on those devices.

https://developer.android.com/training/data-storage/use-cases

[pmahend1]

@Mertsch Thanks . yes, looks like setting requestLegacyExternalStorage doesnt do any good for apps running on Android 11. Looks like its only for devices with API=<29 which have been built with API 30.

[dellis1972]

More investigation https://developer.android.com/reference/android/os/Environment#getExternalStorageDirectory().

This method was deprecated in API level 29.
To improve user privacy, direct access to shared/external storage devices is deprecated. When an app targets Build.VERSION_CODES.Q, the path returned from this method is no longer directly accessible to apps. Apps can continue to access content stored on shared/external storage by migrating to alternatives such as Context#getExternalFilesDir(String), MediaStore, or Intent#ACTION_OPEN_DOCUMENT.

So when targeting Q, the external storage directory will not work.

This will be addressed in PR https://github.com/xamarin/xamarin-android/pull/4690 with the new Fast Deployment system which does not use the external storage directory.

The best thing to do at this time is to disable Fast Deployment when targeting Android Q :(.

[dellis1972]

@pmahend1 So I've been doing some more investigation. I have a version of our shared runtime which used newer api's to get the external storage path.
If you have time would you mind testing out the following packages [1]. Its a zip file which contains all the Mono.Android.DebugRuntime-*.apk files. you will need to extract them and replace the ones in your d16-7 installation directory. Be sure to backup the old ones just in case.

The directory should be in C:\Program Files (x86)\Microsoft Visual Studio\2019\<edition>\MSBuild\Xamarin\Android

Let me know if this works. If it does then I might be able to get this into the next d16-8 preview.

[1] https://1drv.ms/u/s!Amsoudv0Ow8JnXGGZbXp_-kPHtT9?e=v2qEZc

[pmahend1]

@dellis1972 sure. Will do and get back to you.

[pmahend1]

@dellis1972 I did a brief testing with new binaries you have provided. Seems its working well. Even Hot Reload seems to work fine.
I think this can be released in Visual Studio preview.

One more question, will it same for Visual Studio for Mac as well. Will you add it ti VS Mac Preview channel release?

[dellis1972]

@pmahend1 ok that's good to hear. I'll sort out a PR. I suspect however that we have missed the next Preview cut off I'm afraid. But I will try to get this on the next available release.

[brendanzagaeski]

_Release status update_

A new Preview version of Xamarin.Android has now been published that includes the fix for this item. The fix is not yet included in a Release version. I will update this item again when a Release version is available that includes the fix.

Fix included in Xamarin.Android SDK version 11.1.0.3.

Fix included on Windows in Visual Studio 2019 version 16.8 Preview 3. To try the Preview version that includes the fix, check for the latest updates in Visual Studio Preview.

Fix included on macOS in Visual Studio 2019 for Mac version 8.8 Preview 3. To try the Preview version that includes the fix, check for the latest updates on the Preview updater channel.

[tranb3r]

@brendanzagaeski @pmahend1
I've just tested with VS 2019 16.8 pre3. It looks like the issue is still here.
Could you please confirm if this works for you ?

[pmahend1]

@tranb3r Yes. I downloaded preview and tested. Reading from Xamarin.Android 11 and Visual Studio preview notes doesnt seem like the changes were migrated in the last preview.

@brendanzagaeski @dellis1972 Could you please check and confirm?

[brendanzagaeski]

I can help answer the question of whether the specific fix that was provided in a test build earlier has been published. The answer to that is yes, the specific fix from that earlier test build is indeed included in Xamarin.Android SDK version 11.1.0.3 Preview, which is part of Visual Studio 2019 version 16.8 Preview 3 and Visual Studio 2019 for Mac version 8.8 Preview 3. The commit on the d16-8 development branch that provides that fix is https://github.com/xamarin/xamarin-android/commit/ff57bc83b1c2d5e74dc13ac8f192bbbbba079dc9.

This issue (#4996) is accordingly listed under the _Issues fixed_ section in the Xamarin.Android 11.1 Preview release notes on docs.microsoft.com and on GitHub

Next steps

It sounds like the particular fix provided in this issue might not have resolved the problem for all devices. If any users are still seeing the problem when using Visual Studio 2019 version 16.8 Preview 3 or higher or Visual Studio 2019 for Mac version 8.8 Preview 3 or higher, then the next best step would be to submit a new issue and attach or send the diagnostic deployment log to dellis1972 as mentioned in https://github.com/xamarin/xamarin-android/issues/4996#issuecomment-672766328. Thanks in advance!

[pmahend1]

@brendanzagaeski Thanks for clarification. I am checking on VS Mac as well now. Seems the error is slightly different, I will consolidate and create a new issue in sometime.

@tranb3r is the error log same as below for you?

Severity    Code    Description Project File    Line    Suppression State
Error       ADB1000: Mono.AndroidTools.AdbException: couldn't create file: Permission denied
   at Mono.AndroidTools.Util.AggregateAsyncResult.CheckError(CancellationToken token) in C:\A\1\104\s\External\androidtools\Mono.AndroidTools\Util\AggregateAsyncResult.cs:line 72
   at Mono.AndroidTools.Adb.AdbSyncClient.EndPushSyncItems(IAsyncResult result) in C:\A\1\104\s\External\androidtools\Mono.AndroidTools\Adb\AdbSyncClient.cs:line 1070
   at Mono.AndroidTools.Internal.AdbClientTaskExtensions.InnerInvoke(TaskCompletionSource`1 tcs, Func`2 endMethod, IAsyncResult l) in C:\A\1\104\s\External\androidtools\Mono.AndroidTools\Internal\AdbClientTaskExtensions.cs:line 256
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Xamarin.AndroidTools.AndroidDeploySession.<InstallAssemblies>d__121.MoveNext() in C:\A\1\104\s\External\androidtools\Xamarin.AndroidTools\Sessions\AndroidDeploySession.cs:line 614           0   

[tranb3r]

@pmahend1 @brendanzagaeski
I'm having the exact same error as described originally in this issue (using VS 2019 16.8-pre3) :
Xamarin.AndroidTools.AndroidDeploymentException: FailedToSynchronizeFastDevAssemblies ---> Mono.AndroidTools.AdbException: secure_mkdirs failed: Permission denied

I don't see any reason to open a new ticket.
Could you please re-open this one, as the issue is obviously not fixed.

[dellis1972]

@tranb3r I don't think we can work around this. The "fix" was committed and shipped in d16-8 pre3. We are using the new api's on API 29+ devices, and if they are still throwing the error then I'm not sure how we work around this. Running without the shared runtime will probably be the only option

We have a PR which completely rewrites the entire fast deployment system #4690. Its a work in progress and we are hoping to target the d16-9 release with that.

In the mean time the best way to work around this is to use the following settings for you debug Configuration

<AndroidUseSharedRuntime>false</AndroidUseSharedRuntime>
<EmbedAssembliesIntoApk>true</EmbedAssembliesIntoApk>

This will disable fast deployment completely. You should still be able to debug its just all the files will be packaged into the apk.

[tranb3r]

@dellis1972 Thanks for the clarification.
Although I'm still confused about what's in the "fix" if this issue cannot be fixed...

[dellis1972]

@tranb3r we "thought" that updating the shared runtime to not use deprecated API's to access the external storage would get rid of this particular error. Apparently this was not the case. It might also be device/emulator related as well since I never managed to repo the issue reliably.

The best option moving forward is to not rely on the use of external storage, which is what PR #4690 will do.

[dellis1972]

We are going to re-open this issue. The fix will be the new fast deployment system on #4690. The work around for now is to not use the shared runtime or fast deployment.

[brendanzagaeski]

_Release status update_

A new Preview version of Xamarin.Android has now been published that includes the fix for this item.

Fix included in Xamarin.Android SDK version 11.1.99.168.

Fix included on Windows in Visual Studio 2019 version 16.9 Preview 1. To try the Preview version that includes the fix, check for the latest updates in Visual Studio Preview.

Fix included on macOS in Visual Studio 2019 for Mac version 8.9 Preview 1. To try the Preview version that includes the fix, check for the latest updates on the Preview updater channel.

[JR2Media]

Having the same problem with VS19 Enterprise (Version 16.8.4). Turning off fast debugging worked for me.

Build details:

Microsoft Visual Studio Enterprise 2019
Version 16.8.4
VisualStudio.16.Release/16.8.4+30907.101
Microsoft .NET Framework
Version 4.8.03752

Installed Version: Enterprise

Visual C++ 2019   00433-90004-23584-AA490
Microsoft Visual C++ 2019

ADL Tools Service Provider   1.0
This package contains services used by Data Lake tools

ASA Service Provider   1.0

ASP.NET and Web Tools 2019   16.8.557.25636
ASP.NET and Web Tools 2019

ASP.NET Core Razor Language Services   16.1.0.2052803+84e121f1403378489b842e1797df2f3f5a49ac3c
Provides languages services for ASP.NET Core Razor.

ASP.NET Web Frameworks and Tools 2019   16.8.557.25636
For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0   16.8.557.25636
Azure App Service Tools v3.0.0

Azure Data Lake Node   1.0
This package contains the Data Lake integration nodes for Server Explorer.

Azure Data Lake Tools for Visual Studio   2.6.1000.0
Microsoft Azure Data Lake Tools for Visual Studio

Azure Functions and Web Jobs Tools   16.8.557.25636
Azure Functions and Web Jobs Tools

Azure Stream Analytics Tools for Visual Studio   2.6.1000.0
Microsoft Azure Stream Analytics Tools for Visual Studio

C# Tools   3.8.0-5.20604.10+9ed4b774d20940880de8df1ca8b07508aa01c8cd
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Common Azure Tools   1.10
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Extensibility Message Bus   1.2.6 (master@34d6af2)
Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

Fabric.DiagnosticEvents   1.0
Fabric Diagnostic Events

IntelliCode Extension   1.0
IntelliCode Visual Studio Extension Detailed Info

Microsoft Azure HDInsight Azure Node   2.6.1000.0
HDInsight Node under Azure Node

Microsoft Azure Hive Query Language Service   2.6.1000.0
Language service for Hive query

Microsoft Azure Service Fabric Tools for Visual Studio   16.0
Microsoft Azure Service Fabric Tools for Visual Studio

Microsoft Azure Stream Analytics Language Service   2.6.1000.0
Language service for Azure Stream Analytics

Microsoft Azure Stream Analytics Node   1.0
Azure Stream Analytics Node under Azure Node

Microsoft Azure Tools   2.9
Microsoft Azure Tools for Microsoft Visual Studio 2019 - v2.9.30924.1

Microsoft Continuous Delivery Tools for Visual Studio   0.4
Simplifying the configuration of Azure DevOps pipelines from within the Visual Studio IDE.

Microsoft JVM Debugger   1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Microsoft Library Manager   2.1.113+g422d40002e.RR
Install client-side libraries easily to any web project

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual C++ Wizards   1.0
Microsoft Visual C++ Wizards

Microsoft Visual Studio Tools for Containers   1.1
Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.

Microsoft Visual Studio VC Package   1.0
Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio   16.8.43 (00471f8)
Support for debugging Mono processes with Visual Studio.

NuGet Package Manager   5.8.1
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

ProjectServicesPackage Extension   1.0
ProjectServicesPackage Visual Studio Extension Detailed Info

Snapshot Debugging Extension   1.0
Snapshot Debugging Visual Studio Extension Detailed Info

SQL Server Data Tools   16.0.62012.31170
Microsoft SQL Server Data Tools

Test Adapter for Boost.Test   1.0
Enables Visual Studio's testing tools with unit tests written for Boost.Test.  The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test   1.0
Enables Visual Studio's testing tools with unit tests written for Google Test.  The use terms and Third Party Notices are available in the extension installation directory.

ToolWindowHostedEditor   1.0
Hosting json editor into a tool window

TypeScript Tools   16.0.21016.2001
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools   3.8.0-5.20604.10+9ed4b774d20940880de8df1ca8b07508aa01c8cd
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools   16.8.0-beta.20507.4+da6be68280c89131cdba2045525b80890401defd
Microsoft Visual F# Tools

Visual Studio Code Debug Adapter Host Package   1.0
Interop layer for hosting Visual Studio Code debug adapters in Visual Studio

Visual Studio Container Tools Extensions   1.0
View, manage, and diagnose containers within Visual Studio.

Visual Studio Tools for CMake   1.0
Visual Studio Tools for CMake

Visual Studio Tools for Containers   1.0
Visual Studio Tools for Containers

Visual Studio Tools for Kubernetes   1.0
Visual Studio Tools for Kubernetes

Visual Studio Tools for Unity   4.8.2.0
Visual Studio Tools for Unity

VisualStudio.DeviceLog   1.0
Information about my package

VisualStudio.Foo   1.0
Information about my package

VisualStudio.Mac   1.0
Mac Extension for Visual Studio

Xamarin   16.8.000.261 (d16-8@bb99248)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer   16.8.0.507 (remotes/origin/d16-8@e87b24884)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin Templates   16.8.112 (86385a3)
Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms.

Xamarin.Android SDK   11.1.0.26 (d16-8/a36ce73)
Xamarin.Android Reference Assemblies and MSBuild support.
    Mono: 5e9cb6d
    Java.Interop: xamarin/java.interop/d16-8@79d9533
    ProGuard: Guardsquare/proguard/proguard6.2.2@ebe9000
    SQLite: xamarin/sqlite/3.32.1@1a3276b
    Xamarin.Android Tools: xamarin/xamarin-android-tools/d16-8@2fb1cbc


Xamarin.iOS and Xamarin.Mac SDK   14.8.0.3 (c51fabee8)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

[saamerm]

A Summary for anyone interested:

Even though changes to this were made and merged into master in October 2020, they didn't make it to the November Production release, and so they are still in Preview.
You don't need to switch update channels to preview in your Visual Studio, just go to the Xamarin.Android Github, and tap on the "Commercial Xamarin.Android 11.2 (d16-9) Preview for ..." link to download the package to install the version with the amazing changes made, and your app should just start building (super fast) with no changes!

[maxkoshevoi]

go to the Xamarin.Android Github, and tap on the "Commercial Xamarin.Android 11.2 (d16-9) Preview for ..." link to download

Did that and my localization logic broke. https://github.com/xamarin/xamarin-android/issues/5659
Looks like it's in preview for a reason 😄

[Paul-DotNET]

Hi guys, is this bug fixed in any specific VS 2019 for Windows release? I'm facing this issue in VS 2019 v16.8.4

[dellis1972]

@Paul-DotNET Try 16.9.4 we completely re-wrote the Fast Deployment system for that release.

[Paul-DotNET]

Thank you @dellis1972 ! As per my understanding, this issue (related to Fast Deployment) is only for the debug configuration, correct ? If so, then I shouldn't have any problems with my app in release configuration right?

If so, I don't want to upgrade Visual Studio to v16.9.4 now and take risk at the time of launch. Your suggestion please ? Thanks

[Paul-DotNET]

Just wanted to know if I can uncheck Fast deployment for now, target my app to Android 11 and then deploy to play store without upgrading VS2019 to v16.9.4

[dellis1972]

This issue does only effect debug and has no effect on release.

[Paul-DotNET]

Sounds good, thanks 👍