Ml-agents: IL2CPP build fails

Created on 19 Sep 2019 · 10Comments · Source: Unity-Technologies/ml-agents

Describe the bug
IL2CPP build fails in Unity 2018.4.9f1

To Reproduce
Steps to reproduce the behavior:
Download and import ML-Agents .9 import unity package into project
Edit->Project Settings->Player-> Scripting Backend_>IL2CPP

Console logs / stack traces
Please wrap in [triple backticks ()](https://help.github.com/en/articles/creating-and-highlighting-code-blocks) to make it easier to read.Failed running C:\Program Files\Unity\Hub\Editor\2018.4.9f1\Editor\Data\il2cpp/build/il2cpp.exe --convert-to-cpp --emit-null-checks --enable-array-bounds-check --dotnetprofile="unityaot" --compile-cpp --libil2cpp-static --platform="WindowsDesktop" --architecture="x64" --configuration="Release" --outputpath="C:\Users\Documents\ML Test 4\Temp/StagingArea/Data\Native\GameAssembly.dll" --cachedirectory="C:\Users\Documents\ML Test 4\Assets..\Library/il2cpp_cache" --map-file-parser="C:\Program Files\Unity\Hub\Editor\2018.4.9f1\Editor\Data\Tools\MapFileParser\MapFileParser.exe" --directory="C:\Users\Documents\ML Test 4\Temp\StagingArea\Data\Managed" --generatedcppdir="C:\Users\Documents\ML Test 4\Temp\StagingArea\Data\il2cppOutput"

stdout:
Building GameAssembly.dll with MsvcDesktopToolChain
Msvc Install Version: 15.0
Msvc Install SDK Directory: C:\Program Files (x86)\Windows Kits\10
Msvc Linker Path: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\link.exe
Msvc Compiler Path: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\cl.exe

Output directory: C:\Users\\Documents\ML Test 4\Temp\StagingArea\Data\Native
Cache directory: C:\Users\\Documents\ML Test 4\Library\il2cpp_cache

ObjectFiles: 598 of which compiled: 149
Time Compile: 43679 milliseconds Bulk_mscorlib_5.cpp
Time Compile: 16536 milliseconds Bulk_System_1.cpp
Time Compile: 13679 milliseconds Bulk_Google.Protobuf_0.cpp
Time Compile: 10146 milliseconds Bulk_mscorlib_8.cpp
Time Compile: 9795 milliseconds Bulk_System.Core_3.cpp
Time Compile: 9783 milliseconds Il2CppAttributes.cpp
Time Compile: 9615 milliseconds Il2CppTypeDefinitions.cpp
Time Compile: 9419 milliseconds Bulk_mscorlib_2.cpp
Time Compile: 9033 milliseconds GenericMethods1.cpp
Time Compile: 8832 milliseconds Bulk_Generics_18.cpp
Total compilation time: 104664 milliseconds.
il2cpp.exe didn't catch exception: Unity.IL2CPP.Building.BuilderFailedException: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\link.exe /out:"C:\Users\Documents\ML Test 4\Library\il2cpp_cache\linkresult_88786DB1151802B90B7F04768E94A75B\GameAssembly.dll" /DEBUG /INCREMENTAL:NO /LARGEADDRESSAWARE /NXCOMPAT /DYNAMICBASE /NOLOGO /TLBID:1 /OPT:REF /OPT:ICF /HIGHENTROPYVA /DLL /NODEFAULTLIB:uuid.lib "kernel32.lib" "user32.lib" "advapi32.lib" "ole32.lib" "oleaut32.lib" "Shell32.lib" "Crypt32.lib" "psapi.lib" "version.lib" "MsWSock.lib" "ws2_32.lib" "Iphlpapi.lib" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.17763.0\um\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.17763.0\ucrt\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" /LIBPATH:"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\lib\x64" /SUBSYSTEM:WINDOWS @"C:\Users\AppData\Local\Temp\tmp9D7B.tmp"

Creating library C:\Users\Documents\ML Test 4\Library\il2cpp_cache\linkresult_88786DB1151802B90B7F04768E94A75B\GameAssembly.lib and object C:\Users\Documents\ML Test 4\Library\il2cpp_cache\linkresult_88786DB1151802B90B7F04768E94A75B\GameAssembly.exp
62FF50AF98AA89ADF49D98BEBF2725B4.obj : error LNK2019: unresolved external symbol dlopen referenced in function Mono_dlopen_m99E4CF2695C06AB46A063D3973F134EE95112719
62FF50AF98AA89ADF49D98BEBF2725B4.obj : error LNK2019: unresolved external symbol dlsym referenced in function Mono_dlsym_m203A90F70D2F6211B298FFFB55FE63DC5B0BEA3F
C:\Users\Documents\ML Test 4\Library\il2cpp_cache\linkresult_88786DB1151802B90B7F04768E94A75B\GameAssembly.dll : fatal error LNK1120: 2 unresolved externals

at Unity.IL2CPP.Building.CppProgramBuilder.PostprocessObjectFiles(HashSet`1 objectFiles, CppToolChainContext toolChainContext)
at Unity.IL2CPP.Building.CppProgramBuilder.Build(IBuildStatistics& statistics)
at il2cpp.Program.DoRun(String[] args)
at il2cpp.Program.Run(String[] args)
at il2cpp.Program.Main(String[] args)
stderr:

Unhandled Exception: Unity.IL2CPP.Building.BuilderFailedException: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\link.exe /out:"C:\Users\Documents\ML Test 4\Library\il2cpp_cache\linkresult_88786DB1151802B90B7F04768E94A75B\GameAssembly.dll" /DEBUG /INCREMENTAL:NO /LARGEADDRESSAWARE /NXCOMPAT /DYNAMICBASE /NOLOGO /TLBID:1 /OPT:REF /OPT:ICF /HIGHENTROPYVA /DLL /NODEFAULTLIB:uuid.lib "kernel32.lib" "user32.lib" "advapi32.lib" "ole32.lib" "oleaut32.lib" "Shell32.lib" "Crypt32.lib" "psapi.lib" "version.lib" "MsWSock.lib" "ws2_32.lib" "Iphlpapi.lib" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.17763.0\um\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.17763.0\ucrt\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" /LIBPATH:"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\lib\x64" /SUBSYSTEM:WINDOWS @"C:\Users\AppData\Local\Temp\tmp9D7B.tmp"

UnityEngine.Debug:LogError(Object)
UnityEditorInternal.Runner:RunProgram(Program, String, String, String, CompilerOutputParserBase) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:128)
UnityEditorInternal.Runner:RunManagedProgram(String, String, String, CompilerOutputParserBase, Action1) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:73) UnityEditorInternal.IL2CPPBuilder:RunIl2CppWithArguments(List1, Action1, String) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:370) UnityEditorInternal.IL2CPPBuilder:ConvertPlayerDlltoCpp(String, String, String, Boolean) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:351) UnityEditorInternal.IL2CPPBuilder:Run() (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:211) UnityEditorInternal.IL2CPPUtils:RunIl2Cpp(String, IIl2CppPlatformProvider, Action1, RuntimeClassRegistry) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:42)
DesktopStandalonePostProcessor:SetupStagingArea(BuildPostProcessArgs, HashSet`1) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:245)
DesktopStandalonePostProcessor:PostProcess(BuildPostProcessArgs) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:46)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

```Exception: C:\Program Files\Unity\Hub\Editor\2018.4.9f1\Editor\Data\il2cpp/build/il2cpp.exe did not run properly!
UnityEditorInternal.Runner.RunProgram (UnityEditor.Utils.Program p, System.String exe, System.String args, System.String workingDirectory, UnityEditor.Scripting.Compilers.CompilerOutputParserBase parser) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:130)
UnityEditorInternal.Runner.RunManagedProgram (System.String exe, System.String args, System.String workingDirectory, UnityEditor.Scripting.Compilers.CompilerOutputParserBase parser, System.Action`1[T] setupStartInfo) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildUtils.cs:73)
UnityEditorInternal.IL2CPPBuilder.RunIl2CppWithArguments (System.Collections.Generic.List`1[T] arguments, System.Action`1[T] setupStartInfo, System.String workingDirectory) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:370)
UnityEditorInternal.IL2CPPBuilder.ConvertPlayerDlltoCpp (System.String inputDirectory, System.String outputDirectory, System.String workingDirectory, System.Boolean platformSupportsManagedDebugging) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:351)
UnityEditorInternal.IL2CPPBuilder.Run () (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:211)
UnityEditorInternal.IL2CPPUtils.RunIl2Cpp (System.String stagingAreaData, UnityEditorInternal.IIl2CppPlatformProvider platformProvider, System.Action`1[T] modifyOutputBeforeCompile, UnityEditor.RuntimeClassRegistry runtimeClassRegistry) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/Il2Cpp/IL2CPPUtils.cs:42)
DesktopStandalonePostProcessor.SetupStagingArea (UnityEditor.Modules.BuildPostProcessArgs args, System.Collections.Generic.HashSet`1[T] filesToNotOverwrite) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:245)
DesktopStandalonePostProcessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/DesktopStandalonePostProcessor.cs:46)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

Environment (please complete the following information):

Windows 10
_ML-Agents version_: ML-Agents .9
_Environment_: Empty scene

bug

Source

wightwhale

Most helpful comment

Issue still exist with ML-Agents .12

GabHook on 22 Jan 2020

👍2

All 10 comments

Can confirm. Also happened in 2019.1.14f

MHDante on 27 Sep 2019

The issue seems to be around gRPC's declaring of an [DllImport("__Internal")] native function. This is elaborated on this forum post:
https://forum.unity.com/threads/solved-ml-agents-break-when-using-il2cpp-as-scripting-backend.577075/

The issue seems to be somewhat anknowledged by the gRPC team. I believe that it was adressed for Android on these PRs:
https://github.com/grpc/grpc/pull/18039
https://github.com/grpc/grpc/pull/18327

I'm not sure that GRPC is set to be IL2CPP compiled for windows, and it's possible that it will require additional stubs. Perhaps @jtattermusch can comment more on this. I'll see if this issue happens on Android as well.

MHDante on 27 Sep 2019

This solution in the post works for building Windows IL2CPP, however if you build for Linux (Mono) it'll compile but the library will say it can't communicate with Unity.

UnityAgentsException: The Communicator was unable to connect. Please make sure the External process is ready to accept communication with Unity.
  at MLAgents.Batcher.SendAcademyParameters (MLAgents.CommunicatorObjects.UnityRLInitializationOutput academyParameters) [0x0002d] in <82ac4562db4940c9b5f1b06d4a9be1a1>:0                  at MLAgents.Academy.InitializeEnvironment () [0x00228] in <82ac4562db4940c9b5f1b06d4a9be1a1>:0
  at MLAgents.Academy.Awake () [0x00000] in <82ac4562db4940c9b5f1b06d4a9be1a1>:0

Rolling back the changes makes Linux Mono work again.

AcelisWeaven on 27 Sep 2019

So the issue with your workaround is that the fake stubs for dlopen and dlsym should only be provided on windows (I understood that providing them on windows fixes the build). On Linux, they would mask the real dlopen and dlsym, which obviously breaks things.

jtattermusch on 1 Oct 2019

Adding that the same behavior happens on a Mac when targeting IL2CPP.

surfnerd on 3 Oct 2019

I've logged this bug in our internal tracking system as ID MLA-94. We will update this issue as we gather more information. Thank you for reporting this!

surfnerd on 3 Oct 2019

@surfnerd
Any news on this issue? It would be amazing to have il2cpp working properly when training with complex environments.