Godot: Linux server builds cannot instance nodes with mono C# scripts attached

[Classy11]

Godot version:
Godot Engine v3.2.2.stable.mono.official

OS/device including version:
Building on Windows 10 64 bit with visual studio 2019 installed
Building for Linux/Debian10, with Linux Server Binary, Mono software installed.

Issue description:
When I export any project to Linux from my windows PC with the official Linux server binary downloaded from the Godot website I keep on getting the exact same error when I run the server with all my (test/reproduction) projects:
ERROR: can_instance: Method failed. Returning: __null At: modules/mono/csharp_script.cpp:2939.

My projects run perfectly fine when exporting for Windows but for Linux server builds, the projects work like halfway, for all Linux server build projects that I export and use the server binary with GDScript and other nodes without C# scripts attached are present and functional, but all nodes with C# scripts attached just aren't instanced and not present in the server on run-time.

I looked through the Godot engine code and it is this that is returning the error:
// The project assembly is not loaded ERR_FAIL_V_MSG(false, "Cannot instance script because the project assembly is not loaded. Script: '" + get_path() + "'.");

So the project assembly is not loaded apparently? How would I go about fixing this for my projects?

To be clear, no project I work with works when exporting it to Linux with the server binary. Brand new fresh projects return the same error as the real projects I work on when trying to export and run them on Linux as a server.

Related issues:

34765 (This is for the editor with a project name change and the solutions for this also did not work for me)

38268 (This issue does not explicitly focus on this specific error and the OP had his issue resolved but none of the mentioned fixes there works for my case)

I'm pretty new to C# setups and my package and assembly management knowledge is beginner level.

Thank you for your time.

Steps to reproduce:

1. Create a new project on Windows 10
2. Add any node to the scene tree]
3. Attach a mono script with GD.Print("Hello world") in the ready method.
4. Export for Linux
5. Download official Godot server binary and paste it over the exported binary.
6. Transfer files to the Linux machine and run the project and watch the error appear on launch.

Minimal reproduction project:

linuxBugProject.zip

[Classy11]

If anyone is interested, the error persists with 3.2.3RC3 Mono with the RC3 Linux server binary. But still works when exporting for Windows.

[opl-]

The fix is to export the project for the X11 platform without the Debug option, i.e. in release mode (bottom of the file selection dialog shown after clicking "Export Project"). This causes the exported .pck file to use Release in its paths, allowing the server build to find the assemblies.

This is caused by two factors:

• the Mono export plugin changes the export paths based on whether it's a debug build or not,
• the Mono module includes a path in the search_dirs list by guessing what the path will be based on the config of the build itself.

Alternatively, if you need the information you get from having debug builds, you can build the server yourself, specifying platform=server target=release_debug, though in that case it seems from a quick test that you'll need to figure out which BCL (Base Class Library) DLLs you need to use to avoid getting ERROR: _load_api_assemblies: The assembly 'GodotSharp' is out of sync. errors.

---

Notes I took as I was looking into this issue. Figured I might as well post them to show how I came to this solution:

Running the server binary with the --verbose flag prints some helpful logs that reveal that at least in my case the problem is Godot not being able to find mscorlib.dll: Mono: Skipping runtime initialization because 'mscorlib.dll' could not be found (source)

GDMonoAssembly::find_assembly is (only ever?) used to search for the mscorlib.dll assembly, which in turn checks these directories on my Debian: ".../issue-40805-mono-assembly/exports/data_linuxBugProject/Mono/lib/mono/4.5", ".../issue-40805-mono-assembly/exports/data_linuxBugProject/Mono/lib/mono/4.5/Facades", ".../issue-40805-mono-assembly/exports/data_linuxBugProject/Assemblies", "res://.mono/temp/bin/ExportRelease", "res://.mono/assemblies/Release", "res://.mono/assemblies", "", ".../issue-40805-mono-assembly/exports" (where ... is the path to the project's exports directory from which I'm running the server binary named linuxBugProject.mono)

Unfortunately setting PATH to include the system's Mono directory doesn't seem to change anything.

Since mscorlib.dll is not included in the libraries exported with the project nor with the server build, the condition fails. Disabling the condition using find_assembly results in the following error (coming from Mono), which includes a path matching that returned by GodotSharpDirs::get_data_mono_lib_dir():

The assembly mscorlib.dll was not found or could not be loaded.
It should have been installed in the `.../issue-40805-mono-assembly/exports/data_linuxBugProject/Mono/lib/mono/4.5/mscorlib.dll' directory.

I tried to see which directories Mono considers using their debug logging, but unfortunately the flags didn't seem to have any effect, so I tried digging in the Mono source instead.

The log we get from method includes a path which as it turns out is set by Godot here to determine that path. However, that path is not the only place Mono looks for the corlib assembly. Mono actually calls mono_assembly_load_corlib when trying to load the mscorlib.dll, which in turn calls a user set hook which Godot implements here. It uses the basically the same algorithm and paths as GDMonoAssembly::find_assembly, which we already established is static.

mono_assembly_load_corlib also mentions in a comment that MONO_PATH or mono_set_assemblies_path can be used to specify the mscorlib.dll directory. I tried setting MONO_PATH. It crashed with ERROR: _load_api_assemblies: FATAL: Method failed. At: modules/mono/mono_gd/gd_mono.cpp:967.: source.

So let's go back to the search_dirs list which is initialized here. We once again see the appearance of mono_assembly_getrootdir() that Mono used for that log message, which this time can optionally be overridden. Seems the override is used for the editor to figure out the assembly dependencies during export. Hold on, the export plugin references the BCL ("Base Class Library"). Sure enough, it's right there in the export templates for Mono under bcl/net_4_x/. Is the export plugin not copying the libraries on export as it should? Is it disabled by a flag?

The BCL used for server and X11 should both be the same, so why don't we see them in the export directory? The project setting mono/export/export_assemblies_inside_pck seems to default to true, so maybe that's where they are? Sure enough, quick export as zip later I can see it in .mono/assemblies/Debug/mscorlib.dll together with some other dlls like the project's linuxBugProject.dll.

Is this the problem? The search_dirs list includes res://.mono/assemblies/Release, but not res://.mono/assemblies/Debug, which this project is using. Using a hex viewer, the .pck seems to include only Debug paths too...

Bingo!

[Classy11]

@opl- Wow, exporting it to Linux without debugging works. Thank you very much for posting the solution here, appreciate the effective digging you have done! Great work!

[Classy11]

In case of anyone wondering, I can confirm that exported Godot Mono 3.2.3 projects for Linux Servers with debug tools still cannot instance nodes with C# scripts attached. Without debug tools however, it's working well.