Aws-lambda-dotnet: Support `libdl` on AWS Lambda with .NET Core

[augustoproiete]

I have an ASP .NET Core Web API that uses an OSS library to generate files, and this library makes use of System.Drawing.

It runs fine locally, but, but when deployed to an AWS Serverless app, produces the error below:

System.DllNotFoundException: Unable to load shared library 'libdl' or one of its dependencies

System.TypeInitializationException: The type initializer for 'Gdip' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'libdl' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibdl: cannot open shared object file: No such file or directory
   at IntPtr Interop+Libdl.dlopen(string fileName, int flag)
   at IntPtr System.Drawing.SafeNativeMethods+Gdip.LoadNativeLibrary()
   at static System.Drawing.SafeNativeMethods+Gdip()
   --- End of inner exception stack trace ---
   at int System.Drawing.SafeNativeMethods+Gdip.GdipGetGenericFontFamilySansSerif(out IntPtr fontfamily)
   at IntPtr System.Drawing.FontFamily.GetGdipGenericSansSerif()
   at FontFamily System.Drawing.FontFamily.get_GenericSansSerif()
   at void System.Drawing.Font.CreateFont(string familyName, float emSize, FontStyle style, GraphicsUnit unit, byte charSet, bool isVertical)

[klaytaybai]

Hi @caioproiete, could you please try some of the guidance suggested in https://stackoverflow.com/questions/50615704/unable-to-load-dll-libdl-when-using-system-drawing-common-nuget-package-on-aws? I think the CoreCompat.System.Drawing solution might work. Let us know how this goes.

[augustoproiete]

@klaytaybai Thanks for taking a look at this one.

I came across that SO post before opening this issue and using the packages CoreCompat.System.Drawing.v2 and runtime.linux-x64.CoreCompat.System.Drawing and it made no difference... I still get the same error.

[sarkarstanmoy]

I am also using same issue. I have followed https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package/ article without much success. I have created layer with all the dependent so packages including libgdiplus.so.0 but not working

[aguschirico]

Any news on this issue? I'm also interested in any possible solution. I've also tried creating a layer with the corecompat linux runtime without success.

[jon144]

Any update on this problem? I am experiencing the same thing running an AWS Lambda function using .Net Core 3.1

[agarcian]

This issue is still happening. With .Net Core 3.1 and an AWS Lambda still fails.

[ash-manos]

Also failing for me on .NET Core 3.1 within AWS Lambda.

@klaytaybai or any other AWS peeps - is there an update on this? It's preventing us from upgrading to .NET Core 3.1.

[rv-dtomaz]

I finally found a way to get it running on .Net Core 3.1 Lambda.

[aguschirico]

Could you explain how?

[rv-dtomaz]

Sure @aguschirico , I forgot to commit the entire response.
I will detail all steps if anyone wants to do it.
I have stared a bash on amazonlinux image.
So:

#1) Install libgdiplus
Before this step I have installed using yum to get all packages installed with libgdiplus.

yum install -y yum-utils rpmdevtools

rpmkeys --import "http://pool.sks-keyservers.net/pks/lookup?op=get&search=0x3fa7e0328081bff6a14da29aa6a19b38d3d831ef"

curl https://download.mono-project.com/repo/centos7-stable.repo | tee /etc/yum.repos.d/mono-centos7-stable.repo

Download RPM packages

yumdownloader libgdiplus0.x86_64 acl.x86_64 cracklib.x86_64 dbus.x86_64 dejavu-sans-fonts.noarch elfutils-default-yama-scope.noarch fontpackages-filesystem.noarch gzip.x86_64 kmod.x86_64 libSM.x86_64 libXau.x86_64 libXfixes.x86_64 libcap-ng.x86_64 libfdisk.x86_64 libglvnd-glx.x86_64 libpng.x86_64 libsmartcols.x86_64 libwayland-client.x86_64 libxshmfence.x86_64 mesa-libGL.x86_64 pam.x86_64 shadow-utils.x86_64 ustr.x86_64 audit-libs.x86_64 cracklib-dicts.x86_64 dbus-libs.x86_64 device-mapper.x86_64 elfutils-libs.x86_64 freetype.x86_64 hwdata.x86_64 kmod-libs.x86_64 libX11.x86_64 libXdamage.x86_64 libXrender.x86_64 libdrm.x86_64 libglvnd.x86_64 libjpeg-turbo.x86_64 libpwquality.x86_64 libtiff.x86_64 libwayland-server.x86_64 lz4.x86_64 mesa-libgbm.x86_64 pixman.x86_64 systemd.x86_64 util-linux.x86_64 cairo.x86_64 cryptsetup-libs.x86_64 dejavu-fonts-common.noarch device-mapper-libs.x86_64 fontconfig.x86_64 giflib.x86_64 jbigkit-libs.x86_64 libICE.x86_64 libX11-common.noarch libXext.x86_64 libXxf86vm.x86_64 libexif.x86_64 libglvnd-egl.x86_64 libpciaccess.x86_64 libsemanage.x86_64 libutempter.x86_64 libxcb.x86_64 mesa-libEGL.x86_64 mesa-libglapi.x86_64 qrencode-libs.x86_64 systemd-libs.x86_64

Extract Packages

rpmdev-extract -C ./libgdi-runtime *rpm

Copy dependencies to a shared volume

cp ./libgdi-runtime/*/usr/lib64/* /var/task/lib

This step take me some hours to figure out. We need to copy fontconfig fonts to shared volume too

cp -avr /etc/fonts/ /var/task/lib/etc

cp -avr /usr/share/fonts /var/task/lib/share

Edit file to fix fonts path

Edit /var/task/lib/etc/fonts.conf setting the path to fonts like this:

<dir>/var/task/lib/share/fonts</dir>

Zip all files on lib folder and create a lambda layer

Remeber to set this layer to your lambda

*Set Environment variable in your lambda *

FONTCONFIG_PATH=/opt/lib/etc

It is done!

[rv-dtomaz]

Lambda layer ready to use is here: https://github.com/rv-dtomaz/lambda-layer-system-drawing

[aguschirico]

@rv-dtomaz your solution worked for me. Thanks!

[rv-dtomaz]

Glad to know !

[odqp]

Thanks for your help @rv-dtomaz

I'm getting the next error:

START RequestId: 62b77ba7-9a1a-49dd-90be-74318b942576 Version: $LATEST
/var/runtime/bootstrap: error while loading shared libraries: liblambdaio.so: cannot open shared object file: No such file or directory

could it be related to the environment variable?

[rv-dtomaz]

@odqp
I think that it is related.
Remove this variable LD_LIBRARY_PATH=/opt/lib from Lambda environment and let me know if the error continues

[odqp]

@rv-dtomaz , thanks for your quick response.
Without LD_LIBRARY_PATH=/opt/lib, I have again the error about the libraries.

[rv-dtomaz]

@odqp Can you send the entire stack of the error ?

[odqp]

Sure,
this is the stack:

{
"errorType": "TypeInitializationException",
"errorMessage": "The type initializer for 'DevExpress.XtraPrinting.BrickStyle' threw an exception.",
"stackTrace": [
"at DevExpress.XtraPrinting.BrickStyle..ctor()",
"at DevExpress.XtraReports.UI.XRControlStyle..ctor()",
"at DevExpress.XtraReports.UI.XRControl.Initialize(RectangleF bounds)",
"at DevExpress.XtraReports.UI.XRControl..ctor()",
"at DevExpress.XtraReports.UI.Band..ctor()",
"at DevExpress.XtraReports.UI.XtraReportBase..ctor()",
"at DevExpress.XtraReports.UI.XtraReport..ctor()",
"at AWSLambdaReportingPOC.Reports.TestReport..ctor() in C:\Users\Developer\source\repos\AWSLambdaReportingPOC\AWSLambdaReportingPOC\Reports\TestReport.cs:line 10",
"at AWSLambdaReportingPOC.Function.Print() in C:\Users\Developer\source\repos\AWSLambdaReportingPOC\AWSLambdaReportingPOC\Function.cs:line 34",
"at lambda_method(Closure , Stream , Stream , LambdaContextInternal )"
],
"cause": {
"errorType": "TypeInitializationException",
"errorMessage": "The type initializer for 'Gdip' threw an exception.",
"stackTrace": [
"at System.Drawing.SafeNativeMethods.Gdip.GdipGetGenericFontFamilySansSerif(IntPtr& fontfamily)",
"at System.Drawing.FontFamily.GetGdipGenericSansSerif()",
"at System.Drawing.FontFamily.get_GenericSansSerif()",
"at System.Drawing.Font.CreateFont(String familyName, Single emSize, FontStyle style, GraphicsUnit unit, Byte charSet, Boolean isVertical)",
"at System.Drawing.Font..ctor(String familyName, Single emSize, FontStyle style, GraphicsUnit unit, Byte gdiCharSet, Boolean gdiVerticalFont)",
"at System.Drawing.Font..ctor(String familyName, Single emSize)",
"at DevExpress.XtraPrinting.BrickStyle..cctor()"
],
"cause": {
"errorType": "DllNotFoundException",
"errorMessage": "Unable to load shared library 'libdl' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibdl: cannot open shared object file: No such file or directory",
"stackTrace": [
"at Interop.Libdl.dlopen(String fileName, Int32 flag)",
"at System.Drawing.SafeNativeMethods.Gdip.LoadNativeLibrary()",
"at System.Drawing.SafeNativeMethods.Gdip..cctor()"
]
}
}
}

[rv-dtomaz]

@odqp Thanks!
What is your .NET Core Version ?
I got libdl Error using .Net Core 2.1.
The solution above works just for .NET COre 3.1

[odqp]

Same, .NET Core 3.1
this is from the AWS lambda UI:

I'm thinking if I did something wrong with the layer creation.

[rv-dtomaz]

@odqp Maybe you solution needs other libs than that on my package.
I will try to reproduce it here.
Is it possible to share some sample to make easy to reproduce?

[rv-dtomaz]

@odqp I didn`t find a way to reproduce this error.
If you could share your solution (clean all sensitive data, send just pieces of code required do reproduce the error)

[odqp]

Thanks for your support @rv-dtomaz

here is a solution sample
AWSLambdaReportingPOC.zip

[aguschirico]

I was getting that error with setting LD_LIBRARY_PATH but I got rid of it
as soon as I remove it.
However, I can't make it work with Gembox.Document as it says it can't find
any fonts in the os and neither accepts me setting any path as described in
this example
https://www.gemboxsoftware.com/document/examples/private-fonts/103
I'm now trying to figure it out, but couldn't yet. If anyone has similar
issues or any clue on how to fix it please let me know.

El jue., 20 ago. 2020 a las 22:26, rv-dtomaz ([e-mail])
escribió:

@odqp https://github.com/odqp I didn`t find a way to reproduce this
error.
If you could share your solution (clean all sensitive data, send just
pieces of code required do reproduce the error)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-lambda-dotnet/issues/516#issuecomment-677885490,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAVW6ME3OLP4UFOOFST4IUTSBWBGFANCNFSM4ISFTZWQ
.

[rv-dtomaz]

I was getting that error with setting LD_LIBRARY_PATH but I got rid of it as soon as I remove it. However, I can't make it work with Gembox.Document as it says it can't find any fonts in the os and neither accepts me setting any path as described in this example https://www.gemboxsoftware.com/document/examples/private-fonts/103 I'm now trying to figure it out, but couldn't yet. If anyone has similar issues or any clue on how to fix it please let me know. El jue., 20 ago. 2020 a las 22:26, rv-dtomaz ([e-mail]) escribió:
…
@odqp https://github.com/odqp I didn`t find a way to reproduce this error. If you could share your solution (clean all sensitive data, send just pieces of code required do reproduce the error) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#516 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVW6ME3OLP4UFOOFST4IUTSBWBGFANCNFSM4ISFTZWQ .

@aguschirico , my bad sorry. When I have described my solution I asked to change LD_LIBRARY_PATH, how ever you must set FONTCONFIG_PATH=/opt/lib/etc to set fonts folder to that on Lambda layer. Remeber to set this ENVIRONMENT on lambda

[rv-dtomaz]

Thanks for your support @rv-dtomaz

here is a solution sample
AWSLambdaReportingPOC.zip

I will try to reproduce this error

[aguschirico]

@aguschirico , my bad sorry. When I have described my solution I asked to change LD_LIBRARY_PATH, how ever you must set FONTCONFIG_PATH=/opt/lib/etc to set fonts folder to that on Lambda layer. Remeber to set this ENVIRONMENT on lambda

Setting FONTCONFIG_PATH doesn't seem to work for me. The lambda still doesn't find the fonts. Perhaps the path is wrong?

[odqp]

Thanks for your support @rv-dtomaz
here is a solution sample
AWSLambdaReportingPOC.zip

I will try to reproduce this error

Thank you @rv-dtomaz , I have been looking for a solution, but I have not had good luck yet.

[SenthilRG]

@aguschirico, Please update "FONTCONFIG_PATH" value as "opt/lib/share/fonts".

[odqp]

Hi @rv-dtomaz

I have the next scenario:
I have been working on this on a docker container to have my project working on a local environment.
As a reference, I'm using this docker image: https://hub.docker.com/r/lambci/lambda. I can see this it the image recommended reproduce lambda scenarios.
I'm creating the layer using this image, and it is working now on my local environment. But when I try on AWS using the same layer just created, it's not working.
Next is the layer I created:
layer.zip

Could you please take a look at this layer and check if I'm missing something? (maybe an environment variable or wrong folder structure)

next is the error that I'm getting:

{
"errorType": "TypeInitializationException",
"errorMessage": "The type initializer for 'DevExpress.XtraPrinting.BrickStyle' threw an exception.",
"stackTrace": [
"at DevExpress.XtraPrinting.BrickStyle..ctor()",
"at DevExpress.XtraReports.UI.XRControlStyle..ctor()",
"at DevExpress.XtraReports.UI.XRControl.Initialize(RectangleF bounds)",
"at DevExpress.XtraReports.UI.XRControl..ctor()",
"at DevExpress.XtraReports.UI.Band..ctor()",
"at DevExpress.XtraReports.UI.XtraReportBase..ctor()",
"at DevExpress.XtraReports.UI.XtraReport..ctor()",
"at AWSLambdaReportingPOC.Reports.TestReport..ctor() in C:\Users\Developer\Downloads\AWSLambdaReportingPOC-Chino\AWSLambdaReportingPOC\AWSLambdaReportingPOC\Reports\TestReport.cs:line 10",
"at AWSLambdaReportingPOC.Function.Print() in C:\Users\Developer\Downloads\AWSLambdaReportingPOC-Chino\AWSLambdaReportingPOC\AWSLambdaReportingPOC\Function.cs:line 28",
"at lambda_method(Closure , Stream , Stream , LambdaContextInternal )"
],
"cause": {
"errorType": "TypeInitializationException",
"errorMessage": "The type initializer for 'Gdip' threw an exception.",
"stackTrace": [
"at System.Drawing.SafeNativeMethods.Gdip.GdipGetGenericFontFamilySansSerif(IntPtr& fontfamily)",
"at System.Drawing.FontFamily.GetGdipGenericSansSerif()",
"at System.Drawing.FontFamily.get_GenericSansSerif()",
"at System.Drawing.Font.CreateFont(String familyName, Single emSize, FontStyle style, GraphicsUnit unit, Byte charSet, Boolean isVertical)",
"at System.Drawing.Font..ctor(String familyName, Single emSize, FontStyle style, GraphicsUnit unit, Byte gdiCharSet, Boolean gdiVerticalFont)",
"at System.Drawing.Font..ctor(String familyName, Single emSize)",
"at DevExpress.XtraPrinting.BrickStyle..cctor()"
],
"cause": {
"errorType": "DllNotFoundException",
"errorMessage": "Unable to load shared library 'libgdiplus' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibgdiplus: cannot open shared object file: No such file or directory",
"stackTrace": [
"at System.Drawing.SafeNativeMethods.Gdip.GdiplusStartup(IntPtr& token, StartupInput& input, StartupOutput& output)",
"at System.Drawing.SafeNativeMethods.Gdip..cctor()"
]
}
}
}

Thanks!

[odqp]

As update, I can see I just have this issue when I use a RichText elements from devexpress reporting .net core.

[rv-dtomaz]

Hi @odqp , sorry for delay. Bad days due this pandemy.
I will check and back to you soon.
Please put a code on your lambda to run a Process that runs ldd libgdiplus , and print the results. This will give us if any dependency is missing.

```var cmd = "ldd ";
var escapedArgs = cmd.Replace("\"", "\\"");

        var process = new Process()
        {
            StartInfo = new ProcessStartInfo
            {
                FileName = "/bin/bash",
                Arguments = $"-c \"{escapedArgs}\"",
                RedirectStandardOutput = true,
                UseShellExecute = false,
                CreateNoWindow = true,
            }
        };
        process.Start();
        string result = process.StandardOutput.ReadToEnd();
        process.WaitForExit();
        Console.WriteLine(result);```

[ashishdhingra]

Hi @augustoproiete,

Good afternoon.

I was going through the issue backlog and came across this one. Looking at the comments, @rv-dtomaz proposed workaround appears to work for most users. Please confirm if this is still an issue or if it could be closed.

Thanks,
Ashish

[augustoproiete]

Hey @ashishdhingra. It's been over a year since I first reported this issue and I ended up giving up on using AWS Lambda with .NET Core altogether, thus I no longer have an easy way to reproduce the above scenario.

Feel free to close this issue if you think the workaround above is good enough for other users.

[aguschirico]

It wasn't enough for me but I can't blame the suggested solution. I'm trying to make an specific library (Gembox) to work in lambda and the issue with the fonts keeps blocking me from moving forward no matter what I do, so I can't confirm or deny the solution works. Thanks for the support anyway.

[MichaelKovich]

It wasn't enough for me but I can't blame the suggested solution. I'm trying to make an specific library (Gembox) to work in lambda and the issue with the fonts keeps blocking me from moving forward no matter what I do, so I can't confirm or deny the solution works. Thanks for the support anyway.

In case anyone stumbles upon this in the future, I was able to find a solution:

(1) Implement the lambda layer solution discussed here.

(2) Add your font file to your project and ensure that it is copied to your output directory.

(3) In the code for your lambda function, read the bytes of your font file and write them to /tmp/filename.ttf

(4) Set your GemBox FontSettings.FontBaseDirectory to /tmp/

And you're good to go!