Arcade: Invalid ELF BuildID '<null>' for runtime in Maestro Build Promotion

Created on 27 Oct 2020 · 26Comments · Source: dotnet/arcade

[ ] This issue is blocking
[ ] This issue is causing unreasonable pain
[x] This issue gives the appearance of a failure

Link to failing pipeline: https://dnceng.visualstudio.com/internal/_build/results?buildId=866684&view=logs&j=43d2a893-8a64-53c8-f353-9d032c280600&t=567cfe83-9037-5882-498b-95841a0c95da

D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-arm/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-arm/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-arm/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-arm/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-arm64/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-arm64/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-arm64/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-arm64/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-x64/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-x64/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-x64/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-x64/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-x86/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-x86/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/android-x86/native/libmonosgen-2.0.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/android-x86/native/libmonosgen-2.0.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-musl-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-musl-x64/native/libcoreclr.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-musl-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-musl-x64/native/libcoreclr.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.20520.8\tools\SdkTasks\PublishToSymbolServers.proj(101,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) Invalid ELF BuildID '<null>' for runtimes/linux-x64/native/libcoreclr.so

Runtime pipeline that kicked off this build: https://dev.azure.com/dnceng/internal/_build/results?buildId=865594&view=results

Source

missymessa

Most helpful comment

For your third point, we have https://github.com/dotnet/arcade/issues/6403 already filed and being looked at.

What we could do in the interim is detect the result of the promotion build. Then it would say: "the promotion build failed, but the build was still assigned to the target channels". Technically it might not be an error, so you could emit a warning. That would make this case more visible until we fix the root cause (that the symbol publishing isn't integrated into regular publishing)

riarenas on 28 Oct 2020

👍2

All 26 comments

@epananth is this related to issue arcade #6449?

ilyas1974 on 27 Oct 2020

@ilyas1974 I don't think they are related. These are failures seen when using the symbolUploader task, not during the publishing of blobs to storage accounts.

riarenas on 27 Oct 2020

Looking into 3 aspects of this:

[ ] Why do those files have an Invalid ELF BuildID and when did it start?
[ ] Do we care that the symbols were not published?
[ ] Should darc at least warn in the build that kicks off the promotion build that not all jobs completed successfully? Something like "The build was successfully added to the target channel, but there were errors in the promotion build"

adiaaida on 28 Oct 2020

👍1

For your third point, we have https://github.com/dotnet/arcade/issues/6403 already filed and being looked at.

What we could do in the interim is detect the result of the promotion build. Then it would say: "the promotion build failed, but the build was still assigned to the target channels". Technically it might not be an error, so you could emit a warning. That would make this case more visible until we fix the root cause (that the symbol publishing isn't integrated into regular publishing)

riarenas on 28 Oct 2020

👍2

Awesome, thanks Ricardo. I think that covers both 2 & 3. I'll look into the first bullet point.

adiaaida on 28 Oct 2020

This is the first build where we start seeing these failures: https://dnceng.visualstudio.com/internal/_build/results?buildId=863910&view=results.

Based simply on the name of the files that have errors and the fact that the related commit touches mono files, I have a strong suspicion that this this a product bug. Not sure how changing paths would affect build ids in ELF files though.

Nope. I am wrong. My Kusto query failed me.

adiaaida on 28 Oct 2020

Ok, THIS is the first time we saw it for runtime: https://dnceng.visualstudio.com/internal/_build/results?buildId=850796&view=results

Associated runtime build: https://dev.azure.com/dnceng/internal/_build/results?buildId=850462&view=results

adiaaida on 28 Oct 2020

And this is the first time we saw it for more than just mono files: https://dnceng.visualstudio.com/internal/_build/results?buildId=860848&view=results

Which corresponds to this runtime build: https://dev.azure.com/dnceng/internal/_build/results?buildId=860691&view=results

adiaaida on 28 Oct 2020

@mikem8361 I'm having a hard time understanding runtime's build process. Do you know what adds the BuildId to ELF/symbol files? In particular, this is only failing for libmonosgen-2.0.so on various architectures and libcoreclr.so on linux-x64 and linux-musl-x64.

Any details on this process that you have would be super helpful!

adiaaida on 28 Oct 2020

There linker options to enable the build id. All the coreclr binaries should have this enabled (libcoreclr.so on any architecture/platform) so I don’t know why you are getting that message for those. The libmonogen-2.0.so (and other mono binaries) need this option too. I know nothing about the mono build to help.

@janvorli might know of the top of his head what the linker option is.

mikem8361 on 28 Oct 2020

I think all of these are actually Mono runtime files, it's just that for Desktop builds we need to rename libmonosgen-2.0.so to libcoreclr.so (fixing that is tracked with https://github.com/dotnet/runtime/issues/34202).

I know the reason for the issue, will fix it.

akoeplinger on 28 Oct 2020

I opened a PR with a fix.

@adiaaida can we log the name of the .nupkg that caused the error in PublishToSymbolServers.proj? That'd have made it easier to figure out.

akoeplinger on 28 Oct 2020

This recent promotion for dotnet/runtime passed so we should be good: https://dnceng.visualstudio.com/internal/_build/results?buildId=869933&view=results

akoeplinger on 29 Oct 2020

Thanks @akoeplinger! I will look into logging the nupkg when we fail

adiaaida on 29 Oct 2020

It looks like the only place to add additional logging is in dotnet-symuploader.

@mikem8361 is the best way to go about testing this to pass in the --dry-run flag and using the symuploader command line tool?

adiaaida on 29 Oct 2020

Yes, that would be a good way to do it. The “-vd” option enable verbose logging.

mikem8361 on 29 Oct 2020

PR: https://dev.azure.com/dnceng/internal/_git/dotnet-symuploader/pullrequest/11080

adiaaida on 30 Oct 2020

This change has been checked in. As the rest of the work mentioned in this issue already has its own issue, closing.

adiaaida on 30 Oct 2020

Arcade will need it's version of type symbol uploader build task updated to 1.1.153002.

mikem8361 on 30 Oct 2020

We would be getting this update automatically on Monday, but I triggered the subscription manually to expedite this a little: https://github.com/dotnet/arcade/pull/6498

riarenas on 30 Oct 2020

Looks like there were unintended consequences to the error I introduced. I'll look into it.

adiaaida on 30 Oct 2020

I should have catch this. Tracer.Error will fail the build and CreateFileInfo does return null for valid cases that are not errors.

mikem8361 on 30 Oct 2020

Yeah. I also don't want to inundate the user with warnings, but when there ARE errors, we want to let the user know which package the error came from.

Let's back out this change for now, as there's no reason to have broken code checked in.

adiaaida on 30 Oct 2020

Revert PR: https://dev.azure.com/dnceng/internal/_git/dotnet-symuploader/pullrequest/11082

adiaaida on 30 Oct 2020

Triage: The main things in this issue are either already fixed (the error in runtime) or already are being tracked by the publishing epic (making sure we warn/error the starting build when the symbols upload fails). I think the logging change is going to require a much more complicated change to how we handle CreateFileInfo that I think is out of scope of FR, given that null sometimes means that there was an error, but other times just means we've already got that file in the list of file infos.

adiaaida on 3 Nov 2020

Triage: We think we have everything tracked and completed. Closing.