Source-build: Errors while building on 16.04 Xenial Xerus

[sergiusens]

This is what I get with a fresh clone (first time I attempt a build as well) with a call to ./build.sh

"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:3) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:5) ->
(Build target) -> 
  EXEC : error : Could not read profile: Unsupported format version [/home/sergiusens/dotnet-source-build/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/targets/repository.proj(72,5): error MSB3073: The command "/home/sergiusens/dotnet-source-build/src/coreclr/build.sh x64 Release skiptests -PortableBuild=false " exited with code 1.


"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:15) ->
  /home/sergiusens/dotnet-source-build/targets/repository.proj(72,5): error MSB3073: The command "/home/sergiusens/dotnet-source-build/src/websdk/build-core.sh" exited with code 126.

    553 Warning(s)
    3 Error(s)

I was successful in the past in just building coreclr by calling ./build.sh on its own from the repo though.

[omajid]

Can you include more of the log? Maybe upload it somewhere? In my experience, the real cause is often hidden in the middle of thousands of lines of noise.

Also, can you building coreclr using build.sh x64 Release skiptests -PortableBuild=false? Does that lead to any errors?

[sergiusens]

Here's a gist for both a regular build of coreclr and one with VERBOSE=1 added to the make command.

And just in case, this is where the code is standing:

sergiusens@builder:~/dotnet-source-build/src/coreclr⟫ git log -1
commit 9679ded412d1421f3d54ee3082e8b5f900217a4a
Author: Mike McLaughlin <[email protected]>
Date:   Wed Jun 7 07:20:55 2017 -0700

    Fix portable build sos plugin problems. (#12130)

    Removing the explicit reference to liblldb. Since the lldb program has
    already loaded this lib, our will now load regardless of the distro.

    Issue #12098.

[sergiusens]

I have a successful build with an unpatched coreclr on

commit bb1c4a48f9c6d49d5c75628394b7c822f96c57be
Merge: 20de3a9 acfcd31
Author: Brian Sullivan <[email protected]>
Date:   Tue Aug 8 18:22:34 2017 -0700

    Merge pull request #13276 from dotnet-bot/from-tfs

    Merge changes from TFS

But not with an unpatched

sergiusens@builder:~/dotnet-source-build/src/coreclr⟫ git log -1
commit 9679ded412d1421f3d54ee3082e8b5f900217a4a
Author: Mike McLaughlin <[email protected]>
Date:   Wed Jun 7 07:20:55 2017 -0700

    Fix portable build sos plugin problems. (#12130)

    Removing the explicit reference to liblldb. Since the lldb program has
    already loaded this lib, our will now load regardless of the distro.

    Issue #12098.

[sergiusens]

If I take coreclr to a new commit

sergiusens@builder:~/dotnet-source-build⟫ git diff src/coreclr/
diff --git a/src/coreclr b/src/coreclr
index 9679ded..bb1c4a4 160000
--- a/src/coreclr
+++ b/src/coreclr
@@ -1 +1 @@
-Subproject commit 9679ded412d1421f3d54ee3082e8b5f900217a4a
+Subproject commit bb1c4a48f9c6d49d5c75628394b7c822f96c57be-dirty

I get a different error

[crummel]

Hi Sergio,
For the first error you got, error: Could not read profile: Unsupported format version:
Your BuildVersion-<VersionNumber>.props might be missing or malformed - I've seen this happen when there was an issue running git during the build. You could also build with /v:diag and search for CreateVersionInfoFile in the output. I think the second CMakeFiles/Makefile2:1733: recipe for target 'src/jit/standalone/CMakeFiles/clrjit.dir/all' failed error on that gist might be the same issue with some added CMake weirdness from the rebuild attempt.

The inaccessible due to its protection level errors when you update CoreCLR look like that version had some incompatible changes - that commit is a lot newer, maybe try something closer to 9679ded like 02b76a06a63221eaed1b3f7d5fd99ddd4071706b?

[sergiusens]

With regards to the commit being newer I was just trying to make sure my build environment was sane.

If 02b76a06a63221eaed1b3f7d5fd99ddd4071706b is the right commit to be on, let me hop on to that and build with /v:diag

[sergiusens]

Here's the log for the full build with coreclr on 02b76a06a63221eaed1b3f7d5fd99ddd4071706b
log.txt

There is no mention of CreateVersionInfoFile in there.

[sergiusens]

Oh and btw,

sergiusens@builder:~/dotnet-source-build⟫ cat src/coreclr/bin/obj/BuildVersion-20170607-01.props

produces:

<?xml version="1.0" encoding="utf-8"?>
<!-- This is a generated file.  Seed Date is 20170607-01. -->
<Project ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <VersionPropsImported>true</VersionPropsImported>
    <BuildNumberMajor Condition="'$(BuildNumberMajor)' == ''">25407</BuildNumberMajor>
    <BuildNumberMinor Condition="'$(BuildNumberMinor)' == ''">01</BuildNumberMinor>
    <LatestCommit Condition="'$(LatestCommit)' == ''">9679ded412d1421f3d54ee3082e8b5f900217a4a</LatestCommit>
    <BuiltByString Condition="'$(BuiltByString)' == ''"> built by: sergiusens-builder</BuiltByString>
  </PropertyGroup>
</Project>

My naive eyes cannot seem to find a problem.

[crummel]

Yeah, that looks fine to me. I'm going through your log and setting up a repro now.

[crummel]

There's not much useful in the log, I've got a build going now.

[crummel]

I got a new error: /bin/sh: 2: /tmp/tmp1d2d2cc4abda4dc1b5726983d5a750d5.exec.cmd: /home/chris/source-build/src/websdk/build-core.sh: Permission denied. Chmod'ing this and restarting the build.

[sergiusens]

I have a sense you got further than me as I don't have EXEC on that script either and never reached it.

[sergiusens]

I see this as per our conversation in some of the xmls

130 sergiusens@builder:~/dotnet-source-build/src/msbuild? git diff

diff --git a/NuGet.Config b/NuGet.Config
index 4385d98..fc4cd5f 100644
--- a/NuGet.Config
+++ b/NuGet.Config
@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8"?>
+<U+FEFF><?xml version="1.0" encoding="utf-8"?>
 <configuration>
   <packageRestore>
     <add key="enabled" value="True" />
@@ -7,6 +7,7 @@
        Should be removed if restore does not regress-->
   <packageSources>
     <clear />
+    <add key="source-built" value="/home/sergiusens/dotnet-source-build/bin/obj/x64/Release/source-built/" />
     <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
     <add key="nuget.org v2" value="https://nuget.org/api/v2/" />
     <add key="myget.org roslyn nightly" value="https://dotnet.myget.org/F/roslyn/api/v3/index.json" />
@@ -16,4 +17,4 @@
     <add key="myget.org aspnet vnext v3" value="https://www.myget.org/F/aspnetvnext/api/v3/index.json" />
     <add key="myget.org aspnet vnext" value="https://www.myget.org/F/aspnetvnext/" />
   </packageSources>
-</configuration>
+</configuration>
\ No newline at end of file
diff --git a/src/Shared/Constants.cs b/src/Shared/Constants.cs
index b58b0d4..29c8f49 100644
--- a/src/Shared/Constants.cs
+++ b/src/Shared/Constants.cs
@@ -78,7 +78,7 @@ namespace Microsoft.Build.Shared
         internal const string CurrentAssemblyVersion = Microsoft.VisualStudio.Internal.BrandNames.VSGeneralAssemblyVersion;
 #endif

-        internal const string CurrentAssemblyFileVersion = "15.3.0.0";
+        internal const string CurrentAssemblyFileVersion = "15.3.378.6360";

         /// <summary>
         /// Current version of this MSBuild Engine assembly in the form, e.g, "12.0"

[crummel]

Ok, the EXEC : error : Could not read profile: Unsupported format version is due to:
-fprofile-instr-use=/home/chris/source-build/packages/optimization.linux-x64.pgo.coreclr/2.0.0-release-20170531-3000/data/coreclr.profdata in the compile command. Looking into this further...

[sergiusens]

On mié, 9 ago, 2017 at 7:46 PM, Chris Rummel
[e-mail] wrote:

Ok, the EXEC : error : Could not read profile: Unsupported format
version is due to:
-fprofile-instr-use=/home/chris/source-build/packages/optimization.linux-x64.pgo.coreclr/2.0.0-release-20170531-3000/data/coreclr.profdata
in the compile command. Looking into this further...

Thanks for that find!

[sergiusens]

FWIW, I tried building coreclr at that commit with not patches applied and still get error: Could not read profile: Unsupported format version

[omajid]

Would it be possible to try a different version of llvm/clang? I wonder if this will fix things. I understood that PGO data is not compatible between different versions of compilers (let alone between different compilers) so I am not sure how this works, generally.

[crummel]

Nice call, using clang-3.9 works. Now to figure out how to get the generated Makefile to use 3.9 instead of 3.5.

[sergiusens]

On jue, 10 ago, 2017 at 2:27 PM, Chris Rummel
[e-mail] wrote:

Nice call, using clang-3.9 works. Now to figure out how to get the
generated Makefile to use 3.9 instead of 3.5.

Seems most of the work was done in #773, we just need to add 3.9

[sergiusens]

I was eager to reply on that last one, its already there.

[crummel]

Yeah, I just added clang3.9 to https://github.com/dotnet/source-build/blob/b914e2fa16f35fe5b58fb18795872ac7c97030be/targets/coreclr.props#L5 and that seems to be working, the build is in CoreFX now

[sergiusens]

On jue, 10 ago, 2017 at 4:47 PM, Chris Rummel
[e-mail] wrote:

Yeah, I just added clang3.9 to
https://github.com/dotnet/source-build/blob/b914e2fa16f35fe5b58fb18795872ac7c97030be/targets/coreclr.props#L5
and that seems to be working, the build is in CoreFX now

Nice, I've finally reached this now, which we saw already:

  /home/sergiusens/dotnet-source-build/targets/repository.proj(72,5): 
error MSB3073: The command 
"/home/sergiusens/dotnet-source-build/src/websdk/build-core.sh" exited 
with code 126.

By creating a patch, will link soon

[sergiusens]

Hm, my dumb patch breaks 14.04 builds, while I could see the logs due to permissions I did get redirected to what seems to be a 14.04 url

[crummel]

Yeah, that looks like the problem.

15:15:38   Commencing CoreCLR Repo build
15:15:38   Setting up directories for build
15:15:38   Checking prerequisites...
15:15:38   Please install clang-3.9 before running this script

I'm not sure that it would work without this though, that PR build hasn't passed since June 9th. I'm not sure clang 3.9 is available on 14.04 without adding a package source, I'll have to ask someone who knows about the Jenkins images to see what we can do there.

[sergiusens]

I don't think so either, I'll make the patch smarter and pick 3.9 as a default on 16.04 only

[sergiusens]

arm is failing and the PR doesn't even get close to arm as it is a different code path, can't confirm, but I am highly suspicious in being correct :stuck_out_tongue:

[crummel]

Yeah, that one is a very early issue with the docker image:

15:14:51 + ./arm-ci.sh arm ./build.sh /p:Platform=arm /p:Configuration=Release /p:IsJenkinsBuild=true
15:14:52 Unable to find image 'ARM_Release_prtest:/opt/code:latest' locally
15:14:52 Error response from daemon: unable to ping registry endpoint https://ARM_Release_prtest:/v0/
15:14:52 v2 ping attempt failed with error: Get https://ARM_Release_prtest:/v2/: dial tcp: lookup ARM_Release_prtest: no such host
15:14:52  v1 ping attempt failed with error: Get https://ARM_Release_prtest:/v1/_ping: dial tcp: lookup ARM_Release_prtest: no such host

[sergiusens]

I've also created #98 to fix that chmod issue, my local build hasn't completed to verify it works, but I think it should.

[crummel]

This is the fix I'm testing out for the chmod issue: https://github.com/crummel/dotnet_source-build/commit/0703b939be49c37368d170aec4c198dc8c0057f9. I think yours is changing the Windows .cmd file instead of the Linux one.

[sergiusens]

oh, you are indeed right! But wouldn't it be better to create the file with the right permissions instead of changing it in the build?

[crummel]

It would, but I think it's supposed to already going by the patch file (unless I'm misunderstanding git file modes). I'm just trying to brute-force it for now.

[sergiusens]

yeah, I just checked, something is reverting it during the build...

diff --git a/build-core.sh b/build-core.sh
new file mode 100755
index 0000000..4aaec8c

The 755 there is indeed what we want (octal for owner/group/others rwxr-xr-x)

[omajid]

I have a fix for build-core.sh and a few other fixes I needed for Fedora 26 here: https://pagure.io/fedora-dotnet/tree/f26. This is against Preview 2 version of this repo. Not sure how many patches were redundant after Preview 2.

[sergiusens]

Nice, well would be nice that the permission would be kept, so @crummel this will end up in the form of patches/websdk/XXXX-...patch right?

[sergiusens]

I updated my branch on clang, but haven't tested yet (doing that now).

[crummel]

If we find a websdk fix for the permissions issue that would be a patch, if we stick with just the targets file that's in the source-build repo so we can update it directly.

[sergiusens]

@crummel sorry again, didn't pay attention to your target fix. I'll cherry-pick that.

Byt the way, my latest update to #97 works fine on 16.04. Did the ci spit out anything different that the clang-3.9 package not being found?

[crummel]

Yes, now we've got:

17:07:59   mscorlib.forwards.cs(9,73): error CS0122: 'CultureAwareComparer' is inaccessible due to its protection level [/mnt/j/workspace/Private/dotnet_source-build/master/dotnet_source-build_Ubuntu14.04_Release_prtest/src/corefx/src/shims/manual/mscorlib.csproj] [/mnt/j/workspace/Private/dotnet_source-build/master/dotnet_source-build_Ubuntu14.04_Release_prtest/targets/repository.proj]
17:07:59   mscorlib.forwards.cs(10,73): error CS0122: 'OrdinalComparer' is inaccessible due to its protection level [/mnt/j/workspace/Private/dotnet_source-build/master/dotnet_source-build_Ubuntu14.04_Release_prtest/src/corefx/src/shims/manual/mscorlib.csproj] [/mnt/j/workspace/Private/dotnet_source-build/master/dotnet_source-build_Ubuntu14.04_Release_prtest/targets/repository.proj]

It looks like the versions are still mismatched. I think https://github.com/karajas/source-build/commit/c45e0f1c8398e5a54090c36980a8f91b85291c5b has all the versions set correctly but it does have an FSharp issue that Karthik is still working on, I'm trying our fixes out with those versions now.

[sergiusens]

I've got much further than before with the clang version change and your chmod project patch, still failing though:

Done executing task "Exec". Done building target "Build" in project "repository.proj". Target "Package" skipped, due to false condition; ('$(BuildPackagesCommand)' != '') was evaluated as ('' != ''). Target "CopyPackage" in project "/home/sergiusens/dotnet-source-build/targets/repository.proj" (target "BuildRepositoryAndDependencies" depends on it): Task "Copy" Copying file from "/home/sergiusens/dotnet-source-build/src/cli-deps-satellites/bin/Release/CliDeps.Satellites.FSharp.4.4.1-pre-20170712-01.nupkg" to "/home/sergiusens/dotnet-source-build/bin/obj/x64/Release/source-built/CliDeps.Satellites.FSharp.4.4.1-pre-20170712-01.nupkg". Copying file from "/home/sergiusens/dotnet-source-build/src/cli-deps-satellites/bin/Release/CliDeps.Satellites.Roslyn.2.3.0-pre-20170712-01.nupkg" to "/home/sergiusens/dotnet-source-build/bin/obj/x64/Release/source-built/CliDeps.Satellites.Roslyn.2.3.0-pre-20170712-01.nupkg". Done executing task "Copy". Done building target "CopyPackage" in project "repository.proj". Target "WriteVersions" in project "/home/sergiusens/dotnet-source-build/targets/repository.proj" (target "BuildRepositoryAndDependencies" depends on it): Using "WriteVersionsFile" task from assembly "/home/sergiusens/dotnet-source-build/tasks/Microsoft.DotNet.SourceBuild.Tasks/bin/Debug/netstandard1.5/Microsoft.DotNet.SourceBuild.Tasks.dll". Task "WriteVersionsFile" Done executing task "WriteVersionsFile". Done building target "WriteVersions" in project "repository.proj". Target "BuildRepositoryAndDependencies" in project "/home/sergiusens/dotnet-source-build/targets/repository.proj" (entry point): Done building target "BuildRepositoryAndDependencies" in project "repository.proj". Done Building Project "/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)). Done executing task "MSBuild" -- FAILED. Done building target "BuildRepositoryReferences" in project "repository.proj" -- FAILED. Done Building Project "/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target(s)) -- FAILED. Done executing task "MSBuild" -- FAILED. Done building target "Build" in project "build.proj" -- FAILED. Done Building Project "/home/sergiusens/dotnet-source-build/build.proj" (default targets) -- FAILED.

Going to dig through the logs for a bit and see what this failure is all about

[sergiusens]

Ah, this is just a side-effect of removing clang-3.5 from my system and corefx failing due to that

[sergiusens]

Oh, I missed that previous comment about https://github.com/karajas/source-build/commit/c45e0f1c8398e5a54090c36980a8f91b85291c5b
I'll wait on that then.

[crummel]

https://matell.blob.core.windows.net/cli-src/dotnet-2.0.0-11.tar.gz is the tarball produced from that - it still has an F# issue but C# is working if that's enough to get you unblocked.

[sergiusens]

It would be a great start, yes, but given that that commit is built up on others, do you mind giving me a diff of what you ended up applying?

[crummel]

Here's the diff: https://gist.github.com/crummel/0b482c67ac3482e3dbd73b71f7794c21

[sergiusens]

Oh wait, I thought you applied changes from karajas/source-build@c45e0f1
We should be mostly good then.

[sergiusens]

@crummel while I wait for the build, let me ask how did you get by

17:07:59   mscorlib.forwards.cs(9,73): error CS0122: 'CultureAwareComparer' is inaccessible due to its protection level
17:07:59   mscorlib.forwards.cs(10,73): error CS0122: 'OrdinalComparer' is inaccessible due to its protection level [/mnt/j/workspace/Private/dotnet_source-build/master/dotnet_source-build_Ubuntu14.04_Release_prtest/src/corefx/src/shims/manual/mscorlib.csproj] [/mnt/j/workspace/Private/dotnet_source-build/master/dotnet_source-build_Ubuntu14.04_Release_prtest/targets/repository.proj]

?

As of last night I had a bunch of those. Is your diff missing anything applied from karajas/source-build@c45e0f1 or did you straight out switch/checkout fixOnMattBranch?

[crummel]

I just checked out fixOnMattBranch (FixingFsharp should also work as they have the same versions, and F# is included in that branch but still not working). You might have to reset your submodules (git submodule sync --recursive/git submodule update --recursive) if you're still getting those mismatched errors.

[sergiusens]

@crummel any idea when that might get merged? Even if not perfect it should provide a better alternative that broken master, right?

[crummel]

I'm not sure yet, we're still hoping to have a complete fix today.

[sergiusens]

That is good to hear!

[sergiusens]

So I still see issues on fixOnMattBranch:

"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:3) ->
(Build target) -> 
  /home/sergiusens/dotnet-source-build/src/core-setup/src/sharedFramework/sharedFramework.proj(67,5): error MSB
3030: Could not copy the file "/home/sergiusens/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release
/corehost/locked/libhostfxr.so" because it was not found. [/home/sergiusens/dotnet-source-build/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/src/core-setup/src/sharedFramework/sharedFramework.proj(67,5): error MSB3030: Could not copy the file "/home/sergiusens/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/locked/libhostfxr.so" because it was not found. [/home/sergiusens/dotnet-source-build/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/targets/repository.proj(75,5): error MSB3073: The command "/home/sergiusens/dotnet-source-build/src/core-setup/build.sh -ConfigurationGroup=Release -PortableBuild=false -SkipTests=true -- /p:BuildDebPackage=false" exited with code 1.


"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:10) -
>
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:12) -
>
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:14) -
>
(ApplyPatches target) -> 
  /home/sergiusens/dotnet-source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace --binary -i /home/sergiusens/dotnet-source-build/patches/common/0001-Run-dotnet-migrate-on-M.E
.CommandLineUtils.patch" exited with code 1.


"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:34) -
>
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:36) -
>
  /home/sergiusens/dotnet-source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace --binary -i /home/sergiusens/dotnet-source-build/patches/symreader/0001-Add-project-to-build-with-dotnet.patch" exited with code 1.

[sergiusens]

wrt the missing file, maybe the intention was to copy the dll:

file /home/sergiusens/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/locked/lockedhost.dll
/home/sergiusens/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/locked/lockedhost.dll: PE32+ executable (DLL) (console) x86-64 Mono/.Net assembly, for MS Windows

[crummel]

@karajas Did you ever find a way around the libhostfxr issue? I'm also still seeing it on the source-build repo, I've been working from the tarball.

[sergiusens]

It seems to be deleted on purpose, this is grepped output for msbuild.log in core-setup

msbuild.log:         [ 59%] Linking CXX shared library libhostfxr.so
msbuild.log:         Copying file from "/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/corehost/cmake/cli/fxr/libhostfxr.so" to "/home/sergiusens/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/libhostfxr.so".
msbuild.log:         To delete: /home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/framework;/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/framework.exe;/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/framework.dll;/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/framework.pdb;/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/framework.runtimeconfig.json;/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/apphost;/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/libhostfxr.so
msbuild.log:         Deleting file "/home/sergiusens/dotnet-source-build/src/core-setup/Bin/obj/ubuntu.16.04-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/2.0.0/libhostfxr.so".
msbuild.log:    25>/home/sergiusens/dotnet-source-build/src/core-setup/src/sharedFramework/sharedFramework.proj(67,5): error MSB3030: Could not copy the file "/home/sergiusens/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/locked/libhostfxr.so" because it was not found.
msbuild.log:         /home/sergiusens/dotnet-source-build/src/core-setup/src/sharedFramework/sharedFramework.proj(67,5): error MSB3030: Could not copy the file "/home/sergiusens/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/locked/libhostfxr.so" because it was not found.

[sergiusens]

And found here /home/sergiusens/dotnet-source-build/src/core-setup/src/sharedFramework/sharedFramework.proj:

    <!-- Clean up artifacts that dotnet-publish generates which we don't need -->
    <ItemGroup>
      <ToDelete Include="$(SharedFrameworkNameAndVersionRoot)\framework" />
      <ToDelete Include="$(SharedFrameworkNameAndVersionRoot)\framework.exe" />
      <ToDelete Include="$(SharedFrameworkNameAndVersionRoot)\framework.dll" />
      <ToDelete Include="$(SharedFrameworkNameAndVersionRoot)\framework.pdb" />
      <ToDelete Include="$(SharedFrameworkNameAndVersionRoot)\framework.runtimeconfig.json" />
      <ToDelete Include="$(SharedFrameworkNameAndVersionRoot)\apphost$(ExeSuffix)" />
      <ToDelete Include="$(SharedFrameworkNameAndVersionRoot)\$(LibPrefix)hostfxr$(LibSuffix)" />
    </ItemGroup>

[karajas]

Sorry about the confusion, Matt had opened a PR that we haven't merged in yet - master is supposed reflect what built the tarballs we produce. The last two days we've been making some spot fixes that haven't been merged yet to produce a good source tarball, but getting master updated with the right things should be the next top priority.

• We need to rebase Matt's current branch onto master and merge it.
• We need to create a PR for the Test Host and FSharp fixes and merge it.
• Chris, is there an fix that needs to be applied for websdk? (I noticed that it also had something version specific - which may/may not break other things?)

I will try to get master in a good state by Monday.

[karajas]

@karajas Did you ever find a way around the libhostfxr issue? I'm also still seeing it on the source-build repo, I've been working from the tarball.

I haven't had the chance to investigate this yet. It does seem to produce the right core-setup.
@sergiusens I wonder if it's just a missing Condition check for file exists on the ToDelete itemgroup and follow up delete command.

[crummel]

The new tarball is https://matell.blob.core.windows.net/cli-src/dotnet-2.0.0-12.tar.gz, I think you'll still need the websdk chmod fix and the coreclr clang3.9 fix, I'm still looking at getting those fixes into master.

[sergiusens]

@crummel that new tarball is with fixOnMattsBranch? FWIW, I failed to build that branch as was (with clang, websdk chmod fixes and skipping the deletion of files).

[crummel]

I think master should be in good shape now other than the libhostfxr.so issue - @karajas, do you just run build-source-tarball.sh, ignore the libhostfxr error (and then we don't see it building from that tarball)? Is that the error you're still seeing?

[karajas]

Sorry just catching up on this thread, so from the dockerfiles we have for .NET Core 2.0 we took a dependency on clang3.5, it seems like the PGO data build requires 3.9. Should we have a default and then just condition the build command for a pgo build to add that extra parameter?

[karajas]

For the libfxr error, still trying to repro consistently.

[crummel]

That sounds reasonable on the PGO issue - there's a nopgooptimize flag we can add.

[crummel]

Actually is looks like this should already be happening: https://github.com/dotnet/coreclr/blob/7f70e548bb22d7e551dd7167863c83e984d0755e/pgosupport.cmake#L45, unless this was added after the version of CoreCLR we're using. I'll look into it.

[omajid]

from the dockerfiles we have for .NET Core 2.0 we took a dependency on clang3.5, it seems like the PGO data build requires 3.9. Should we have a default and then just condition the build command for a pgo build to add that extra parameter?

Most *nix software wiil have some sort of configure script that figures out which dependencies are available (eg, is clag version 3.5 or 3.9) and which optional compilation features can be used (can PGO data be used). It prints the default detected values to the user, with optional flags that a user can use to override them - for example, user can ask for PGO data to be used and build will error out because the required version of clang is missing.

[sergiusens]

I am getting much further ahead in the build now.

[sergiusens]

I started from a clean repo and this is my current status
msbuild.txt

[crummel]

error : Failed to retrieve information about 'System.Threading' from remote source
This one is odd, it almost looks like a network issue during your build, is that possible? @karajas, have you ever seen that error?
The command "patch -p1 --ignore-whitespace --binary -i /home/sergiusens/dotnet-source-build/patches/nuget-client/0001-Revert-38949ce93081016167dfd8e37e4e897bedf47d65-DOS-.patch" exited with code 1
I just merged #103 which should fix this issue, we were patching in binary mode instead of text mode and depending on git settings the wrong line endings could have been checked out.

[sergiusens]

Thanks @crummel
Yeah, the network one is interesting, as I thought the patchset forced everything to be local.

Also, the patch that failed to apply (unless this file gets dynamically generated at build time) does not exist:

130 sergiusens@builder:~/dotnet-source-build/src/nuget-client? ls src/NuGet.Core/NuGet.Build.Tasks.Pack/NuGet.Build.Tasks.Pack.csproj                                                                                         
ls: cannot access 'src/NuGet.Core/NuGet.Build.Tasks.Pack/NuGet.Build.Tasks.Pack.csproj': No such file or directory                                                                                                            
2 sergiusens@builder:~/dotnet-source-build/src/nuget-client? git log -1                                        
commit 52c4d2b1d6b5334a27c7e4c93cb899e7395bdedd                                                                
Author: Davis Goodin <[email protected]>                                                                    
Date:   Mon Aug 7 16:28:19 2017 -0500                                                                          

    Use dagood fork to fetch lost FileSystem branch    

[crummel]

Ah, I had missed that, sorry. It's odd that we're only seeing this now, @dagood forked the NuGet repo to recover a lost submodule last week and I don't think there should have been any changes since then. I'm going to try building without that part of the patch and see what happens.

[crummel]

Actually it looked like this patch changed in https://github.com/dotnet/source-build/commit/4edf81379fba3caa5a98a82b4a864f5dc472d132 - @karajas, do you know what's going on there?

[sergiusens]

Any updates on this?

[crummel]

Hi Sergio, sorry for the delay. I think I have a fix at https://github.com/crummel/dotnet_source-build/tree/fixSdkPatch if you want to give that a try - I've only tested the patching so far, I've got it building now.

[sergiusens]

On jue, 24 ago, 2017 at 5:06 PM, Chris Rummel
[e-mail] wrote:

Hi Sergio, sorry for the delay. I think I have a fix at
https://github.com/crummel/dotnet_source-build/tree/fixSdkPatch if
you want to give that a try - I've only tested the patching so far,
I've got it building now.

Thanks! I'll give it a go later in the evening when I am back at my
desk and report back.

[sergiusens]

hi @crummel , on that branch I get:

Unable to checkout '884b3d8f10656137babc06db85dac63d10757bbb' in submodule path 'src/nuget-client'

[dagood]

It looks like in Chris' commit fb630d2a3bccb7b5120d6aa7053072a0d925f569, .gitmodules is pointing to my NuGet fork, but the submodule points to a commit that's not in my fork. I've updated my fork's dev branch to bring in that commit, as a short-term fix.

[sergiusens]

Thanks @dagood!

Unfortunately I am still seeing the usual error:

"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepository
AndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepository
AndDependencies target) (2:3) ->
(Build target) -> 
  /home/sergiusens/dotnet-source-build/src/core-setup/src/sharedFramework/share
dFramework.proj(67,5): error MSB3030: Could not copy the file "/home/sergiusens
/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/locke
d/libhostfxr.so" because it was not found. [/home/sergiusens/dotnet-source-buil
d/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/src/core-setup/src/sharedFramework/share
dFramework.proj(67,5): error MSB3030: Could not copy the file "/home/sergiusens
/dotnet-source-build/src/core-setup/Bin/ubuntu.16.04-x64.Release/corehost/locke
d/libhostfxr.so" because it was not found. [/home/sergiusens/dotnet-source-buil
d/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/targets/repository.proj(75,5): error MSB
3073: The command "/home/sergiusens/dotnet-source-build/src/core-setup/build.sh
 -ConfigurationGroup=Release -PortableBuild=false -strip-symbols -SkipTests=tru
e  -- /p:BuildDebPackage=false" exited with code 1.

And these:

"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepository
AndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepository
AndDependencies target) (2:10) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepository
AndDependencies target) (2:12) ->
(ApplyPatches target) -> 
  /home/sergiusens/dotnet-source-build/targets/repository.proj(49,5): error MSB
3073: The command "patch -p1 --ignore-whitespace -i /home/sergiusens/dotnet-sou
rce-build/patches/nuget-client/0001-Revert-38949ce93081016167dfd8e37e4e897bedf4
7d65-DOS-.patch" exited with code 1.


"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:18) ->
  /home/sergiusens/dotnet-source-build/targets/repository.proj(49,5): error MSB3073: The command "patch -p1 --ignore-whitespace -i /home/sergiusens/dotnet-source-build/patches/templating/0002-Add-a-target-that-just-build-packages-CLI-needs.patch" exited with code 1.



"/home/sergiusens/dotnet-source-build/build.proj" (default target) (1) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2) ->
"/home/sergiusens/dotnet-source-build/targets/repository.proj" (BuildRepositoryAndDependencies target) (2:38) ->
(Build target) -> 
  /home/sergiusens/dotnet-source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(141,5): error : The file '/home/sergiusens/dotnet-source-build/src/xliff-tasks/bin/Release/XliffTasks/netcoreapp1.1/XliffTasks.dll' to be packed was not found on disk. [/home/sergiusens/dotnet-source-build/src/xliff-tasks/src/XliffTasks/XliffTasks.csproj] [/home/sergiusens/dotnet-source-build/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/Tools/dotnetcli/sdk/2.0.0-preview3-006845/Sdks/NuGet.Build.Tasks.Pack/buildCrossTargeting/NuGet.Build.Tasks.Pack.targets(141,5): error : The file '/home/sergiusens/dotnet-source-build/src/xliff-tasks/bin/Release/XliffTasks/netcoreapp1.1/XliffTasks.dll' to be packed was not found on disk. [/home/sergiusens/dotnet-source-build/src/xliff-tasks/src/XliffTasks/XliffTasks.csproj] [/home/sergiusens/dotnet-source-build/targets/repository.proj]
  /home/sergiusens/dotnet-source-build/targets/xliff-tasks.targets(8,5): error MSB3073: The command "/home/sergiusens/dotnet-source-build/Tools/dotnetcli/dotnet build /home/sergiusens/dotnet-source-build/src/xliff-tasks/build/build.proj /t:Pack /v:normal /flp:Verbosity=Diag;LogFile=/home/sergiusens/dotnet-source-build/src/xliff-tasks/msbuild.log" exited with code -1. [/home/sergiusens/dotnet-source-build/targets/repository.proj]

[weshaggard]

Depending on lsb_release isn't going to work see the conversation at https://github.com/dotnet/corefx/pull/24289.

[crummel]

I think we determined in #256 that Ubuntu 16.04 is working again - could you let me know if you still see this issue?

[dseefeld]

Assuming this is fixed. Closing.