Lnd: Impossible to build the latest release tagged version 0.5.1-beta

[NicolasDorier]

I am trying to build on top of tag v0.5.1-beta from my CircleCI server, but blocked by this error.

(38/43) Wrote golang.org/x/sys@ebe1bf3edb3325c393447059974de898d5133eb8
(39/43) Wrote google.golang.org/grpc@8dea3dc473e90c8179e519d91302d0597c0ca1d1
(40/43) Wrote github.com/miekg/dns@79bfde677fa81ff8d27c4330c35bda075d360641
(41/43) Failed to write github.com/ltcsuite/ltcd@cdab10132e8c6e4a3ffd112dba54791946d28906
(42/43) Failed to write google.golang.org/genproto@df60624c1e9b9d2973e889c7a1cff73155da81c4
(43/43) Failed to write golang.org/x/[email protected]
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/ltcsuite/ltcd: fatal: failed to unpack tree object cdab10132e8c6e4a3ffd112dba54791946d28906
: exit status 128
make: *** [Makefile:117: dep] Error 1

The exact same code was working fine before.

[Xian001]

I ran into something similar recently. Have you updated to the latest GoLang from Google's site ?

[NicolasDorier]

@Xian001 I am building from golang:stretch and golang:alpine docker images. Which was working fine before.
I briefly tried with an older pinned version, without luck.

[PWKad]

Just peeking around briefly but I wonder if this is related to LTCD or LND? The error indicates it's missing an object reference with a hash, I wonder if somehow a version of LTCD was published improperly or something?

[Roasbeef]

We don't use dep. We use go modules now. Check out the latest install instructions, you should also remove the old vendor directory if you still have it.

[NicolasDorier]

@Roasbeef please reopen. I want to build 0.5.1-beta which depends on dep are you telling me that LND does not support building the latest official release, and that if I need to build LND I need to use master?

[Roasbeef]

Ah I see what you're trying to do. A force push at ltcd (they didn't tag a
new version) broke a pinned commit hash in that dependancy for dep. In
order to build using dep, you'll need to update that commit hash. We no
longer use dep.

On Thu, Jan 17, 2019, 8:29 PM Nicolas Dorier <[email protected]
wrote:

@Roasbeef https://github.com/Roasbeef please reopen. I want to build
0.5.1-beta are you telling me that LND does not support building the
latest official release?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lightningnetwork/lnd/issues/2492#issuecomment-455424071,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA87LmHLabo8_66ZHbUJviu7zX_5PtZ_ks5vEU2ZgaJpZM4aEvWA
.

[Roasbeef]

Future releases will include a full vendor directory to avoid issues like this in the future.

[NicolasDorier]

@Roasbeef I am fine to update a commit hash, can you point how should I do? I am quite noob in go.

[NicolasDorier]

btw it is strange, I checked that the commit was existing, and it does.

[NicolasDorier]

see https://github.com/ltcsuite/ltcd/commits/cdab10132e8c6e4a3ffd112dba54791946d28906 is still here

[NicolasDorier]

as you say they forced pushed... but still it seems strange to me that it stop working, the commit still exist.

[NicolasDorier]

Ok so it seems that indeed, cloning the repo leave out this commit... even if it exists in the repo remote.

[NicolasDorier]

Maybe if they tag it, a git clone would retrieve it... asking them..

[NicolasDorier]

Ah because they have a fork, I can't post issue there.... who should I ping?

[NicolasDorier]

@Roasbeef please reopen, I changed the title, this is a painful issue.

[halseth]

You can change the commit hash in Gopkg.yaml

[NicolasDorier]

To what? The master probably have some breaking changes, and if not, then I am using a version nobody tested :(

[NicolasDorier]

@halseth I think the best would be that you add a tag from github to the commit in question so it can get pulled during the clone. (if you have access to the repo)

[NicolasDorier]

Tried

[[constraint]]
  name = "github.com/ltcsuite/ltcd"
  revision = "cdab10132e8c6e4a3ffd112dba54791946d28906"
+ source = "https://github.com/ltcsuite-archive/ltcd.git"

no luck...

[NicolasDorier]

This did the trick...

[[projects]]
  digest = "1:3402f61a37575d808e4499dbfda1ab756d570c66cdfed492b84a936bbb909533"
  name = "github.com/ltcsuite/ltcd"
  packages = [
    "chaincfg",
    "chaincfg/chainhash",
    "wire",
  ]
  pruneopts = "UT"
  revision = "cdab10132e8c6e4a3ffd112dba54791946d28906"
+ source = "github.com/ltcsuite-archive/ltcd"

I hope the new dependency system is sane enough to not make the build breakable by a force push on some third party repo....

[molxyz]

@NicolasDorier This issue has been solved with this PR: https://github.com/lightningnetwork/lnd/pull/2395

[NicolasDorier]

@molxyz it would not have solved my issue, as your PR depends on the new go modules, which are unavailable for 0.5.1-beta. I don't want to build upon master or some unreleased version.

[halseth]

Most dependency systems cannot handle the dependency disappearing from the source... I do find it a bit weird though that it doesn't attempt to fetch the whole repo to find the commit, but I guess there are performance tradeoffs.

In any case, this is not a problem for users running the released binary, and for those that want to compile from source, there are workarounds.

We do want to fix this by potentially vendoring all dependencies, but glad you were able to fix the issue!

[molxyz]

@NicolasDorier I tested on my system a few times and it worked fine. Maybe reinstall LND?
Here's what I did:

cd $GOPATH/src/github.com/lightningnetwork/
rm -rf lnd
git clone https://github.com/lightningnetwork/lnd
cd lnd
ls #note you will see go.mod and go.sum in this lnd download
git checkout v0.5.1-beta
ls #note there's no go.mod and go.sum they're gone, BUT..
make
ls #LOOK! go.mod and go.sum are back
make install

[NicolasDorier]

@molxyz it is probably because you built before they did the force push.
If you do it on a blank new machine, it won't work.

[molxyz]

@NicolasDorier Yes, you are correct. Thanks for this pointer. I found this out by testing on another node that was created after that ltcd force push which happened around July last year... And I'm glad you got it figured out how to fix it (which I still don't know.. lol!) How I got it fixed for me is running that commit that fixed it: git checkout 6e2762ba8c80c2a18548384cc5ae24b1e5b3f3c6 .

Also, I have been running lnd master branch on both testnet and mainnet nodes, they are fine, you shouldn't worry. I wouldn't be running v0.5.1 now because a lot of bugs in this release have been fixed in many commits. (For example one of the fixes is my old stuck dust balances in a mainnet nodes got swept nicely.. LND can run a year old testnet datadirs I kept and swept all stuck wallets..)

Anyways, thanks again! :-D

[PWKad]

Running in to this as well now I cannot run 5.1 with a docker container. Where is a Gopkg.yaml file to add this line to?

When cloning I see on the container there is a GoPkg.toml file - is that the file? If so if someone can tell me which file needs to be fixed I can PR it and request a patch release, as-is the latest release isn't able to build for me on a fresh system.

[NicolasDorier]

@PWKad https://github.com/lightningnetwork/lnd/blob/v0.5.1-beta/Gopkg.lock#L301

[PWKad]

Thanks for pointing this out @NicolasDorier going to figure out a way to insert that line with docker and I'll share here. sed not working yet I'll figure something out though.

[PWKad]

Well that didn't work out well, if anyone else runs in to this and trying to use the 5.1 release on Docker I just copied the file Nicolas shared above beside the Dockerfile and after cloning the repo I just used -

COPY replacement-Gopkg.lock $GOPATH/src/github.com/lightningnetwork/lnd/Gopkg.lock

in the docker file with the file to replace of course being named replacement-Gopkg.lock

[NicolasDorier]

@PWKad what I am using to build is this commit https://github.com/btcpayserver/lnd/commit/3c4df267a05a0fae53e80b0627aa0c05205ecb48 basically. (I have my own fork of lnd)