Kops: git history is huge (2.7GB) - switch to go dep?

[integrii]

1. What commands did you run? What is the simplest way to reproduce this issue?
   go get k8s.io/kops

2. What happened after the commands executed?
   2.7GB of data came down, 1.3GB of which is in the .git directory.

3. What did you expect to happen?
   A reasonable amount of code to be downloaded.

It is possible to go back in time and purge this stuff out. Is there a reason nobody has done this yet? Is there really something that big in the history that needs kept?

I am happy to do some cleanup here in a branch, if there is no reason not to downsize this.

This causes problems for us because we use some kops packages in one of our projects. Whenever we build our docker container for that project, we run a go get to fetch the required code to build our tool. This causes the kops repo to be cloned, which is huge. We then build our binary and manually delete the massive kops project directory from our path before uploading our image for production use. It would be nice not to download and delete the extra couple gigs of stuff :-)

[integrii]

Looks like this really involves how the project uses vendoring, which must be solved before this is fixed.

We should consider using go dep and not including the vendor directory into the codebase.

[justinsb]

We have tried repeatedly and continue to try, go dep currently crashes out while trying to resolve the dependencies.

The git submodule approach works in the meantime, although it's not great, just the best we've got that works. The biggest downside is what you've identified - that _vendor has the git submodules, and that go get fetches git submodules (and does a full fetch, not a shallow clone)

If instead you git clone, that doesn't (normally) fetch the submodules by default, and then it is only 160MB of code, which is not great, but much more reasonable IMO:

> git clone https://github.com/kubernetes/kops
Cloning into 'kops'...
remote: Counting objects: 156068, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 156068 (delta 3), reused 1 (delta 1), pack-reused 156049
Receiving objects: 100% (156068/156068), 162.84 MiB | 40.74 MiB/s, done.
Resolving deltas: 100% (90346/90346), done.
Checking out files: 100% (31187/31187), done.

We'd welcome help on adopting go dep.

• https://github.com/kubernetes/kops/pull/4094 started the process
• https://github.com/kubernetes/kops/pull/4152 is a script to output the correct dependencies for go dep; we want to use the same versions that the kubernetes project does, and it maintains them in Godeps.json

[justinsb]

Actually ... I may have got this working with a few version overrides. So hopefully in kops 1.9 / 1.10. But not cloning the submodules will remain the big win!

[justinsb]

Half way there ... we're using dep now as of #4382. Now just to remove _vendor ...

[chrislovecnm]

I have a PR in to remove _vendor ... yay

[chrislovecnm]

Hopefully the submodules helps!

[integrii]

I just did a test checkout - the code on disk is 287MB. Thats a lot better than 2.7GB! Thanks @chrislovecnm

[chrislovecnm]

👍 - most of the thanks needs to go to @justinsb and others that helped