Test-infra: prow: clonerefs: speed up cloning of large repos through shallow cloning / pre-cloned repo in base image

[AviKndr]

What would you like to be added:
I would like to add some feature to alleviate the overhead of cloning on every job for very large repos (> 1 GB).

Given that we only need the specific revision and shallow cloning for our repo takes about half of the time, we'd like to add a "--shallow" arg to clonerefs for cloning without tags/history (depth of 1) as referenced here: #6865 .

Alternatively, a way of preserving the source by having a checkout image with a pre-cloned repo used as the base on which to fetch new changes for jobs.

Why is this needed:
Currently our presubmit jobs take ~13 minutes in which ~10 are a result of cloning the entire repo (Linux Kernel). This would be very inefficient in terms of both speed and network usage with a large number of concurrent presubmit jobs for any kernel related jobs.

/kind feature
/area prow

[AviKndr]

@BenTheElder

[stevekuznetsov]

There are a large number of tools inside of repos that expect to have deep clones -- the k8s versioning bash stuff is a prime example that needs to know about tags, commits between them, etc. A shallow clone option might be nice but would start to leak infra state into the job which isn't ideal.

On our end we do use a cache container image to make cloning faster, but that's entirely external to the utilities and just something to set up in the job configuration.

@BenTheElder had some ideas about cloning into a volume that held a cache, but that imposes limits on job concurrency while maintaining the caching. Not sure what the final idea was there

[BenTheElder]

I suggested shallow cloning would be a reasonable and simple option, it should not be the default but we could allow enabling shallow clones for projects that don't need full clones.

AFAICT, they could get away with shallow clones, other projects probably could too, and flipping a switch to enable that will be simpler than the pre-cloned repo support.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stevekuznetsov]

/remove-lifecycle stale

An option to do a shallow clone would be great

/help

[k8s-ci-robot]

@stevekuznetsov:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/remove-lifecycle stale

An option to do a shallow clone would be great

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vishleshpatel]

@stevekuznetsov if its okay for a newbie to learn system and work on it, then I would like to go ahead and pick this one up.

[stevekuznetsov]

@vishleshpatel hi! Totally :)

Job configuration holds information about the references that are under test and need to be cloned as well as any extra references that might be necessary for the job. Options for the the refs under test are in the UtilityConfig:

https://github.com/kubernetes/test-infra/blob/143a95a43bb70d540e021e23fd50db5020b30089/prow/config/jobs.go#L395-L406

You could add something like:

// CloneDepth is the depth of the clone that will be used. A depth of zero will do a full clone.
CloneDepth int `json:"clone_depth,omitempty"`

However, to add the same option for the refs that are not under test, you'd need to add it to Refs here. The program we use to clone is clonerefs and it will read the Refs, where you can decide how deep to clone.

Check out this commit for an example of adding a change like this!

[vishleshpatel]

Thanks @stevekuznetsov for the pointers. I will take a look at it this week and ask questions if in need.

[vishleshpatel]

Hi @stevekuznetsov

so i now have changes put together. How to test the changes? Also, What tests needs to be run?

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stevekuznetsov]

This got implemented in #12499

/close

[k8s-ci-robot]

@stevekuznetsov: Closing this issue.

In response to this:

This got implemented in #12499

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.