Venus: [bug] CircleCI Forked Build Resource Class Issues

[ognots]

Describe the bug
When Pull Requests are created from forked repositories, in some cases the build_linux container is downgraded to a Medium resource class and the Test step ultimately fails. Failures log out of memory errors - affected build:

/usr/local/go/pkg/tool/linux_amd64/link: flushing /tmp/go-link-680255609/go.o: write /tmp/go-link-680255609/go.o: cannot allocate memory

The build_linux job requests an xlarge build container by default.

To Reproduce
Steps to reproduce the behavior:

1. Fork https://github.com/filecoin-project/go-filecoin from a github account not linked to a CircleCI account
2. Make a change and create a pull request
3. Build will fail

Expected behavior
The build_linux job should pass on forked builds

Additional Context

• Go test parallelization increases resource requirements. Investigate using CircleCI parellelization instead. Parellelization might scale better because it utilizes parellelization settings in one's account. Forked builds will take much longer if the user hasn't paid for parellelization, but will at least pass.
• There is no documented way to grant forked builds xlarge resource class. Might be worth investigating further if it's possible, but could have cost of goods implications even if it is possible.

[ognots]

A temporary way to unblock community contributions experiencing this issue is for a go-filecoin committer to fork and push a new branch from an affected PR.

Use the following steps:

git fetch origin pull/####/head:pr-####
git push origin pr-####

The git push will trigger the CircleCI build within the original forked PR.

[anorth]

@ognots you suggested some changes to Circle and parallelism that might help us here. Could you outline that? We agreed that we want contributors to have a first-class experience submitting PRs.

[ognots]

The design outline:

• reduce go test parallel setting to 2 (from 8) in order to run in a medium container class without running out of memory
• set CircleCI container parallelism to 4 (from 1)
• using new go test test classification tags, break up build_linux in to smaller jobs that can be run in parallel in CircleCI workflow

The following is testing results from Implemented design:

Build time from a go-filecoin committer is on-par with existing builds (~7m30s).
https://circleci.com/workflow-run/19c2526e-8002-498e-b506-deafe0a51b99

Build from fork (15m) takes longer but now completes with medium container class and reduced circleci parellelism.
https://circleci.com/workflow-run/2bbac585-3a25-4729-8766-8006c381f2ee

All in all, this new design seems to address the problem with no loss of performance.

Tomorrow gong to investigate if any additional tweaking can speed up the builds for go-filecoin committers and do some final clean up.

[anorth]

Can go test parallelism be sourced from environment so that local
development can opt in to 8?

On Tue, 16 Apr 2019 at 13:43, Oggy Nots [e-mail] wrote:

The design outline:

• reduce go test parallel setting to 2 (from 8) in order to run in a
  medium container class without running out of memory
• set CircleCI container parallelism to 4 (from 1)
• using new go test test classification tags, break up build_linux in
  to smaller jobs that can be run in parallel in CircleCI workflow

The following is testing results from Implemented design:

Build time from a go-filecoin committer is on-par with existing builds
(~7m30s).
https://circleci.com/workflow-run/19c2526e-8002-498e-b506-deafe0a51b99

Build from fork (15m) takes longer but now completes with medium container
class and reduced circleci parellelism.
https://circleci.com/workflow-run/2bbac585-3a25-4729-8766-8006c381f2ee

All in all, this new design seems to address the problem with no loss of
performance.

Tomorrow gong to investigate if any additional tweaking can speed up the
builds for go-filecoin committers and do some final clean up.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/filecoin-project/go-filecoin/issues/2516#issuecomment-483499814,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAbLeqHui9yqdpu95Dk13tkwdeADSxf5ks5vhUb0gaJpZM4ci38h
.

[ognots]

https://circleci.com/workflow-run/c789099c-6c0a-463f-9132-3f5dbeeb3253 throttles unit, integration and functional tests at medium resource class and passes (ran several times to verify). 8:56

https://circleci.com/workflow-run/988aa450-e3bb-445d-8199-6202c8b3c07b runs at optimal settings. 7:41

[ognots]

Closed by https://github.com/filecoin-project/go-filecoin/pull/2609