Buildx: [output clipped, log limit 1MiB reached] and then exit code 7

[jessehui]

Docker build on my local machine works fine. But when I use GitHub actions to build the docker image, I got error.
First I see this log: [output clipped, log limit 1MiB reached], and after a few minutes, I got error code 7.
After multiple times of retry, the results are the same.

The log snippet is here (for full log, check this out: https://github.com/occlum/occlum/runs/1573929172?check_suite_focus=true):

2020-12-18T00:48:25.5043289Z #10 272.4 ../include/libunwind-common.h:39:31: note: in expansion of macro 'UNW_PASTE'
2020-12-18T00:48:25.5044095Z #10 272.4  # define UNW_PREFIX UNW_PASTE(UNW_PASTE(_UL,UNW_TARGET),_)
2020-12-18T00:48:25.5044612Z #10 272.4                                ^~~~~~~~~
2020-12-18T00:48:25.5045321Z #10 272.4 ../include/libunwind-common.h:35:31: not
2020-12-18T00:48:25.5045945Z #10 272.4 [output clipped, log limit 1MiB reached]
2020-12-18T00:52:18.2647230Z #10 ERROR: executor failed running [/bin/sh -c git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx]: exit code: 7
2020-12-18T00:52:18.2680001Z ------
2020-12-18T00:52:18.2682323Z  > [stage-1  6/30] RUN git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx:
2020-12-18T00:52:18.2684280Z ------
2020-12-18T00:52:18.2686098Z failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx]: exit code: 7
2020-12-18T00:52:24.8045851Z ##[error]buildx call failed with: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx]: exit code: 7
2020-12-18T00:52:24.8230306Z Post job cleanup.

Github actions yaml: https://github.com/occlum/occlum/actions/runs/428064733/workflow
dockerfile: https://github.com/occlum/occlum/blob/master/tools/docker/Dockerfile.centos8.1

[jiayunwei]

i met the problem too

[leimao]

Any way to increase the log limit cache size?

[tonistiigi]

clipping can be disabled with buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=-1 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=-1 but this should have no effect on the outcome of the build. If it does then please post a reproducer.

[Mr-K10]

I am facing the same issue, Can someone tell me how to stop logging or increase log limits from 1 MiB to at least 10MiB.

[k-hashimoto]

I have same issue. I want to how to stop logging or increase limits.

[tonistiigi]

increase log limits from 1 MiB to at least 10MiB.

for 10MB --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760

[jessehui]

For those who use buildx github actions for image building, just need extra with to prevent this:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          driver-opts: env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760

[OmryAr]

I'm struggling with this myself, running a buildkit enabled docker build from a linux command line - pulling hairs on how to pass those params... can anyone please point me to the right direction?
(docker does not accept --driver-opt, and restarting dockerd with BUILDKIT_STEP_LOG_MAX_SIZE=-1 and BUILDKIT_STEP_LOG_MAX_SPEED=-1 environment vars doesn't have any effect).

[jessehui]

@OmryAr I believe these parameters are provided by buildx which means you should build image with docker buildx build. And these parameters are used when creating a buildx instance. For more info, please read the buildx doc.

[OmryAr]

Thanks @jessehui ! So after a long while of digging into the docs and experimenting - I've managed to create a buildx instance set with these configuration params. Something on the lines of:

docker buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=1000000,env.BUILDKIT_STEP_LOG_MAX_SPEED=100000000 --use

So now my revised question would be - how can I configure the default instance (without creating another one - which I wasn't intending to... and seems wasteful)?

• When running docker buildx build, out-of-the-box, I am running the default instance, right?
• How can I pass --driver-opts to the default instance?

[yardenshoham]

I'm using a regular docker build, how can I increase the log limit?

[OmryAr]

Hi @yardenshoham !
After researching this quite deeply, I can try to explain this:

• At the time of this writing, log limits cannot be arbitrarily set using normal params to the docker build command.
• To set log limits globally, you need set appropriate environment params for the dockerd service, and restart it for the configuration to take effect. Assuming a systemd based Linux-distro server, you'll need to add the following lines to /etc/sysconfig/docker:
  BUILDKIT_STEP_LOG_MAX_SIZE=10000000
  BUILDKIT_STEP_LOG_MAX_SPEED=10000000
• Alternatively you might create a builder instance and set it up with your desired options, as shown in my comment above.

HTH!

[phdoerfler]

for convenience, here's a bash one-liner:
docker buildx build . --builder "$(docker buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000)"

[LukeIreland1]

clipping can be disabled with buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=-1 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=-1 but this should have no effect on the outcome of the build. If it does then please post a reproducer.

[output clipped, log limit -1B reached]