Repo2docker: failing docker tests locally on macOS (case 2)

[hwine]

Bug description

With the latest versions of macOS (Catalina 10.15.4) and Docker CE (2.2.0.5), many of the tests fail to run locally. _(This issue was split from #881 after the approach in #882 did not seem to apply to these failures.)_

Expected behaviour

Running the test suite locally on a released version of repo2docker should have no errors or failures.

Actual behaviour

A number of tests which require running docker fail. There appear to be a number of different failure messages, many appear to be related to failures using conda or pip to install packages inside the docker container during self.build() invocations. Typical message:

>                           raise docker.errors.BuildError(l["error"], build_log="")
E                           docker.errors.BuildError: The command '/bin/sh -c apt-get -qq update &&     apt-get -qq install --yes        libapparmo
r1        lsb-release        psmisc        sudo        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lis
ts/*' returned a non-zero code: 100

repo2docker/app.py:729: BuildError
__________________________________________________________________ test session ___________________________________________________________________

    def test():
        app = make_r2d(args)
        app.initialize()
        if app.run_cmd:
            # verify test, run it
            app.start()
            return
        # no run_cmd given, starting notebook server
        app.run = False
>       app.start()  # This just build the image and does not run it.

tests/conftest.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
repo2docker/app.py:743: in start
    self.build()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Step 1 will be to enumerate and categorized those messages.

How to reproduce

1. On a macOS box, checkout a released version of repo2docker
2. Setup the development environment.
3. invoke pytest tests/
4. observe failures.

Your personal set up

• OS: macOS
• Docker version: CE 2.2.0.5
• repo2docker version: 0.11.4

[hwine]

This one is proving a bit challenging. I have a large list of failing tests (gist forthcoming), and I'm starting at the top with tests/base/node10/verify, which passes on ubuntu, but fails on macOS.

repo2docker has some custom test collectors to work with pytest, and tests/base/node10 is one of the special test cases it supports. That makes debugging a bit more challenging for pytest-users-but-not-gurus. In this case, the actual test ends up invoking repo2docker in the tests/base/node10 directory. Of course, there is no real repo there, this is some sort of "edge case" for repo2docker.

I then tried running repo2docker on both ubuntu and macOS (Catalina) in empty directories, and noticed the same behavior (works on ubuntu, doesn't on macOS). Then I happened to notice the docker image on ubunto was 61GB, and the max image size I had configured on macOS was 51GB. Bumping the macOS image size to 64GB fixed that test. (Configuring a specific volume size is not needed on linux systems.)

Moving on to the next failing tests, but I'll try to see if there's a nicer way to detect and report this issue.

Updates:

• _2020-05-05 Update 1: using an image size of 64 GB, my "failed test" count reduces to 16 failed._
• _2020-05-05 Update 2: using an image size of 128 GB, "failed tests" are down to 5._ :tada:

[betatim]

Making an image of 61GB seems like a (serious) bug :-/ Just to double check, you cd tests/base/node10 and then ran repo2docker .?

[hwine]

Just to double check, you cd tests/base/node10 and then ran repo2docker .?

Correct. And I got the 61GB number from doing that on my ubuntu (wsl2) system, where the test executes successfully.

After the test ran successfully, I re-ran the image via:

docker run --rm -it {container_id} /bin/bash
df -h /

And using the number reported as "used".

[hwine]

Working on test tests/unit/test_subdir.py, which fails at step:

Step 41/51 : RUN ${KERNEL_PYTHON_PREFIX}/bin/pip install --no-cache-dir -r "requirements.txt"

with "No space on device". Running bash in the prior image yields:

$ docker run --rm -it 7d5ca7f83331 /bin/bash
hal@e06a95d3d109:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay          63G   62G     0 100% /

I'll try upping the image size significantly and re-running to see how many tests are disk space limited.

Updates:

• _Upping image size to 128GB allowed tests/unit/test_subdir.py to pass_
• _Only 5 failing tests with 128GB image!_ 🎉

[hwine]

woot! Those 5 failures disappear with the installation of git-lfs, called out in #888 (as I think it will be a different documentation fix). _[Edit: nope, this doc fix catches that as well]_

So: current status is complete green run

=== 297 passed in 1904.19 seconds ====

with two changes:

1. bump docker image size to 128GB
2. install git lfs on host (development) machine.

[hwine]

_[moving followup discussion from merged #890]_
_Originally posted by @betatim in https://github.com/jupyter/repo2docker/pull/890#issuecomment-629216485_

I am on OSX and I don't think I've had to change the disk size of my docker VM. I wonder what it could be, I have no good idea though or place to start digging

What is the disk size allocated on your machine? If it isn't greater than 64 GB by some amount, then my analysis is wrong. There would be some definite debugging to do then 😀

[betatim]

Docker says "64 GB (61.9 GB used)".

When you run id in a terminal, what does it say? We had reports a good while back that some people (on OSX) basically couldn't run repo2docker locally because it would use up "all space" and/or take a very long time. I think we narrowed it down to this being people who have a laptop which is somehow centrally managed. A indication for that was that they had a (very) large user ID. Instead of something in the hundreds or so (I have 501). If that is the case for you too, that might explain why you need to grow the disk space so much. Doesn't yet explain why for users with a managed laptop this is the case.

[manics]

That bug should've been fixed by https://github.com/jupyter/repo2docker/pull/804

[hwine]

TIL docker preferences tells you the "max used" in the GUI. I have "128 GB (76.6 GB used)".

That's only ~16GB difference, which is almost noise. That could be a side effect of some other Docker setting I have - perhaps inherited from years ago.

@betatim - I see several ways to proceed:

1. continue investigation for the ~16GB difference
2. tweak the docs to say "80GB" (rather than 128), and add "your mileage may vary" language.
3. leave as is (aka "perfection is the enemy of done")

Your thoughts?

---

_(fwiw, my mac is not centrally managed, and my uid is also 501.)_

[betatim]

I vote for "leave as is". Seems like our time is better spent doing other things than tracking down differences in storage usage, especially if the likely outcome is that we continue to recommend "YMMV, but try making the disk bigger" :D