Dvc: Unit testing experience is broken in Mac OS

[dmpetrov]

OS: Mac OS (might be other)

It is hard to run unit-test out of the box. It looks like some tribal knowledge is needed to run it - set up ulimits. It might prevent people from contributing code for DVC.

Ideally, it should work in any machine (with installed git, python and pip) as simple as:

$ git clone https://github.com/iterative/dvc.git
$ cd dvc
$ pip install -r requirements.txt
$ pip install -r test-requirements.txt
$ python -m tests

All the tests should pass.

[shcheklein]

@dmpetrov could you provide more context? what exactly is broken? Is it related to this one - https://github.com/iterative/dvc/issues/1534

[efiop]

@shcheklein it is not. @dmpetrov is talking about low ulimits on his machine :)

[shcheklein]

Is it related to the http server? I didn’t experience any problems with limits on my mac last time I checked.

[efiop]

@shcheklein Btw, those tests are already fixed with a temporary hardlink hack(needed it for macos release). The state patch from @pared will fix them permanently even on reflinks :slightly_smiling_face:

@dmpetrov I don't think it is worthy of a bug label though :slightly_smiling_face: I've provided you with sysctl commands to increase maxprocperuid limit and our tests/__init__.py handles calling ulimit. Are you still trying to run it with pycharm? Or python -mtests doesn't work too?

[shcheklein]

@efiop why do we hit them in the first place? too many files? too many connections?

[efiop]

@shcheklein Too many files and too many processes. Might optimize both in the future, but it is not worth bothering with right now.

[shcheklein]

@efiop is it possible to run a clean after each test?

[efiop]

@shcheklein Yep.

[efiop]

@shcheklein basically what it comes to is limiting the number of tests that run at the same time and also making sure that all of the fds are closed as soon as possible(not only in dvc, but in all dependencies that it is using). Open fds problem is obviously also affected by the number of processes. The problem really affects macs because they have some really low ulimits (e.g. -u is ~700 processes on mac and 64K on linux :slightly_smiling_face: ).

[dmpetrov]

The issues that I saw so far:

1. The ulimits issue as @efiop said.
2. nosetests works only in virtual env - not documented.
3. Errors OUT.txt

[efiop]

@dmpetrov

1. The ulimits issue as @efiop said.

Let's just add that to our contribution guide for now.

2. nosetests works only in virtual env - not documented.

Do you mean we should document that they are not properly handled by pycharm automatically? Or are you talking about not being able to run them in your system python? We specifically mention virtualenv in our contribution guide https://dvc.org/doc/user-guide/contributing .

3. Errors OUT.txt

Your tests pass successfully. Or are you talking about some test error logs finding their way into the primary log?

[dmpetrov]

@efiop none of the issues is related to PyCharm.

Yes, the errors in the output look very confusing. A guide is definitely needed if we don't want to fix the output.

[ghost]

@dmpetrov , I agree with having the errors on the log is very confusing, is also broken on Linux.

There's a --logging-filter and --logging-clear-handlers feature that I didn't knew about.

As a side note, while running pytest, I'm hitting with the following error during TestCheckoutHook.test:

E       OSError: reading from stdin while output is captured

Can we change the bug label to a refactoring one?

[dmpetrov]

Closing since the docs were improved and output issue was fixed.