Fastai: Continous Integration tests for Fast AI

[PiotrCzapla]

Jeremy
What would you say for adding some integration tests (https://circleci.com/ or https://travis-ci.org/) so that we can save on manual testing? It will make it easier to introduce small fixes and it will give us some ground to write some proper unit test in the future.

We could start with simply executing some notebooks and watching for errors, it won't check for GPU errors but at least this is a start and we can use this test on our workstations to execute GPU tests.

Later we could try to flush out the API in unit test to show the relations between different methods.

Let me know what you think and if it is ok for you I will the setup a CI on my fork and create a pull request explaining how to plug Travis or Circle CI to your account and repo.

[hamelsmu]

I am on board to help with this, if approved.

[jph00]

Yeah that would be good. I'd rather start with proper python tests rather than executing notebooks - I think starting the "right way" is important to provide a model for other contributors to follow. E.g this is what we're trying to do for docs right now - show a really good role model for the fastai.transforms module, and then let the community follow that for the other modules.

I think that means picking one piece of functionality, or the functionality displayed in one lesson, or picking one class, and trying to show how to test it in a reasonably light-weight but effective way. I do agree that integration tests should be the focus. A few thoughts:

• Tests should check correctness both with and without the GPU (assuming one is available); you can just toggle fastai.core.USE_GPU to check both backends
• CI integrations tests should run quickly. That probably means we can't really know if an algorithm is fully working (i.e that it actually makes the loss go down when run on real data) since that takes a while for any meaningful dataset. But we could at least do a single batch and check loss isn't NaN
• We could have a longer integration test for each major piece of functionality (e.g. 'fine tune an image classifier CNN') that runs for 30 seconds or so. With a carefully chosen dataset this should be enough to confirm the model is doing something reasonable
• We should include at least some unit tests, so folks know what our expectations for unit tests are.

If people submit a PR that changes core functionality like the DataLoader, I'd like to see unit tests with that, since there's a lot of edge cases that are easy to miss and the integration tests won't always identify those issues. E.g if padding isn't done correctly in DataLoader, integration tests might still run, but NLP models might become less accurate.

[PiotrCzapla]

I think we are in alignment here. Dummy integration tests are easier (just executing a notebook that is prepared), but I see your point with proper unit-tests, but let's give it a go. I would pick the transformations as this will benefit the most from the tests and I know the code quite well.

I will get first steps implemented in a separated branch.

[PiotrCzapla]

Here is the first version of travis-ci integration.
https://travis-ci.org/PiotrCzapla/fastai/builds/363774572

It was still building when I was leaving if this is red or yellow then it means the test are working (the current test fail so the button should be red)

[PiotrCzapla]

I'm getting segmentation faults probably due to memory limits on travis-ci (4GB per docker container).

[radekosmulski]

Can't verify as I'm on mobile but the job log seems to indicate we are trying to activate env fastai whereas the name of the CPU only env that we are installing is fastai-cpu

[apiltamang]

Nice ... despite being a dev. for several years now, I haven't yet had any familiarity with a CI system. Will be good to track this. Would you mind tagging me in future developments?

Thanks much!

[Gokkulnath]

@PiotrCzapla I Would like to help in writing tests. Though i am not totally aware how it is done but interested in spending sometime learning it.
Misc : Can we have a separate Environment File for Test Env ? I believe we don't need some of packages eg Jupyter, Ipython etc.. Guess this will reduce the time to run the test.
Tried figuring out few of unnecessary stuffs : Check out the Gist here

[PiotrCzapla]

@Gokkulnath the gist seems to be broken I see just the JS with plenty of document write :/ (after some fiddling I guess you meant this url: https://gist.github.com/Gokkulnath/31b91f1db29d171471269be07d9177ff)

The point of CI is to check that the dependencies don't collide so having them there may help us in the future, and should not hinder you when you write tests, as the test should be mostly executed locally on your pc /aws.

to run the tests you simply install pip install pytest then run python -m pytest tests. I'm playing with a docker instance to reporduce the crash we can observe at travis-ci and the test are failing so I can't show you how they look like but you will be able to see if they are working or not.

[PiotrCzapla]

update. I've managed to reproduce the crash in travis-ci locally in a docker instance I'm trying to see what I can do with that. Filtering tests only seem to solve the crash but introduce another one:

from PyQt5 import QtCore, QtGui, QtWidgets
E   ImportError: dlopen: cannot load any more object with static TLS

https://github.com/pytorch/pytorch/issues/2575

[jph00]

Importing torch first normally fixes that problem FYI.

We don't need qt - so agree that maybe a test-only env might help make

this faster and easier?

Jeremy Howard

On Tue, Apr 10, 2018, at 1:20 PM, Piotr Czapla wrote:

update. I've managed to reproduce the crash in travis-ci locally in a
docker instance I'm trying to see what I can do with that. Filtering
tests only seems to solve the crash but introduce another one:

from PyQt5 import QtCore, QtGui, QtWidgets E ImportError: dlopen:
cannot load any more object with static TLS

pytorch/pytorch#2575[1]

— You are receiving this because you commented. Reply to this email
directly, view it on GitHub[2], or mute the thread[3].>

Links:

1. https://github.com/pytorch/pytorch/issues/2575
2. https://github.com/fastai/fastai/issues/286#issuecomment-380233710
3. https://github.com/notifications/unsubscribe-auth/AAVLd0d1VmzAQB1wRd6jKyRZSSuBfSm_ks5tnRQjgaJpZM4THck4

[PiotrCzapla]

@jph00 I've seen you talking about this on pytroch issue tracker. I've fixed it by setting matplotlib to use Agg as backend and importing the torch and cv in a right order.

[PiotrCzapla]

The build is looking good after some spelling fixes (thx @Gokkulnath) we have first successful build on travis-ci. I've updated the transforms.py pull request #312 so that there are no conflicts, but I still want to get the minimal version of transforms2.py working so that the branch can be merged and the new API can be played within it separate space.

[PiotrCzapla]

@jph00 the ci it is working fine, It is bundled with the changes to transform.py but it can be extracted to a new branch if you wish.

[devforfu]

I wonder how usually people test deep learning algorithms implementations? I mean, if we're talking about performance metrics and model accuracy verifications, not asserting correct properties values? Quite time consuming process to train model and assert its quality I guess. I mostly trying to do it analytically, but probably someone knows possible alternatives?

[PiotrCzapla]

@devforfu I think Jeremy would prefer to talk about this on forum. Here is one article that maybe of interest to you, William implemented some tests for SWA:
https://medium.com/@hortonhearsafoo/adding-a-cutting-edge-deep-learning-training-technique-to-the-fast-ai-library-2cd1dba90a49

The tests aren't yet merged to Fast AI because they are bundled with other changes that @jph00 is reluctant to merge, so I will split the pull request to two and once it is merged maybe we start a thread on the forum to organise some ppl to write more tests to the library.

I wouldn't be too worried with testing the deep-learning parts yet as we are too early in the process. The Fast.ai does not have any test yet and bugs in regular code like transformations can easily break your code the same way as bugs in deep-learning parts.

[PiotrCzapla]

@jph00 once the pull request above is merged you need to get a https://travis-ci.org account for fastai, it is free for opensource it uses your github credentials to pull the right repositories, then you just need to go to your profile and select fastai for tests then you should be able to see the tests running at this url:
https://travis-ci.org/fastai/fastai

Here is how it should look like: