Ignite: Master branch nightly build depending on torch_nightly

[zasdfgbnm]

We need a build of ignite to allow people try nightly version of PyTorch together with ignite. Since the PyTorch is nightly built, it might be better to call this package pytorch-ignite-nightly which is also a nightly build of master branch, but with the dependencies on torch changed to torch_nightly.

cc: @vfdev-5

NB: I'm not sure the dependency should be torch_nightly or torch-nightly, because the install command has torch_nightly but the output of pip says torch-nightly:

gaoxiang@pc ~ $ sudo pip install torch_nightly -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
Looking in links: https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
Collecting torch_nightly
  Downloading https://download.pytorch.org/whl/nightly/cpu/torch_nightly-1.0.0.dev20181029-cp37-cp37m-linux_x86_64.whl (85.9MB)
    100% |████████████████████████████████| 85.9MB 698kB/s 
Installing collected packages: torch-nightly
Successfully installed torch-nightly-1.0.0.dev20181029

@soumith Do you think we should change these two names to make them consistent?

[vfdev-5]

Updates:
Conda nightly packages:
https://anaconda.org/pytorch/ignite-nightly/
PyPI nightly packages:
https://pypi.org/project/pytorch-ignite-nightly/

@zasdfgbnm could you please check if this works for your purposes ?

[zasdfgbnm]

Thanks a lot!

[vfdev-5]

To close this one, we need to improve the following part
https://github.com/pytorch/ignite/blob/e3299b6c03e9b7f15cf294fa2b378116628ca951/.travis.yml#L174-L182

The problem is that git push did not update nightly branch on the github.

EDIT: this idea with push has potential problem multiple travis building of nightly branch. We need to redo this part

[vfdev-5]

@zasdfgbnm as they just released 1.0 stable maybe we do not need this issue anymore ?

[zasdfgbnm]

@vfdev-5 Hmm... Actually I don't know. Shouldn't there be users that always use nightly pytorch, no matter whether it is 1.0 or 1.1 or something else?

[zasdfgbnm]

@vfdev-5 I don't understand why do we need a separate nightly branch. Could it be something like this? https://github.com/pytorch/ignite/pull/363
I don't know travis well, so my this question might be stupid...

Is this because travis's pypi deployer always re-download things from a branch from scratch, instead of using the one just built on their CI environment?

[vfdev-5]

@zasdfgbnm we have a CRON travis event that is used for building and uploading configured on nightly branch. Otherwise every master merge will produce a build/upload. Secondly, I do not want to expose nightly folder in master.

[zasdfgbnm]

@vfdev-5 Thanks. So could we also do the merge with master operation on the cron job? This would solve the problem of multiple job trying to push to nightly branch.

[vfdev-5]

However I agree that all stuff with git merge, push is a non-sense. I failed to test everything until the end. The problem with push stuff is that it will trigger another build/upload etc and loop...

[vfdev-5]

Maybe we could setup CRON on master, probably there is a env variable to verify whether it is a CRON build and we should deploy or it is a simple merge and we do not need to deploy.
However in this case we need to setup new dependency torch-nightly, version etc using sed. This is not evident for me.

[zasdfgbnm]

It is possible to test if in cron:
https://docs.travis-ci.com/user/conditions-v1#integration

For the versioning, I am not sure if I get your point. Is it possible that, we use cron on master to push to nightly (and trigger a commit build on nightly branch), and use commit's build on nightly to deploy?

Something like:

stage: push nightly
  if: branch = master && type = cron
  # push to nightly
stage: deploy nightly
  if: branch = nightly && type = push
  # deploy nightly

I don't understand why we need to do these using sed...

[vfdev-5]

Is it possible that, we use cron on master to push to nightly (and trigger a commit build on nightly branch), and use commit's build on nightly to deploy?

I think this can be a good idea ! Maybe we can think more about this. The problem is that in this case there are 2 tests executions instead of a single execution:

• first execution when CRON
• second execution when pushed into nightly

I don't understand why we need to do these using sed...

I thought you would like to remove nightly branch.

[zasdfgbnm]

If the cron is per day, then one extra execution per day would not be a big problem. Otherwise, can we just add if: branch = master && type != cron to all tests?

[vfdev-5]

Otherwise, can we just add if: branch = master to all tests?

There is a problem with that when tests are defined as it is done in our travis and not as a proper stage... They somehow become invisible when I tested in my fork..

[zasdfgbnm]

So maybe just keep the duplicate test there? It shouldn't be long.

[vfdev-5]

Otherwise I thought about making nightly branch as gh-pages without common origin to master and put there only nightly folder and its own travis file and execute git clone master and then deploy....

In this case we do not need to make nightly branch ~even with master.

However I have a doubt whether we could build such branch in travis...

So maybe just keep the duplicate test there? It shouldn't be long.

Yes, we can take this solution if we could not find something else.
Anyway, today nighlty builds are (it seems to me) passing correctly and built packages should reflect master version. Just nightly branch on the github becomes old as no more merge into it.

[zasdfgbnm]

Can we do something like this? https://github.com/pytorch/ignite/pull/364

[vfdev-5]

A link to keep it here: https://github.com/pytorch/pytorch/blob/master/tools/aten_mirror.sh
May help to keep nightly branch update to master...

[vfdev-5]

Closed in favor of #595