Glow: [Improvement] Testing infrastracture

[ayermolo]

How hard would it be to add windows build/unit tests as pre-merge check?

Thank You

[nadavrot]

This is a good idea. Is windows supported by the CI systems that we use? Can we add an additional external windows CI system? Do you know, @rdzhabarov ?

[rdzhabarov]

How hard would it be to add windows build/unit tests as pre-merge check?

for that we'd need to run continuous-integration/appveyor/pr similar to what onnx does. I do not think circleCI supports windows unfortunately (which would be ideal).

I need to see how much effort it's going to be.

[ayermolo]

@rdzhabarov any update on this?

[ayermolo]

@rdzhabarov @nadavrot to continue conversation from yesterday.
Looking at pytorch they have jenkings server hooked in to github to build on windows.
Looking around github looks like for CI it also offers AppVeyor
https://github.com/marketplace/appveyor
Poking around it does support c++ builds.
https://www.appveyor.com/docs/lang/cpp/

Would you guys be willing to provide a server, or would want us to provide one?
Do you need help setting up at least the build? If yes how would details of that look like?

Thank You
Alex

[nadavrot]

Is this a cloud service that we could buy? Is this what Pytorch uses?

[ayermolo]

Pytorch just uses jenkings server. Looking at URL looks like it's theirs, that they connected to github somehow: https://ci.pytorch.org/jenkins/job/caffe2-builds/job/py2-cuda9.0-cudnn7-windows-trigger-build/

Do you have contacts on Pytorch side to see how they did it?

The appveyor is something I found on github marketplace, along with Travis CI, so presumably integration that works similar to how Travis was integrated in to glow github. From Infrastructure perspective.

[nadavrot]

We can pay for appveyor. What do you say @rdzhabarov ?

[rdzhabarov]

I want to make sure that if we enable this we are not bottlenecked in our CI on Windows builds (like much smaller pool of machines comparing to capacity of CircleCI with linux images).

Likely we (fb) should have something in works with appveyor. I'll talk to few internal folks tomorrow and give an update. And I apologize for all the breakage we had with Windows builds.

[rdzhabarov]

@ayermolo I synced with some internal folks.
It seems that our options are:

• AppVeyor (Windows/Linux support), most popular one (have some minor issues, probably fine)
• Travis support is not mature, not a good candidate
• Azure devops, promising, https://andrewlock.net/building-an-open-source-github.../, supports windows/linux/macos but harder to configure.

I think AppVeyor would be a good option. If you can configure windows build on it - would be great! Procurement can be done by fb.

[ayermolo]

@rdzhabarov Thanks for looking in to it. Yeah looks like AppVeyor is the best candidate at the moment.
I can experiment with it. It's not clear to me the logistics of it. I can do it on our github fork and create how to, since I don't have access to your github. What do you think?

[rdzhabarov]

I can do it on our github fork and create how to, since I don't have access to your github

This sounds good. Once we have working config for it it should be easy to integrate here.
Thanks you for looking into this!

[ayermolo]

So tried out appveyor. Got it installed on our fork and was playing around with build scripts.
Big blocker right now is that they don't have llvm 7.0.1 libraries that are part of the windows image. Building it from scratch will take too long, plus with their free version build times out after 60 minutes.
Alternative is to download pre-build libraries, but open question is where to store them. They are 3gig, and too big for github.

[nadavrot]

I think that we can store the pre-built binaries in our amazon bucket. @artemrakhov

[ayermolo]

That would be awesome. @artemrakhov how would that work?
I also filed issue with appveyor, will see where that leads.
https://github.com/appveyor/ci/issues/2909

[artemrakhov-glow]

@ayermolo: sure, I can add pre-built LLVM binaries to our Amazon bucket. The bucket is owned by Facebook, but is world readable. We already use this bucket to store some models (http://fb-glow-assets.s3.amazonaws.com/models/).
We only need to find a way for you to send the binaries to us. I can suggest Dropbox, but anything will work.

[ayermolo]

Dropbox has 2gb limit, for free one, full zipped library with Debug/release is 3.5gig. Trying to see if there is internal storage I can put it on to share it.

[ayermolo]

@artemrakhov OK. I managed to get internal FTP account, and uploaded compressed debug/release libraries there. It requires username/password to access. I can email them directly to you. Would that work?

[artemrakhov-glow]

@ayermolo: yes. You can email me to the address shown in my GitHub profile.

[ayermolo]

cool sent. thanks

[artemrakhov-glow]

The package is now accessible at http://fb-glow-assets.s3.amazonaws.com/bin/llvmbuilddir64.7z

[ayermolo]

great, thanks.

[ayermolo]

Quick update.
Streamlined building supporting librarries, and created repo with pre-build ones:
https://github.com/ayermolo/glowWindowsSupportLibraries
Did PR to ONNX with minor change that would allow building all configs with one solution
https://github.com/onnx/onnx/pull/1966

On appveyor side, it greatly simplified things. Now it fails due to resource constraint. I think big part of it is giant llvm library folder. Cleaned it up locally so it reduces size considerably. Need to do some testing, and give new version to @artemrakhov

So slowly progressing as I have time.

[ayermolo]

@artemrakhov Did you see my email?

[artemrakhov-glow]

Yes, sorry for the delay.
Updated the package at the previous location:
http://fb-glow-assets.s3.amazonaws.com/bin/llvmbuilddir64.7z

[ayermolo]

@artemrakhov Thanks!
Got a proof of concept up and running on my test fork. https://github.com/ayermolo/glow/pull/3
Builds and runs unit tests in release mode.
Example of appveyor.xml

appveyor.zip

[ayermolo]

So what would be next steps to "productise" it on your github?
Also forgot to mention two caveats.
OCL backend is disabled, and I had to set warning level to 3. First one is we don't use OCL backend and I am not familiar with how to set it up, and at least free version was timing out with /Wall. It has 1hour build limit. Didn't seem like setting parallel build made a difference.

[ayermolo]

@nadavrot Do you still want to proceed with this?

[nickgg]

This looks good, what's the status on the FB end? @rdzhabarov