Gensim: Missing wheels for Windows

[piskvorky]

Lately, there's been a stream of issues from Windows users, e.g. here.

Looking at PyPI, it seems binary wheels are completely missing from the last few distributions (incl. the latest 3.8.0):
https://pypi.org/project/gensim/#files

This means Windows users don't get to install the optimized Gensim easily, and unless they have a compiler properly set up, they'll install the slow (uncompiled) version of word2vec, fastttext etc.

We definitely want those binary wheels on PyPI, especially for Windows. Windows users are the least tech-savvy, least likely to have a C compiler, and most likely to raise confused support tickets.

[piskvorky]

The last distribution with Windows wheels seems to be 3.7.1, from January 2019:
https://pypi.org/project/gensim/3.7.1/#files

@mpenkov IIRC this coincides with you taking over from @menshikh-iv – did you perhaps forget some release steps?

[mpenkov]

@piskvorky I had a look, the problem is caused by failing AppVeyor builds (we use Travis CI for Linux and Mac, and AppVeyor for Windows). During the handover, I spent a lot of time dealing with problems in Travis CI, and neglected to check AppVeyor. Since the builds were failing, the Windows wheels weren't being built.

I'll investigate why the builds are failing - it seems to be a MemoryError in FastText code (https://ci.appveyor.com/project/piskvorky/gensim-wheels/builds/25836814/job/8yiy55wlf43soy01). Since the tests pass everywhere else, I suspect the AppVeyor machine is genuinely running out of memory, but I will need to double-check this.

Once that's fixed, I'll add some failsafes to prevent this from happening again:

1. Update documentation to explicitly focus on both Travis CI and AppVeyor (right now, it just mentions CI vaguely)
2. Add automatic checks for wheels for all platforms before finalizing the release

[piskvorky]

OK, thanks. This is important, definitely warrants a new "bugfix release".

[piskvorky]

@mpenkov do you have an ETA on those wheels being on PyPI?

Also needed here https://groups.google.com/d/msg/gensim/3ms1iH_Miuc/Wej0Go7EAQAJ

Having an internal automated checking process will be nice, but not as urgent as having the wheels actually available for Windows users.

[mpenkov]

@piskvorky No ETA at the moment, but I'm working on it. I'm going to try skip the tests altogether and rebuild the wheels for the current release. Ideally, we'd figure out what's causing those MemoryErrors on appveyor, but that's going to take substantially longer.

[piskvorky]

OK, thanks. The sooner the better, this is awkward (another request today and one more).

[mpenkov]

I got the wheels to build (see https://ci.appveyor.com/project/piskvorky/gensim-wheels), and those wheels should be uploaded to rackspace. I'm having trouble downloading them, because they don't appear to be where I'm expecting them (see http://b153eb958f4da6029aca-3f9dff7fe564350f10153d8c7bfc5ab6.r54.cf2.rackcdn.com/, no Windows wheels for 3.8.0 there).

@menshikh-iv Any ideas where I can get the built wheels?

[mpenkov]

@piskvorky I couldn't find the new wheels, so I'm going to make a bugfix 3.8.1 release, and hope that the wheels get built properly as part of the release process. I'll let you know how that goes within a day or so, when the CI chews through the jobs.

[mpenkov]

@piskvorky I still couldn't get the CI to publish the wheels, but I was able to download them manually and include them in the 3.8.1 release.

[mpenkov]

Opened https://github.com/MacPython/gensim-wheels/issues/15. We'll have to deal with that issue separately.

[menshikh-iv]

So, we found a reason, this will be fixed soon

[mpenkov]

Yay, thanks @menshikh-iv !