Cellprofiler: Error running welcome screen pipeline from master

[braymp]

Using fccab88f73f9df9b490e3950595598006c1e232b to load the welcome screen pipeline (and correcting it as noted in https://github.com/CellProfiler/CellProfiler/issues/2614). Running it yields the following error:

Traceback (most recent call last):
  File "cellprofiler\gui\pipelinecontroller.py", line 2847, in do_step
    self.__pipeline.run_module(module, workspace)
  File "cellprofiler\pipeline.py", line 1997, in run_module
    module.run(workspace)
  File "cellprofiler\modules\identifysecondaryobjects.py", line 479, in run
    (labels_in[0, :], labels_in[-1, :], labels_in[:, 0], labels_in[:, -1]))
  File "C:\Python27\lib\site-packages\numpy\core\shape_base.py", line 280, in hstack
    return _nx.concatenate(arrs, 1)
ValueError: all the input array dimensions except for the concatenation axis must match exactly

[mcquin]

Huh. I'm not able to duplicate.

Are your dependencies up to date? If you're using pip you can run pip install --upgrade --process-dependency-links --editable .. I'm not sure what the equivalent conda command is.

[braymp]

After upgrading (which unfortunately, I had via wheels from pythonlibs), I ran pip install --upgrade --process-dependency-links --editable . to see if everything worked and I got this error:

Obtaining file:///C:/Users/brayma2/Desktop/github/cellprofiler
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\brayma2\Desktop\github\cellprofiler\setup.py", line 84, in <module>
        os.environ["PATH"] += os.path.pathsep + os.path.split(zmq.__file__)[0]
    NameError: name 'zmq' is not defined

If it helps, 'pip freeze' gives me:

appdirs==1.4.1
cellh5==1.3.0
centrosome==1.0.5
colorama==0.3.7
cycler==0.10.0
Cython==0.23.4
dask==0.13.0
decorator==4.0.11
functools32==3.2.3.post2
h5py==2.6.0
hmmlearn==0.2.0
inflect==0.2.5
javabridge==1.0.14
libtiff==0.4.0
lxml==3.7.3
mahotas==1.4.3
matplotlib==2.0.0
MySQL-python==1.2.5
networkx==1.11
numpy==1.12.0
olefile==0.44
packaging==16.8
pandas==0.19.2
Pillow==4.0.0
prokaryote==1.0.11
py==1.4.32
pyparsing==2.1.10
pytest==3.0.6
python-bioformats==1.1.0
python-dateutil==2.6.0
pytz==2016.10
PyWavelets==0.5.1
pyzmq==16.0.2
scikit-image==0.13.dev0
scikit-learn==0.18.1
scipy==0.18.1
seaborn==0.7.1
six==1.10.0
toolz==0.8.2
verlib==0.1
wxPython==3.0.2.0
wxPython-common==3.0.2.0
zmq==0.0.0

[mcquin]

Huh. zmq==0.0.0 looks weird. pyzmq==16.0.2 is correct, though. Can you pip uninstall zmq and try again?

[braymp]

Just gave it a try; same error.

[mcquin]

Hm. Does zmq import cleanly?

Open up a terminal and run:

> python

import zmq

[braymp]

It does.

>>> import zmq
>>> zmq.__version__
'16.0.2'

[braymp]

At the python prompt, I also tried (from setup.py)

>>> import importlib
>>> importlib.import_module(zmq)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'zmq' is not defined
>>> import zmq
>>> zmq.__version__
'16.0.2'

Weird.

[mcquin]

You'll need to add quotes: importlib.import_module("zmq")

[braymp]

duh, thanks.

importlib.import_module("zmq")
<module 'zmq' from 'C:\Python27\lib\site-packages\zmq\__init__.pyc'>

[mcquin]

Try adding import zmq on line 83 (or anywhere in this block, before the call to zmq.__file__).

[braymp]

Got it, I think. The try/except at https://github.com/CellProfiler/CellProfiler/blob/master/setup.py#L56 fails on import scipy.sparse.csgraph._validation, so it never gets to import zmq

The fault is

>>> import scipy.sparse.csgraph._validation
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\site-packages\scipy\__init__.py", line 61, in <module>
    from numpy._distributor_init import NUMPY_MKL  # requires numpy+mkl
ImportError: cannot import name NUMPY_MKL

[braymp]

More details: Attempting to pip install scipy when updating had failed for me, so I fell back to d/l'ing the wheel from the pythonlibs site (as I always do). Apparently, that version requires numpy+mkl.

[0x00b1]

@braymp I recommend using the pre-compiled binary available from PyPI, e.g.

pip install --upgrade pip
pip install numpy

It will download and install numpy-1.12.0-cp27-none-win_amd64.whl.

[braymp]

@0x00b1: Any other packages (other than scipy) that I should uninstall first before re-installing numpy?

It's been an open question in my head for some time as to whether mkl is actually needed....

[braymp]

Performing pip install numpy, then pip install scipy leads to the compilation error described here: http://stackoverflow.com/questions/33368261/what-is-the-easiest-way-to-install-blas-and-lapack-for-scipy.
I've come across this before, and now I remember why I started using wheels in the first place...

When I've googled this in the past, the solutions seem to be either:

• Use the MKL wheels from pythonlibs site
• Use distros (e.g., Anaconda) which have it pre-packaged.

I've always gone to option 1, but are there other options? Having been down this road already, I feel like the numpy/scipy MKL-or-not issue is the main blocker for a fully PyPI accessible approach (at least, for me)

[mcquin]

Duplicate of #2647 resolved by #2650