pandas-0.19.0rc1: 'global name tempita is not defined'

[jgerardsimcock]

I am running a Travis CI build in a python 2.7 min environment. Not sure why it is pulling from an unreleased version of pandas but it looks like the tempita needs to be imported.

[jreback]

you must have a really really old version of cython (or maybe cython is not installed). you are building from source so you need to have the development requirements. http://pandas.pydata.org/pandas-docs/stable/install.html#installing-from-source

This is not recommended unless you are developing pandas. there are wheels and conda packages available.

[jreback]

acutally this should be a better reported error (meaning setup should fail)

when cython is NOT installed (as is required), we cannot run tempita.
So need to put a check in.

[Sun Sep 11 19:47:16 ~/pandas]$ rm pandas/src/algos_groupby_helper.pxi
[Sun Sep 11 19:47:36 ~/pandas]$ make
python setup.py build_ext --inplace
running build_ext
Traceback (most recent call last):
  File "setup.py", line 94, in <module>
    if cython:
  File "/Users/jreback/miniconda/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/jreback/miniconda/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/jreback/miniconda/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/jreback/miniconda/lib/python2.7/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "setup.py", line 368, in build_extensions
    build_ext.build_extensions(self)
  File "setup.py", line 140, in build_extensions
    pyxcontent = tempita.sub(tmpl)
NameError: global name 'tempita' is not defined
make: *** [tseries] Error 1

[fungi]

It's unclear to me why this was implemented with the equivalent of an assertion in setup.py rather than adding cython to the setup_requires list. Pandas 0.19.0 upgrades via pip on Linux (due to lack of Linux wheels support on PyPI) collapse catastrophically, requiring someone to manually pip install cython into the env first.

[jreback]

@fungi not sure what you are taking about

this issue is fixed on 0.19.0
and wheels are supoorted on linux

[fungi]

Wheels are supported on Linux, but Linux wheels (the manylinux1 platform PEP aside) are not supported on PyPI. Thus, pip install pandas on Linux throws "ImportError: Building pandas requires cython" instead of installing cython automatically. I'm in the process of opening a separate issue now for this and will gladly submit a PR updating setup.py to use setup_requires for cython unless there's a good reason it shouldn't.

[jorisvandenbossche]

@fungi Have a look at PyPI, the linux wheels are definitely there: https://pypi.python.org/pypi/pandas. Maybe you need to update pip to download them?

[jreback]

@fungi before you open an issue
please show an authoritative source that says setup_requires is a good idea -
and is a standard way to do this

development building has always required cython

[fungi]

Those are the manylinux1 platform wheels to which I was referring. Support for manylinux1 was first added in pip 8.1.0 (March 2016), so yes this should work for anyone using a pip release from the past 7 months. I'm just curious why this was not added as a setup_requires so that it can work on a broader set of platforms which either may not automatically retrieve manylinux1 wheels or need to operate on sdists for other reasons. In contrast, you already have a setup_requires declared on numpy rather than simply aborting setup.py when you don't see it available in the environment.

[jreback]

hmm, we are using setup_requires already. ok, then guess would take a PR for adding cython there. Not really sure why we don't have that already, though we do have a slightly different process than some other libraries. IOW, we _don't_ checkin in .c and instead build them in the dev version.

[fungi]

As to "development building has always required cython," prior to 0.19.0 development building must not have included installs from sdist since pip install 'pandas<0.19.0' on a system with, e.g., pip 7.1.0 successfully retrieves pandas-0.18.1.tar.gz from PyPI and installs it into the environment.

[lmeyerov]

FWIW, we're hitting this in our various builds and are not sure how to treat. We're seeing the same error for both our internal jenkins & public usages, e.g., https://readthedocs.org/projects/pygraphistry/builds/4425847/ .

[jorisvandenbossche]

The error message can be improved (but that is fixed in 0.19.0, which is released in the meantime), but, as explained in the above comments, this just means that you are trying to build pandas from source, which requires cython to be installed. And this has not changed in comparison to previous versions, this has always been the case.

However, normally there should be wheels available. Not sure why readthedocs is downloading the zip source instead of the binary wheels.

[fungi]

I think there's probably still some confusion here. The "fix" in 0.19.0 makes it impossible to install the pandas sdist (.tar.gz or .zip) packages published on PyPI without manually preinstalling cython in the environment. The sdists of prior releases up to and including 0.18.1 could be installed without cython present. Digging deeper, it seems this worked because you were (and still are, in fact) shipping the cython-generated files in your sdists, which made them installable without cython present. So yes, _creating_ the sdist does require cython (and this is what I assume you mean by "building from source," e.g. setup.py install or setup.py sdist or similar), but _installing_ the created sdist did not require cython until the additional raise ImportError was added in 461e0e9. I get that it made "installs from source" fail with a slightly more obvious error, but it also broke "installs from sdist" at the same time.

[frol]

There is no wheel for Alpine Linux, so installing Pandas 0.19.0 from PyPi without Cython installed manually, the exception is raised and the installation breaks: https://hub.docker.com/r/frolvlad/alpine-python-machinelearning/builds/bfih8p7usyqibefhnsjk5be/

[jorisvandenbossche]

@fungi Aha, thanks, that clarifies the issue. We should try to solve that for 0.19.1