Geonode: setup.py has hardcoded versions

[ingenieroariel]

This makes it harder to use GeoNode with some depedencies coming from a PPA and others coming from PIP.

I believe setup.py should instead only limit minimum or maximum versions that come with incompatibilities. For reproducible builds we can already specify == on requirements.txt so there is no needed to make setup.py more restrictive than it should.

/cc @kalxas @afabiani @francbartoli @simod

[kalxas]

I agree with @ingenieroariel
setup.py should not have strict version numbers

[simod]

Yes, agree.

[capooti]

I am not sure about this. Please remember me the reason why we are using ppa packages vs pypi ones in debian setup

[afabiani]

@ingenieroariel @kalxas @simod @capooti @francbartoli

Accordingly to the issue statement, we can start from here
https://github.com/GeoNode/geonode/pull/3240

can you please take a look?

[cezio]

Could we use one single source point for dependency? It can be requirements.txt, setup.py can just read it, parse to list and put to install_requires.

As for strict versions, it could be relaxed, especially for patchlevel versions.

[ingenieroariel]

The reason there are two is because we need one for reproducible builds
(known configuration that works) with exact version numbers in order to be
able to do releases. Those exact packages can live in a text file (called
requirements.txt, and used by pip directly) or can live in a PPA as debian
packages to simplify installation.

The other use case is running GeoNode as a dependency in a bigger system.
For example downstream projects, in this case you want to be able to use
GeoNode which whatever versions you are using downstream (an older
pinax-theme because you liked it more or a newer version of a package
because it fixes an issue you had)l In that case, when you do pip install
geonode or setup.py install GeoNode should not go and try to download the
latest versions of everything if stuff if the packages are already
available.

What this points to is a setup.py that just lists the package names that
are strictly needed to be able to run GeoNode, not including the
dependencies of contrib modules and only excludes major versions that are
known to be incompatible because there was a breaking change. And a
requirements.txt file that is super strict and we can use to make sure we
are all testing the same thing during the release cycles.

On Fri, Sep 29, 2017 at 4:18 AM, Cezary Statkiewicz <
[email protected]> wrote:

Could we use one single source point for dependency? It can be
requirements.txt, setup.py can just read it, parse to list and put to
install_requires.

As for strict versions, it could be relaxed, especially for patchlevel
versions.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GeoNode/geonode/issues/3227#issuecomment-333076257,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADW13Nt3VLTesMndxI752PFsKnA1PVGks5snLX_gaJpZM4PAJ5D
.

[ingenieroariel]

One good example is the one below. GeoNode will work with literally any
paver version for the past 7 seven years, but setup.py insists in
downloading a particular version (not higher, not lower). This is
unnecessary and potentially problematic if you need to use a particular
paver version for any reason. GeoNode should not care, but by putting that
in setup.py we are making it care.

In setup.py
"Paver==1.2.4", # python-paver (1.2.4)

On Fri, Sep 29, 2017 at 8:43 AM, Ariel Nunez [e-mail]
wrote:

The reason there are two is because we need one for reproducible builds
(known configuration that works) with exact version numbers in order to be
able to do releases. Those exact packages can live in a text file (called
requirements.txt, and used by pip directly) or can live in a PPA as debian
packages to simplify installation.

The other use case is running GeoNode as a dependency in a bigger system.
For example downstream projects, in this case you want to be able to use
GeoNode which whatever versions you are using downstream (an older
pinax-theme because you liked it more or a newer version of a package
because it fixes an issue you had)l In that case, when you do pip install
geonode or setup.py install GeoNode should not go and try to download the
latest versions of everything if stuff if the packages are already
available.

What this points to is a setup.py that just lists the package names that
are strictly needed to be able to run GeoNode, not including the
dependencies of contrib modules and only excludes major versions that are
known to be incompatible because there was a breaking change. And a
requirements.txt file that is super strict and we can use to make sure we
are all testing the same thing during the release cycles.

On Fri, Sep 29, 2017 at 4:18 AM, Cezary Statkiewicz <
[email protected]> wrote:

Could we use one single source point for dependency? It can be
requirements.txt, setup.py can just read it, parse to list and put to
install_requires.

As for strict versions, it could be relaxed, especially for patchlevel
versions.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GeoNode/geonode/issues/3227#issuecomment-333076257,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADW13Nt3VLTesMndxI752PFsKnA1PVGks5snLX_gaJpZM4PAJ5D
.

[cezio]

What this points to is a setup.py that just lists the package names that
are strictly needed to be able to run GeoNode, not including the
dependencies of contrib modules and only excludes major versions that are
known to be incompatible because there was a breaking change. And a
requirements.txt file that is super strict and we can use to make sure we
are all testing the same thing during the release cycles.

Still this doesn't negate that we should have one list of dependencies. setup.py can include requirements.txt and strip version information. Also, if you need specific versions that are not aligned with, let's say, general requirements lists, requirements file can be parametrised and passed to reading function as, for example, env var.

something like this:

import os
from pkg_resource

def read_requirements():
      fname = os.environ.get('GEONODE_REQUIREMENTS_FILE', 'requirements.txt')
      fpath = os.path.join(os.path.dirname(__file__), fname)
      with open(fpath, 'rt') as f:
            _requirements = pkg_resource.parse_requirements(f.read())
           return [r.name for r in _requirements]

setup(...
          install_requires=read_requirements(),
          ..)

[ingenieroariel]

I think this is a great idea Cezary.

On Mon, Oct 2, 2017 at 8:32 AM, Cezary Statkiewicz <[email protected]

wrote:

What this points to is a setup.py that just lists the package names that
are strictly needed to be able to run GeoNode, not including the
dependencies of contrib modules and only excludes major versions that are
known to be incompatible because there was a breaking change. And a
requirements.txt file that is super strict and we can use to make sure we
are all testing the same thing during the release cycles.

Still this doesn't negate that we should have one list of dependencies.
setup.py can include requirements.txt and strip version information. Also,
if you need specific versions that are not aligned with, let's say, general
requirements lists, requirements file can be parametrised and passed to
reading function as, for example, env var.

something like this:

import os
from pkg_resource

def read_requirements():
fname = os.environ.get('GEONODE_REQUIREMENTS_FILE', 'requirements.txt')
fpath = os.path.join(os.path.dirname(__file__), fname)
with open(fpath, 'rt') as f:
_requirements = pkg_resource.parse_requirements(f.read())
return [r.name for r in _requirements]

setup(...
install_requires=read_requirements(),
..)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GeoNode/geonode/issues/3227#issuecomment-333535219,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADW1wIZKcA9k6z_ljBO6F8inac6F_iXks5soOYAgaJpZM4PAJ5D
.

[olivierdalang]

Major versions changes of libraries usually break the api, and minor versions changes sometimes do, so dependencies like django-polymorphic>=0.9.2 are due to break at some point (which they do, and which currently makes it impossible to rely on pip to install geonode in a reproducible way).

I think there must be a max version set for all deps, at least sticking to major release (==x.*), or even more conservatively sticking to compatible releases (`~=x.y.*).

I like the idea of reusing requirements.txt in setup.py. It will help making sure setup.py actually installs compatible libraries.