autogluon has too many dependencies

[leezu]

autogluon lists

requirements = [
    'Pillow<=6.2.1',
    'numpy>=1.16.0',
    'scipy>=1.3.3',
    'cython',
    'tornado>=5.0.1',
    'requests',
    'matplotlib',
    'tqdm>=4.38.0',
    'paramiko>=2.5.0',
    'dask==2.6.0',
    'cryptography>=2.8',
    'distributed==2.6.0',
    'ConfigSpace<=0.4.10',
    'gluoncv>=0.5.0',
    'gluonnlp==0.8.1',
    'graphviz',
    'scikit-optimize',
    'catboost',
    'boto3',
    'lightgbm==2.3.0',
    'pandas>=0.24.0,<1.0',
    'psutil>=5.0.0',
    'scikit-learn>=0.20.0',
]

as requirements.

This makes it essentially infeasible for other projects to introduce a dependency on autogluon and make use of it's tuning facilities.

It would be good for autogluon to provide a minimal core that allows external projects to integrate with it easily.

[Innixma]

I can see this being quite beneficial and healthy long term for users especially as AutoGluon gains more components.

An interesting point here is the complexity in regards to development once the package has been modularized. I'm not very familiar with handling projects split into various modules in terms of code-refactors, but it seems to me like it adds complexity it terms of ensuring the code is working across the package.

I feel like it may be too early in AutoGluon's development to introduce such complexities to its developers, thus potentially slowing down progress. This opinion primarily stems from my lack of experience in this matter, so I am happy to be shown counterexamples where it works well, and what kind of external integration becomes feasible upon modularization that was previously infeasible.

[mseeger]

This is an important request, but also not something simple to do quickly.

The best would be to think of a way to split between the core scheduling and optimization capabilities, and then the code needed to get important use cases going (e.g., GluonCV, GluonNLP, different boosting methods, ...).

On the other hand, I feel it needs to remain simple and robust to install the "sum of all important parts".

[leezu]

Would extras_require option in setup.py help solve this problem? For example, to use autogluon tabular, users wculd then be required to install pip install autogluon[tabular].

[Innixma]

What other packages make use of this? I think it would be good to take examples from other popular open source projects.

[leezu]

This is the standard way to declare optional dependencies https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies

I suppose any python project with optional dependencies uses it.

[Innixma]

Seems like it could be a nice solve, although I haven't seen other packages use this before, which is why I was wondering what other packages use it.

Can we do the reverse? pip install autogluon installs everything but we have pip install autogluon[tabular] which only installs tabular dependencies and pip install autogluon[core] which only installs core dependencies etc.?

[szha]

GluonNLP does: https://github.com/dmlc/gluon-nlp/blob/a0d7fbb1b9be4b155c9b03ecdf56ae4c844389c8/setup.py#L64-L93

and so does GluonTS: https://github.com/awslabs/gluon-ts/blob/master/setup.py#L212-L217

[szha]

@Innixma The feature is for extra dependencies so the semantics of autogluon[tabular] means the required dependencies plus the extra dependencies for tabular. What you described sounds more like having a separate package for the tabular part (e.g. autogluon-tabular), which is a feasible alternative.

[mseeger]

Let us separate two issues here:
1) Different built-in large features of AutoGluon (e.g., tabular)
2) ML frameworks we'd want AutoGluon to apply to (for tuning parameters) (e.g., GluonCV,
GluonNLP, )

I personally would have 1) always be part of the core, unless it becomes really large. I think we want a strategy for dealing with 2). Not only will many users not care, but others who do care, will have their favorite version and not be happy with our dependency, etc.

And for newby users, we should certainly have a smooth way to deal with 2), so they can get up and running quickly. Not an easy task ...

[jwmueller]

Can we do the reverse? pip install autogluon installs everything but we have pip install autogluon[tabular] which only installs tabular dependencies and pip install autogluon[core] which only installs core dependencies etc.?

I also like this solution the best but would be good to see examples.
It would be nice to keep it as simple as possible for users who just want to quickly try out various autogluon functionalities (single pip install). Users who've already tried autogluon and decided they like it enough to integrate it into a bigger system can then install only the dependencies for components they want to integrate.

[leezu]

You can provide pip install autogluon[all]. I don't think the reverse is supported, but maybe I'm wrong