Ulauncher: Feature: Automatic install of extension dependencies

[brpaz]

Hello.
I think the most common issue people have when installing extensions is because of missing dependencies.

It would be nice if Ulauncher could install dependencies automatically.
Probably the simplest and more native way, since Ulauncher is built with Python would be to use requirements.txt file. Ulauncher could detect if the requirements.txt file is present In the extension soure code and if yes it would run pip install -r requirements.txt automatically before starting the extension.

[gornostal]

Hi @brpaz

Thanks for the suggestion. I understand the problem with the missing dependencies and approach you've described should work.

[goodwillcoding]

@brpaz @gornostal one possible way of doing this that creating virtualenvs on the fly for each exensions with additional pathing to Ulauncher libs (not hard, but not a common case). Doing that would avoid having collisions between extensions requiring different versions of same library. Unfortunately that opens another can of worms wen it comes to upgrades.

[friday]

For a CLI dependency the below may work. Not really optimal for more complex dependencies, or if it's critical that they get the most recent version, though you could probably adapt it to work for some of these cases too.

import os
from distutils.spawn import find_executable

cmd = 'binary_name'
extDir = os.path.dirname(os.path.realpath(__file__))

# test if client is installed on system and is in PATH or change to local version
if not find_executable(cmd):
    cmd = '{}/{}'.format(extDir, cmd)

# test/get local version
if not find_executable(cmd):
    # Simple example assuming the url is directly to a binary. More likely you'll get an archive or a .deb which needs to be extracted
    urllib.urlretrieve('https://...', cmd)
    os.chmod(cmd, 0o755)

[brpaz]

Think this is an important issue. Most of the issues opened in my extensions repos are because of missing dependencies. Its getting a little bit annoying to just say "you need to install this using pip" in many issues.

I try to keep the dependencies required listed in the README but it doesn't seem to help much. ;(

We need a more automatic way.

[friday]

Yeah, the average user doesn't bother reading instructions it seem. Just posts straight to Disqus that it doesn't work, even if the dependency is mentioned in the title. :P

Would it be feasible to include it in the source? Your Lorem Ipsum extension for example has a tiny dependency, which is not likely that people have. For GPL dependencies you may have to use GPL too this way though. Not sure.

Also, when failing to meet a dependency, you could perhaps show a notification so users become aware of the issue: https://github.com/friday/ulauncher-clipboard/pull/8

[friday]

For pip this should work:

import subprocess
import sys

def ensure_import(package):
    try:
        return __import__(package)
    except ImportError:
        subprocess.call([sys.executable, "-m", "pip", "install", "--user", package])
    return __import__(package)

Example use:

lorem = ensure_import("lorem")
print(lorem.sentence())

It will use the correct pip binary for the system.

ensure_import() should not be run on demand within the KeywordQueryEventListener since it may take a second or so to install the module.

Binaries installed this way will end up in $HOME/.local/bin/. This is not likely in path, but you don't really need it to be. You can just run them with that prefix.

pip isn't a dependency of Ulauncher though, so maybe that needs to be handled too.

[Riezebos]

Even if I follow the prereqs of an extension I often still get errors because some packages are assumed to be installed (probably because they are common). I don't use Python 2 at all normally, so I haven't installed any packages there. Usually if I want to use an extension I first have to open main.py and manually run the import statements to see which packages pop up as missing (e.g. requests). To the outsider the Ulauncher extension ecosystem can look like it's an unmaintained mess because half the extensions throw a weird API error (on standard Ubuntu 18.04 at least).

After debugging I personally like the extensions ecosystem a lot!

[friday]

Even if I follow the prereqs of an extension I often still get errors because some packages are assumed to be installed (probably because they are common)

That's their mistake imo. Installing packages globally means you need sudo access to install extension, and also indirectly introducing conflicts with packages in the distro. :/

To the outsider the Ulauncher extension ecosystem can look like it's an unmaintained mess because half the extensions throw a weird API error (on standard Ubuntu 18.04 at least).

I think the problem here is made worse by failing silently or just logging the error. That is, not doing this: https://github.com/friday/ulauncher-clipboard/pull/8.

[Riezebos]

I agree on both points, but even if it's their fault it reflects badly on Ulauncher if it occurs often. Another solution for this would be to test every extension in a controlled environment with just the minimal requirements + stated prerequisites before the extension is added to the website.

[friday]

I think a simple solution that works for most extensions is better than spending a lot of time on this and adding much code that needs to be maintained.

Suggestion:

• Add a key/value list or multiple arrays for dependencies in the json manifest. It needs to separate python dependencies from binaries, and perhaps python modules from python binaries bundled as modules.
• Install the python dependencies locally when installing the plugin, unless the manifest specifies that it shouldn't be installed locally.
• Show a warning for any extension that have unmet dependencies, or binary dependencies which couldn't be detected on the system, rather than assuming extension devs will handle it (evidently they don't, and as you say, this reflects back on ulauncher). Perhaps even show a cli example how to install them (for python modules).

[brpaz]

Even if I follow the prereqs of an extension I often still get errors because some packages are assumed to be installed (probably because they are common)

Yes, I am guilty of this. I dont have many experience with Python and some modules are used everywhere so i think they are from the standard lib, when in fact they are not.

Add a key/value list or multiple arrays for dependencies in the json manifest. It needs to separate python dependencies from binaries, and perhaps python modules from python binaries bundled as modules.

Why not using requirements.txt, which is the standard way to define dependencies in Python, instead of adding to json manifest? it wont work for binaries tough. so yeah, I guess your solution of using the manifest would be more complete.

[friday]

Why not using requirements.txt, which is the standard way to define dependencies in Python, instead of adding to json manifest? it wont work for binaries tough. so yeah, I guess your solution of using the manifest would be more complete.

If that's better for devs and/or easier to implement/maintain I think it's reason enough. My main concern is actually showing a warning if a dependency is unmet so they can address the problem themselves. That could be an entry in the manifest while also supporting requirements.txt

[brpaz]

Just a compilation of issues opened in my Extensions repositories in the last weeks, all related to missing dependencies.

• KeyError · Issue #1 · brpaz/ulauncher-cloudflare
• Unexpected API error. · Issue #1 · brpaz/ulauncher-devhints
• Dependency confusion / Unexpected API error · Issue #3 · brpaz/ulauncher-gitlab
• no options available... · Issue #3 · brpaz/ulauncher-pwgen
• crashed instantly · Issue #5 · brpaz/ulauncher-pwgen
• UnhandledError in com.github.brpaz.ulauncher-docker extension · Issue #403 · Ulauncher/Ulauncher

This is getting out of control.

We need to find a way to handle this.

And while a Python dev can probably figure it out looking the extension logs, It is much harder for end users of Ulauncher that just go to the Extensions Website and install from there.

[ptandler]

While the installation of requirements is not yet automated, it would be helpful if the requirements are at least listed on the extension page, see #581
If I know that there are additional requirements, I can install them manually ...

[ptandler]

Well: And an error message complaining about missing requirements would also help of course. Ideally with hints of what is missing and how to install it. I don't remember exactly, but I think I just got a message saying something like "extension terminated" without anything else.