Super-linter: How to install Python dependencies

[yhoiseth]

In https://github.com/yhoiseth/python-prediction-scorer/runs/786096970, pylint is complaining about missing imports. They are already installed in a previous step, but it doesn’t seem like they are “kept”. How can I run e.g. python -m pip install --upgrade pip pip-tools setuptools wheel && pip-sync so that it works with Super-Linter?

[admiralAwkbar]

This is a great question!
I know we have had to add the lines like below before...

import click # pylint: disable=import-error

But I think it may stem from where the linter actually runs, like directory wise...
I need to do some more testing to help understand this...
I know there is a way to tell it where to find the import like: import click # pylint: import-path: /tmp/here or something like that but can't find the documentation... maybe someone more versed in pylint could help out?

[kevinfealey]

I came here looking for an answer to this same question - just wanted to bump it for visibility.

I had tried the following:
- name: Pip Installer (application dependencies) uses: BSFishy/pip-action@v1 with: requirements: requirements.txt
But I still get import errors.

[dang3r]

I attempted to handle this by disabling import-errors using a custom pylint configuration. Prepending import-error to the list of disabled errors at https://github.com/github/super-linter/blob/master/TEMPLATES/.python-lint#L57 prevents this error from being caught.

However, this project runs pylint using the -E/--errors-only flag in addition to using the custom configuration file. See here. The use of this flag seems to override the disabled errors present in the pylint configuration file.

The one workaround I found was to use --disable=R,C,W instead of the -E flag. This _seems_ to disable all other types of checks except for errors, and it properly handles the disabled checks in the config file. I'm not sure if this is the right solution, but it could work.

[jgaffiot]

I finally found a workaround (works with pip + requirements.txt or pipenv). The problem is to find a way to provide your dependencies to the super-linter container, and then to tell pylint where they are.

For the first step, we can install our dependencies in the container running the action, but the super-linter runs in a separate container. So we have to share the dependencies with the super-linter container, something we can do with container volumes. But we can not declare new volumes... So let's look at the volume already mounted when the super-linter runs:

-v "/var/run/docker.sock":"/var/run/docker.sock"
-v "/home/runner/work/_temp/_github_home":"/github/home"
-v "/home/runner/work/_temp/_github_workflow":"/github/workflow"
-v "/home/runner/work/<your_repo>/<your_repo>":"/github/workspace"

We could put our dependencies in /home/runner/work/<your_repo>/<your_repo>, but they will be linted too, so let's use /home/runner/work/_temp/_github_workflow.

Then we can tell pylint where are the dependencies by using the environment variable PYTHONPATH. Right, all together we get:

---
# Beginning is the same than the official documentation
name: Lint Code Base
on:
  push:
    branches-ignore:
      - 'master'
jobs:
  build:
    name: Lint Code Base
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2
      # Here I use the system Python, which is 3.6.
      # You may install another Python version with the proper action
      - name: Install missing python packages
        run: sudo apt-get install -y --no-install-recommends python3-venv python3-setuptools
      # USE ONLY ONE THE 2 STEPS BELOW
      ## Dependencies with pip + requirements.txt
      - name: Install dependencies with pip + requirements.txt
        run: |
          python3 -m venv .venv
          .venv/bin/pip install -r requirements.txt
      ## Dependencies with pipenv
      - name: Install dependencies with pipenv
        run: |
          python3 -m pip install pipenv
          python3 -m pipenv sync --python 3.6
        env:
          PIPENV_VENV_IN_PROJECT: 1
      # Now we move the dependencies where super-linter can see them
      - name: Move the dependencies
        run: mv .venv /home/runner/work/_temp/_github_workflow
      # Now we set the PYTHONPATH to the path of the dependencies *inside* the container
      - name: Lint Code Base
        uses: docker://github/super-linter:v2.2.0
        env:
          VALIDATE_ALL_CODEBASE: true
          PYTHONPATH: "/github/workspace/:\
            /github/workflow/.venv/lib/python3.6/site-packages"
...

Hope this helps!

[bazzadp]

The other workaround is to copy the template into .github/linters/.python-lint and then add these to ignored modules at line 246. For example:

# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
ignored-modules=flask,
                flask_talisman,
                matplotlib,
                matplotlib.pyplot,
                mistune,
                pandas,
                scour,
                werkzeug.routing,
                werkzeug.http

[elopez]

@jgaffiot I'm struggling to get your workaround to work, I suspect it is because some of the dependencies I need have binary components (.so libraries), and the super-linter docker image uses Alpine Python, which has musl libc, in contrast to the Ubuntu host where the venv is created, which uses glibc. Do you have any ideas on how we could make it work? :thinking: Thanks!

[jgaffiot]

If the problem really comes from incompatibilities between glibc and musl, I suppose you would have to build your Python dependencies in an Alpine-based image. Unfortunately, it seems that Github does not provide Alpine-based image for actions, neither natively and neither with self-hosted runner. Perhaps by building your own action based on an image taken on DockerHub ?
The other way would be to convince Github to provide another super-linter action, based on Ubuntu, which I suspect would not be much larger than an Alpine-based one once Python is installed.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[alerque]

@github-actions The problem is not fixed, ergo the issue should not be closed.