Elpy: Project management with ELPY + Skeletor + Projectile

[montaropdf]

Hi,

What is the usual workflow when working with the packages mentioned in the title and the tools from the python ecosystem (setuptools, pipenv, virtualenv, ...)?

I am working sporadically, with python since version 2.3, and I have never used setuptools, nor virtualenv. So, now that I want to work on some serious project, I want to explore there usage and how to combine them with Emacs and the tools it provides to work efficiently with python.

I am currently updating the skeleton to create python projects with skeletor, and I am wondering how to ensure it blends properly with ELPY and the current packaging and standard of python.

The following sections show what I have done so far.

My first questions are:

1. Is a makefile a proper way to initialise the project environment?
2. I read that pipenv is meant to replace setuptools, what is the perception of seasoned python developpers and is it (or will it be) supported by ELPY?

The skeleton I have created so far

  drwxr-xr-x. 2 roland roland 4096 19 déc.  19:27 .
  drwxrwxr-x. 4 roland roland 4096 18 déc.  20:11 ..
  -rw-r--r--. 1 roland roland  764 17 déc.  22:09 CONTRIBUTING.md
  -rw-r--r--. 1 roland roland  541 17 déc.  22:09 __DOT__gitignore
  -rw-r--r--. 1 roland roland    0 17 déc.  22:09 __init__.py
  -rw-r--r--. 1 roland roland    0 17 déc.  22:09 INSTALL
  -rw-r--r--. 1 roland roland  465 19 déc.  19:27 Makefile
  -rw-r--r--. 1 roland roland  467 18 déc.  20:18 Makefile~
  -rw-r--r--. 1 roland roland  254 17 déc.  22:09 README.md

The contant of the Makefile

PYTHON_BIN = __PYTHON-BIN__
VIRTUALENV = __VENV-DIR__/__PROJECT-NAME__

all : $(VIRTUALENV)

# Configure virtualenv for this project.
$(VIRTUALENV) :
    __VENV-BIN__ -p $(PYTHON_BIN) $(VIRTUALENV)
    source $(VIRTUALENV)/bin/activate

# Install editor tooling.
.PHONY: tooling
tooling : $(VIRTUALENV)
    pip3 install --user jedi
    pip3 install --user epc
    pip3 install --user flake8
    pip3 install --user black
    pip3 install --user autopep8
    pip3 install --user yapf
    make

The elisp code written to populate some of the skeletor variables

(defun get-python-bin-list ()
  "Return a list of python executables."
  (let ((python-bin-list))
    (dolist (python-bin-path skeletor-python-bin-search-path)
      (setq python-bin-list (append python-bin-list
                                    (directory-files python-bin-path t "^python.*")))
      )
    (print python-bin-list)))

(defun get-venv-bin-list ()
  "Return a list of virtualenv executables."
  (let ((venv-bin-list))
    (dolist (python-bin-path skeletor-python-bin-search-path)
      (setq venv-bin-list (append venv-bin-list
                                  (directory-files python-bin-path t "^virtualenv.*")))
      )
    (print venv-bin-list)))

(add-to-list 'skeletor-global-substitutions
             (cons "__PYTHON-BIN__" (lambda () (completing-read "Python Binary to use: "
                                                                (get-python-bin-list)
                                                                nil
                                                                nil
                                                                nil
                                                                'minibuffer-history))))

(add-to-list 'skeletor-global-substitutions
             (cons "__VENV-BIN__" (lambda () (completing-read "Virtualenv Binary to use: "
                                                              (get-venv-bin-list)
                                                              nil
                                                              nil
                                                              nil
                                                              'minibuffer-history))))

(add-to-list 'skeletor-global-substitutions
             (cons "__VENV-DIR__" (lambda () (read-string "Location of the Python Virtual Environments: " nil 'pyvenv-workon-home))))

[jorgenschaefer]

What is the usual workflow when working with the packages mentioned in the title and the tools from the python ecosystem (setuptools, pipenv, virtualenv, ...)?

I do not know skeletor.

Elpy mimics the virtualenvwrapper workflow. So usually:

$ mkvirtualenv newproject
M-x pyvenv-workon RET newproject RET
M-x elpy-config RET

The last will prompt you for packages you might or might not want, and provide buttons to install them.

You probably want a project directory, ideally version controlled with git. You then just start editing there.

pipenv is newer, but Elpy should work with it just the same, as it also just creates virtualenvs. There's documentation for a good workflow here: https://docs.python-guide.org/dev/virtualenvs/

(If anyone's interested in updating elpy or pyvenv to be more pipenv-compatible, I'd be happy :-) )

[montaropdf]

Skeletor creates the directory and initialize a git repository in it.
On top of that, shell commands and elisp code can be embedded in the creation of the project, so, creating an ELPY python template should be possible.

[gopar]

Honestly, there's not much to add. I always add a .dir-locals.el file in the root of the project so elpy and pick up the variables and apply them whenever it's inside a file of that directory.

Here's the typical example I have in mostly all of the projects:

((nil
  ;; first create virtualenv named `my-project`
  (pyvenv-workon . "my-project)

  (eval setq flycheck-python-mypy-args
        `("--config-file" ,(format "%ssetup.cfg"
                                   (elpy-project-root)))) ;; use the setup.cfg file for configuration

  (eval setq py-isort-options
        `(,(format "-sp=%ssetup.cfg"
                   (elpy-project-root)))) ;; use setup.cfg file for configuration

  (eval setenv "PYTHONPATH" "/Users/gopar/Downloads/google-cloud-sdk/platform/google_appengine/"))) ;; change Python path, append to it, etc

With this, I don't have to configure elpy whenever I visit a different project, since that proj will also have it's own dir-locals.el that elpy will pick up.

Also, just a little tip, you might want to name virtual envs with the version of python you are using to make it easier for your self.

eg, if I'm using python 3.7 then I usually go for my-project37 so right from the start I know what version of python this venv uses

[montaropdf]

@gopar, thanks for the tip about .dir-local.el, I use emacs for almost 20 years and it is the first time I read about its existence.

By the way, to avoid a long thread on the basics of python project creation and management, can someone provide one or more articles on these topics, using virtualenv, pip, setuptools, etc?

I have read the documentation for virtualenv and setuptools, but they are just considering their perimeter of activity and not the management of one or more projects as a whole, and my head has still difficulties to see how they help the developer to manage its projects.

@jorgenschaefer, their is already an emacs package for pipenv, call pipenv.el, maybe its maintainer could see to integrate its tools with ELPY.

[jorgenschaefer]

I'm happy to integrate existing software. I suspect the work needed is more on the Elpy side of things, though. Elpy assumes the existing virtualenv workflow. But part of "supporting pipenv as the default" would be to find out what needs to be done! :-)

[montaropdf]

For what I understand, pipenv manage both pip packages add virtualenv. I think it is worth having a talk with the creator of pipenv.el, @pwalsh. He is, obviously more informed than me on the subject.

[lljr]

I'm an Elpy user but would like to see more integration with pyenv and poetry. I like how Poetry is simple to use and only requires pyenv to manage the python versions. Maybe Elpy could integrate pyenv and poetry in the future?

[galaunay]

@avocadoras What kind of integration do you have in mind ?

It should be relatively easy to switch to the right virtualenv when editing a file from a poetry project.
Being able to run poetry commands from Elpy would be more difficult though.

[lljr]

@galaunay nothing too fancy. Just something that would allow me to change the pyenv version and a function that would allow me to create new projects with poetry. I know I could probably do this myself but I'm very new to elisp. Therefore, I do not have enough knowledge to add changes myself.

I know that the way I use Python with pyenv and poetry is very simple and probably don't need these extra functions in Emacs, but, nevertheless, it would be neat to see some integration with tools other than pyvenv or virtualenvwrapper.sh. Since the venv module comes with Python 3.4+, wouldn't it make sense to reduce the tooling by favoring built-in modules? That's how I see my Python usage when I use tools like pyenv, to manage my versions, and poetry, to start new projects.

[galaunay]

There is already a pyenv-mode package, that we should be able to integrate as an Elpy module.

For poetry, I couldn't find any emacs package.
In my opinion, it should be a package on its own, that Elpy could use as a dependency.
It should not be too difficult to do so with something like transient (note to myself).

I will give it a try, see if it is an easy one or not.

[galaunay]

@avocadoras You can find my attempt at integrating poetry in Emacs here.
There is still a bit of work, but the basic functionalities should work.

Would be glad to have your feedback :).