Pipx: Inquiry regarding pipx best practices for Singer.io data platform

[aaronsteers]

_Disclaimer: This is more of an inquiry than a feature request._

The singer.io platform uses a number of (primarily) pip installable libraries in order to extract and ingest data. The community in general is using virtual environments but the process is cumbersome. It seems we could encourage the community to switch to pipx and this could make adoption a lot easier for end users, while also just reducing the amount of code and orchestration needed for installs.

The general paradigm is tap-somesource | target-somedestination such as: tap-exchangeratesapi | target-csv. Because each tap and target is it's own pip-installable executable, it's common for conflicts to occur amongst taps and targets, or amongst multiple taps which might be installed simultaneously to support a multitude of data sources.

The same install paradigm also carries for orchestrators such as meltano.com and pipelinewise, since these are generally also pip installs.

• Is there any reason you could foresee why we _should not_ use pipx as a replacement for pip virtualenvs?
• OR any other warnings/gotchas we should watch out for?

Usage examples:

https://meltano.com/ example:

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install Meltano
pip3 install meltano

https://help.deputy.com/en/articles/3413067-singer-data-tap-build-your-own-data-integration

Step 2 - Create and activate a Python 3 virtual environment for the Tap, which we'll call tap-deputy
Run the following from the command line:

python3 -m venv ~/.virtualenvs/tap-deputy

source ~/.virtualenvs/tap-deputy/bin/activate

Step 3 - Install the Tap using pip
Run the following from the command line:

pip install tap-deputy

More info about the singer platform:

https://www.singer.io/

[itsayellow]

If I understand the use case correctly this seems like exactly the type of situation pipx is intended for.

My assumption is that you are only interested in installing utilities that can be run as apps (i.e. executing from the command-line), and they are python-based and pip-installable. If this is the case pipx is perfect.

pipx automates the creation of separate venvs for each separate pip-package. Different packages do not have to worry about overlapping dependencies with different versions, for example.

pipx also puts all the executables in the same directory, so that you only need to add one directory to your PATH (and not the bin directories of multiple venvs).

For your examples, the pipx equivalent would be:

pipx install meltano

and

pipx install tap-deputy

That's it! Now both meltano and tap-deputy can be run from the command-line if ~/.local/bin is in the user's path.

I also tested it on my own computer and had no problems.

[cs01]

Agreed, this looks like the exact use case for pipx. The only gotcha is that pipx does not allow multiple installations in a single command. pipx install X Y won't work. You need to do pipx install X then pipx install Y. I would actually like to revisit this since it seems like an unnecessary limitation of pipx.

[aaronsteers]

Awesome. Yes, I did notice that. Installing multiple simultaneously would be great of course, but either way, it's still much more abbreviated than manually spec'ing out a full virtualenv each time.

Three follow-ups:

1. It would be nice to be able to "uninstall" via the same ref that was used in the install. I notice that we can install with any pip-identifiable reference, including specific git repos or specific git refs/branches. During the install process, it seems that pipx auto-detects a base name (tap-csv from something like git+https://github.com/aaronsteers/tap-csv...). But during uninstall, we have to remember the base name in order to run the install. For automated use cases, it would be very nice to be able to run an uninstall using the original ref. Does that make sense, and perhaps is it already support and I missed it?

2. For docker use cases, are there specific paths we can map as volumes and/or cache to get a "portable" install experience? This is not critical but I'm wondering if this is "already in the box" because of the way in which the shared path and virtualenv experience is under-the-hood.

3. We occasionally have many forks of the same repo, so the executable name "tap-csv", for instance may have many variants. In "real life" use cases (ignoring plugin test/dev), I've seen people toggle between forks but I've not seen any use case to have multiple forks installed simultaneously on the same machine or same docker image. Still, I'm curious what behavior we would expect if someone wanted to pipx install a second git fork that had the same "tap-csv" executable but with slightly different functionality.

None of these questions are show-stoppers but wanted to get your thoughts.

Thanks in advance.

[itsayellow]

For #3 see the --suffix option to pipx install
https://pipxproject.github.io/pipx/docs/#pipx-install

It's a new feature, and allows for a suffix to be added to both the executable and the venv. It allows and differentiates between simultaneously-installed different versions of the same package.

[cs01]

Re 1: This is not an existing feature. It probably could be, by inspecting the pipx metadata file.

Re 2: If you want to tell pip to use a specific cache, pipx allows you to pass arbitrary arguments to pip via --pip-args.

Some other environment variables that can be set, described in pipx --help:

Optional Environment Variables:
PIPX_HOME: Overrides default pipx location. Virtual Environments
will be installed to $PIPX_HOME/venvs.
PIPX_BIN_DIR: Overrides location of app installations. Apps are symlinked
or copied here.
USE_EMOJI: Override emoji behavior. Default value varies based on platform.

If these do not cover your use case, could you give a specific example of what you are trying to do?

Re 3: Agreed with Matthew

[aaronsteers]

@cs01 and @itsayellow - This is very helpful guidance - thank you, both. Honestly, I don't see any blockers here at all. I think it's worth sharing this thread back and starting to socialize pipx with members of the Singer community.

• Number 1 ("uninstall from git ref") is kind of a nice-to-have, and if it every became an issue, it's probably a PR I/we could help contribute.

• Number 2 ("caching and xcopy-like folder volumes") sounds like it could be certainly possible with a little trial-and-error - but certainly not something we rely on today anyway.

• Number 3 ("allowing multiple installs via --suffix") sounds like a fantastic solution with the existing capability. I can imagine advanced users taking advantage of this --suffix as they test a new version or as they want to support side-by-side installs.

[aaronsteers]

@cs01 - This is a bit of a tangent, but I saw your comments on #490 and I was thinking in general about pipx adoption. I saw that there's a pipx brew formula for not yet one for chocolatey, which is basically the Windows equivalent. Would ya'll have any interest in publishing to the chocolatey packages repo?

Similar to brew install pipx on Mac, this would allow Windows users to run choco install pipx. I maintain "dataops quickstart" docs for windows and mac here and here, respectively. Both of these pages make heavy use of the respective package managers because I can lose people very quickly if anything takes multiple steps.

[cs01]

Both of these pages make heavy use of the respective package managers because I can lose people very quickly if anything takes multiple steps.

I guess the implication is that installing pipx with pip takes multiple steps? I agree it is safe to say there are plenty of landmines with pip, so I see where you're coming from.

I know @uranusjr made pipx-standalone for Scoop. I am not a windows user, so I'm not sure if there is a big difference between Scoop or Chocolately. Is Scoop sufficient? https://github.com/pipxproject/pipx/issues/338 may be a better issue to continue the discussion.

[aaronsteers]

Great, thanks. I'll follow up there on #338 regarding windows installs.

[xinbinhuang]

@aaronsteers @cs01 @itsayellow Thanks for the helpful discussion here. I just found out pipx today, and soon realize that it's a perfect fit for the singer.io platform. (I previously wrote a bash script to abstract away the virtualenv steps). Thanks for the great work!

[aaronsteers]

@xinbinhuang - Fantastic. Glad you found it useful. I've been using and recommending pipx for singer.io work (and other python tools) for the past 4 months now and everything just works _beautifully_.

pip install pipx
pipx ensurepath
pipx install tap-salesforce

Related re: DBT:

• I've also used it for tools like dbt, but for dbt you have to specify pipx install dbt --include-deps since the _actual_ executable is in a dependent python module (dbt-core).

[xinbinhuang]

Related re: DBT:

• I've also used it for tools like dbt, but for dbt you have to specify pipx install dbt --include-deps since the _actual_ executable is in a dependent python module (dbt-core).

Thanks for the recommendation! I will try it out with dbt too - singer and dbt kind of go hand in hand these days.

[itsayellow]

Hi @aaronsteers we're glad you seem set up with pipx! Can we close this issue now? Do you have any other questions?

[aaronsteers]

@itsayellow - your help has been great and I think we have a very good solution - feel free to close this. And thank you so much!