Salt: Feature request podman support

[aboe76]

Feature request salstack state/module support for podman

Saltstack has already support for docker please add podman support

Development Information to get started:
https://medium.com/cri-o/python3-support-for-podman-a0a2395c3b4

podman is already build for suse and kubic-project so it should fit well with suse manager 3

[Ch3LL]

will approve as a feature request

[deuscapturus]

I just starting looking into creating some podman modules.

I think this is a good starting point: https://github.com/containers/libpod/tree/master/contrib/python/podman

[fignew]

Now that both Fedora and RHEL/CentOS 8 no longer have Docker packages in their native repos I think that podman support is very much needed.

[aboe76]

The podman python has it's own module:
https://github.com/containers/python-podman

[defensivedepth]

I agree with @fignew, this becomes a more pressing issue with the lack of Docker packages with Fedora and RHEL/CentOS 8.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[aboe76]

Please keep this open, podman support would be beneficial to saltstack.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[ankitola]

Yes, It will be great if SaltStack provide Podman support. Please keep open this.

[max-arnold]

I did some initial research and found this:

https://github.com/containers/python-podman/issues/70#issuecomment-572175842
https://podman.io/blogs/2020/01/17/podman-new-api.html

A couple of important highlights:

python-podman implementation is not fully compatible with libpod varlink defined interface

The bottom line we are moving away from varlink to a similar Protocol to the Docker.API

Thanks for the heads up, I will take a look to the APIv2 to avoid to spend time on something who will be deprecated in the next releases.

But we heard from users that varlink was a hurdle for libpod adoption especially for those who were using the Docker API and its bindings. They simply could not or did not want to rewrite their custom applications for libpod’s new, varlink-based API.

The new API is a simpler implementation based on HTTP/REST. We provide two basic groups of endpoints. The first one is for libpod; the second is for Docker compatibility, to ease adoption. The two endpoints are namespaced to keep them separate. Our goal with implementing a portion of the Docker API, is to be as compatible as possible; while similar calls in the libpod API might bring back additional libpod specific information.

In our proof of concepts, we have tested our endpoint with the docker-py project. There are of course subtle differences which we are still working on. And there are compatibility endpoints that we can not support like swarm which Podman does not support.

As for the existing varlink code, it has been in maintenance mode already. We will continue to address bugs but no new functionality will be developed. Once the new API is fully implemented, we plan to make a deprecation announcement.

This potentially means that some of the existing Salt docker modules might be somewhat compatible with the upcoming Podman APIv2 endpoint.

However, I'm not really familiar with Podman and not sure if APIv2 could provide a viable shortcut for Salt to support Podman containers eventually.

[doesitblend]

ZD-4685

[max-arnold]

I made some progress trying to run podman containers using its new Docker-compatible API and the existing dockermod execution module.

Installation instructions (Ubuntu 20.04, Salt 3001):

Pip pkg:
  pkg.installed:
    - name: python3-pip

Podman repo:
  pkgrepo.managed:
    - name: deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ /
    - key_url: https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_20.04/Release.key
    - file: /etc/apt/sources.list.d/podman.list

Podman pkg:
  pkg.installed:
    - name: podman

Podman service:
  file.managed:
    - name: /etc/systemd/system/podman.service
    - source: salt://podman/podman.service

Podman socket:
  file.managed:
    - name: /etc/systemd/system/podman.socket
    - source: salt://podman/podman.socket
  service.running:
    - name: podman.socket
    - enable: true

Docker socket:
  file.symlink:
    - name: /var/run/docker.sock
    - target: /var/run/podman/podman.sock

Docker python:
  pip.installed:
    - bin_env: /usr/bin/pip3
    - reload_modules: true
    - pkgs:
        - certifi==2019.11.28
        - chardet==3.0.4
        - docker==4.2.1
        - idna==2.9
        # - requests==2.23.0
        - six==1.14.0
        - urllib3==1.25.8
        - websocket-client==0.57.0

restart_salt_minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: true
    - onchanges:
      - pip: Docker python

podman.service

[Unit]
Description=Podman API Service
Requires=podman.socket
After=podman.socket
Documentation=man:podman-api(1)
StartLimitIntervalSec=0

[Service]
Type=oneshot
Environment=REGISTRIES_CONFIG_PATH=/etc/containers/registries.conf
ExecStart=/usr/bin/podman system service
TimeoutStopSec=30
KillMode=process

[Install]
WantedBy=multi-user.target
Also=podman.socket

podman.socket

[Unit]
Description=Podman API Socket
Documentation=man:podman-api(1)

[Socket]
ListenStream=%t/podman/podman.sock
SocketMode=0660

[Install]
WantedBy=sockets.target

By default any docker.* functions will fail due to the hardcoded version check:

docker.errors.InvalidVersion: API versions below 1.21 are no longer supported by this library

To fix that, run the following command and restart salt-minion:

echo "docker.version: '1.35'" > /etc/salt/minion.d/docker.conf

Here is an incomplete list of docker functions that work (i.e., do not crash and output something meaningful):

• salt-call docker.version
• salt-call docker.info
• salt-call docker.images (only if there is no images)
• salt-call docker.list_containers
• salt-call docker.ps
• salt-call docker.pull ubuntu:18.04

There is one minor blocker that prevents docker.images from succeeding: https://github.com/containers/podman/issues/6796#issuecomment-668359421 (it should be fixed, but for some strange reason still exists in podman 2.0.4). UPD: it will be released in 2.0.5

• salt-call docker.run_container docker.io/library/ubuntu:18.04 name=ubuntu replace=true bg=true will fail due to incompatible network settings format (this is the issue I'm currently testing)
• docker.{login,logout,copy_from,load,layers,save} will fail because they depend on docker cli command (some of them will work if you run ln -s /usr/bin/podman /usr/bin/docker, but I didn't test this thoroughly)

If you want to help, testing other docker functions/states and reporting any issues is appreciated. Podman developers tend to ask for shell-based reproducers, you can find a couple of examples here: https://github.com/containers/podman/issues/5553. Please do not ask them to run any Salt commands to reproduce an issue :)

To trace the HTTP requests that are made by Salt docker module, patch the /usr/lib/python3/dist-packages/requests/sessions.py file (the debug output will be visible when you use salt-call):

--- sessions.py.orig    2020-08-04 00:28:13.977552983 -0700
+++ sessions.py 2020-08-04 00:20:30.776382799 -0700
@@ -530,7 +530,9 @@
             'allow_redirects': allow_redirects,
         }
         send_kwargs.update(settings)
+        print('REQ', prep.method, prep.url, prep.body)
         resp = self.send(prep, **send_kwargs)
+        print('RES', resp.text)
         return resp

     def get(self, url, **kwargs):

The Docker API reference can be useful in comparing and reporting any inconsistencies: https://docs.docker.com/engine/api/v1.35/

[mchugh19]

Is the goal to make the current docker modules work with both docker and podman, or to introduce a new podman module with a docker virtualname?

[max-arnold]

The current goal is to make the existing modules work with minimal or no changes (it was stated that docker-py should just work with Podman APIv2).

If it turns out to be unachievable, then the only option left is to write a new set of Podman-specific state/execution modules using the new podman-py module that supports APIv2 (not to be confused with the obsolete python-podman that uses Podman Varlink protocol).

Right now I'm only considering the easy path :)