Podman: Request: Stable releases, keep previous versions

[barseghyanartur]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

It's a second time already that the latest and greatest fresh release (1.9.0 in this case) breaks everything (within past 6 month).

Listing versions (sudo dnf list --showduplicate podman) gives me this:

podman.x86_64                             2:1.9.0-1.fc31
podman.x86_64                             2:1.6.2-2.fc31

I want to install 1.8.2, which worked just fine for me.

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora 31.

[abitrolly]

@barseghyanartur do you have specific issues in mind? I want to compare with the problems I've got.

[barseghyanartur]

@abitrolly:

No, besides that fresh podman releases often break things (as I mentioned already, this happened twice within past half a year) and I want to have an option to stick with older versions for a little longer. If your entire development stack is very much dependant on containers, it's pretty harsh to have it unavailable even for a half of the day.

[barseghyanartur]

@abitrolly:

P. S.

This time I got the following error Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd. However, I don't want to relate specific errors to this issue. Instead, just have an option to roll back the update to the version I have successfully used.

[abitrolly]

@barseghyanartur have you tried dnf downgrade with previous .rpm from koji?

[abitrolly]

@barseghyanartur and the specific issue with invalid configuration is tracked in #5903.

I've got problems with toolbox, but I haven't figured yet if they are related to podman update.

[barseghyanartur]

@abitrolly:

dnf downgrade will downgrade to 1.6.2 and that's not what I want. I want to downgrade to 1.8.2.

As for the specific issue, I managed to solve it yesterday by removing the ~/.config/containers/libpod.conf file.

[mheon]

@lsm5 PTAL

[lsm5]

@barseghyanartur RE: downgrade, there were some issues with selinux dependencies because of which 1.8.2 never landed into stable. This hopefully was just a 1-off occurrence.

You can still get 1.8.2 from https://koji.fedoraproject.org/koji/buildinfo?buildID=1479552 . HTH.

[barseghyanartur]

@lsm5:

Great. This is perhaps worth mentioning in troubleshooting section, if there's one. As for myself, I'll memorize that in gist. :) Thank you!

[lsm5]

@lsm5:

Great. This is perhaps worth mentioning in troubleshooting section, if there's one. As for myself, I'll memorize that in gist. :) Thank you!

Ack, we'll update the website with instructions for this. I'll close this then...

[aleks-mariusz]

@barseghyanartur RE: downgrade, there were some issues with selinux dependencies because of which 1.8.2 never landed into stable. This hopefully was just a 1-off occurrence.

You can still get 1.8.2 from https://koji.fedoraproject.org/koji/buildinfo?buildID=1479552 . HTH.

@lsm5 do you have a link for an older CentOS 7 rpm i can downgrade with too please? :-)

[abitrolly]

@aleks-mariusz if everybody is downgrading, then who will report the issues? :D

[abitrolly]

@aleks-mariusz the koji for CentOS lists 1.7.0 at the latest version https://cbs.centos.org/koji/packageinfo?packageID=6853

Going to 1.8.2 doesn't look like an upgrade there. )

[barseghyanartur]

I certainly would/will still report an issue (if not yet reported), but in the meantime will continue to use working version. :)

[lsm5]

@aleks-mariusz the koji for CentOS lists 1.7.0 at the latest version https://cbs.centos.org/koji/packageinfo?packageID=6853

Going to 1.8.2 doesn't look like an upgrade there. )

@abitrolly fyi, I'm not building on CBS anymore. Everything goes to OBS. https://build.opensuse.org/package/show/devel:kubic:libcontainers:stable/podman

[aleks-mariusz]

@lsm5 i see no older RPM there than the broken (re rootless) 1.9.0-1 there, 1.8.2 (from the kubic libcontainer stable repo) worked nicely before upgrading all my CentOS hosts this weekend :-/

although i'm starting to question the word stable in that repo name tho :-)

[lsm5]

@lsm5 i see no older RPM there than the broken (re rootless) 1.9.0-1 there, 1.8.2 (from the kubic libcontainer stable repo) worked nicely before upgrading all my CentOS hosts this weekend :-/

Yup, in the case of OBS, it gets rid of older packages.

although i'm starting to question the word stable in that repo name tho :-)

RE: stable, it usually means the latest upstream non-RC tag.
Upstream is aware of the issues with 1.9.0 and is working on those so stable will really mean stable going fwd :) /cc @mheon @baude

[lsm5]

The fedora source at https://src.fedoraproject.org/rpms/podman/tree/f31 is used for the CentOS builds on OBS as well. So, in case you need an older build you could clone the source, check out an older commit and build it locally.

[lsm5]

or rather, get the f31 srpm from https://koji.fedoraproject.org/koji/buildinfo?buildID=1479552 and rebuild it for centos7

[aleks-mariusz]

or rather, get the f31 srpm from https://koji.fedoraproject.org/koji/buildinfo?buildID=1479552 and rebuild it for centos7

Thanks for the idea, this is the work-around i ultimately went with until we get a working podman 1.9.0 running properly in rootless mode.

[ocroz]

Dear podman developers,

Could you please revise the definition of stable and what number of last package versions should be kept available?
I think _you are not helping the adoption of podman_ by the community by providing only one package version unfortunately broken.

I have been testing podman version 1.8.2 for several weeks, and I was happy with it.
On Monday I was about to upgrade one production server with this podman version.
Unfortunately I got podman version 1.9.0 and it failed in a root-less environment on my CentOS 7 system.
I did not want to leave a comment before trying all the solutions you provided.
So I came to compile podman. PS: gcc is required to compile too.
The executable I got worked for a simple ping but failed to run a node.js application.
It appeared that the size of the compiled executable is 55MB whereas the size of the packaged executable is 75MB.
Also, what about the compatible dependencies?
Eventually I tried to install conmon and runc from the stable Kubic project, and to copy the packaged executable from another CentOS 7 machine I had.
And it seems to work, but...

This is not very production proof as I am not using official images. I am not a podman developer.
PS: It took me 2 days to find the solution. I was even about to come back to docker which looks much more _stable and trustable_. Also reading here that such a problem already happened in the past.

So why I think you should:

1. Always provide a working package version of podman, and keep it available at least until you are sure all major problems are fixed in the next version.
2. Keep several package versions available, not forcing the users to take the latest version if they are happy with a previous one.

For me this is not incompatible with reporting issues.
One can try latest version on dev environments, and use stable version on prod environments.

Hope you understand my point,
Anyway thanks you a lot for the great work that you are doing

[mheon]

Is there a reason you're unable to use the official CentOS podman package in the extras repo? That is what we would consider our stable, supported Podman on CentOS 7 and 8. The OBS repos are provided as a rolling snapshot of Podman's latest release for those that want to try something more recent, but the intent is that most people should use the official packages.

We do recognize that this is a problem for other distributions without official packages. We can support the OBS repos because automatically building tags is very easy and takes very few resources on our part, but I don't believe they can easily support multiple simultaneous versions. If someone wants to maintain a stable Podman release repository, we'd love to collaborate and assist, but I don't think we have the resources to do it ourselves.

[ocroz]

True I forgot this point.

Unfortunately the server where I am using podman is CentOS 7 for which I am getting version 1.4.4 from the official CentOS podman package in the extras repo.
With this version I am not able to share a volume with the host, making it writable from the podman container.

On CentOS 8 I am getting version 1.6.4 which is better for my purpose. At least I can share a volume with the host, making it writable from the podman container, thanks to the run parameter mount and the bind option relabel=shared.
This option does not exists in podman version 1.4.4.

At least with the podman version 1.8.2, I can use either the run parameter mount with the bind option relabel=shared, or even better a standard run parameter volume as with docker.

I am fine if you update the official CentOS podman package in the extras repo with a version that is better for me.
Or may be I am wrong with my analysis...
Anyway I felt that using a more up-to-date version could help me for this fast moving software.

[rhatdan]

The Centos7/RHEL7 version of Podman will be updated in RHEL7.8 to podman 1.6.4 (I believe) very soon, but this will be the final official release for RHEL7, unless there are CVEs. Going forward the supported version of Podman for RHEL8 and therefore Centos will be updated every 12 weeks or so. RHEL8.2 release happening this week will be podman 1.6.4 and 12 weeks from now, we plan to release podman-1.9.* for RHEL8.2.1 release. Our stretch goal for RHEL8.3 is to get podman 2.0.* out for the fall release.