Gnome-shell-extension-gsconnect: PackageKit not working well across distributions

[ferdnyc]

This relates to #293, and indirectly to things I'd detailed in my fixes for #263.

Even though the broken python2-nautilus package on F29 was replaced in an update with a nautilus-python package that fixes the issue, the original python2-nautilus still exists in the "fedora" repo for Fedora 29, which is frozen at its release state. (The replacement package lives in the "updates" repo.)

So, although nautilus-python obsoletes python2-nautilus, and any attempt to install python2-nautilus will simply report "Package nautilus-python-1.2.2-2.fc29.x86_64 is already installed.", our PackageKit code currently just sees a matching dependency name that's not installed, and considers the File Integration support incomplete.

I'll have to see if I can get the PackageKit lookup to account for obsoletes, in checking availability.

Otherwise, the right-er approach is probably to have it actually resolve the dependency set, to determine whether there are any unsatisfied dependencies. Rather than just blindly comparing available package names vs, installed package names, like it does now. There are likely to be other scenarios where it's correct to leave some available packages uninstalled, as long as the installed set satisfies all of the dependencies.

[andyholmes]

All good points. There are functions in PackageKit for "provides" (but I think this is actually only mime-types, eg. image/jpeg), and "search_files" (but I think distros only list these for installed packages). So both of those I think are unfortunately not helpful.

What would be nice is if we can determine the 'top-level' package for each distribution, and like you say let PackageKit resolve the rest.

One problem that is becoming a whack-a-mole is different distributions offering multiple package names and and named versions (foo-python2 etc) or even the same package name which does something different or is not required on that distribution (eg. #322 and python2-gobject).

We could probably be using PkControl to check the backend-name or better yet distro-id to make this less painful. I don't know these properties are guaranteed to be populated and that probably wouldn't solve problems like the recent Fedora 29 thing by itself, but I bet it would it make it easier. Probably we just check these once, or create another singleton like we do here:

https://github.com/andyholmes/gnome-shell-extension-gsconnect/blob/7aae6ca13330807c9e32b1fa7b49ca7d7acc1f97/src/service/ui/packagekit.js#L7-L12

If that actually works, PackageGroup would become more a dictionary of distros, and if the properties are NULL we could just disable PackageKit, which is something I "plan" offering as a meson option so packagers (especially NixOS) can disable it when appropriate.

Thoughts on that?

[ferdnyc]

All good points. There are functions in PackageKit for "provides" (but I think this is actually only mime-types, eg. image/jpeg),

It _shouldn't_ be. At least in rpm/dnf, Provides are the fundamental unit of published "capability", basically — the complement to Requires. (Actually, RPM now has soft dependencies, so "Requires" is supplemented by things like "Recommends" and "Suggests", as well. While on the other side, "Provides" go hand in hand with "Obsoletes", "Conflicts", etc. to define how a set of packages can be used to satisfy a required capability.) Since RedHat people also wrote PackageKit, I would expect it to operate on a similar model — at least when dnf is the backend, if nothing else.

So, even when it was called python2-nautilus, the package was always tagged "Provides: nautilus-python", and from the command line you could actually install it by typing sudo dnf install nautilus-python, even in Fedora 28. The resolver figures out that installing the package python2-nautilus satisfies the request for nautilus-python, so it does, and it Just Works™. If we can get our PackageKit checks to use that same notion of _capabilities_-lookup, rather than literal package _name_ search, it should be able to do the same, and we could remove distro-quirky names like python2-nautilus from the list of required _capabilities_ for GSConnect's optional features.

The reason I avoided going down that road so far is, I just don't know how the other distros' backends are implemented, and whether the same operational logic applies. If it _does_, then we're great, but I guess the only way to know is to just try it and see what happens. I know for a fact it can't be _assumed_ to be true, because...

PackageKit does not do dependency resolution. This problem has already been solved by the backend systems and we don't really want to re-invent the wheel.

So we're at the mercy of the various backends, to a degree.

Worst case, maybe some backends either don't have any concept of capabilities-type Provides, or some distros don't tag their packages with the more common names. But every package still Provides its own name, at least (that's a fundamental enough notion in RPM/dnf that I'm confident in assuming Hughsie built it into PackageKit as well), so a resolver-/capability-based lookup should still work to pull in named packages. And as long as we look up our list as "wanted capabilities", not "required packages", then the distros that don't have those packages should just ignore them.

One problem that is becoming a whack-a-mole is different distributions offering multiple package names and and named versions (foo-python2 etc) or even the _same_ package name which does something different or is not required on that distribution (eg. #322 and python2-gobject).

Yeah, I just saw that one, and it's definitely an instance of the same thing. If they have a python2-nautilus alongside a python-nautilus on Arch... well, then either they built their nautilus-python as two different plugins to support either Python generation, depending which you install, or they may have broken it in the same exact way that Fedora 29 did, because you can't really have two different Python generations supported _simultaneously_. (The fact that they're defined as conflicting makes me think the former, and means they got this right-er than Fedora.)

Like Fedora's, Arch's packages are built off the same nautilus-python base package. So as long as one (or both!) of those python*-nautilus Arch packages Provides nautilus-python, then switching from package names to wanted capabilities should mean that's enough. ...If _neither_ of them do, and if it can't be added to the metadata on their end, then we have to keep python-nautilus on the list as a quirk for Arch. Which is fine, because we can still drop python2-nautilus by name, since if we run our checks through the resolver then PackageKit on Fedora 28 will pick it up automatically for nautilus-python.

We could probably be using PkControl to check the backend-name or better yet distro-id to make this less painful.

Hmmm, I hadn't seen those. As you'd said back when, I'd _prefer_ to do this distro-agnostically, and I don't feel like we're quite at the point of having to completely abandon that just yet. I think it should at least cover _most_ of the distros to take that approach. And if it doesn't cover every case because some backends don't publish/resolve the same "standard" capabilities, then I can see having some additional "quirks" checks. But I think I'd rather approach it that way: agnostic general case + specific distro/backend quirks as needed. Because there are a lot of distros, and code that tries to account for them all separately feels like it should be in a function called _runKobayashiMaru().

[andyholmes]

Provides are the fundamental unit of published "capability"...

Hmm, that's what I thought, but I feel like I tried "provides" when I first did this and it didn't work. Ubuntu/Debian definitely lists attributes like these (provides, requires, conflicts...), so maybe I just did it wrong.

I'd prefer to do this distro-agnostically
agnostic general case + specific distro/backend quirks as needed.

Definitely preferable. If PackageKit decided against it (generally speaking) it seems silly for us to do it. I guess the best bet is really to get testers and give the "provides"/"capabilities" thing a shot; Ubuntu and Fedora are both covered by us, I'm sure we can dig up an Arch user to test something unstable ;)

I also know NixOS wants PackageKit out of the picture (but they'll go either way on the informative rows), I just haven't gotten around to that since I've been wrestling this new messaging UI. I also think we're getting specific enough that we can safely hard-code the widgets a little more if it saves us trouble - no reason to over-abstract anymore.

On that note, I realized today (when working on the messaging UI) that the CPU spin-up when you open that page isn't actually the PackageKit operations; it's the GtkSpinner widgets :disappointed:. I was told that each frame-tick can (does?) cause the whole window to be to be invalidated and redrawn. Sometimes I forget how old Gtk3 is now, but we can probably make that situation better.

[ferdnyc]

Provides are the fundamental unit of published "capability"...

Hmm, that's what I thought, but I feel like I tried "provides" when I first did this and it didn't work. Ubuntu/Debian definitely lists attributes like these (provides, requires, conflicts...), so maybe I just did it wrong.

Well, no, I may have been really, really overoptimistic. I've been playing around with it for an hour or so (after re-teaching myself enough JavaScript to write a "simple" console-output script), and even on Fedora it's... weird.

resolve() doesn't do provides, it just translates package names into specific IDs. It'll ignore unknown packages, which is nice, but anything named will get returned IFF it's a match.

• So, on Fedora 29
  
  
  
  • call: resolve(flags, ['nautilus-python', 'python-nautilus', 'python2-gobject'], ...)
  
  
  • result: ['nautilus-python', 'python2-gobject']
  
  

which is great, but...

• call: resolve(flags, ['nautilus-python', 'python2-nautilus', 'pygobject3'], ...)
• result: ['nautilus-python', 'python2-nautilus']

even though we don't want python2-nautilus, _and_ python2-gobject Provides: pygobject3.

what_provides(), OTOH, totally pulls up packages based on provides, but instead of solving for them, it'll pull back the _entire_ list of packages that provide that capability.

• So, again, Fedora 29:
  
  
  
  • call: what_provides(flags, ['nautilus-python', 'pygobject3'], ...)
  
  
  • result: ['nautilus-python', 'python2-nautilus', 'python2-gobject']
  
  

which is great in that it picks up python2-gobject, but in terms of nautilus-python it's already going the _opposite_ direction from what we want. And if we were to add more package names, we'd get more results, when we _want_ fewer.

There don't seem to be any functions geared towards _reducing_ the set of potential candidates, unless we wanted to either build our own dependency solver, or we just blindly ran install_packages() in simulate mode... but I think even still we'd potentially have to do it multiple times in multiple different ways, to determine what we want.

(Like, say we check what_provides(['nautilus-python']) and get back ['nautilus-python', 'python2-nautilus'], then simulating an install gets rejected because they conflict, so then we remove one package, re-run the simulated install, and we get an "already installed" result. OK, so now that sorts out that _one_ dependency. Repeat for every other individual dependency on the list? Ugh.)

I don't know, I'll keep poking at it some more. My (incredibly bare-bones and embarrassingly ugly) test script is in my repo on the packagekit-3 branch, src/service/ui/test-packagekit.js if you're interested in poking at it any.

[andyholmes]

Hmm, that is indeed not very promising (but that script is actually pretty clean :wink:). It's starting to look like checking the distro-id/backend-name (probably the former) is going to be the most...reliable I guess is the word.

I mean, there aren't actually that many "effective" distros since most deb based will use Debian's package guidelines (except openSUSE I think) and likewise for RPM->Fedora. I'm pretty sure all Arch-derived are just user-friendly wrappers and Gentoo almost always does everything right anyways. Everyone else doesn't want anything to do with PackageKit (eg. NixOS, FreeBSD, etc).

---

FYI, if all you need is a script that keeps running, you can just do:

// All stuff above, since run() will block until Ctrl-C
let loop = new GLib.MainLoop(null, true);
loop.run();

On the other hand, GApplication is really handy for:

let gaction = new Gio.SimpleAction({
    name: 'foobar',
    parameter_type: new GLib.VariantType('(ss)')
});
gaction.connect('activate', (action, parameter) => {
    let [mystring1, mystring2] = parameter.deep_unpack();
});
this.application.add_action(gaction);

// Now you can call this action from DBus/D-Feet (just give it an app-id so you can find it)

[andyholmes]

I think it's time we consolidate all PackageKit related problems into this thread, so I closed #322.

@ferdnyc Since I set November 26th as the release date for v17, what are your thoughts on PackageKit right now? ie. Do you think we have time to "fix" it, should we leave it as-is or should we just disable it in stable releases until we can fix it (or drop it)?

[andyholmes]

So at this point, I'm starting to remove all use of PackageKit. Given all the problems it's causing, how unlikely it is to find an elegant solution or get the testing done across distributions, this seems like best option. This certainly won't be finished by v17, but I think I can safely remove "Sound Effects" and "Desktop Contacts".

All of these "dependencies" are in fact optional, since no plugins are required for GSConnect to function, and it's only one (SFTP; "soon" to be replaced by GMount) and a half (Mousepad; not working great anyways) that need them at all. This is how all other applications function with regard to optional dependencies, with the only obvious use of PackageKit being codec downloads.

With the new Android Contacts plugin, libfolks is not likely to see much use since it will likely be aggregating the exact same address book with slower resolution. It's already been designed to fallback to collecting the necessary information from packets anyways.

There is now a sufficiently reliable fallback for canberra-gtk-play/GSound (ie. I haven't been able to make it break on purpose), and losing your desktop or laptop seems pretty hard to do within the range of your WiFi anyways (if you're losing a wired PC on your LAN, you have problems GSConnect can't help you with :wink:).

The Nautilus extension is even more optional-y, and the current extension architecture is due to be replaced anyways. No functionality will be lost at all if it's not available, only a minor inconvenience, and users who are even aware of Nautilus extensions are likely to have it installed already.

[ferdnyc]

Whoops, sorry, missed the previous email in all the chatter. I'm beginning to regret a tiny bit that I let GitHub's Collaborator status auto-subscribe me to every Issue email for the project. :laughing: (I _could_ change that, of course, so really it's on me.)

So at this point, I'm starting to remove all use of PackageKit. Given all the problems it's causing, how unlikely it is to find an elegant solution _or_ get the testing done across distributions, this seems like best option.

Given what I/we have learned about PackageKit's design and function since the beginning, I have to regretfully say that it seems like that's probably the best solution. It'd be great if PackageKit _was_ able to be a completely distro-agnostic, automated general dependency-management system for Linux applications... but I don't see any way that it can be with its current design.

Everything about how it works is geared towards user-driven package management, where it doesn't matter if package names aren't the same as on other distros, or there are multiple ways to satisfy the same dependency. The backend resolver will sort that out for you at install time, and beyond interactive install sessions there's no need to worry about any of that.

...All of which is fine, as I said, for interactive, user-driven installs. And makes it completely useless for our purposes. :disappointed:

[andyholmes]

All gone, here's what you get now:

I think that's like the fourth feature to be removed this cycle now :stuck_out_tongue_closed_eyes:

[ferdnyc]

Well... hmm, we could say "misfeature" instead, does that make it better or worse? :wink: