Gnome-shell-extension-gsconnect: Make GSConnect more friendly to traditional packaging

[jtojnar]

We would like to create a GSConnect package for NixOS, unfortunately the runtime creation of the service files is not very compatible with distribution packages. Would it be possible to add meson option to install the files to systemdsystemunitdir and session_bus_services_dir (see for example fwupd source) instead?

cc @etu

[andyholmes]

Hi, I'm aware of this issue, but just haven't had time to come back to it since I hacked in a band-aid solution for an Arch packager.

I've had a glance at the linked meson.build and it seems pretty straightforward, however I'll have to include some code in GSConnect itself to determine where the various support files are during run-time, since user extensions keep everything under a top-level directory including gschema and i18n. I've also not completely tested the new systemd DBus activation, so I need to make sure no problems can occur since a failed systemd DBus unit can hang the system in some situations.

I'll try and get to this sometime this week, and I'll make sure it's in the next release, v12. This will probably be a good place for NixOS to start packaging anyways, since it will be the first version with dbus-broker support, which I believe you use.

[jtojnar]

Using ext.datadir as is done in master should work good enough – we do not need locale files to be in --localedir, extensiondir is also fine, as long as they are managed by the package manager. Though you could do something like https://gitlab.gnome.org/GNOME/gnome-shell-extensions/blob/6746061898376ff39dd8034c22469d1ce4aadf9d/lib/convenience.js#L42-54 if you want to be super orthodox.

Looking at the source code some more, the native messaging hosts and a nautilus extension should not be installed during runtime either.

We do not use dbus-broker by default, I think, but there will still be one more modification needed. We do not really have FHS (each package is installed into its own prefix under /nix/store) so the packages need to link to absolute paths of other dependencies. For run-time dependencies, we rely on wrappers – we move the original file to .original-wrapped and create a wrapper like the following:

#! /nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/bash -e
export GIO_EXTRA_MODULES='/nix/store/m616rmk0i0pl1ff1wv9f7naj4w1951w4-dconf-0.28.0-lib/lib/gio/modules'${GIO_EXTRA_MODULES:+':'}$GIO_EXTRA_MODULES
export GIO_EXTRA_MODULES='/nix/store/m616rmk0i0pl1ff1wv9f7naj4w1951w4-dconf-0.28.0-lib/lib/gio/modules'${GIO_EXTRA_MODULES:+':'}$GIO_EXTRA_MODULES
export GDK_PIXBUF_MODULE_FILE='/nix/store/sl3qhym76dnag1y98gi1rzmvs79lfvfv-librsvg-2.42.2/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache'
export XDG_DATA_DIRS='/nix/store/zms902y572jn2mndasd9bqzw0zld205r-hicolor-icon-theme-0.17/share:/nix/store/cvl545hn3r7cv2307pw7rfkj5pnhs9jc-cups-2.2.6/share:/nix/store/zms902y572jn2mndasd9bqzw0zld205r-hicolor-icon-theme-0.17/share:/nix/store/cvl545hn3r7cv2307pw7rfkj5pnhs9jc-cups-2.2.6/share:/nix/store/zms902y572jn2mndasd9bqzw0zld205r-hicolor-icon-theme-0.17/share:/nix/store/cvl545hn3r7cv2307pw7rfkj5pnhs9jc-cups-2.2.6/share'${XDG_DATA_DIRS:+':'}$XDG_DATA_DIRS
export XDG_DATA_DIRS='/nix/store/c8i4flxb5qc31v6zi1jpmd7512amyms8-gtk+3-3.22.29/share/gsettings-schemas/gtk+3-3.22.29:/nix/store/47pw4n4fw6gagg5z9fp9w8g1mi4yyxsl-gsettings-desktop-schemas-3.28.0/share/gsettings-schemas/gsettings-desktop-schemas-3.28.0:/nix/store/c8i4flxb5qc31v6zi1jpmd7512amyms8-gtk+3-3.22.29/share/gsettings-schemas/gtk+3-3.22.29:/nix/store/c8i4flxb5qc31v6zi1jpmd7512amyms8-gtk+3-3.22.29/share/gsettings-schemas/gtk+3-3.22.29:/nix/store/47pw4n4fw6gagg5z9fp9w8g1mi4yyxsl-gsettings-desktop-schemas-3.28.0/share/gsettings-schemas/gsettings-desktop-schemas-3.28.0:/nix/store/c8i4flxb5qc31v6zi1jpmd7512amyms8-gtk+3-3.22.29/share/gsettings-schemas/gtk+3-3.22.29'${XDG_DATA_DIRS:+':'}$XDG_DATA_DIRS
export XDG_DATA_DIRS='/nix/store/0zdr8zfmi8pw1nsbqry73y2mlszmflfq-gcolor3-2.2/share'${XDG_DATA_DIRS:+':'}$XDG_DATA_DIRS
exec -a "$0" "/nix/store/0zdr8zfmi8pw1nsbqry73y2mlszmflfq-gcolor3-2.2/bin/.gcolor3-wrapped"  "${extraFlagsArray[@]}" "$@"

Unfortunately, the dbus service has gjs in Exec=, so it would run the bash wrapper as JS file. It should just rely on the file’s shebang. Though I am not sure what to do about it if zip does not support executable bit. (Edit: it should support it)

[andyholmes]

Using ext.datadir as is done in master should work good enough – we do not need locale files to be in --localedir, extensiondir is also fine, as long as they are managed by the package manager.

The location of these files is actually modified after build by the make-zip/install-zip targets, so if accommodating the standard ninja install target I'll have to address that anyways. I've also had requests from other distros for system install options.

Looking at the source code some more, the native messaging hosts and a nautilus extension should not be installed during runtime either.

With regard to the native messaging host, that's probably possible, but also more code will have to be written to account for that. For the Nautilus extension, the python script will have to be modified to respect the user settings.

Unfortunately, the dbus service has gjs in Exec=, so it would run the bash wrapper as JS file.

Although Zip files support an executable bit, the extensions website modifies extensions after upload which might remove that bit. I haven't tested that, so I don't know for sure, but if true then Exec= and the executable bit will just have to be set during the meson target.

Since extensions are primarily distributed on e.g.o the GSConnect is written with local installation in mind, and there are a number of situations here that will require modifying the way the Extension settings function to accommodate packaging in this way while maintaining compatibility. That's probably going to mean a slightly longer wait, but I'll get to it as soon as I can.

[andyholmes]

@jtojnar

I'm just getting around to this, but won't make it until the rewrite is merged due to refactoring required for non-runtime installs and I'll probably do one more release from master before that. I'd like to keep the meson.build fairly simple (eg. there are really no "build time" requirements technically), but does this look roughly sufficient for what you need? (not committed yet)

dbus = dependency('dbus-1', required: false)

if get_option('session_bus_services_dir') != ''
  dbus_dir = get_option('session_bus_services_dir')
elif dbus.found()
  dbus_dir = dbus.get_pkgconfig_variable('session_bus_services_dir')
else
  dbus_dir = join_paths(datadir, 'dbus-1', 'services')
endif

configure_file(
  input: 'data/org.gnome.Shell.Extensions.GSConnect.service',
  output: 'org.gnome.Shell.Extensions.GSConnect.service',
  configuration: extconfig,
  install_dir: dbus_dir
)

I've also changed the service, desktop entries, etc to use the shebang as you requested (I can just deal with the exec bit at runtime for user installs if necessary), and added some meson options for nautilus/webextension. AFAIK the Gnome APIs I use defer to XDG env variables and others; are there any other environment considerations I should know about? One example is daemon.js adding a search path for Gvc library that's shipped with Gnome Shell instead of standard libdirs; is this sort of thing an issue or can your wrappers account for this?

[jtojnar]

does this look roughly sufficient for what you need?

Yeah, that should work.

AFAIK the Gnome APIs I use defer to XDG env variables and others; are there any other environment considerations I should know about?

The main things we had problems with in the past are typelibs and gsettings schemas. For executables (e.g. the daemon) we just wrap them, but for extensions we have to patch the code (typelibs & schemas). I am not sure if these things should be done upstream, they are pretty inelegant.

I would think that Gvc is available to the extensions. After all, GNOME Shell's rpath should contain its pkglibdir.

Another thing that might cause trouble is the daemon relying on the argv[0] for something like loading resource files (https://gitlab.gnome.org/GNOME/gnome-characters/issues/29) but that probably is not the case here.

[andyholmes]

The main things we had problems with in the past are typelibs and gsettings schemas.

I think I can solve this using replacements like @somedir@ in metadata.json without any inconvenience. For GSettings the default schema source doesn't work, so the schema source just always needs to be created from a variable directory? Would datadir from meson work here, such as @datadir@/glib-2.0/schemas work or would you a need an explicit @gschemadir@?

Is it a problem for the extension to read from other desktop GSettings? This is done a few places to respect desktop settings.

I would think that Gvc is available to the extensions.

The only difference here is I use Gvc in a module of daemon.js which is "outside" the regular gnome-shell process unlike extension.js. Would @libdir@/gnome-shell work or do you need a specific @gslibdir@?

...the daemon relying on the argv[0]...but that probably is not the case here.

No problem here, I just store the app id as a string and don't rely on any inference like package.js does.

[jtojnar]

For GSettings the default schema source doesn't work, so the schema source just always needs to be created from a variable directory?

I think the common extension practice of keeping gsettings schemas in separate directory is good enough.

Would datadir from meson work here, such as @datadir@/glib-2.0/schemas work or would you a need an explicit @gschemadir@?

@datadir@/glib-2.0/schemas should be okay. We can patch it ourselves for now, in the future we want to do something different anyway.

Is it a problem for the extension to read from other desktop GSettings? This is done a few places to respect desktop settings.

We do have separate gschemas.compiled files per package but the standard API respects XDG_DATA_DIRS so it should be fine.

Would @libdir@/gnome-shell work or do you need a specific @gslibdir@?

@libdir@/gnome-shell would not work, as each package uses a different prefix, but we can solve that by wrapping with GI_TYPELIB_PATH.

[andyholmes]

Hi, I know it's been a long time coming, but I've recently merged the rewrite into master. Fairly soon I'll tag a release candidate at which point I'd like to do an informal freeze.

Now would be convenient time to request or submit any changes to the build system. Hopefully, we can achieve a system that works for all distributions without any downstream patches. The only addition I have planned is a translator's build target for the WebExtension, which shouldn't affect packaging.

There is a new page in the Wiki called Packaging that may answer some questions.

[jtojnar]

Wow, this is quite comprehensive page; thank you. At a glannce, I see just one problem for NixOS:

• --libdir LIBDIR
  Should be the LIBDIR that gnome-shell uses. GSConnect uses this to look for the PulseAudio bindings shipped with gnome-shell (Gvc-1.0.typelib) that are usually in PREFIX/LIBDIR/gnome-shell/.

Nix has separate, read-only paths for each package, therefore a single libdir is not enough. Perhaps you could add gvc_path extra meson option?

Installing under prefix is fine for us.

[andyholmes]

Sure, I can do that. I'll add options for any directories gsconnect might use and just use fallbacks.

[andyholmes]

I've just comitted dccdfa1 which adds two options --gnome_shell_libdir and --mozilla_libdir that override LIBDIR, eg /PREFIX/GNOME_SHELL_LIBDIR/gnome-shell/. Does that work?

Also, do you need the same for the Nautilus script or Chrome & Chromium manifests?

[jtojnar]

We should only need configurable paths for locations we want to get something from – for installation default paths work for us, as long as they respect prefix. (Extra option like mozilla_libdir cannot hurt but I really wish there was a meson feature that would allow more granular control over install paths like make has.)

One thing that does not work for us is:

https://github.com/andyholmes/gnome-shell-extension-gsconnect/blob/dccdfa1a63e33f505e38f7d2667c79ff6addd8b1/meson/nmh.sh#L3

We do not have DESTDIR set, and sysconfdir is the meson default (etc/, i.e. relative to prefix). The line will try to write to unwritable /etc instead of $prefix/etc, but this is yet another meson limitation – We would want to be able to use different directories for build and for installing.

Though, in this case, we can actually set --sysconfdir=$prefix/etc since it is not really used during build.

[andyholmes]

The meson build is mostly a simple copy script, so I don't mind adding explicit options for everything. How about if I add --chrome_nmhdir, --chromium_nmhdir and --mozilla_nmhdir which would still default to:

SYSCONFDIR/opt/chrome/native-messaging-hosts/
SYSCONFDIR/chromium/native-messaging-hosts/
PREFIX/LIBDIR/mozilla/native-messaging-hosts/

But for each --browser_nmhdir replace the configurable prefix with BROWSER_NMHDIR. So:

CHROME_NMHDIR/opt/chrome/native-messaging-hosts/
CHROMIUM_NMHDIR/chromium/native-messaging-hosts/
MOZILLA_NMHDIR/mozilla/native-messaging-hosts/

I believe the hardcoded end of the path is fixed for each browser, so would this give you enough flexibility?

[jtojnar]

That would work as well.

[andyholmes]

Pushed: 46273a8

Okay, I think I understand better what the problem was. I was clobbering relative paths if DESTDIR was unset. The script now does a check like so:

if [ -n "${DESTDIR}" ]; then
    CHROME_NMHDIR=${DESTDIR}/${1}
else
    CHROME_NMHDIR=${1}
fi

I still added the options --gnome_shell_libdir, --chrome_nmhdir, --chromium_nmhdir and --mozilla_nmhdir which work out like this (with/without):

PREFIX/LIBDIR/gnome-shell/Gvc-1.0.typelib
GNOME_SHELL_LIBDIR/gnome-shell/Gvc-1.0.typelib

SYSCONFDIR/opt/chrome/native-messaging-hosts/
SYSCONFDIR/chromium/native-messaging-hosts/
PREFIX/LIBDIR/mozilla/native-messaging-hosts/

CHROME_NMHDIR/opt/chrome/native-messaging-hosts/
CHROMIUM_NMHDIR/chromium/native-messaging-hosts/
MOZILLA_NMHDIR/mozilla/native-messaging-hosts/

I think that should give you full control?

[jtojnar]

This looks fine, apart from the typos I reported in https://github.com/andyholmes/gnome-shell-extension-gsconnect/commit/46273a8ac5902a638a6a6ed732c537c518db16b4.

The meson option names are little weird, though: they are named foo_nmhdir but they do not set the location of the native-messaging-hosts directory, only its base/prefix. Maybe the options should be named foo_nmhdir_base. The descriptions in meson_options.txt also suggest the options would set the full path.

[jtojnar]

Would it be also possible to make gschemadir configurable? We need to move it to a subdirectory to avoid conflicts when merging packages into a single directory tree.

[andyholmes]

Thanks for the commits. I've also updated *_nmhdir to take full paths. It makes more sense and there's no reason to harcode the end of the path really. I'll update the Wiki shortly.

[etu]

Hey, @andyholmes

I found an issue: https://github.com/andyholmes/gnome-shell-extension-gsconnect/blob/master/meson.build#L132-L137

The lines before that either takes an option for chrome_nhmdir and chromium_nhmdir that is full path or default. Then it takes that path and append the default path to it :)

[andyholmes]

Hi packagers, v13 release is approaching (October 18th or so), so I wanted to check everything you need has been included before that happens. Let me know if there are any extra build options or changes you need!

[andyholmes]

I'm going to mark this as closed for now. Please feel free to open a new issue for anything more you need.

[andyholmes]

@jtojnar @etu

I thought I'd just ping you from here quickly, to let you know we're adding Caribou as another optional dependency in v14 for keyboard modifier and unicode support.

Since there's more distributions packaging GSConnect now, between v14-v15 I'm going to revisit separating the packagekit UI and adding a build option to disable it for distros handling dependencies themselves. That way you still get the Android app and browser links, but hide the irrelevant packagekit stuff.

[etu]

@andyholmes Ok, good to know! Thanks for all the help so far. Unfortunately I haven't had the time to finish up our packaging of it yet. But it's there and I'm using the bits that are packaged up so far.

[andyholmes]

No problem, I think the main thing is we have everything setup you need to package. Then you should just be able to version-bump without any changes.