Jabref: Get snapcraft build approved

[LyzardKing]

Can somebody add the snapcraft secrets to the admin repo for the github actions?
Then we can uncomment the line in the build script and push the jabref snap to the store

as described here: https://forum.snapcraft.io/t/building-and-pushing-snap-packages-from-gitlab/9537
The tldr is:
snapcraft login
snapcraft export-login snapcraft.login
base64 snapcraft.login | xsel --clipboard

paste to the SNAPCRAFT_LOGIN_FILE in secrets

[LyzardKing]

I wonder if (to simplify things) the snap should not be built in the github actions after all..
it seems to complicate things, and it might instead just automatically trigger (build.snapcraft can do that) once the url for the latest build.jabref file is updated.
To enable this it would have to live in a separate repo (under the jabref organization, like JabRef/jabref-snap)
So without having to change the url a succesful build upstream would trigger a snapcraft build...without the docker addition..
@koppor @tobiasdiez what do you think?

[koppor]

This refs https://github.com/JabRef/jabref/pull/5379.

[koppor]

I think, we should stick building it here. That way, we can ship daily builds. - Otherweise, we had to change our built files. Refs https://github.com/JabRef/jabref/issues/5399

[LyzardKing]

It would ship daily builds anyway, ending up in the --edge channel.
But there would only ever be one daily build available.
That's the tradeoff..
A simple and automated build system, with 4 levels of "available" images: edge (continuously built, beta, candidate and stable).
Anyway, I think the snap is coming along nicely, with the new browser integration :)

[tobiasdiez]

I'm also in favor of build it here on github. This way we are more flexible (what triggers a build or prepare files before building). Moreover, we have everything in one place and treat different app stores in a similar way. As building the snap is working now, I also don't see any advantages in using build.snapcraft.io.

Mr @koppor just needs to login in snapcraft and provide github with the necessary secrets ;-)

[LyzardKing]

Building on builds.jabref is good.
It works and you can build different branches easily.
Just as a note, #5399 is not an issue with snapcraft, since it would refresh from master even if the name/version are the same.
In the previous snap the version was always set to 5.0.0-dev

[koppor]

I followed https://github.com/JabRef/jabref/pull/5379#issuecomment-537887016 and provided SNAPCRAFT_LOGIN_FILE as secret in this repository.

[koppor]

Build upload activated at https://github.com/JabRef/jabref/commit/fd2d32b41c702cf4d69f578812d3780943ec890f in the master branch. Added || true to allow failures of this step.

[LyzardKing]

Great! Now let's see if it builds.. I noticed in the script the other day it was set to push to beta...did we set it to edge?
(I'm not on my computer now..)

It should say push --release=edge for the daily build

https://github.com/JabRef/jabref/blob/fd2d32b41c702cf4d69f578812d3780943ec890f/.github/workflows/deployment.yml#L92

[tobiasdiez]

Done: https://github.com/JabRef/jabref/commit/372c3ec80dddb74efe25d1fac38acc04ce1d60df

[LyzardKing]

I'm sorry...I can't type...in the same line there's mdkir that should be mkdir

[koppor]

And we cannot read either 🤣🤣. Fixed at https://github.com/JabRef/jabref/commit/2e796e998a881fd1e50b184c329be4e22b4c0fbf

[LyzardKing]

The upload works now, the only thing missing is to ask for the review.
Since we used a special permission to add the json file needed for the native-messaging-hosts part,
you need to ask on the snapcraft forum for permission.
It only has to be done one time, and if we link the discussion I had related to getting native-messaging-hosts to work in the snap it should be fairly straight forward

EDIT: It's better if the request is official (it starts from one you) but if you want to tag me and the discussion I had on the forum I'm happy to intervene if it's needed :)

[koppor]

@LyzardKing Please provide me the link to the discussion so that I can comment on the release review (https://dashboard.snapcraft.io/snaps/jabref/revisions/551/feedback/) accordingly. Link to the discussion: https://forum.snapcraft.io/t/personal-file-under-mozilla-native-messaging-hosts/13627

[LyzardKing]

I'm not sure what the procedure is, but once the request is on the forum they can redirect the proper people to look at it.
Thanks!

[LyzardKing]

@koppor
You could try opening an explicit request like this one: https://forum.snapcraft.io/t/personal-files-request-for-kontena-lens/13504
The permission is system-files, with access to /usr/lib/mozilla/native-messaging-hosts/org.jabref.jabref.json

[Siedlerchr]

Btw, the snapcraft build fails at build.snapcraft.io:
Do we need to remove the snapcraft.yml file?

Pulling jabref 
Failed to pull source: 'build/distribution/JabRef-portable_linux.tar.gz'.
Please ensure the source path is correct and that it is accessible.
See `snapcraft help sources` for more information.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 267, in run
    self.pull()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 233, in pull
    env=env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 104, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 502, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-bionic-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-51911051", "build-request-timestamp": "2019-10-13T08:47:39Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/800385119bd8f8b023fe0dab7fbce0bc/+build/702124"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/jabref && linux64 snapcraft pull']' returned non-zero exit status 2
Revoking proxy token...

RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=bionic --arch=amd64 SNAPBUILD-702124
Scanning for processes to kill in build SNAPBUILD-702124

[koppor]

@Siedlerchr Can you provide the link to the build? I disabled the build 12 hours ago so that we build using GitHub workflows (as decided above).

[koppor]

Reading https://forum.snapcraft.io/t/personal-files-request-for-kontena-lens/13504/4?u=koppor it seems to be required that we are patient. Since I asked at the thread our supporter for an approval, I would suggest that we wait until Sunday with an explicit request. - Then I would ask @LyzardKing to write down the request since I am not aware of the details and cannot properly join the discussion.

[Siedlerchr]

https://build.snapcraft.io/user/JabRef/jabref/702124
(a day ago ) Did not know that you disabled the build there.

[LyzardKing]

It takes some time for the request to be approved.
I'll write a request, then link it here, so you can keep an eye on it.
Then I'll link the previous forum post with the methodology followed in developing the feature (https://forum.snapcraft.io/t/personal-file-under-mozilla-native-messaging-hosts/13627) in the new request.
Is this acceptable?

[koppor]

Sure!

[LyzardKing]

Great! So I'll open the request now :)
https://forum.snapcraft.io/t/system-files-permission-for-jabref-native-messaging-hosts-integration/13702

[LyzardKing]

By the way.. the terminal fix in the docker image worked... the apt log is still very long, but the snapcraft stuff is very limited and readable

 Pulling jabref 
Building gnome-extension 
make -j2
gcc -Wall -O2 -o bindtextdomain.so -fPIC -shared ./../src/bindtextdomain.c -ldl
make install DESTDIR=/home/runner/work/jabref/jabref/parts/gnome-extension/install
install -d /home/runner/work/jabref/jabref/parts/gnome-extension/install/gnome-platform
install -d /home/runner/work/jabref/jabref/parts/gnome-extension/install/data-dir
install -d /home/runner/work/jabref/jabref/parts/gnome-extension/install/data-dir/icons
install -d /home/runner/work/jabref/jabref/parts/gnome-extension/install/data-dir/sounds
install -d /home/runner/work/jabref/jabref/parts/gnome-extension/install/data-dir/themes
install -D -m755 desktop-launch "/home/runner/work/jabref/jabref/parts/gnome-extension/install/snap/command-chain"/desktop-launch
install -D -m644 bindtextdomain.so "/home/runner/work/jabref/jabref/parts/gnome-extension/install/lib"/bindtextdomain.so

Building jabref 
Staging gnome-extension 
Staging jabref 
Priming gnome-extension 
Priming jabref 
Snapping 'jabref' ...

Snapped jabref_5.0.0_amd64.snap
Preparing to push 'jabref_5.0.0_amd64.snap'.
After pushing, an attempt will be made to release to 'edge'

[LyzardKing]

There's still other parts of the log that can be cut down (regarding the upload), but that's up to the snapcraft tool to remove on "dumb" terminals :)

[koppor]

I received following email:

Rejecting use of system-files as per https://forum.snapcraft.io/t/system-files-permission-for-jabref-native-messaging-hosts-integration/13702/2

To check review details, go to https://dashboard.snapcraft.io/snaps/jabref/revisions/551/review/

Upload status: Rejected

[LyzardKing]

I can't view the page, but I replied to the comment on the forum..
On the other post there was one positive vote..

[LyzardKing]

The discussion is now open again and proceeding.
I uploaded a branch with a fix suggested by the snapcraft forum:
the extension was calling the jabref app from outside the confinement.
Now it's fixed and it works.
It would be nice to automatically test the snap.., but that's for the future I guess
The thing to do now would be (once it builds) to merge the branch https://github.com/JabRef/jabref/pull/5512
It's fine since the voting is not over.
The other issue is that some distros (like Fedora) use different paths to the snaps
(/var/lib/snapd/snaps/... instead of /snap/...)
so it would not work there.
They could always "sudo" edit the json file... so it's not a complete blocker for now..

[Siedlerchr]

@LyzardKing What is the status here? Can we get the snap build running again?

[koppor]

I triggered another manual review today:

[koppor]

The thread at https://forum.snapcraft.io/t/system-files-permission-for-jabref-native-messaging-hosts-integration/13702/13 does not read that positive:

With this, jabref (or extensions) obviously run under the firefox confinement and this has the potential to break firefox, so we would want to maintain the default for auto-connection of the content interface and not allow other publishers to auto-connect. A process could be put in place for allowing auto-connection of certain snaps, but mozilla would likely need to be involved.

[LyzardKing]

That reply regards snap versions of the browsers..
unfortunately the discussion didn't go far..
I explicitly said that for now the supported browsers are non snaps

[LyzardKing]

I wonder if we should try for a classic confinement instead.
That would give access to the whole system (like deb/rpm have) and could be used
also to push to the tex editor (not possible in the strict confinement).
The snap might be larger...and it would still depend on the permission being accepted..
Another idea is to keep the strict confinement without the special permission and use the shell script to add the proper json file (KeepassXC does this for the snap: https://keepassxc.org/download/#linux)

[tobiasdiez]

@LyzardKing can you ask them again what the current status is? (I know you already did it a few times without response...). @koppor can you trigger a re-review or something like this?

Reference: https://forum.snapcraft.io/t/system-files-permission-for-jabref-native-messaging-hosts-integration/13702

[koppor]

---------- Forwarded message ---------
Von: Snap Store [e-mail]
Date: Fr., 22. Nov. 2019 um 21:43 Uhr
Subject: Status update for version 5.0.0 (637) of JabRef

Version 5.0.0 (637) upload for jabref to the Snap Store has been rejected.

The reviewer provided the following feedback:

Rejecting for now while this is still being discussed in the forum.

[koppor]

I am not deep into the topic - thus, I cannot support @LyzardKing in the forum.

[LyzardKing]

They should be reaching out to verify that the account and the name on the snap store is owned by you (the proper upstream for jabref)...
EDIT: I tried pinging the proper users on the forum..

[LyzardKing]

The snapcraft build has a couple of positive votes, so the next step is for someone from snapcraft to contact the jabref upstream to verify that the snap is official.
I have asked to include the chrome/chromium permissions

[koppor]

Follow-up discussion at https://github.com/JabRef/jabref/pull/5811#issuecomment-571931338

[AEgit]

Any news on this? It would be nice to have a working snap image that can be installed from the snap store.

[tobiasdiez]

Still in progress: https://forum.snapcraft.io/t/system-files-under-mozilla-native-messaging-hosts/13627/31 :(

[AEgit]

I reckon, there is nothing that can be done, is there? Or would it help if users were to create an account at snapcraft.io and ask for the vetting to proceed (basically supporting the request)?

[LyzardKing]

I don't think that adding comments requesting the same vetting to proceed would make much difference..
I tried contacting someone from the vetting team directly.. let's wait and see

[AEgit]

Ok, cheers!

[LyzardKing]

Accepted! The vetting has gone through.
I updated the snap in a branch and tested it (with firefox/jabfox) locally

[AEgit]

I can confirm, that the JabRef snap is now available again. I installed the --edge version (sudo snap install --edge jabref). As far as I can tell, however, this does not represent the current developer version (according to snapcraft.io/jabref this version is from 6th October 2019) and, indeed, it does not work at the moment, but gives the following error message, when started from the terminal:

/snap/jabref/550/bin/desktop-launch: line 574: /snap/jabref/550/jabref/bin/JabRefMain: No such file or directory

I reckon the available snap version only needs to be updated and then things will work again (?).

[AEgit]

JabRef 5.0--2020-03-08--93de138
Linux 5.3.0-40-generic amd64
Java 13.0.2

I updated the snap version today using snap refresh --devmode jabref. The snap version appears to be working now (it still throws some errors, but at least the snap version starts now). Well done to everyone involved!

Can this issue now be closed?

[LyzardKing]

The snap works.
@koppor @tobiasdiez Do you think it could be promoted to stable?
That would give more visibility in the app stores

[koppor]

Current issue: all releases show the same JabRef version:

This should be fixed for 5.1. I think, we can use the same version as JabRef uses internally in the buildinfo file. See https://github.com/JabRef/jabref/blob/c2bd8988d37049b51013f86639e3d5b37c195c06/.github/workflows/deployment.yml#L73 for the embedding in the GitHub action thing.

I am working on the snapcraft branch to build and publish the 5.0 release of JabRef. Still shows 5.0.44 as version, but is OK... (Currently installing Ubuntu 20.04 to test the image).

[koppor]

Not sure how I can release a version as full release.

Do we need to change the grade to something else?

https://github.com/JabRef/jabref/blob/c2bd8988d37049b51013f86639e3d5b37c195c06/snap/snapcraft.yaml#L8

Feel free to submit a PR to the snapcraft branch.

[LyzardKing]

Ah yes, the grade needs to be set to stable

[koppor]

beta channel runs fine:

[LyzardKing]

I am using it daily on Ubuntu Mate 20.04, so I can confirm it works.
But if we change to grade: stable the new versions can be automatically shown in the app store and be installed without adding --beta, --edge (which can be reserved for actual beta versions)

[koppor]

Some command line parameters are passed to JabRef instead of the JVM:

[koppor]

But if we change to grade: stable the new versions can be automatically shown in the app store and be installed without adding --beta, --edge (which can be reserved for actual beta versions)

I think, we should keep --edge for the daily builds and use the release channel only for real releases. To keep the releases in Ubuntu conistent with our releases. --beta should also point to JabRef beta releases. -- (Decision from our devcall: https://github.com/JabRef/jabref/wiki/Minutes#2020-03-17-8oo)

[koppor]

Released as stable version. I will close the issue. Regarding the parameters, we could still discuss here. If a user finds it strange, they can open a new issue ^^.

[tobiasdiez]

Right now grade is set to stable in the master branch. Is this correct in view of https://github.com/JabRef/jabref/issues/5417#issuecomment-605712350 ?

[LyzardKing]

Yes, now jabref can be searched in the snap store without issues!

[koppor]

@tobiasdiez Yes. I think, I understood it: grade: stable enables me to release that specific version to stable. If not, I do not have this button (as indicated at https://github.com/JabRef/jabref/issues/5417#issuecomment-605710351). Thus, I "just" have to take care at a release which verson to promote as stable. - No other side effects known.

[tobiasdiez]

Ok, I just wanted to make sure that our daily builds don't end up as stable builds...

[LyzardKing]

@tobiasdiez If it's not automated (and it has to be done manually)
The daily uploads should only go in the edge channel.

[koppor]

Yes, they automatically are added to the edge channel.