Appimagekit: linuxdeployqt -g guesses wrong at times

[probonopd]

-g is supposed to check TRAVIS_TAG and if it is empty or "continuous", then it should set the update information to "continous"; in all other cases to "latest".

According to https://docs.travis-ci.com/user/environment-variables/

TRAVIS_TAG: If the current build is for a git tag, this variable is set to the tag’s name.

So why is it that we get

Guessing update information based on $TRAVIS_TAG=(null)

https://travis-ci.org/AppImage/NCSA-Mosaic-AppImage/builds/297852566#L5379
but

Guessing update information based on $TRAVIS_TAG=[secure]

in https://travis-ci.org/Subsurface-divelog/subsurface/builds/298113422#L7050?

And why would Travis CI even replace the value of $TRAVIS_TAG with [secure]?

As a result, we get

me@host:~$ /home/me/Downloads/Subsurface-988d5bc-x86_64.AppImage --appimage-updateinformation
gh-releases-zsync|Subsurface-divelog|subsurface|latest|Subsurface-_*-x86_64.AppImage.zsync

where latest is wrong and should be continuous.

[TheAssassin]

And why would Travis CI even replace the value of $TRAVIS_TAG with [secure]?

The docs describe it like so:

TRAVIS_TAG: If the current build is for a git tag, this variable is set to the tag’s name.

So yes, should probably be empty (or (null), actually). Probably an issue in Subsurface's Travis CI configuration.

[dirkhh]

Any idea what might be set up incorrectly? I haven't done anything that I'm aware of to mess with the Travis environment variables...

[probonopd]

I can't see anything strange in your .travis.yml. Can you throw in an echo "TRAVIS_TAG: $TRAVIS_TAG" please?

[TheAssassin]

Well, Travis CI has that feature that the values of environment variables are hidden in the logs.
So this might be just a log formatting thing, [secure] might not be the real value.

[dirkhh]

BTW - i asked you about this elsewhere... why don't I see the result of the updatetool in the build log?
Where would you like me to add the echo $TRAVIS_TAG ?

[dirkhh]

@TheAssassin do I need to somehow explicitly enable that variable?

[TheAssassin]

No, it's one of the variables Travis CI creates.

About uploadtool (it's not updatetool, so maybe a in-browser search might not have worked because of that), see e.g., https://travis-ci.org/Subsurface-divelog/subsurface/builds/298113422#L7156.
Travis' after_success section's commands are collapsed by default, you'll have to hit the little triangle on the left side.

Travis CI provides a switch in the job settings in their web interface where you can un-hide the values. However, I wouldn't recommend it, since the more confidential values are then exposed, too.

[probonopd]

Well, Travis CI has that feature that the values of environment variables are hidden in the logs.
So this might be just a log formatting thing, [secure] might not be the real value.

Variables that are entered as "secure variables" in the Travis CI UI are supposed __not__ to show up in the build logs and are hence censored with [secure]. What I don't understand is why Travis CI thinks $TRAVIS_TAG is a secure variable, and censors its content.

@dirkhh could it be, by pure chance, that you have configured a "secure variable" in the Travis CI UI with the name TRAVIS_TAG, or one with an empty value or a value of continuous? Just speculating wildly here.

[TheAssassin]

@probonopd you either switch it on or off, I don't remember a blacklist feature. But yes, that might be a Travis bug.

[probonopd]

See that "Display value in build log" there? Maybe it is somehow related to that. But beware, do NOT inadvertently show you private stuff like encryption keys, API keys, etc. in the build logs!

[probonopd]

Btw, @dirkhh it's not clear at all whether the bug has to do with anything in your repository or Travis CI setup at all. This feature is really very fresh, may well be our bug.

[dirkhh]

OK, thanks for the hint with the build log @TheAssassin
@probonopd - I don't have any environment variables except for the GITHUB_TOKEN

[TheAssassin]

@dirkhh to make sure it's not an issue with the settings on Travis, we're running a build for a repository fork at the moment. We'll report back as soon as possible.

[probonopd]

No clue why, but when I clone the Subsurface-divelog/subsurface repo and build it in __probonopd/subsurface__ (without changing anything) then I get "continuous":

https://travis-ci.org/probonopd/subsurface/builds/298187785#L4845-L4846

Guessing update information based on $TRAVIS_TAG=(null) and $TRAVIS_REPO_SLUG=probonopd/subsurface
gh-releases-zsync|probonopd|subsurface|continuous|Subsurface-_*-x86_64.AppImage.zsync

which is __correct__.

Whereas the exact same commit in the __Subsurface-divelog/subsurface__ repo gives

https://travis-ci.org/Subsurface-divelog/subsurface/builds/298175465#L7059-L7060

Guessing update information based on $TRAVIS_TAG=[secure] and $TRAVIS_REPO_SLUG=Subsurface-divelog/subsurface

gh-releases-zsync|Subsurface-divelog|subsurface|latest|Subsurface-_*-x86_64.AppImage.zsync

which is __wrong__.

Both Travis CI pages say Commit d09d12e at the top.

Travis CI bug?

[probonopd]

Everyone who wasted time on this: I owe you a beer. Get it at AppImagePizzaParty.

It's my own stupid bug, so stupid that it hurts. Sorry! Will keep you posted when fixed.

[dirkhh]

I wonder if this is because one is YOUR repo .../probonopd/subsurface whereas the other one is ours .../Subsurface-divelog/subsurface - maybe it tries to hide something from you?
But then again, in my build log here https://travis-ci.org/Subsurface-divelog/subsurface/builds/298191953#L7165-L7167 it seems that the variable is empty for me as well...

[probonopd]

Once the above builds, I need to trigger a rebuild of linuxdeployqt and thereafter your next builds should hopefully be golden. Once we start seeing "continuous" in your repo, it's a good time to start uploading the .zsync file as well.

Sorry for having wasted your time on this incredible stupidity of mine.

[dirkhh]

Cute bug - I make those all the time. I still wonder why the 'echo $TRAVIS_TAG' resulted in an empty output...

[TheAssassin]

@dirkhh because the variable _is_ actually empty. @probonopd just used the value of another environment variable, which then showed up as [secure] in the logs, and as it wasn't empty, didn't use continuous but latest.

Luckily, the environment variable wasn't leaked in any of the binaries, @dirkhh, but just to make sure, I'd suggest to generate a new GitHub token.

[probonopd]

@dirkhh please attempt another build, and upload the *.AppImage.zsync, too, to GitHub Releases.

[dirkhh]

a build is just about to complete and I assume that this requires a travis.yml change... so same base name, just a '.zsync' suffix?

[probonopd]

Renaming the AppImage file in this way breaks the magic. Give me some time as I figure out what to do here.

[dirkhh]

I don't need to rename it. Let me remove that part

[probonopd]

Take the files exactly as they come out of linuxdeployqt. If you want other filenames, let me know. But don't rename them "after the fact" because the zsync file contains the filename of the AppImage.

[dirkhh]

The reason why I liked having the SHA in the name is that one can immediately tell which SHA was built to create it, without running or unpacking it... so yes, I'd love to have the naming pattern that is there right now, but I'll remove that so we can continue to experiment with the .zsync

[probonopd]

Thanks. I will think about a solution.

[dirkhh]

In the meantime you can track https://travis-ci.org/Subsurface-divelog/subsurface/builds/298230443
(I need to run a couple of errands, will be back in an hour or so)

[TheAssassin]

The actual filename doesn't really matter too much for the GitHub Releases implementation of the update information. It accepts wildcards as per the spec, and the new implementation uses fnmatch.h to find a matching artifact. So, you can safely add a hash in the filename. However, this comes with the drawback that if the file is updated, it won't be renamed.

Therefore, my AppImage builds always include the Git commit hash ID, and print it out when -V/--version is specified.

[probonopd]

You may be right @TheAssassin that's what the * is meant for in the update information. But let's make first sure it really works. Also, I was thinking that we might add the SHA to CI builds automatically in appimagetool even, so that it "just works" without having to rename stuff in the .travis.yml.

[dirkhh]

ok, now I have a .zsync file... what do I do with that? :-)

[TheAssassin]

@dirkhh there was a bug in appimagetool causing it to generate the wrong wildcard (a few characters too much), which is fixed, build is on the way. We'll post an update once it's built, then you can trigger a rebuild, too. And then finally we should be good to go.

[TheAssassin]

@dirkhh meanwhile you can get AppImageUpdate from https://github.com/AppImage/AppImageUpdate/releases.

[dirkhh]

The README of AppImageUpdate is tantalizingly vague in that it _almost_ tells me what it does and how it is used... I'm guessing I am missing a few pages of discussion on some mailing list to fully understand how I am supposed to use this :-)

[probonopd]

Please restart a build now @dirkhh then the wrong wildcard (a few characters too much) should be fixed.

[TheAssassin]

@dirkhh it's a pretty young rewrite, it lacks quite some documentation, and a few features. At the end of the week I'll update the README to the current state.

[dirkhh]

I'm just about to push the GStream magic to master which will start a build. One sec

[dirkhh]

Let's hope that I folded this into my .travis.yml correctly :-)
Build is over here: https://travis-ci.org/Subsurface-divelog/subsurface/builds/298273799

[dirkhh]

So what I'm hearing is that by preparing for the .zsync files we are ahead of the times :-)
That's cool - you guys have been incredibly helpful to us, I'm happy to pay it back by being the guinea pig for some interesting innovation!

[probonopd]

./appimageupdatetool-x86_64.AppImage Downloads/Subsurface-x86_64.AppImage 
Checking for updates...
... done!
Update not required, exiting.

Now, if you do another build, and then run the same command on the _old_ Appimage again, then it should do a delta update...

[TheAssassin]

@dirkhh by the way, at the end of the week, it is planned that some API docs for AppImageUpdate's core library will be available, and a little tutorial how to integrate it into other applications. Then you might be able to implement a self-update feature for Linux.

If you want to contribute a bit to us -- you could send us a Qt expert who's willing to rewrite AppImageUpdate's (currently FLTK) UI in Qt.

Just kidding. If you have feedback for AppImageUpdate, you're invited to open an issue or visit us on IRC (#AppImage on Freenode). UX is the next big topic for this application.

Meanwhile, Subsurface finished, and it works fine! Please try it out!

[probonopd]

we are ahead of the times :-)

@dirkhh iirc you told me some time ago that Subsurface is a _proving ground_ for innovation ;-)

[dirkhh]

@probonopd happy to be a _proving ground_
The appimagetool - how does it deal with two dozen builds being skipped. I.e., does this only work if I run build n-1 and now build n is up?
@TheAssassin finding good contributors is HARD.
Especially people with UI experience. This is one of our biggest challenges as well.

[probonopd]

@dirkhh by the way, at the end of the week, it is planned that some API docs for AppImageUpdate's core library will be available, and a little tutorial how to integrate it into other applications. Then you might be able to implement a self-update feature for Linux.

Actually Subsurface already has a version check, written in Qt. Possibly this can hook into libappimageupdate in the not too distant future. But that's the topic of https://github.com/Subsurface-divelog/subsurface/issues/770 ;-)

[probonopd]

The appimagetool - how does it deal with two dozen builds being skipped. I.e., does this only work if I run build n-1 and now build n is up?

No problem. Will work just fine. It does block based hashes, and downloads the blocks that have changed using HTTP range requests.

[TheAssassin]

Well, I will have to improve AppImageUpdate and zsync2 dependency wise first. It's all a bit alpha still. It works well however.

[dirkhh]

Actually Subsurface already has a version check, written in Qt. Possibly this can hook into libappimageupdate in the not too distant future. But that's the topic of Subsurface-divelog/subsurface#770 ;-)

Yes, absolutely - that's why I'm asking about n/n-1 vs multi-version update.
Many divers start Subsurface a few times in a row (during a dive trip), but then not for several months...

[TheAssassin]

No problem. Will work just fine. It does block based hashes, and downloads the blocks that have changed using HTTP range requests.

That's the magic of zsync(2). There's no real work server wise, you just publish two files, and the rest is calculated client side. Version and build numbers are a relic, as @zyga once said, they're "just tags" and "they mean nothing [to the update processes]".

[probonopd]

Also, continuous builds have "continuous" in the update information string, whereas releases should have "latest" - which means releases update to releases, and continous builds to continuous builds.

[TheAssassin]

Right, latest maps to GitHub's Releases API, which returns the latest real release. Continuous builds uploaded with uploadtool are marked as prereleases, hence you need to hardcode the tag name continuous in the update information.

See the spec: https://github.com/AppImage/AppImageSpec/blob/master/draft.md#github-releases

[dirkhh]

brilliant :-)
I have to admit that this sounds really cool.
In the meantime I'm rebuilding QtWebKit to actually shrink it by disabling all the crud I don't need (including GStreamer) and hoping that this makes things smaller, faster, and less in need of hacks...

[dirkhh]

one thing that I find challenging... if I want to play with this on a feature branch (I'm working on debugging a problem, I need AppImages to deal with that)... if I push to that branch, that becomes the latest pre-release binary.
Would it be possible to use continuous for master and continuous-branchname for other branches and offer them separately? Or is this beyond what GitHub/Travis can do?

[probonopd]

Sounds entirely doable, if one has the time and patience to massage uploadtool and appimagetool (which linuxdeployqt internally uses) a bit. I'll open a feature request on it.

[TheAssassin]

Sounds good. I'd love to receive some feedback for my APIs. I've designed dozens, but I'm always curious whether I do my job well or not.

Re gstreamer: Desktop Linux Platform Issues strike again! Someone should really set up a working group or committee in a larger organization some day where this is discussed with a broader audience. I've watched a video today from 10 years ago where the same points are discussed, and the majority still applies today...

[TheAssassin]

Would it be possible to use continuous for master and continuous-branchname for other branches and offer them separately? Or is this beyond what GitHub/Travis can do?

This is entirely doable, maybe not with @probonopd automagic update information detection but by setting the update information manually with a more narrow filename parameter containing the branch name as well. It's just some work, but not at all complicated.

[probonopd]

Even _with_ automagic. Travis CI is kind enough to supply all needed information in the form of environment variables.

[TheAssassin]

Right, but then you'd have to include a branch by default. You can never know what branch is set as default (or don't want to rely on it). And I think then the filename will be cluttered. This is out of scope of automagic.

[probonopd]

Closing here since bug seems fixed. Continuing in https://github.com/AppImage/AppImageKit/issues/519