Etcher: Creating Debian Linux installation package

Created on 8 Aug 2016  ·  86Comments  ·  Source: balena-io/etcher

Create install images as Debian/Ubuntu deb files, and possible other types like rpm as well.

The idea would be to avoid the "this is how you make it executable, this is how you run it from the file manager", etc issues, but getting as simple as possible to install Etcher on common distros. Reported by @limator

linux packaging

Most helpful comment

Everything up and running now! apt-get update errors should be gone now.

All 86 comments

I tried packaging this for Debian/Ubuntu about a month ago since I have a bit of experience packaging software, but I quickly gave up. Debian policy is to build _everything_ from source, so the packaging infrastructure is geared towards this.

The easiest thing to do is to have http://lanuchpad.net build the packages and host the ppa. However, the launchpad build servers do no allow Internet access during the build. So, we cannot fetch node and bower packages during the build process. This means in order to use launchpad, we also would have to package all of the node and bower packages as debian packages. This sounds like too much work for me.

I suppose we could do like the Arch Linux package and just throw the AppImage into a debian package. There seems like a reason why I decided this was not a good idea and didn't try it, but I can't remember what is was right now.

Hi @dlech thanks for the comments! Yeah, our Launchpad setup has stalled a bit, and good to hear your comments, makes sense.

Debian/Ubuntu

As for packaging, as far as I remember, Ubuntu/Debian packages have the sources in there, so you think the build path could be adjusted by splitting the package downloads and the build to two clearly defined steps. Then could prepare a zip with all sources, which launchpad then builds? As I have no experience with their build system yet, would love to hear if this makes any sense or there are any gaps in the process. And whether @jviotti thinks that the build can be split like that :)

Arch

For ArchLinux, you made me think that indeed building it from source would make a lot of sense, will check it out, now that I have more experience building it locally (and some node bugs that were affecting the build were fixed up).

so you think the build path could be adjusted by splitting the package downloads and the build to two clearly defined steps. Then could prepare a zip with all sources, which launchpad then builds?

This could work. I don't know anything about how node and bower work though, so it would take me quite a long time to figure out how to do this. If someone can figure out how to get all of the source files zipped up, I will be glad to do the rest of the debian packaging.

i guess you can attach the source tarballs to github release page as well:
https://github.com/resin-io/etcher/releases

i'm also interested building from source the rpm package. but considering this is electron app, the node dependencies need to be at least included in the source package.

Hi @imrehg , @dlech

After some discussion, mainly with @alexandrosm , we decided the Etcher GUI _will not_ provide official standard distribution packages (e.g: .deb, .rpm), and will focus entirely on AppImages, at least for the time being, since it aligns much better with desktop applications, and making .deb packages for an Electron app violates a lot of the "debian" rules (not sure if this holds for .rpm).

As a solution for the AppImages execution permissions problems, we can ship a zip containing the AppImage and potentially the user documentation file. Zip will make sure the execution bit is re-added when expanding.

That being said, _we do want_ to provide packages for the Etcher CLI, so I'll close this issue and we can continue discussing Etcher CLI packaging on https://github.com/resin-io/etcher/issues/355.

@glensc

i guess you can attach the source tarballs to github release page as well:
https://github.com/resin-io/etcher/releases

We've considered this as well, however the analytics that we get from GitHub Releases are not enough for us at the time being.

Thinking about it a bit more, it would be nice to have etcher in package
managers, but only if it can be done without a large impact on our release
flow, at least for the time being.

On 25 Sep 2016 5:16 p.m., "Juan Cruz Viotti" [email protected]
wrote:

Hi @imrehg https://github.com/imrehg , @dlech https://github.com/dlech

After some discussion, mainly with @alexandrosm
https://github.com/alexandrosm , we decided the Etcher GUI _will not_
provide official standard distribution packages (e.g: .deb, .rpm), and
will focus entirely on AppImages, at least for the time being.

As a solution for the AppImages execution permissions problems, we can
ship a zip containing the AppImage and potentially the user documentation
file. Zip will make sure the execution bit is re-added when expanding.

That being said, _we do want_ to provide packages for the Etcher CLI, so
I'll close this issue and we can continue discussing Etcher CLI packaging
on #355 https://github.com/resin-io/etcher/issues/355.

@glensc https://github.com/glensc

i guess you can attach the source tarballs to github release page as well:
https://github.com/resin-io/etcher/releases

We've considered this as well, however the analytics that we get from
GitHub Releases are not enough for us at the time being.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/632#issuecomment-249456042,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCOEB3Z2-ipCnOsPW07Mot51AvAQjks5qtw7kgaJpZM4Je3im
.

@alexandrosm I thought that was the discussion above, figuring out how to create packages with minimal-to-no changes in existing setup. :) Hence it would have been nice to just keep the issue open as a place to collect ideas. I guess it won't stop people working on it, anyways.

I think we can reconsider the closure, especially since people are
organically interested in stepping in.

On 25 Sep 2016 9:02 p.m., "Gergely Imreh" [email protected] wrote:

@alexandrosm https://github.com/alexandrosm I thought that was the
discussion above, figuring out how to create packages with minimal-to-no
changes in existing setup. :) Hence it would have been nice to just keep
the issue open as a place to collect ideas. I guess it won't stop people
working on it, anyways.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/resin-io/etcher/issues/632#issuecomment-249474933,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLUCKEj-j4n5JMiVsMr-w488pyiPw5qks5qt0PBgaJpZM4Je3im
.

Sounds good to me, lets keep it open then!

:tada: An unofficial debian package is now availible: https://github.com/dlech/etcher/wiki

Feel free to open issues at https://github.com/dlech/etcher/issues with suggestions or problems or edit the wiki.

Hi @dlech,

Amazing news! I'll give it a go tomorrow! How do you feel about sending a PR with your changes to the main repo, so we can publish .deb packages as part of every release?

Works great in my Ubuntu 16.04 👍

FYI, I have updated the etcher-electron package on in bintray repo for beta 16. I did this using the changes that have been merged into official Etcher.

So, if resin.io want to start publishing debian packages, I think it is good to go. If you need help with a deployment script, let me know.

@dlech Awesome, thanks a lot!

So, if resin.io want to start publishing debian packages, I think it is good to go. If you need help with a deployment script, let me know.

Yeah, sounds good. We have a Bintray account, so if you could give me a hand with a standalone deployment script, I'd really appreciate it!

There are some administrative things to take care of first. I'm looking at https://bintray.com/resin-io, is this correct?

You will need to manually setup a repository and then a package in the repository (this is a one-time setup). What to name things will have an effect on what people will have to use in apt. There is the line to add to /etc/apt/sources.list.d/etcher.list. Mine looks like this:

deb https://dl.bintray.com/dlech/etcher-deb precise main

And here is where each name comes from:

deb https://dl.bintray.com/<user/org>/<repository> <distribution> <component>

So, for you, <user/org> will be resin-io (if I have guessed the correct account).

I think it would make sense to just have a single Debian repository for resin.io as a whole. So, you will need to create a new repository of type Debian. I would recommend naming it debian.

You will need to decide if you want to use the bintray GPG key or your own key. If you want to use your own, you need to create the key and set it up in bintray.

Then, in this new repository, you will need to create a new package. I would suggest calling it etcher-electron to match the name of the Debian package itself.

Once you have done that, there will be a "Set me up" button on the page for the package that you just created. This will have the curl commands needed to upload the .deb to bintray. This is where you choose the <distribution> and <component>. You can pick anything you like for these. <distribution> is traditionally a name like jessie or xenial. So, you could pick something like wheezy or precise to indicate the lowest supported OS version. Or, I suppose you could make up your own name, for example resin. The <component> is usually main, but I would suggest making it etcher. This way, resin.io could use this repository for other project besides Etcher, but people could opt into only the projects they want.

So, in the end, you might have the following the people will need to save as /etc/apt/sources.list.d/etcher.list:

deb https://dl.bintray.com/resin-io/debian wheezy etcher

I know almost nothing about Debian packaging, and even less about sources.list ;-)
Would choosing a <distribution> of wheezy allow all Ubuntu users to install it too? Does this relate to PPAs on Ubuntu, or are they something different entirely?

Edit: And where do the different arches (x86 / x64) fit into this?

Would choosing a <distribution> of wheezy allow all Ubuntu users to install it too?

<distribution> is just a name that has to be matched, so it can be anything. You can use wheezy in your sources.list file and it will work just fine on Ubuntu.

Does this relate to PPAs on Ubuntu, or are they something different entirely?

I suppose technically, PPAs are debian package repositories that are hosted on launchpad.net (and therefore targeted for Ubuntu), but some people use the term PPA in a more general sense to mean any debian package archive that host just a few packages rather than a whole linux distribution.

So, this relates to PPA in that we are hosting our own package repository, just not on launchpad.

And where do the different arches (x86 / x64) fit into this?

The arch is inherent in the debian package, so we will be uploading one *_amd64.deb and one *_i386.dev for each release. apt knows the arch of your computer, so it knows which package to use. Or if you want to install the 32-bit version on a 64-bit machine for some reason, you can apt-get install etcher-electron:i386.

I'll take care of this today.

is just a name that has to be matched, so it can be anything. You can use wheezy in your sources.list file and it will work just fine on Ubuntu.

What about naming it "stable"? Putting a distribution release name here is very confusing, and using a naming convention like "stable"/"development"/"alpha" or something like that would allow us to push continuous builds for testers.

Something else that occurred to me last night: if/when Etcher implements auto-update functionality, I guess that ought to be disabled in the Debian package, as we'll want people to update using apt-get?

@lurch Exactly. Currently the debian packages @dlech built define ETCHER_DISABLE_UPDATES=1 to disable update checks.

deb https://dl.bintray.com/<user/org>/<repository> <distribution> <component>

Since we will be publishing an etcher CLI package soon, I propose the following convention:

# GUI
 deb https://dl.bintray.com/resin-io/etcher stable etcher-electron

# CLI
 deb https://dl.bintray.com/resin-io/etcher stable etcher-cli

I think its cleaner to create different repositories for different applications we might want to publish.

IMHO (and again with the caveat that I don't know much about Debian packaging) it does make sense to have separate .deb files for etcher-electron and etcher-cli, but I don't think it makes sense to store them in separate repositories.

@lurch I might be confusing things, but in the way I outlined above, we create a single etcher repository containing all packages. The user can pick which they like to install using the corresponding component, so the etcher package contains two components: etcher-electron and etcher-cli.

And regarding the distro, I guess Etcher is unusual in that the same binary works on multiple different distros, and doesn't need different versions of the binary to link against different versions of system libraries on different distros?

@lurch I might be confusing things, but in the way I outlined above, we create a single etcher repository containing all packages. The user can pick which they like to install using the corresponding component.

Sorry, I got confused by your "I think its cleaner to create different repositories for different applications we might want to publish." statement, where I guess you _actually_ meant components rather than repositories. Even then, I'm still not sure it's worth having etcher-electron and etcher-cli in different components. Perhaps component is a better place for the "stable"/"development"/"alpha" ? shrug
I defer to @dlech 's greater knowledge.

And regarding the distro, I guess Etcher is unusual in that the same binary works on multiple different distros, and doesn't need different versions of the binary to link against different versions of system libraries on different distros?

Yeah, exactly. This is the reason why I prefer making "distribution" something like "stable"/"development" rather than an actual GNU/Linux distribution name.

Sorry, I got confused by your "I think its cleaner to create different repositories for different applications we might want to publish." statement, where I guess you actually meant components rather than repositories. Even then, I'm still not sure it's worth having etcher-electron and etcher-cli in different components. Perhaps component is a better place for the "stable"/"development"/"alpha" ? _shrug_
I defer to @dlech 's greater knowledge.

Hm, I guess having them in different components is indeed a bit weird, however if we publish both etcher-electron and etcher-cli under the same component, the component "history" will be even more confusing, since it will contain entries from two different packages, and we will also need to somehow clarify which is which on each version description.

As far as I can see, the component is what its used to signal the release type (e.g: stable/development). In that case, we could adopt something like:

deb https://dl.bintray.com/resin-io/etcher generic stable

A "generic" distribution name would mean it works everywhere.

the component "history" will be even more confusing

Is there even such a thing as "component history"? Surely the only thing that matters is the package history.

:+1: for <distribution> names of stable, beta, alpha, etc. :-1: for separate components for the two different Etcher packages. I think many people with want to install both packages, and it will be annoying to have to specify two components.

Then you can provide a etcher.list file as follows:

deb https://dl.bintray.com/resin-io/debian stable etcher
# uncomment the line below to enable the beta channel
#deb https://dl.bintray.com/resin-io/debian beta etcher

Or if you want separate repositories, then...

Then you can provide a etcher.list file as follows:

deb https://dl.bintray.com/resin-io/etcher stable main
# uncomment the line below to enable the beta channel
#deb https://dl.bintray.com/resin-io/etcher beta main

The problem with using "etcher" as the repository name though is that it will be Debian only. If you ever want to add another "etcher" repository on bintray (for docker or npm or whatever), it will have to have a different name. This is why I named the repository etcher-deb on my account.

I guess the only reason to have etcher-electron and etcher-cli in different components / distributions would be if etcher-cli _does_ have version-specific dependencies on external system libraries (which I guess is a possibility, given that #355 hasn't been nailed down yet).

The problem with using "etcher" as the repository name though is that it will be Debian only. If you ever want to add another "etcher" repository on bintray (for docker or npm or whatever), it will have to have a different name. This is why I named the repository etcher-deb on my account.

Fair enough. You can give this a go by adding the following to /etc/apt/sources.list:

deb https://dl.bintray.com/resin-io/debian stable etcher

I'll PR the script to publish to Bintray very soon.

@dlech Do you know if there is a way Bintray can automatically infer the package version and architecture from the file name?

When I did sudo apt-get update

W: GPG error: https://dl.bintray.com stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 379CE192D401AB61

got printed at the bottom.

When I did sudo apt-get install etcher-electron:

The following NEW packages will be installed
  etcher-electron
0 to upgrade, 1 to newly install, 0 to remove and 0 not to upgrade.
Need to get 48.7 MB of archives.
After this operation, 223 MB of additional disk space will be used.

Wow, is it really supposed to be that big??!

But it all installed and seems to work fine :+1:

W: GPG error: https://dl.bintray.com stable Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 379CE192D401AB61

Hm, weird. I configured Bintray to sign using its own GPG signature, but that might not be working very well, however I don't get that error on my machine at all. I'll investigate.

Wow, is it really supposed to be that big??!

Yeah, size is one of the biggest complaints about Etcher. All of it comes from the Electron framework. There are usually on going conversations on that project about how to reduce the final size, but there is not much we can do at the moment.

Do you know if there is a way Bintray can automatically infer the package version and architecture from the file name?

The version and arch are part of the file name, but Bintray does not look at these. Instead, you specify all of the information in the upload command. For example....

curl -T <FILE.deb> -udlech:<API_KEY> https://api.bintray.com/content/dlech/etcher-deb/etcher-electron/<VERSION_NAME>/<FILE_TARGET_PATH>;deb_distribution=<DISTRIBUTIONS>;deb_component=<COMPONENTS>;deb_architecture=<ARCHITECTURES>

I guess I probably need to import the bintray GPG key, @dlech ?

@lurch, yes, instructions are here.

That sucks, since it means we have to either write code to inspect the .deb package, or make the publish script take options for the architecture and the version.

Thanks @dlech , no warnings from sudo apt-get update now :-)

@jviotti Can't the publish script determine the architecture and version from the filename? (sorry if I'm misunderstanding things!)

@lurch Yeah, looks like we're going to have to do that.

It's easy, file name is etcher-electron_1.0.0~beta.16_amd64.deb, so remove the .deb and split on _.

oh, and change ~ back to - if you like.

oh, and change ~ back to - if you like.

I was just going to ask about that :) Do you think we should use the tilde or the hyphen? Is the tilde the "right" way to do it in the debian world?

FILENAME=etcher-electron_1.0.0~beta.16_amd64.deb
echo $FILENAME
BASENAME=`basename $FILENAME .deb`
VERSION=`echo $BASENAME|cut -d_ -f2`
ARCH=`echo $BASENAME|cut -d_ -f3`
echo $VERSION
echo $ARCH

:-)

The ~ is needed in the debian package to make sure beta versions are < non- beta versions. But when it comes to publishing on bintray, it doesn't matter what the bintray version is, so if you want to change it back to - for <VERSION_NAME> in the upload command, it is not problem - just don't change it in the file name.

I am in favor of - since that is the way the version is written everywhere else.

@lurch Oh, your cut approach is way better than the tr + awk one I was currently developing :)

The ~ is needed in the debian package to make sure beta versions are < non beta versions. But when it comes to publishing on bintray, it doesn't matter what the bintray version is, so if you want to change it back to - for in the upload command, it is not problem - just don't change it in the file name.

Sounds good, I'll change it to a hyphen!

Why does /usr/share/etcher-electron/LICENSE say:

Copyright (c) 2016 GitHub Inc.

...

?!

Maybe Electron its adding its own license (Electron its made by GitHub)?

I'm currently re-publishing without the tildes...

Yeah, it is picking up the license from etcher-release/Etcher-linux-x64/. This is just how electon-installer-debian works.

Is that worth "fixing"? We probably don't want to give the impression that Etcher is owned by GitHub?

Is that worth "fixing"? We probably don't want to give the impression that Etcher is owned by GitHub?

Sounds like a good topic for a new issue :wink:

Running apt-get update is now giving me:

Get:26 https://dl.bintray.com stable/etcher amd64 Packages
Err https://dl.bintray.com stable/etcher amd64 Packages
  HttpError404
Get:27 https://dl.bintray.com stable/etcher i386 Packages
Err https://dl.bintray.com stable/etcher i386 Packages
  HttpError404
W: Failed to fetch https://dl.bintray.com/resin-io/debian/dists/stable/etcher/binary-amd64/Packages  HttpError404

W: Failed to fetch https://dl.bintray.com/resin-io/debian/dists/stable/etcher/binary-i386/Packages  HttpError404

E: Some index files failed to download. They have been ignored, or old ones used instead.

Is that just a temporary failure because @jviotti is re-publishing, or has something more serious gone wrong?

@lurch Its probably because I'm re-publishing. I just finished uploading the amd64 one so it should work fine now.

After this operation, 223 MB of additional disk space will be used.

Wow, is it really supposed to be that big??!

That is how big Etcher really is. I removed the upx compression because I was previously building the debian package a different way. We could add it back now without causing problems. It seems to noticeably increase the startup time though.

Hmm... On my bintray page, it shows "Debian Details", but on the resin.io page, it does not. This can be useful for seeing the list of dependences.

I wonder if this is because on the resin.io bintray the bintray package name ("etcher") does not match the debian package name "etcher-electron"?

@dlech Could it be because you only published amd64? Since I published both amd64 and i386, there is not a single file to show the "Debian Details" for.

Hmmm, I just installed the asar command from npm, and running asar list /usr/share/etcher-electron/resources/app.asar (which is a whopping 102MB) shows that it still includes the entirety of the scripts directory, which AFAIK is only needed for development, not runtime?

I wonder if there are any other dev-but-not-runtime files being included too?

...and I'm still getting the same apt-get update issues BTW.

I think I looked before you had published the i386 file.

I suppose it could be an option in the bintray package settings as well.

I wonder if there are any other dev-but-not-runtime files being included too?

Good catch. I didn't update the "packageIgnore" list in a while, and there are some things that can be left aside. I'll send a PR in a moment.

...and I'm still getting the same apt-get update issues BTW.

Weird, let me check here again.

Ah, I see now, that bintray is not publishing the debian meta-data. This explains both what @lurch is seeing and what I am seeing.

Ah, I see now, that bintray is not publishing the debian meta-data. This explains both what @lurch is seeing and what I am seeing.

What do you mean by "not publishing the debian meta-data"? Is there anything I can do from my side?

If you look at http://dl.bintray.com/dlech/etcher-deb/, you can see I have a dists directory. Bintray creates this automatically. It is missing from http://dl.bintray.com/resin-io/debian/ for some reason.

It could have something to do with the way you published the files, or you might have to contact Bintray support.

@lurch I created https://github.com/resin-io/etcher/issues/820 to discuss ideas on how to reduce the final package size.

@dlech

If you look at http://dl.bintray.com/dlech/etcher-deb/, you can see I have a dists directory. Bintray creates this automatically. It is missing from http://dl.bintray.com/resin-io/debian/ for some reason.

Hm, let me see what I can do.

...and yet weird that I _was_ able to successfully run apt-get update earlier :-S

@dlech @lurch Its also failing for me as well. The only thing that changed is that I replaced the tilde with an hyphen. Maybe that is required for Bintray to analyze the files correctly?

I'm re-publishing once more with the tilde, and we'll see if that fixes it.

Maybe that is required for Bintray to analyze the files correctly?

No, I did the same thing on my bintray.

I think it just could be the fact that you re-uploaded the same files and it threw off the automatic part of bintray. I suppose you could try deleting the version and files manually on bintray and re-upload again.

I did delete the packages before re-uploading. Let me try once more, making sure I delete everything I can.

The Bintray dashboard says there are no files at all: https://bintray.com/resin-io/debian/etcher#files

Okay, I think I found the problem. I had a small bug in my script that caused the .deb extension to be omitted from the final uploaded path. Re-publishing, once more :)

Everything up and running now! apt-get update errors should be gone now.

And the "Debian Details" are there too.

Yup, all seems to be fine again :-) The description field on https://bintray.com/resin-io/debian/etcher# is duplicated though:

An image flasher with support for Windows, OS X and GNU/Linux. An image flasher with support for Windows, OS X and GNU/Linux.

I assume now that it's all working you'll be publishing the i386 build too?

EDIT: now done ;)

The descriptions can be changed by adding "description" and "productDescription" to https://github.com/resin-io/etcher/blob/master/scripts/build/debian/config.json

If the current description is OK, then just "productDescription" with a long description can be added.

Yup, all seems to be fine again :-) The description field on https://bintray.com/resin-io/debian/etcher# is duplicated though:

Any clues about this, @dlech ? As far as I understand, electron-installer-debian is fetching the description from package.json, right? If so, why is it duplicated? I'd prefer to have the tool grab the description from there so we don't keep duplicating this in any more places.

From @dlech 's last comment, it sounds like electron-installer-debian is possibly copying the description from package.json to both of description and productDescription in the .deb?

From the electron-installer-debian docs:

options.description
Type: String Default: package.description
Short description of the application, used in the Description field of the control specification.

options.productDescription
Type: String Default: package.productDescription || package.description
Long description of the application, used in the Description field of the control specification.

So, basically, if you don't want this repeated, you need to add a productDescription to package.json.

Thanks guys! I've sent a PR that declares productDescription, and
confirm that fixes the issue.

On Thu, Nov 03, 2016 at 09:27:48AM -0700, David Lechner wrote:

From the electron-installer-debian docs:

options.description
Type: String Default: package.description
Short description of the application, used in the Description field of the control specification.

options.productDescription
Type: String Default: package.productDescription || package.description
Long description of the application, used in the Description field of the control specification.

So, basically, if you don't want this repeated, you need to add a productDescription to package.json.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/resin-io/etcher/issues/632#issuecomment-258195579

Juan Cruz Viotti
Software Engineer

Since the debian discussion took over here, I think we should close this issue and we can open new ones for rpm, aur or other linux package types.

Totally agree. Thank you very much for your awesome work on this!

Was this page helpful?
0 / 5 - 0 ratings