Gitea: Provide releases and deb/rpm packages via Bintray

Created on 3 Nov 2016  Â·  110Comments  Â·  Source: go-gitea/gitea

I would like to integrate a proper process to create real system packages and distribute them via Bintray, than users can just add a deb/rpm repository and got a clean upgrade path.

Beside that we can also publish our releases to a static page or also to Bintray for download.

kinbuild kindeployment revieweconfirmed

Most helpful comment

Both binary and docker are needed.

All 110 comments

Do we really need deb/rpm? Aren't just docker images enough? Idk, prove me wrong! :)

drone only has docker images, right?

Gogs is using packager.io, why not reuse that?

Because not everybody is/will use docker.

Packager.io only supports lts versions, beside that the golang packaging can be done much easier than with the overhead of Packager.io.

Okay I don't know enough about Go. But having a solution for Ubuntu 16.04 would be great, as https://github.com/gogits/gogs/pull/3617 didn't get merged :(

To provide packages for any ubuntu/debian/whatever version we can wrap the same binary into the package and just need to adjust the init (sysvinit, systemd) script.

Can we merge the 16.04 packager.io PR and enable gitea on it for the time being though?

Both binary and docker are needed.

Can we merge the 16.04 packager.io PR and enable gitea on it for the time being though?

We don't have enabled anything over there yet, but since this issue is estimated for 1.0.0 I would like to introduce it for our first release.

I also would like binary releases - I don't have a go development environment setup to compile things.

Also, will I be able to upgrade to gitea from gogs? I'd hate to loose my issues etc.

We will have binary releases and hopefully also packages for the major Linux distributions. You don't need to compile anything.

We will also have binaries for the latest master version.

You will be able to directly launch gitea with the Gogs database. If we got database changes they are getting automatically migrated.

Ooh sounds awesome! I'm looking forward to it then ☺

-- Starbeamrainbowlabs (keybase.io/sbrl)

https://packagecloud.io (OSS — 25 GB Storage / 250 GB Bandwidth + CDN bintray.com 1TB/m)

@tboerger https://packagecloud.io/docs#os_distro_version LTS and non-LTS deb/rpm/etc…
elementary OS / Raspbian / Ubuntu / Debian / SUSE Linux Enterprise Server / openSUSE / Fedora / LinuxMint / poky distro / Oracle Linux / Scientific Linux / Enterprise Linux (CentOS, RedHat, Amazon Linux)

@denji yeah I know packagecloud, maybe we will take it, but than we need a drone plugin for it :)

This won't happen for 1.0.0, so I will move it to 1.1.0.

Gogs already provides deb and rpm packages, so this would be a real pity if it won't happen for 1.0.0.

Will I be able to update Gogs 0.9.99.0903 to Gitea 1.1.0 without problems?

Gogs is using a service that only supports lts versions and which does some strange binary wrapping. We want to have a real solution.

I think you could upgrade from Gogs 0.9.99.0903 to Gitea 1.1.0.

@tboerger what about doing something like this? https://blog.codeship.com/using-docker-build-debian-packages/

@lunny I'm guessing @jhasse means upgrade by apt-get upgrade

A plain apt-get upgrade won't work since we will have a different package name. But I think it will work to do apt-get remove gogs && apt-get install gitea :)

Then they have to copy Gogs' app.ini to Gitea.

Then they have to copy Gogs' app.ini to Gitea.

They have to do that before apt-get install gitea though, otherwise the service would have already been started.

@jhasse IIRC debian packaging rules forbid auto-enabling services on install :wink:

You can configure this: http://askubuntu.com/questions/365911/why-the-services-do-not-start-at-installation

And I think the default on Ubuntu is to auto-enable them. At least when I installed Gogs on Ubuntu 14.04 I didn't need to do it manually.

It is true that packager.io does binary wrapping, but it is needed for gogs/gitea. Several environment variables need to be set up, etc. I found the CentOS 6 gogs RPM very simple to install and set up.
Whatever packaging solution you pick for gitea please make sure it is easy to install, configure and upgrade.

Many packages rely on environment variables, still no need to wrap the binary. And easy to install and configure is always the focus of a package or not?

So I actually have gitea packaged up in a .deb. I allow nothing on my server that isn't packaged properly. It's a little bit of a patchwork job at the moment (partly because of this #480).


On the subject of upgrade path... .deb packages allow some files to be declared as "config" files which the package manager will:

  • Keep after uninstalling the package until the user explicitly "purges" the package
  • Check for modification before overwriting with with an upgrade. It will ask the user what to do if the config has changed.

This allows the .deb to include a default app.ini file containing a bunch of defaults that are different to the core gitea defaults (eg: file paths mentioned in #480).

There is also a mechanism for obsoleting other packages (such as gogs) but this may be less useful if gogs was installed without a .deb


For anyone interested in having a poke around my config code is here: https://github.com/couling/gitea/tree/dpkg-config/scripts/dpkg

I build the package (based on the manifest file in my config) using an old script which I haven't yet thrown open-source but I can do that soon if it helps.


@jhasse IIRC debian packaging rules forbid auto-enabling services on install 😉

That's interesting. Every server package (including such things as apache2) have started the service when I installed.

@couling could you send a PR?

@tboerger any update?

No updates so far from my side. IMHO we need to run different docker containers to make sure it's properly matching the target OS (reffering to the glibc issue). Beside that a docker workflow excludes 32bit systems and the other architectures since we can not support that with our buildchain yet.

So let's put this to v1.2?

I'm not sure if this is helpful. But I have an experimental Golang lib/tool to create Debian packages without using debian at-all (just native golang). I also had some "problems" when creating packages for Debian in a "cross-platform" way and I'm not a big fan of dpkg builder. Have a look: https://github.com/xor-gate/debpkg

It is still in experimental stage, but it is able to generate installable packages.

@xor-gate thanks, will have a look when I come back to this issue.

I was going to take a stab at packaging Gitea for Debian, but I see someone else has beaten me to - see Debian Bug 79210.

@mjturner This has been a five month long project for me so far and at my current rate, at least that much longer. The bulk of the work isn't too bad, but I'm stuck in a few key places that are absolutely destroying my ability to make progress.

I need the most help writing package descriptions for all of the gitea dependencies that are not currently packaged in Debian. This has been consuming at least 80% of my time because my Golang skills are weak and many of these deps have little to no description about what they do. I've also been fighting with some more debian-build stuff.

If you, or anyone else wanting 1. install debian 2. apt-get install gitea, want to give me a hand, feel free to hit me up on IRC! MTecknology @ Freenode/OFTC

@MTecknology You beat me to it - I was going to email you and see if I could help in any way. I have a fair amount of Debian packaging experience (I'm a Developer emeritus), although none with packaging Go applications, and a decent amount of Go development experience. Will get in touch via IRC shortly

@mjturner I seem to have scared you off! If you're at all still interested, I'm at a place where this is getting a bit overwhelming and I could really use a hand.
or.. anyone else? Anyone at all that understands Debian packaging and/or Go? Pretty please?... :cry:

Hi guys, debian packaging is not very intuitive because it is very very flexible. Most people just want some files inside a .deb and some meta (packages, version, description). I think the fastest way is to use some frontend tools to build a simple debian package. E.g https://github.com/laher/goxc or https://github.com/jordansissel/fpm.

The guys at Syncthing (golang project) use FPM to build a debian package in their build.go script:
https://github.com/syncthing/syncthing/blob/2579e8f7152c3205691f3798a81d43c1af4e8af6/build.go#L531-L539

Some people have already put some effort into the debian Make build system to have a dh-make-golang (debian helper make for golang), see https://github.com/Debian/dh-make-golang.

It needs some love to package for debian, but it is worth the effort.

@MTecknology Apologies! I definitely haven't been scared off, I've just been so swamped with a ton of other things that I haven't had a chance to give Gitea another look since our last contact a few weeks ago :(

My time is still pretty limited at the moment, but I am still very keen on getting involved.

No worries. This project is more than any reasonable, sane, or well adjusted person should ever attempt to pull off. Right now I think I'm dealing with version incompatibilities where the build succeeds but runtime results is a segfault. At least it's finally a policy-compliant build.

It turns out I need some additional javascript libraries packaged. More... woohoo..... :sob:

@MTecknology Any chance you could share your current WIP packages (eg via public repo)? Might help encourage getting others involved.

@mjturner heheh... you actually posted a link to where I posted a link to my WIP (the debian bug)
--> https://github.com/MTecknology/gitea-wip/blob/master/work_in_progress

There's been no significant change since my last comment on this thread. I managed to get an actual build, but it segfaults. I have a pretty solid hunch the segfaults were because of mismatched library versions. I have most (~90) of the new build deps included in unstable, but I need to wait until after freeze before I can consider updating anything. (sooooon)

At the moment, I need the most help with javascript packages/dependencies.

This is going to be held up by #1484, among other things, but Debian 9 is released and I'm planning to start making some progress again in the near future. There's still a lot to do, but I'll get this sucker closed someday!

hello, has anyone succesfully made rpm's in fedora/centos/rhel?

@tboerger - I really love what you and your team is doing with Gitea! To address folks concerns with Gitea not having RPMs yet, I have written a tool to generate Gitea RPMS for Red Hat derivatives 6.x and 7.x. I've been testing it locally for a while now and I'm really close to releasing it in the next day or two.

Not sure if you are looking for a GO solution to build RPMS or if you just want a process that builds RPMs. My tool (even though it's mostly written in python) will generate RPMS that conform to Redhat's packaging standards.

Not sure if you are open to using such a tool but I do plan on putting on my public gitub in the next day or two.

We are always open to PR even if not merged it could be usefull for other. You could use folder contrib to store your script. Overall we try to keep it to golang but for specific case we make exception and python is a common tool.

@codylane looking forward to it! I was working on doing this using the fedora packaging way but, I barely know how to do that and was going to be a shot in the dark.

Right on, glad to hear this might be useful for people. I'm just finishing up the readme but I should have it on my public github account tonight. I'll share the link as soon as it's available.

I'd be happy to submit a pull request to gitea, just figured the community could help me make sure it works before it goes in your repo. I have a lot of integration tests around the RPM but the best way to make sure it works is get in front of people. :)

Thanks again for creating such an awesome open source tool!

Maybe this can also be some ground work for a drone plugin, already talked to another dev who created a deb plugin for drone.

Cool. I need to checkout drone as well and it's been on my list of things to play with.

As promised, here's an automated CI process for building the Gitea RPM for CentOS 6 and 7. https://github.com/codylane/gitea-rpm.

Please let me know if you questions or concerns or if there are any problems. I've already opened one small issue with systemd. I'll fix it tomorrow.

Have you thought about using Copr?

Maybe who can send a PR to change the drone file to generate deb? @tboerger @bkcsoft @appleboy

@jhasse than we can also use the open buildservice. I don't believe that these tools are the best selection.

I would prefer a drone plugin for rpm and for deb and host the repos on our own or on bintray/packagecloud

@tboerger - Just curious would you like me to take a stab at the drone stuff? The reason I ask is because I was thinking about hooking up my repo into TravisCI. I totally understand if you guys would prefer a self contained build service. :)

If you would like me to look into drone would you be able to share any documentation on how your team is using it? I'm big on standards and don't like re-creating the wheel if I don't have too.

Currently we are using mostly plugins created by drone contributors from the drone-plugins github org, @thomasf started to build drone-deb for Debian based packaging and I would like to start a drone-rpm plugin within the drone-plugins org.

You can see our .drone.yml within the project root, we are hosting our drone instance at drone.gitea.io

Cool. Once I fix this systemd problem i'll take a look as you recommended. Thanks man!

FYI - The systemd issue has been resolved and the RPMs should be in good working order. Still need look into the drone stuff.

Just throwing in another idea to provide a simple means of installation via a "package" under Linux: http://linuxbrew.sh.

Hi, any news about the deb packages? I would really love to have a gitea deb package ;)
Actually I would need them for Ubuntu 16.04 and Ubuntu 18.04, just to be more precise ...

I'm also willing to make the debian package and make it available if this step hasn't been done before, just let me know ;)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

Thanks stalebot for your contribution.

I'm still promoting my github.com/xor-gate/debpkg go package for debian packaging 🎉 .

@xor-gate could you send a PR to add a config file for your library on contrib? Maybe we could release it on a drone plugin.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

Please consider as demand for it...

Only stale because of the lack of a solution.

I don't think any one person has enough energy to keep gitea packaged for
Debian main, there are just too many dependencies with the typical golang
ecosystem problems [1]. I spent over a year at > 30 hr/wk trying to make it
happen; when I gave up, it took less than three months for the libs I
pushed into Debian to be pulled from the archive due to breaking changes in
dependencies.

I could help someone maintain this for non-free (or main if a masochistic
team presents itself), but it would need to come with a warning about lack
of stability, for many reasons.

[1]

  • Some libraries are dead, with security bugs against them
  • A lot of libs have been forked with varying level of maintenance (mostly
    patch and forget)
  • New libraries are arbitrarily added, often with little to no review
  • 'go vendor' (curl | sudo -)
  • The >200 golang deps rarely provide releases (expecting apps to always
    pull latest git, including whatever additional deps libs added)

Additionally-

  • There seems to be no method for coordinating security updates (with the
    mindset that updating keeps you secure... it does the opposite)
  • The >200 golang libs pales in comparison to the >2,000 js libs
  • There is a terrible lack of concern as to what can legally be included in
    open source software
  • Efforts to fix these issues have been met with significant resistance

While it's possible to package gitea for distributions that don't mind
these licensing and compatibility problems, I gave up when I realized it
can never survive in Debian main without a team of masochists with a whole
lot of time available, for more than just the initial release.

Again, I'm happy to help anyone interested, but I've personally moved on to
producing a more minimalistic "clone" that removes these problems. (Slow
going because of little time available)
... If anyone is interested in either my project or working on gitea for
Debian, reach out to MTecknology on Freenode.

>

@MTecknology there is a PR to resolve this problem. see https://github.com/go-gitea/gitea/pull/6671 .

It doesn't resolve any of the problems I mentioned, it just introduces them all into a convenient package that's still unsuitable for the Debian archive.

And what about providing a ppa (which we would host) to provide automatically built debian packages?

Would would have thought that JS libraries would be bundled with gitea itself - I've always avoided the versions of so-called javascript libraries available via apt, because you can't guarantee what version that are.

As for the go libraries, I thought that Gitea released just a single binary? That's what I've been using, at least. Could a simple apt repository not be created that has a package that contains that single binary?

Disclaimer: I'm _not_ a debian maintainer,a nd have rather limited experience packaging things into .deb files. My experience runs as far as fpm

@MTecknology You've done a huge amount of work, but unfortunately I feel like it doesn't necessarily make sense to package gitea for debian. There isn't a goal in the project to support old versions with security or bug fixes which would match debian release cycles. And as you mentioned, the amount of dependencies is unmanageable. The go proverb "A little copying is better than a little dependency." has been ignored. But that's quite understandable as well, as it's a volunteer effort with little resources for stuff like keeping dependencies up to date or refactoring copied code to only include necessary stuff and then backporting fixes. And the JS/NPM world of dependencies is even worse. I believe software like gitea is better offered as docker images that work on several linux distributions. Hopefully the docker images are locked down, since the code still has the issues with dependencies and security fixes that you mentioned.

Provide a yum repo and apt source will be another with an installation shell script will another choice.

Of course it would be cool to have gitea in the official repos of Debian or ubuntu, but as already stated above that results in a lot of work.

To avoid such a packaging nightmare we could simply build deb and rpm packages that are provided via bintray or even via some folder on the download mirror which gets backed by the cloudflare cdn.

The packages itself could be built via fpm, or somebody contributes a drone plugin which could even land in the drone-plugins org if it's written in go and got some standards. There is even a go library that implemented lots of fpm features.

In terms of packaging formally for distributions, although that could and probably should be a long term goal I don't think we should necessarily be doing that at present. Our codebase is still rapidly changing and I think at present we can't commit to maintaining the very old versions that would end up being packaged formally with these distributions.

I'm also concerned about the number of external dependencies that we have - we should definitely think about whether we need these as hard dependencies (e.g. gorgeous, boltdb) and whether some of our smaller dependencies might benefit from being forked or rewritten. (Macaron session and go-ini are both problematic imho.)

We also need to refactor our internal structure to make go plugins a reality. This I think would allow us to make a leaner core-gitea that could be packaged without bringing in all of the other dependencies.

Properly packaging Go projects is on any distribution a PITA. That's why we should just build the binary and provide simple packages that ignore the best practices of the distributions to package every dependency as a separate package.

Speaking as Debian maintainer, it would be very great to have gitea in Debian as package itself. But there are technical (and maybe license) issues with it, so this is out of scope of this issue.

Speaking as Debian user, it would be cool to ship gitea via deb package. As we already have a static go binary, it can be shiped in it. The advance of this is, that we can also include proper infrastructure like unitfiles for getting it started on system start and maybe a sane default config if needed.
For most ppl it means adding a package source, adding a signing key and just install the package. Updates are deployed in a way users already familiar with.

It would be great to get an rpm into a frequently updated dnf/yum repo so rpm distro users can easily stay up-to-date with security updates, etc to gitea.

@waja Any change of your CentOS package being updated for Fedora and possibly added to EPEL if it is tested and stable?

Any news on this? A month ago @techknowlogick in #6671 said that Drone could be used for automated builds of rpms and debs. Did someone manage to find a moment to enable it? If so, could you please update documentation?

@Janhouse I have been trying to get this packaged for Fedora but Go is a pain to get properly packaged. The CentOS version linked above did not actually properly build for me and I noticed some changes that needed to happen with it.

I am still interested in proper official builds though as it ensures the packages are better maintained.

@waja dont know mouch about geting a project into official debain repo but alpine-linux has already a working package: source

We won't get gitea into the official Debian repos, it's a really huge effort which already failed in the past.

How hard is a ppa to maintain?

Not particularly hard, @6543. In fact, I have my own apt repository here: https://apt.starbeamrainbowlabs.com/

The code I use to manage it can be found here: https://git.starbeamrainbowlabs.com/sbrl/aptosaurus

If someone has a Bash script that automatically downloads the release files from GitHub, verifies the SHA256 hashes, and packages them with fpm, I'd be happy to run it myself (though I'm not sure that I'd be able to host previous package versions; Gitea is quite big and the storage space quickly adds up).

Building the package should be done by drone, and than it should be simply uploaded to bintray ;)

@tboerger I'll try make a draft PR with the deb build script ...
& we have dl.gitea.io is it worth to upload the deb there too (and perhaps as atatchment on release)

I personally wouldn't like to manage deb and rpm repos on my own, that's why Bintray would be great for that. Just uploading a deb or rpm file without any repository is IMHO pretty useless.

Just uploading a deb or rpm file without any repository is IMHO pretty useless.

it's completely nonsens!

We could also use nfpm for building deb and rpm files

Somebody has already started some time ago to build a drone fpm plugin... I still want to build one within the plugins org based on the go fpm fork.

Here is a drone plugin that I made for nfpm: https://github.com/techknowlogick/drone-nfpm

If you make packages for FHS systems remember to either use the shadowing script in contrib or build with the LDFLAGS set correctly.

Hi all, I have my own APT packages repository, I can add gitea on it.
See the project already included : https://packages.azlux.fr/
Now it's running, it's easy to add any .deb you want (I can even create it)

Azlux

@azlux it all comes down to the trust. You are probably a cool guy but I would rather install it from some known automation service where build scripts are maintained by the gitea dev team, than get it from some private repo. I have my own packages built as well, but ideally it would be managed by gitea team.

Ho ok
Thank for the info @Janhouse
I've setup my repo with reprepro, I can help if you need to.

Az

@Janhouse After a deeper thinking, The apt repo allows us to verify that the sources have not been modified in the meantime. Ideally, it's better to have gitea maintaining their own repo, but they can also put other repo as "unofficial repository".
Lot of project don't have a problem with that. Like that, People warned.

I know I've said this before but...

If you're providing Deb packages, you should probably build Gitea so that it obeys the FHS by setting the LDFLAGS as described here:

https://docs.gitea.io/en-us/install-from-source/#changing-the-default-custompath-customconf-and-appworkpath

Same here - I have a personal apt repo I can host in

I would also like to see gitea follow the FHS @zeripath , its silly they need a wrapper shell script to do so too.

Any news on RPMs in the meantime? I see a lot of discussion about DEBs but many enterprises will be using CentOS which does not use DEB packages. The BSDs seem to have their own packages (with patches) but their processes are a bit different.

@evitalis you don't need a script if you build it with the appropriate LDFLAGS.

I know, I was referring to their need to even have this.

Just thinking of it. FHS path should be default for v2 as it would be a needed breaking change.

I think we need to break that down a bit. There are four kinds of build at present:

  • development builds - you download the source and you build with make
  • personal builds - you download the source and you build with make but want this a running server
  • release builds - we build with make release and stick on gitea.io and GH
  • docker builds

FHS by default makes sense for up to 2 of those: release and possibly personal - but at present you can't tell if a personal build is a development build or vice versa. Breaking the development builds would be a very very bad idea.

The Docker should also have its defaults embedded - it's nonsensical to have to require "-c" by default on a docker build we build ourselves.

There's a possibility of changing where we look by default depending on where the binary is placed - but I suspect that might be a bad idea. I guess I'd have to check again what is normal practice for this. (hierarchies of config lookups? - ugh complex)

When I first made the LDFLAGS option work I proposed:

  • Make a separate release with FHS paths embedded
  • Make the docker have the default paths for docker embedded - which solves a few other weird issues with running gitea on the docker command line.
  • Add/Expose a make endpoint for FHS builds that sets the flags properly <- Package builders should use this...

These are non-breaking changes that would not necessarily require waiting for 2.0 - we're just adding a new download option.

An alternative is to add a configure script and make install endpoints - as then it would look "normal" to people used to:

# get source by whatever means necessary
cd gitea
./configure
make
sudo make install

It's absolutely normal to search within predefined paths for configs if you do not provide a flag for a custom path. That's as an example already part of https://github.com/spf13/viper to define multiple standard paths. The current fuckedup behavior of Gitea is totally uncommon.

Perhaps it would be worth putting the release builds into the repos first would be a good 1st step?

@sbrl already done on my repo , the build script is store here : https://github.com/azlux/dpkg-deb
As you said, It use directly the release build.

@zeripath I think you are making this more complicated than it needs to be based on your reply, but I may be misinterpreting it. If people are running git clone and building things themselves then they should know problems might happen. For many projects either doing said build from the master branch or using a release is considered stable, anything else is not and can break at any time.

Even in Docker there is no reason to not follow the FHS so that case goes away too. For the rest as noted by @tboerger checking for a standard set of paths for a config is considered normal. Relying only on $PWD or random compile flags that most users won't ever see causes more issues, including with OS packaging. As someone who has packaged more than a few things the less standard the application the more work for me and the more patches I may have to introduce.

With all that said I am still interested in seeing official RPMs at least. I don't use Debian or Ubuntu but I am sure an official DEB would also be welcome. Thanks to everyone in this thread doing their best on the packaging in the interim.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

haytona picture haytona  Â·  3Comments

kifirkin picture kifirkin  Â·  3Comments

Fastidious picture Fastidious  Â·  3Comments

thehowl picture thehowl  Â·  3Comments

jakimfett picture jakimfett  Â·  3Comments