Elasticsearch: Apt repo missing previous versions of Elasticsearch (Packaging)

Created on 2 Dec 2016 · 27Comments · Source: elastic/elasticsearch

It seems that, as soon as a new release of Elasticsearch happens, the previous version is removed from the apt repository (ex. Ubuntu 16.04):

root@ubuntu-1604:~# cat /etc/apt/sources.list.d/elastic-5.x.list
deb      "https://artifacts.elastic.co/packages/5.x/apt" stable main

Could we leave a few recent versions up there, so that if someone needs to reinstall, or run CI/CD that had a previous version, that it still works? If this was a deliberate decision, I apologize in advance -- I'm just having trouble keeping up with the releases and suddenly having my CI break until I bump the version there (this has happened 3 times now, where previous versions disappear from the apt repo, IIRC).

Thank you in advance!

:DeliverPackaging >bug Delivery

Source

martinb3

👍12

Most helpful comment

aptly did the trick. I just did this without trouble:

sudo apt-get update
sudo apt-get install elasticsearch=5.1.1
sudo apt-get install elasticsearch

There is nothing to show in Elasticsearch so I'm going to close without reference to code. I switched the unified release mechanism to building the apt repository with aptly rather than reprepro.

nik9000 on 14 Feb 2017

👍5 ❤2 🎉2 😄2

All 27 comments

This is a side effect of us using reprepro for building the apt repository. My understanding (thanks to @jordansissel) is that it will overwrite with the latest each time we update it. I also understand that apt-ftparchive is supposed to not suffer from this deficiency, so I will look into switching our release process to use that.

rjernst on 2 Dec 2016

👍3

@rjernst aptly (https://www.aptly.info/) has worked excellently for me in the past and AFAIR can maintain multiple versions of the same package in a repo. Bonus points for a sane CLI and solid documentation.

Xylakant on 14 Dec 2016

@rjernst if your apt repo is in an s3 bucket, and I believe it is, you should use deb-s3 (https://github.com/krobertson/deb-s3). It's awesome, and easy to use. Make sure to call it with -p.

jjshoe on 1 Feb 2017

👍2

Is there any chance this will be fixed soon? 5.2.0 just came out and removed my ability to install 5.1.2 :(

jforest on 2 Feb 2017

5.2.0 just came out and removed my ability to install 5.1.2 :(

You can download the deb manually.

jasontedor on 2 Feb 2017

You can download the deb manually.

Yes, but that breaks my automation

jforest on 2 Feb 2017

👍2

Yes, but that breaks my automation

Sure, but you can automate downloading and installing it. 😇

Do not take this to mean that we are not going to fix the problem. We are going to fix the problem. I'm just offering you solutions in the meantime for the problem that exists today.

jasontedor on 2 Feb 2017

@jasontedor I think the frustration for me is that it only takes five minutes to fix what is a relatively painful issue.

jjshoe on 2 Feb 2017

I know that this issue is very frustrating, I'd be frustrated by it too if I were on your side. However:

I think the frustration for me is that it only takes five minutes to fix what is a relatively painful issue.

This assumption is incorrect.

jasontedor on 2 Feb 2017

as a workaround, hosting a private mirror of the repo should be relatively straightforward.

Xylakant on 2 Feb 2017

as a workaround, hosting a private mirror of the repo should be relatively straightforward.

Yes, that is as well, I was just surprised you aren't keeping old versions around. I didn't' want to create my own private mirror for dependencies at this stage, but clearly I have to

jforest on 2 Feb 2017

@Xylakant yeah, it took me five minutes to set up a mirror, that's how I know it's not that difficult to fix.

jjshoe on 2 Feb 2017

1) Create a new bucket in s3
2) Have CI fetch the deb file
3) Have CI use deb-s3 to upload it
4)...
5) Profit.

Settings up deb-s3 on CI is simple, install ruby, gem install deb-s3.
Done.

jjshoe on 2 Feb 2017

Hey! So I'm going to work on this soon-ish. Like, when I finish my queue of things. Probably Monday-ish. I'll have a look at the options people have suggested. If anyone has any other suggestions please post them.

nik9000 on 2 Feb 2017

I was just surprised you aren't keeping old versions around.

@jjshoe you're talking to the wrong person here - I'm just another user that's trying to be helpful.

Xylakant on 2 Feb 2017

you're talking to the wrong person here - I'm just another user that's trying to be helpful.

Sorry @Xylakant I don't know who the players are here :) I am glad to see this is getting some traction though, and yes, I will be creating my own local repository of our dependencies as well

jforest on 2 Feb 2017

I don't know who the players are here

@jforest The ones helpfully marked with "member" or "contributor", though I must admit that github could make that more obvious ;)

Xylakant on 2 Feb 2017

@Xylakant ah yes, I see that light gray tag now. Missed it before. Thanks!

jforest on 2 Feb 2017

yeah, it took me five minutes to set up a mirror, that's how I know it's not that difficult to fix.

We appreciate the suggestion but that is not how we are going to fix this. The production of the repository is built into the unified release process that we have for all products across the stack. To be clear, all products are affected by this right now. Fixing this involves changing our unified release process which requires a fair amount of testing.

jasontedor on 2 Feb 2017

Have you looked into https://www.aptly.info/ ? It has pretty comprehensive repository management, we use it for both mirroring external repos and for our own private packages. S3 is also supported, altho I haven't used that particular feature

XANi on 2 Feb 2017

aptly did the trick. I just did this without trouble:

sudo apt-get update
sudo apt-get install elasticsearch=5.1.1
sudo apt-get install elasticsearch

There is nothing to show in Elasticsearch so I'm going to close without reference to code. I switched the unified release mechanism to building the apt repository with aptly rather than reprepro.

nik9000 on 14 Feb 2017

👍5 ❤2 🎉2 😄2

Confirmed:

% sudo apt-cache policy elasticsearch
elasticsearch:
  Installed: (none)
  Candidate: 5.2.1
  Version table:
     5.2.1 500
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages
     5.2.0 500
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages
     5.1.2 500
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages
     5.1.1 500
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages
     5.0.2 500
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages
     5.0.1 500
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages
     5.0.0 500
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 Packages
        500 https://artifacts.elastic.co/packages/5.x/apt stable/main i386 Packages

jordansissel on 14 Feb 2017

This just happened again with the 6.4.0 release of filebeat.

tvisher@loader-x2:~$ apt-cache policy filebeat
filebeat:
  Installed: 6.1.1
  Candidate: 6.4.0
  Version table:
     6.4.0 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
 *** 6.1.1 0
        100 /var/lib/dpkg/status

Compared to a few days ago:

tvisher@loader-x2:~$ apt-cache policy filebeat
filebeat:
  Installed: 6.1.1
  Candidate: 6.3.2
  Version table:
     6.3.2 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.3.1 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.3.0 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.2.4 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.2.3 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.2.2 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.2.1 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.2.0 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.1.4 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.1.3 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.1.2 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
 *** 6.1.1 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
        100 /var/lib/dpkg/status
     6.1.0 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.0.1 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages
     6.0.0 0
        500 https://artifacts.elastic.co/packages/6.x/apt/ stable/main amd64 Packages

timvisher on 23 Aug 2018

Sorry for the issue @timvisher, we are on it.

jasontedor on 23 Aug 2018

@timvisher This is addressed now.

jasontedor on 23 Aug 2018

👍

timvisher on 27 Aug 2018

Hi, I am facing similar issue with logstash. I want to use 6.3.0 version of logstash. I am using yum install. When I use 'yum install logstash-6.3.0' it says package does not exist. Please help me out.