Ansible: Add 'update' option to ansible-galaxy

[geerlingguy]

Issue Type:

Feature Idea

Ansible Version:

ansible 1.5

Environment:

N/A

Summary:

As mentioned on the mailing list, the ansible-galaxy command has install and remove options, but no update option.

The solution is currently to use ansible-galaxy install [role],[version] --force, but that feels a little strange, and isn't necessarily obvious.

Could we add an update command that works like ansible-galaxy update [role],[version] (where ,[version] is optional, and specifies a particular version to download), and simply updates to the latest version of the module available—or, if it's already the latest, no update is performed?

Steps To Reproduce:

N/A

Expected Results:

N/A

Actual Results:

N/A

[geerlingguy]

Thank you, ansibot :)

[ramondelafuente]

:+1:

Also, I think it would be nice if ansbile-galaxy install didn't download the zipfile before finding out that the folder already exists.

If you add a role to your dependency list you can update -i (ignore errors), but all roles are downloaded.

[nickjj]

Sounds good but I think update would also need an optional version. I have some tooling that generates a rails application along with a play book/inventory to go with it.

I would like to eventually have it update the roles automatically but I wouldn't want to always goto the latest version. If I goto the latest version then I'm forced to update my tool whenever I release a new role or someone releases a breaking change in a role they release.

[steenzout]

@nickjj why not use a requirements.yml and specify your dependencies with version there?
you'll need to tag your role's repo.

- src: git+ssh://git@host:port/repo/myrole.git
  version: v1.0.0-1  # git tag that you want to download
  name: myrole
  scm: git

$ ansible-galaxy install -r requirements.yml

to update you may need to create a script to remove the roles since I didn't manage to do a force install using the requirements file.

#!/bin/bash
# update-roles.sh
grep 'name:' requirements.yml | awk -F: '{print $2}' | xargs -I {} ansible-galaxy remove {}
ansible-galaxy install -r requirements.yml

[damko]

I totally support this feature request, even without versioning at first.

[mchelen]

Update would be very helpful. When using a role with many dependencies they all have to be install --force manually.

[jloh]

+1

[otakup0pe]

:+1:

[geerlingguy]

@nickjj - I've updated the original request to add a note about optionally specifying a [version] after the role name.

[ricardclau]

:+1:

[arturmartins]

+1

[kozejonaz]

+1

[nickjj]

@geerlingguy Sounds good.

[radamanth]

:+1:
Would be nice.

I use install --force bur it systematicly deleted my vars directory even if it's not in the galax role.
It would be nice if we could keep non galaxy file in the path (cleaning only galaxy role file)

[joshukraine]

+1

[radamanth]

Has it happens I found a way of doing something quite close
I install my ansible roles in my roles directory.

Then I create a role that depends on this one, where I put my vars file.
In my playbook I only use the role that depends on the ansible-galaxy one .

to be more specific, let's say I have a galaxy role 'bob/galaxy-mysql'
My requirements.yml would look like this :

- src: bob/galaxy-mysql
  version: v1.0.0
  name: myrole
  path: ./roles

Then I run :

$ ansible-galaxy install -r requirements.yml

Then I create in ./roles/mysql a role that has a dependency on my local galaxy role

roles/mysql/meta/main.yml:

---
dependencies:
  - { role: bob.galaxy-mysql, sudo: yes}

and in roles/mysql/vars/main.yml:

---
bob_mysql_var: "MySpecificValue"

Then in my playbook I never user directly bob.galaxy-mysql but my 'mysql' role

Hope this would help you

[xbeta]

+1

[peterloron]

+1

[miloskroulik]

That's very needed.

[thiagotalma]

:+1:

It would be great if it worked as the Composer.
We changed the requirements.yml file and install/update all roles without using --force or --ignore-errors

[Kuchitama]

:+1:

[mgoodness]

:+1:

[ssbarnea]

+1

[thom8]

:+1:

[rasebo]

+1

[BrianGilbert]

:+1:

[lhoss]

+1
but IMO even more important to fix https://github.com/ansible/ansible/issues/11266

[lukosan]

+1

[JorisVanEijden]

:+1:

[ghost]

+1

[haad]

+1 I see one problem and that's if ansible-galaxy should detect any changes done manually in a role or not.

[craneworks]

+1

[ksingh7]

:+1: +1

[nichenke]

:+1: +1

[dinoamino]

:+1: +2 .. that's a plus for each year this request has been open :)

[krisdante]

I know it is scary, but it works

<?php
$items=[];
exec('ansible-galaxy list', $items);

foreach ($items as $package) {
  preg_match("/^- (.*?),/", $package, $found);
  $package = $found[1];
  system("ansible-galaxy remove $package");
  system("ansible-galaxy install $package");
}

[gniltaws]

+1

[blade2005]

+1 how has this not been implemented yet?

[Wtower]

+1

[skornehl]

+1

[nhooey]

Please stop adding useless +1 messages and notifying everyone. Use GitHub's "reaction" feature instead.
@Wtower @skornehl @gniltaws

[xenithorb]

I don't think anyone believes that anyone is paying attention to the "thumbs", hence +1.

This is necessary feature. Subsequently, there should be an option to include the .git repo as well.

The problem that I'm running across are on @geerlingguy's aging playbooks (no offense, really, this is just the reality) that I need to modify in order to use - I have the original ansible-galaxy role adjacent to a forked's copy of of the git repo that I rsync'd content into. Seeing as how _I could not update the role_ I'm having to spend time sorting out what was updated by me, and what was updated in the repo I assumed was at the same version as the thing I downloaded from ansible-galaxy. This could cause double commits by me, and will generally make it much much more difficult for me to contribute back pull requests.

Going forward _I will NOT use ansible-galaxy during my playbook development_, but instead clone the repos myself in order to mitigate that problem.

[tima]

We care @xenithorb. It's hard to tell whats going on here because it's been littered by those +1's and discussion of additional ideas that weren't part of the PR over an extended period of time. It's not clear if many of the +1 party are aware of some features that have been available in galaxy for some time. I get the feeling many are not. If you are it's even more unclear what is being proposed. (This is why I'm leaning toward having this thread closed not to end conversation, but to reset and focus it.)

I'm not sure why the core ansible team did not act or comment on the original PR early on. I can speculate a bit, but I really don't know.

An "upgrade" feature of sorts has already been implemented through the --force flag as @geerlingguy mentions. It may not be the clearest labelling of the desired function, but it will go out and grab the latest version and replace anything you have locally. Labelling that command line option "upgrade" is just as unclear since galaxy is backed by git/github and there is no contract or requirement to version releases. The tool cannot truly guarantee what you'll get is an upgrade the way its been designed. (Is commit 01d4e before or after ef40ae? We don't know.) Also --force can be used to downgrade a role. This is partially why I'd be opposed to an "upgrade" subcommand.

As for using your own patched/forked version of @geerlingguy's role -- use a requirements.yml as was proposed by steenzout 18 months ago here: https://github.com/ansible/ansible/issues/6466#issuecomment-65454871. With the spec you can have in a roles file, you can go directly at a git repo instead of thru the public galaxy repo. (This is how enterprise customers developing private roles for their in-house proprietary systems use galaxy today.)

Really if you are working with more than one role it's good idea to have a requirements.yml file anyways. As a best practice, you should peg the version you are using to specify a specific tag or commit to avoid getting the rug pulled out from under you unexpectedly. (I learned this the hard way a few times -- use a role from Galaxy, author decided to do some big refactor, breaks backwards compatibility and breaks my production code.)

I'm lazy and hate typing long command line strings repetitively with all the version/source etc spec of a role anyways. So I put it in that roles file and use the -r switch. Bonus is what roles and versions I'm using in my project can be tracked in my project's version control.

A roles file is more in the sprit of Ansible (I define the state I want my externally sourced roles to be in and galaxy gets it that way even if that means changing nothing) than manual per-role management that the PR proposes.

Hope that helps.

[geerlingguy]

See related: https://github.com/ansible/galaxy-issues/issues/168

Currently one problem is that Galaxy can't see 'versions' of roles using something like semver—I think Galaxy's CLI only interacts with version as the idea of a git hash/tag/branch tip (unlike package managers like Composer, NPM, pip, etc. which have the idea of "2.5.1 is higher than 2.5.0".

Besides that, I wholeheartedly +1 @tima's comment above; I switched to explicitly defining role versions in a requirements.yml file a few months ago for Drupal VM and things are SO much more stable/maintainable, even with the galaxy upgrade quirks.

Also, of note, I'll be giving a presentation on the best practices/tips I've gleaned over the past few years at AnsibleFest SF next week, so if you're there, come see me! I think Ansible will post slides (and maybe video too?) soon after. One of the main topics is role use in projects via forks, private repos, requirements.yml, etc.

[xenithorb]

Thanks for the tips! I'll be keeping an eye out for a video of your talk, since I regrettably won't be making the conference. Definitely will take a look at requirements.xml and switching over.

[tima]

Currently one problem is that Galaxy can't see 'versions' of roles...

Right. This is what I meant when I said...

there is no contract or requirement to version releases. The tool cannot truly guarantee what you'll get is an upgrade the way its been designed.

The entire Galaxy backend would need to rewritten (meaning git would need to be replaced) to force a lot of policy, workflow and overhead that a package management system has. (BTW Galaxy is not a package manager.)

I believe I am the MC for your track @geerlingguy and present our essential best practices 2 slots after you. Talks won't be recorded. Nothing from stopping you from doing it on your own. See you in SF.

[ssbarnea]

Unless Ansible (RedHat) starts to spend money managing a curated set of playbooks the community is doomed to deal with galaxy-mess. Galaxy is now nothing more than the kitchensink of roles shared by individuals that had good intentions. Not to say that the sink smells ok when the vegs inside are fresh, but they soon start to rot and nobody is getting things out of this sink.

Still, nobody should expect to have to maintain ansible-roles just because he shared them at some point. That's why one-man open-source projects are most of time failures, as they have a single-point-of-failure which sooner or later will be triggered.

The only way to make Ansible great again! ;) is to assure that role are maintained but a community of ansible power-users (users that do have more experience with ansible than the average user).

[geerlingguy]

@ssbarnea - While we're going a bit off course in terms of this particular thread, I think you pretty much sum up The Cathedral and the Bazaar, though it seems you may not take kindly to the Bazaar side of things :)

Open Source _is_ messy, whether you have one or a thousand developers behind a particular project, but 'cathedral-style' curated sets of roles or proprietary/official/centralized maintained software (akin to the examples in Ansible core) come with their own set of gotchas too... in most OSS communities, there's a constant tug of war between the two ideas.

I, for one, maintain around 100 different small OSS projects (a few with 500+ stars on GitHub), and I welcome PRs, though because of the SPOF you mention, I also encourage people _strongly_ to fork and reuse. I am proud of the fact that many of my most popular projects have almost as many _forks_ as _stars_, meaning people are modifying it to suit their needs exactly. That's open source at work!

The Bazaar-style espoused by Galaxy isn't perfect, but it does mean that there is a role for everything under the sun, and usually a few different ones. Now, _finding the best one_ is another topic, and should be explored outside this issue as well. There's room for improvement in the Galaxy UI and CLI, for sure.

[xenithorb]

While I agree with you (and am also trying to resist dirtying up this bug, but can't) - I think RH, just in good business sense and as a show of professionalism, should pay someone to .... well, basically maintain _your_ roles. (Since you basically have that cornered pretty well). If not paying literally you, they should have a RH staff member maintain them as a full/part time job. In essence, there should be a well-maintained set of playbooks for the top 100 open source projects.

I feel this way because getting started with ansible is difficult (The real kind of ansible, not the one-off ad-hoc commands and simple playbooks they teach in the documentation), and the first impression matters.

So, when I go pull in a heavily starred and clearly popular role to install a tool that I would assume is popular enough to get the attention it deserves - and it doesn't work on the first run - I question whether or not I'm even using the right tool for the right job, or whether I'm doing it for my own motivations because I like the agentless approach and the lack of ruby. It's really that simple. Good working, approachable roles will skyrocket the project and make it more valuable, I think. It's a PR move that makes business sense at the very least.

Edit: also, since I'm talking here anyway and I don't know where else to put it: @geerlingguy would you mind recording your talk and putting it on your blog like you did the other one?

[lowks]

:+1:

[bcoca]

https://github.com/ansible/proposals/issues/23

[webnull]

I'm voting up.

[jimi-c]

As this has been moved to a proposal, we'll go ahead and close this issue now.

Thanks!