Galaxy: Latest version not updated after Travis build notifies Galaxy

[elliotweiser]

Bug Report

SUMMARY

There's a possible issue related to importing Ansible roles (and creating new versions/releases) through travis-ci.org using the Notifications API

STEPS TO REPRODUCE

I'm not entirely certain about how to reproduce this. I suppose you could try to create a new tag on an existing role and see if this imports correctly when notifying Galaxy from Travis CI. In any case, here's the relevant information in this case.

This is the notification received by Ansible Galaxy after this Travis build passed.

EXPECTED RESULTS

The notification should have triggered an import, which would have created the 2.4.3 release.

ACTUAL RESULTS

Nothing happened. 2.4.2 remains the latest release.

See versions in https://galaxy.ansible.com/geerlingguy/homebrew

[chouseknecht]

Looks like it started the import, but there was an error. Here's a link to the import log:

https://galaxy.ansible.com/my-imports?namespace=geerlingguy&repository_name=homebrew&selected=257923&page_size=10

Not sure why it got a duplicate key error though. Investigating...

[chouseknecht]

Here's the traceback:

2018-07-01 00:59:57,226 ERROR tasks: duplicate key value violates unique constraint "main_content_namespace_id_repository__d7de6a83_uniq"
--
  | DETAIL:  Key (namespace_id, repository_id, name, content_type_id)=(1609, 34981, docker, 11) already exists.
  | Traceback (most recent call last):
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/galaxy/worker/tasks.py", line 59, in import_repository
  | _import_repository(import_task, logger)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/utils/decorators.py", line 185, in inner
  | return func(*args, **kwargs)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/galaxy/worker/tasks.py", line 128, in _import_repository
  | content_obj = importer.do_import()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/galaxy/worker/importers/base.py", line 38, in do_import
  | content = self.make_content()
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/galaxy/worker/importers/base.py", line 75, in make_content
  | 'is_valid': False,
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
  | return getattr(self.get_queryset(), name)(*args, **kwargs)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 466, in get_or_create
  | return self._create_object_from_params(lookup, params)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 506, in _create_object_from_params
  | six.reraise(*exc_info)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 498, in _create_object_from_params
  | obj = self.create(**params)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 394, in create
  | obj.save(force_insert=True, using=self.db)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
  | force_update=force_update, update_fields=update_fields)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base
  | updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table
  | result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert
  | using=using, raw=raw)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
  | return getattr(self.get_queryset(), name)(*args, **kwargs)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert
  | return query.get_compiler(using=using).execute_sql(return_id)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql
  | cursor.execute(sql, params)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
  | return self.cursor.execute(sql, params)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
  | six.reraise(dj_exc_type, dj_exc_value, traceback)
  | File "/var/lib/galaxy/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
  | return self.cursor.execute(sql, params)
  | IntegrityError: duplicate key value violates unique constraint "main_content_namespace_id_repository__d7de6a83_uniq"
  | DETAIL:  Key (namespace_id, repository_id, name, content_type_id)=(1609, 34981, docker, 11) already exists.

[chouseknecht]

Somehow I think we're creating new content records, and the new content record of course has '-' converted to '_'.

The unique key on Content is: 'namespace', 'repository', 'name', 'content_type'

During import, in galaxy/worker/importers/base, the get_or_create() is:

obj, is_created = models.Content.objects.get_or_create(
            namespace=ns,
            repository=repo,
            content_type=models.ContentType.get(self.data.content_type),
            original_name=original_name,
            defaults={
                'name': self.translate_content_name(name),
                'is_valid': False,
            }

Was able to reproduce this in local development.

In the case of geerlingguy.homebrew, a content row already exists where the name is 'homebrew', and the original_name is 'homebrew'. During import the original_name being searched for is 'ansible-role-homebrew'.

I think in the above, we need to put name in the search, and move original_name to 'defaults'.

[elliotweiser]

@chouseknecht Thanks for the quick response!

[chouseknecht]

Fixed deployed to https://galaxy-qa.ansible.com - our public QA site. An import in QA should work now, and it should not change '-' to '_'.

[chouseknecht]

The patch will be deployed to production tomorrow morning, btw.

[lae]

@chouseknecht will this be a retroactive fix as well (i.e. will travis_lxc get renamed back to travis-lxc?)?

[cutwater]

@lae travis_lxc is fixed.

[lae]

Thanks!

[chouseknecht]

Closing this issue, as the duplicate key error is fixed, and Travis builds are working. We're still working on NOT renaming existing repo names. See issue #775

[Voronenko]

@chouseknecht What linter is used for roles during the import ? How can I check role locally before releasing ?

Why galaxy UI does not provide easy way for authors to spot roles with problems in import (especially one via travis hook) ?