Robottelo: Error handling for negative tests in hammer

[sghai]

This is the issue specifically for negative scenarios. Currently, we are checking the returncode however, we should actually check the expected error message too. This is required because in hammer if we pass incorrect_id of an entity as an option then some cmd fails like:

]# hammer  -u admin -p changeme hostgroup create --name text --subnet-id 12345
Could not create the hostgroup:
  ERROR:  insert or update on table "hostgroups" violates foreign key constraint "hostgroups_subnet_id_fk"
  DETAIL:  Key (subnet_id)=(12345) is not present in table "subnets".

Since we are just validating returncode which is not sufficient like below where we are just checking returncode, because never know if cmd fails with expected error message or some db error

def test_negative_delete_by_id(self):
        """Create HostGroup then delete it by wrong ID

        @id: 047c9f1a-4dd6-4fdc-b7ed-37cc725c68d3

        @assert: HostGroup is not deleted
        """
        for entity_id in invalid_id_list():
            with self.subTest(entity_id):
                with self.assertRaises(CLIReturnCodeError):
                    HostGroup.delete({'id': entity_id})

This can be done like below:

        subnet_id = gen_string('numeric', 4)
        with self.assertRaises(CLIReturnCodeError) as exception:
            HostGroup.create({
                'name': gen_string('alpha'),
                'subnet-id': subnet_id
            })
        self.assertIs(
            exception.exception.stderr,
            'Could not find subnet {0}'.format(subnet_id)

[renzon]

Once it impacts almost all files from cli, I am gonna create issue for each file so PR are smaller and easier to review

[rplevka]

@renzon - just keep this issue as a tracker, create a list of files with checkboxes so we can keep track of those you've already addressed. e.g.:

• [ ] test_organization.py
• [ ] test_user.py

[elyezer]

One thing to have in mind is if we start to do localization testing this will all break since the messages are going to be translated. If localization tests is something to have in the future then this should be considered before starting working on this issue.

[pondrejk]

Adding a tracking checklist (for lack of a better place):

• [x] test_abrt.py -- no negative cases
• [x] test_activationkey.py -- #3716
• [x] test_architecture.py -- #3747
• [x] test_bootstrap_script.py -- all negative stubbed
• [x] test_capsule.py -- #3768
• [x] test_capsule_installer.py -- all negative stubbed
• [ ] test_classparameters.py #3768
• [ ] test_computeresource.py
• [ ] test_contenthost.py
• [ ] test_contentview.py
• [ ] test_contentviewfilter.py
• [x] test_discoveredhost.py -- no negative cases
• [x] test_docker.py -- no negative cases
• [ ] test_domain.py
• [ ] test_environment.py
• [x] test_errata.py -- all negative stubbed
• [ ] test_fact.py
• [x] test_filter.py -- no negative cases
• [x] test_globalparam.py -- no negative cases
• [ ] test_gpgkey.py
• [x] test_hammer.py -- no negative cases
• [ ] test_host.py
• [ ] test_host_collection.py
• [ ] test_hostgroup.py
• [ ] test_import.py
• [x] test_installer.py -- no negative cases (all stubbed)
• [x] test_lifecycleenvironment.py -- no negative cases
• [ ] test_location.py
• [x] test_medium.py -- no negative cases
• [ ] test_model.py
• [ ] test_myaccount.py
• [ ] test_operatingsystem.py
• [ ] test_organization.py
• [x] test_partitiontable.py -- no negative cases
• [x] test_ping.py -- no negative cases
• [ ] test_product.py
• [x] test_puppetmodule.py -- no negative cases
• [x] test_report.py -- no negative cases
• [ ] test_repository.py
• [x] test_repository_set.py -- no negative cases
• [x] test_role.py -- no negative cases
• [x] test_satellitesync.py -- all negative stubbed
• [x] test_sso.py -- no negative cases
• [x] test_subnet.py -- #3752
• [ ] test_subscription.py -- the only negative test is blocked by BZ#1226425
• [x] test_syncplan.py -- #3752
• [x] test_template.py -- no negative cases
• [x] test_user.py -- #3745
• [ ] test_usergroup.py -- #3743
• [x] test_variables.py -- all stubbed

[rochacbruno]

Lets consider the #3749 before continue with this.

[rochacbruno]

What is the current status of this @renzon @pondrejk ?

[pondrejk]

Well, at some point in past we decided on a meeting to halt this activity, so I'm not sure if we want to proceed or put this to backlog. Maybe a question for a meeting.

[renzon]

@pondrejk +1

[rochacbruno]

@pondrejk @renzon

This issue has been implemented on CLI side, but we need to file a new issue on nailgun to address API, this issue will be opened until we move the list of files to the new issue.

@svtkachenko

[ntkathole]

Bugzilla closed as Wontfix. Please feel free to re-open, if this still needs fixing.