Robottelo: SyncPlan test timeout on waiting for the sync task to get created

[rplevka]

This is happening spontaneously, most probably because of the server load.

the last time it caused these two to fail:
test_positive_synchronize_rh_product_future_sync_date
test_positive_synchronize_rh_product_past_sync_date

• i'll try to dig out the whole tier4 logs to see, whether i can find the list of the tasks.

[san7ket]

On upgrade box, i see these failures,

tests.foreman.cli.test_syncplan.SyncPlanTestCase.test_positive_synchronize_custom_product_daily_recurrence tests.foreman.cli.test_syncplan.SyncPlanTestCase.test_positive_synchronize_custom_product_past_sync_date tests.foreman.api.test_syncplan.SyncPlanSynchronizeTestCase.test_positive_synchronize_rh_product_future_sync_date

Error:
AssertionError: No task was found using query 'resource_type = Katello::Repository and owner.login = foreman_admin and resource_id = 636'

When I fire up the same query now, I can see the task was created and completed successfully.

Like @rplevka said , think this is due to the server load, test is quering , before the actual task is being created.

[san7ket]

Failure count increased for the issue on Upgrade box , should we run these tests in one thread , would that be beneficial ? @rplevka @renzon thoughts?
11 test from UI/CLI/API for 6.2

[renzon]

The ideal scenario would be check for the tasks and waiting them to finish. Otherwise if we keep pushing test cases to one thread our runtime will increase more and more. Since we are already in 35 hours, this is an issue. @rplevka and @lpramuk have been working in a solution to run tests against multiple satellites. In a ideal world we would have 1 host/thread so all our test would "run in one thread". So I think @rplevka is the best person to tell us a good solution ;)

[rplevka]

@renzon @san7ket - running single-threaded is not a good solution here.
@renzon - the tricky bit here is, we timeout on task to appear.

while changing the infra would definitely solve many related issues, we should probably think of a much more realistic solution for now.
There has been already a lot done to test sync plans reliably (e.g. polling for the specific sync task than checking the package counts only). However, there is this gap between reaching the sync date/time and trigerring the sync task. especially on the overpopulated satellite with a lot of things going on.
I'm currently trying to get some feedback from devs on how the scheduling works and how could it be enhanced (perhaps scheduling the very next sync task immediately with the start_date in the future) instead of creating it when the time comes)

[san7ket]

@rplevka what would suggest, how should we go about improving this, It creates lots of failures for upgrade.
@renzon Even when I run standalone automation for single test on a populated satellite, I get no consistent results. :(

[rplevka]

@san7ket, not sure if we can do better than increasing the retries count at this time.
We might also file a bug for this.

@san7ket if you hit this again on a standalone automation - could you ping me? i'd like to look around the environment.

[rplevka]

@san7ket, ok, i think i know a way of how to deal with this (or at least proceed further in finding out whether it's worth waiting or not).

the way sync plan works is that foreman actually creates a synchronization plan in pulp (backend).
Now the pulp task is created silently, without foreman beaing notified. Only after the (pulp) task finishes syncing and starts to publish, it notifies foreman, which creates a (foreman) task.

• there's a possibility to attach to pulp http api and poll for its own sync task - all we need is to know the backend id of the repository we sync.
  this one is being provided by katello api:

GET /katello/api/repositories/112
...
Response: {
    "name" => "Red Hat Satellite Tools 6.2 for RHEL 7 Server RPMs x86_64",
    ...
    "backend_identifier" => "a675f396-d6be-4ad8-9dd0-f68d2844cad5",
    ...
}

now we want to poll pulp tasks by providing this and the task type (pulp.server.managers.repo.sync.sync) as filters:

$ curl -sku admin:sd3LLGEvVzngRa4sUK9cbemWHG32VsnK -d '
{"criteria": {
    "filters": {
        "tags": {
            "$in": ["pulp:repository:a675f396-d6be-4ad8-9dd0-f68d2844cad5"]
        },
        "task_type": {
           "$in": ["pulp.server.managers.repo.sync.sync"]
        },
    }
}}' https://ibm-x3690x5-01.rhts.eng.bos.redhat.com/pulp/api/v2/tasks/search/

et voila:

[
  {
    "exception": null,
    "task_type": "pulp.server.managers.repo.sync.sync",
    "_href": "/pulp/api/v2/tasks/a238bf52-6d6e-4640-a4f7-01b24e72b74f/",
    "task_id": "a238bf52-6d6e-4640-a4f7-01b24e72b74f",
    "tags": [
      "pulp:repository:a675f396-d6be-4ad8-9dd0-f68d2844cad5",
      "pulp:action:sync"
    ],
    "finish_time": "2018-05-31T12:33:31Z",
    "_ns": "task_status",
    "start_time": "2018-05-31T12:33:11Z",
    ...
    "state": "finished",
    ...
}

..this way we can get more detailed insight whether pulp side executed the task and attach to the progress of it. after it is finished, we can carry on by polling the foreman task and then finally assert the synced files.

p.s. password to the pulp api can be obtained by:
ssh.command('# grep "^default_password" /etc/pulp/server.conf | awk \'{print $2}\')

[san7ket]

I am implementing the above solution, the problem with the test is ,

If the delay between creation of Sync Plan and adding Products should not be higher that the Start Time of the Sync Plan. This delay increases when the box is under load and can be problem on slower environments.
We can tackle this by updating the Sync plan Start Time just before polling the tasks and also the above solution.
Will update all relevant tests and raise a PR shortly

[san7ket]

PR merged, closing this.