Robottelo: Many UI tests fails to save screenshot when using saucelabs

Created on 19 Jul 2017 · 17Comments · Source: SatelliteQE/robottelo

many UI tests failed in tearDown when saving screenshot with the same error:

robottelo/test.py:622: in take_screenshot
    self.browser.save_screenshot(path)
../../shiningpanda/jobs/375dbdea/virtualenvs/d41d8cd9/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:758: in get_screenshot_as_file
    png = self.get_screenshot_as_png()
../../shiningpanda/jobs/375dbdea/virtualenvs/d41d8cd9/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:777: in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
../../shiningpanda/jobs/375dbdea/virtualenvs/d41d8cd9/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:787: in get_screenshot_as_base64
    return self.execute(Command.SCREENSHOT)['value']
robottelo/ui/browser.py:44: in execute
    params)
../../shiningpanda/jobs/375dbdea/virtualenvs/d41d8cd9/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py:201: in execute
    self.error_handler.check_response(response)
../../shiningpanda/jobs/375dbdea/virtualenvs/d41d8cd9/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py:102: in check_response
    value = json.loads(value_json)
/usr/lib64/python2.7/json/__init__.py:339: in loads
    return _default_decoder.decode(s)
/usr/lib64/python2.7/json/decoder.py:364: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <json.decoder.JSONDecoder object at 0x7f71580bb590>
s = "The test with session id b08625539bb64e46af50a676d3ad4345 has already finished, and can't receive further commands.
Y.../b08625539bb64e46af50a676d3ad4345
For help, please check https://wiki.saucelabs.com/display/DOCS/Common+Error+Messages"
idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` or ``unicode``
            beginning with a JSON document) and return a 2-tuple of the Python
            representation and the index in ``s`` where the document ended.

            This can be used to decode a JSON document from a string that may
            have extraneous data at the end.

            """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration:
>           raise ValueError("No JSON object could be decoded")
E           ValueError: No JSON object could be decoded

/usr/lib64/python2.7/json/decoder.py:382: ValueError

6.2 6.3 Infrastructure TestFailure

Source

ldjebran

👍3

Most helpful comment

So the "easiest" solution with least amount of framework changes as for me is to move browser start and closure to with Session context manager, as inside this CM is the only place where we actually work with UI and no timeout can happen there, plus context manager allows us to define steps for enter/exit in a similar way to setup/teardown.
It's a pretty serious core change anyway, so let's discuss it on tomorrow's automation meeting.

abalakh on 20 Jul 2017

👍3

All 17 comments

The problem:
By default saucelabs has a session timeout 90 seconds if it receive no commands it close the session
https://wiki.saucelabs.com/display/DOCS/Test+Didn%27t+See+a+New+Command+for+90+Seconds
worked with one of the test on sauselabs and effectively was able to reproduce the error and at saucelabs test task we can see in test meta data:
failed_test
with https://wiki.saucelabs.com/display/DOCS/Test+Configuration+Options
in webdriver_desired_capabilities put idleTimeout=300
so my webdriver_desired_capabilities became:

webdriver_desired_capabilities=platform=macOS 10.12,version=45.0,idleTimeout=300,seleniumVersion=2.48.0,build=build-1234,screenResolution=1600x1200,tunnelIdentifier=activekey

the test in question passed without problem
tests

the problem was in idleTimeout we have to put it in webdriver_desired_capabilities, but that also will not save us from the problem as this value has a max of 1000 seconds
and we have many UI tests that can take more than that to initialize without sending any UI cmd

ldjebran on 19 Jul 2017

👍1

@kbidarkar where we can update the timeout property?

rochacbruno on 19 Jul 2017

and we have many UI tests that can take more than that to initialize without sending any UI cmd

Time to refactor our tests not to initialize session before non-ui prerequisite tasks are being executed/completed - if we have such

rplevka on 19 Jul 2017

👍1

@rochacbruno in robottelo.properties webdriver_desired_capabilities
mine modified to be (notice idleTimeout=300):

webdriver_desired_capabilities=platform=macOS 10.12,version=45.0,idleTimeout=300,seleniumVersion=2.48.0,build=build-1234,screenResolution=1600x1200,tunnelIdentifier=activekey

ldjebran on 19 Jul 2017

@ldjebran yeah this works for local run, but I ask on how to update for CI/Jenkins

rochacbruno on 19 Jul 2017

👍1

For CI/Jenkins it is the jenkins-config repo on gitlab. The file robottelo.properties.
I am raising a PR shortly.

kbidarkar on 19 Jul 2017

Ok, just checked, ignore my previous comment, it is the robottelo-ci project where we need to update it, my bad.

kbidarkar on 19 Jul 2017

This PR should fix this issue for the jenkins jobs, https://github.com/SatelliteQE/robottelo-ci/pull/723

kbidarkar on 19 Jul 2017

👍1

@ldjebran : Excellent Analysis. 💯

Now we need to deal w/ the tests those takes more than 1000secs as max-limit is 1000sec

sghai on 20 Jul 2017

👍1

@sghai mentioned PR increased timeout to 300secs, not 1000 (i'm not sure why).

Time to refactor our tests not to initialize session before non-ui prerequisite tasks are being executed/completed - if we have such

@rplevka yeah, but it may appear not that easy to refactor - the thing is browser is started at the very beginning of test execution, before even executing steps from setUpClass and setUp. So the change is not just about placing Session to the right place, but probably to start browser only when Session is triggered. And it's key change which may affect badly every single UI test, we have to be very careful and re-test everything :)

abalakh on 20 Jul 2017