Sitespeed.io: Chrome mobile emulation visually complete flicker between 99 and 100%

[kkopachev]

Was not sure to what repo to post, but running latest sitespeed.io in docker container and VisuallyComplete metric on mobile emulation (Chrome). So VisuallyComplete happening at around 20 seconds for some reason, where Visuallycomplete85 is around 3-5 seconds or so. Then there is no visual changes happening on the page.

[soulgalore]

Hey @kkopachev are you really sure nothing is changing in the video? VisualMetrics also creates some small screenshots by default for each change, have you checked them too and can confirm?

Best
Peter

[kkopachev]

Checked screenshots and yes, there is a change - chrome shows "Loading blahblah.com" in left bottom corner. VisualMetrics seem to react on that and treat it as a page change. There is no visual changes to the rendered page itself.
These are last 3 screenshots it saved:

[soulgalore]

ok thanks, think we need to check then if we can fine tune the configuration.

[soulgalore]

I'll have a go with this when I'm back from vacation so it will take a week or so before I can start. If anyone else have the time, have a try in Browsertime (needs to be fixed there).

[soulgalore]

Hey @kkopachev now I'm back. Can you show me your config how you run? I tried to reproduce but for me the last part of the screen isn't picked up, but I think then it is different depending on viewport. I tried with default setting mobile to iPhone 6 (I did it in Browsertime so it's easier for me to debug):

docker run --privileged --shm-size=1g --rm -v "$(pwd)":/browsertime-results sitespeedio/browsertime --video --speedIndex -n 1 --chrome.mobileEmulation.deviceName 'iPhone 6' https://www.popsugar.com/ -c cable

Best
Peter

[kkopachev]

Hi @soulgalore I tried your command and it works fine, however, if I add viewport size to it (as sitespeed.io adds) then that "Loading..." thing taken into account calculating last visual change. Here is command I ran to repro:

docker run --privileged --shm-size=1g --rm -v "$(pwd)":/browsertime-results sitespeedio/browsertime --video --speedIndex -n 1 --chrome.mobileEmulation.deviceName 'iPhone 6' --viewPort '360x640' 'https://www.popsugar.com/' -c cable

[soulgalore]

Thanks @kkopachev I missed that, let me have a look asap!

[soulgalore]

Had a go at this today to check if there's something we could do just by configuring VisualMetrics but no look, I'll need to change where we set the offset for the video that adds some testing, let me continue with that.

[soulgalore]

Sorry for not getting back, I'll continue this weekend.

[soulgalore]

I actually tried it out and got a solution for it, but need to do a change in the VisualMetrics script. I'll do the change probably tomorrow but also wanna check if it can be changed upstream, that makes it easier in the future.

[soulgalore]

I made a fix in master for browsertime: https://github.com/sitespeedio/browsertime/commit/a4c0d76e539383b2f19c50673a92fc44afbe4487

It works for me now. Roll it out in the next release (hopefully this weekend).

[soulgalore]

Released this yesterday in sitespeed.io so I hope it will work now :)

[kkopachev]

Hi Peter!
Tried latest sitespeed.io image and it didn't solve the problem, but created other problems.
http://sitespeedio.sugarops.com/www.popsugar.com/2017-09-15-14-17-11/pages/www.popsugar.com/celebrity/Easy-Halloween-Costumes-Women-Color-42592271/query-3b2b8401/data/video/0.mp4

There are no ovelay "First/Last paint" timings on the video (didn't find in changelog if new flag was introduced?)
Visual progress still jumps 99% -> 95%->99%.
At the very end of video page all the sudden scaled up 2x and then back to normal 1x. That's when video ends.
Same issue in screenshots it generated. This was not happening with version 5.4.4.

[soulgalore]

Hi @kkopachev
thanks for getting back. The timer was a mistake by me when I pushed the new version of Browsertime, pushing a fix release now.

The size, I think something has changed in Chrome 61, but I haven't looked into it yet: https://github.com/sitespeedio/browsertime/issues/367 - I'll look later, else I'll rollback the to Chrome 60.

For Visual Progress, hmm it worked when I tried it Browsertime, but I'll try more this weekend, can set the diff to larger.

[soulgalore]

That upscaling, it doesn't happen when I try, it's strange.

I changed the margin again, now it is bigger and it will work for Linux but run on Mac it will still fail, I think the 2x DPI density is the problem. Is it easy for you to test from master (if you run on Linux but I guess you are)?

Best
Peter

[soulgalore]

When I looked closer I could reproduce, I'll rollback to 60 and if that works, I'll report a bug in Chrome.

[kkopachev]

5.6.2 fixed the main issue. Inappropriate scaling at the end of video gone as well. Timings are back.

[soulgalore]

Ok good. I'll increased the the diff now to 25 (that is the same as WebPageTest, it's different values in VisualMetrics and WebPageTest. For the scaling, I'll run a test on CHrome 62 in master for a while and hope that it is gone (there have been multiple problems with the emulated mobile in the past), if that still fails, I'll create a bug in Chrome.

[soulgalore]

No couldn't upgrade to 62 because of https://bugs.chromium.org/p/chromium/issues/detail?id=766674

[soulgalore]

I haven't seen this for our tests at work so I guess this has been fixed in Chrome/Chromdriver some time ago. Please re-open if there still is an issue.