There seems to have been a recent regression in the way rounded borders are rendered. Here is a test case, with screenshots below.
Without WebRender:

With WebRender:

This seems to have changed within the last month, as the circle can be seen rendering properly in this issue created at the end of March.
I couldn't reproduce it to look like your screenshot, but I filed a bug based on your testcase: https://bugzilla.mozilla.org/show_bug.cgi?id=1461513
Edit: I think this test case is just a basic (and easier to reproduce) variant of #2590.
@Darkspirit This could be the case, as the flickering boxes (which now appear if I make the window small enough to scroll) seem to align with some of the missing parts of the border.
That said, something must still have changed as I used to see the dots in the circle on https://rawgit.com/Zirro/03f0e2d19b087db29b1f069e014e3ee5/raw/0f32535d70e56185e01e4f9be860afc4dc4fec79/fireworks.html when I created that issue, and for some time afterwards.
I will have another look at this test case when #2590 has been resolved, to see if they are indeed the same issue or two separate ones.
Hrm. I'm having trouble reproducing this one as well. I do see artifacts in https://rawgit.com/Zirro/03f0e2d19b087db29b1f069e014e3ee5/raw/0f32535d70e56185e01e4f9be860afc4dc4fec79/fireworks.html, though they don't seem to be related to to dot or dash rendering.
Okay. I do see artifacts on this test case when resizing the window. These artifacts seem to affect the borders and the backgrounds equally though -- which are two different code paths.
If it can help to narrow down the specific rendering issues I'm seeing by looking at certain hardware/drivers, I'm on a MacBook Pro (Early 2013) with Intel HD Graphics 4000 and a NVIDIA GeForce GT 650M with High Sierra.
Works fine on my end (Linux/Intel) with or without resizing.
Seeing the same issue on MacBookPro12,1 (Retina, 13-inch, Early 2015) w/ Intel Iris Graphics 6100.

This appears to have been resolved - quite likely through the merging of https://github.com/servo/webrender/pull/2795. This is what I see now:

The border placement could use some improvements to match the non-WebRender version, but I think this issue can be closed at this point.
Latest WebRender has not been merged into Servo yet, but is available in Firefox Nightly.
@MartinRogalla Could you verify whether this is fixed in latest Nightly for you?
Better for me too. The dashed border needs some love and the bottom-right dot of the top-left box looks bitten off.
Looks a bit better, but I see some strange behaviour on Nightly. The elements change if I click on them. I'm not sure if this is related. Please see the attached GIF:

Please let me know if you want me to provide any additional information.
That is strange behaviour indeed. For what it's worth, I can't seem to reproduce it on my machine.
It seems like the original bug has been fixed. We should probably close this bug and open a new one with the behavior mentioned above by @MartinRogalla - I'm also unable to reproduce that issue on any of my local test machines - it may be a driver specific issue.
Most helpful comment
Looks a bit better, but I see some strange behaviour on Nightly. The elements change if I click on them. I'm not sure if this is related. Please see the attached GIF:

Please let me know if you want me to provide any additional information.