Creating a circle body(any size) and putting it on a rectangular body(big enough for circle to roll around on), then slowly increasing the angular velocity to cause the ball to start rolling(I used 0.015 per frame/update) will cause the ball to start bouncing up and down as if it was rolling across a bumpy surface.
I tried different settings for gravity, friction, restitution, etc, but it always results in the same issue. It does not seem to be caused by 'bounciness', it's more like as the ball rotates against the flat rectangle, the engine thinks that it is hitting an edge/bump causing the ball to kind of ramp off of it(perhaps the ball sometimes falls slightly through the floor, then on the next update it bounces off the floor and ramps up a little bit).
Not sure if this counts as a bug or just a side-effect of the engine approximating physics, but I couldn't find anywhere else to report such a problem.
Rather setting the ball's angular velocity, have you tried applying a force to it, with applyForce?
Combining force and torque made the motion look fairly realistic in my bike sim.
It's hard to say without seeing the code on how you're applying this, but as other commenters have mentioned already, I'd also advise against controlling velocity and go with a force / torque approach instead in this case.
Most helpful comment
Rather setting the ball's angular velocity, have you tried applying a force to it, with
applyForce?