Is there a way to toggle whether selection/dragging is enabled?
Not globally, no. You can disable dragging on each object individually via properties like lockMovementX, lockMovementY (or other actions like scaling or rotating via other properties). Take a look at https://github.com/kangax/fabric.js/wiki/Preventing-object-modification for more details.
There's no way to globally disable selection, but it should be easy to extend fabric to allow for that :)
Closing this, now that your other fix is in.