I'm trying to Hook into the block editor components-toolbar, but couldn't find any reference, my idea was to use a filter to remove some of the defaults block editor components toolbar items on all components.

Any hint is more than welcome, thanks
You can use PluginBlockSettingsMenuItem. Here's the docs link : https://developer.wordpress.org/block-editor/developers/slotfills/plugin-block-settings-menu-item/ 馃憤
Sweet! Thanks, @phpbits SlotFills of course!
Most helpful comment
Sweet! Thanks, @phpbits SlotFills of course!