Color controls, like background media, are primary features of this block and should be moved to the toolbar:

Related issue for Group and Columns blocks: #20070.
馃憢 Halloo! I started looking into this today.
Ma goodness! What a journey. It's definitely a challenge, especially if we want to do it cleanly.
I'm up for taking this one! Just want to leave a note for my future self (and to others)
Don't under estimate this one 馃檭
@ItsJonQ Yeah, as I discovered in #20246, the way controls, toolbars, and popovers relate to each other and how you're supposed to use them is rather complicated. Good luck!

Got a quick + dirty version working. (Ignore the W placeholder Icon for now)
One thing I noticed is that there may be cases where the dropdown menu will cover the content (like above). It's a little unsettling (at least for me). I prefer the controls being on the sidebar (as disconnected as they are). At least it allows me to see the Cover in full view as I adjust it.
If anyone is curious, I pushed up my explorations here:
https://github.com/WordPress/gutenberg/tree/try/cover-color-toolbar-experiment
Branch: try/cover-color-toolbar-experiment
@ItsJonQ I think this one should be present in both places. There some work left to do to clean the contents but this is a good first step to exposing the most important tools.
@mtias Great! Thanks for your thoughts :). In that case, I'll proceed with this effort!
@mtias I thought we wanted to avoid duplicate controls in the toolbar/inspector?