Wled: Effect-Intensity based on amplitude (Sound Reactive)

[GrumpyMeow]

Hi,

I'm working on the WLED Sound Reactive fork. I noticed that on that fork specific Sound Reactive effects have been added.
I've made some changes which allows the user to "enable effect intensity to be driven by the sound amplitude". By doing this we achieve actual Sound Reactiveness on the OOTB WLED effects.
For this i've added a checkbox behind the intensity-slider.

I'm in doubt if my current implementation is "in-line" of the base WLED-implementation. This implementation does work, but continuosly changes the intensity. The intensity-slider doesn't continuously "move".

void userLoop() {

  if (millis()-lastTime > delayMs) {
      ....
      if (effectIntensitySR==1) {
        if (sample > effectIntensity ) {
          effectIntensity = sample;
        } else {
          effectIntensity *= 0.99f;
        }
        colorUpdated(NOTIFIER_CALL_MODE_FX_CHANGED);
      }
      ....
    }
  }

uint16_t WS2812FX::mode_static(void) {
  uint16_t brightness = SEGMENT.intensity;
  uint32_t c = color_blend(0,SEGCOLOR(0), brightness);

  for(uint16_t i = 0; i < SEGLEN; i++) {
    setPixelColor(i,c);
  }
  return FRAMETIME;
}

Or would it be better to add to add a "SEGMENTOPTION"?

//Segment option byte bits
#define SEG_OPTION_SELECTED       0
#define SEG_OPTION_REVERSED       1
#define SEG_OPTION_ON             2
#define SEG_OPTION_MIRROR         3            //Indicates that the effect will be mirrored within the segment
#define SEG_OPTION_NONUNITY       4            //Indicates that the effect does not use FRAMETIME or needs getPixelColor
#define SEG_OPTION_FREEZE         5            //Segment contents will not be refreshed
#define SEG_OPTION_INTENSITY_REACTIVE 6       <--- example
#define SEG_OPTION_TRANSITIONAL   7

Could you maybe give me a thumbs-up or thumbs-down on my current implementation? How would you go about when you would implement this Sound Reactive intensity?

greetings, Sander

[huggy-d1]

    if (sample > effectIntensity ) {
      effectIntensity = sample;
    } else {
      effectIntensity *= 0.99f;
    }

Might be better like this:
// DEFINE uMIN_INTENSITY_DELTA = 0.02f might be 0.02 (2%) change before it adjusts the effect intensity
// test to find the correct value (-D uMIN_INTENSITY_DELTA set during build)
if (ABS(sample - effectIntensity) > uMIN_INTENSITY_DELTA ) {
effectIntensity = sample;
} else {
effectIntensity *= 0.99f;
}
This may still hunt because it is constantly reducing the intensity with the *= 0.99f;

[GrumpyMeow]

@huggy-d1 I'm more "seeking" for where to put the "Sound Reactive" flag.
Should it be a "runtime variable like "samplePeak" or "part of the segment" or "part of a favourite" ?
Transmitted by "transmitAudioData" and/or "notify"?

My thoughtprocess:

• it makes no sense continuously transmitting a continuously changing "effectintensity" to Hyprion/TPM2.NET
• the value driving the effectintensity is already being transmitted to other WLED's by "transmitAudioData"
• the "SR flag" should be saved as part of a preset
• the "SR flag" should be might be usefull as part of a segment

[huggy-d1]

Ahh, then I am of no help at all. I'll shut up now and pay attention for the people with useful answers! Have you tried posting an issue on the sound reactive fork's github?

[Aircoookie]

Hi,
first off all I believe it is a very cool idea to also add sound reactivity to the standard WLED effects!
I do believe that the per-segment flag would be the way to go here since that will enable some static and some reactive segments. We can reserve bit 6 for this purpose 👍

Another idea I had is that if it is your goal to make _all_ effects sound reactive, going with intensity is a good starting point, but for some effects, modulating speed or brightness might be more suitable (either because its an effect where the intensity is unused or one where using the intensity slider would cause a "jump" in the effect).
For that I would recommend a flag in FX.h like srMode. In each effect function, SR mode would then be set to the most appropiate for that effect (eg. 0 = brightness, 1 = speed, 2 = intensity, maybe color... that one's a stretch though) and if the "SR flag" is activated that property would be modulated. Up to you though :)

[huggy-d1]

Having a "random" SR mode option (for effects with multiple available SR modes) would be fantastic.

I saw a Tool concert where the visual effects were sound reactive and used a variety of SR effect types for the displayed imagery.

It was not until I saw the same Tool concert at another venue that I realized they had randomized some of the effects.

For ex: first concert an image was reacting to the rhythm (beat) of the drums by what visually felt like an overlay of a beaten drum head with the actual image. At the second concert, it felt like the displayed images colors were modulated by the rhythm and patterns of the various drums used.

I would be amazed if the audio and video mixing was done manually and adjusted each show. I would not be surprised if they randomized which effect would be used through each particular images presentation.

Those choices produced dramatically different feelings and visual effects.

I would appreciate WLED having that SR capability, being able to impact our eyes with what we hear in our ears, and not always in the exact same predictable way.

[GrumpyMeow]

Thanks for the replies..

I did notice that the bit-6-flag was left unused, so I'll use that bit for toggling "Sound Reactivity" on-and-off.

I already did some fiddling with segments and presets. I think I saw that the segment-settings weren't saved by the preset. But I'll have to do some more investigation into this.

My current implementation was continuously changing the effect-intensity based on the sound-amplitude. This has the side-effect of overriding the intensity slider value. It feels a little bit like a dirty-hack. But I think I've come up with a better solution by taking the user-selected effectintensity and multiplying that with a amplitude-based-factor. This would keep the intensityslider functional.

I already implemented "drip" and "popcorn" to react on the detected beat of the sound. It does work, but the results are not that spectaculair. Also changing the effect-speed based on amplitude did not give a visually pleasing result. The best results by far are with changing the intensity of the effect. It made even the most basic of effects quite interesting.

This evening I was working on adding more icons to the iconfont. I've added SVG files and a NPM package which converts SVG files to font-files. I think that for some icons there are more appropriate icons available. On the SR-fork the same icons are used for all the sliders, which i found confusing. I also want to add the icon-symbols to the effect buttons, to indicate which parameters influence the effect.

I'll try to get these changes as-soon-as-possible available for everyone.

Good night

[GrumpyMeow]

Ok. I'm making progress here...

• I now added a checkbox "Reactive effects" to the "Segment panel" (under "mirror" & "reversed")
• I now kept the Effect-intensity-slider functional by replacing SEGMENT.intensity with SEGMENT.getIntensity(). In this function i return the normale-intensity or the reactive-intensity ((intensity * amplitude) / 255).
• I added more appropriate icons and made it easier to add/replace icons

[THATDONFC]

Thanks for the work you've been putting in on this @GrumpyMeow! I like this idea a lot. Are you in our SR Discord? If not, here's an invite so we can discuss this further https://discord.gg/4CQRmfR . I'm sure Andrew would love to discuss this.

Regarding the icons, getting the code functional and stable was our main priority. I've always wanted to clean up the code and UI to make it more polished. I never got around to figuring out the icons. How have you been adding icons to WLED?

[jtheg777]

Loving all this talk about all animations being sound reactive in one form or the other. Just amazing👍😃👍

[GACGAMA]

@GrumpyMeow, how do you process audio via PWM? Which sensors are you using?
I'm currently using my windows with LedFX with a lot of success, as it turns audio spectrum captured via MIC/Virtual Cable into a 1d matrix to display effects, but a built-in option in the ESP32 would be way better, faster and more reproducible.

I'm also tinkering with a raspberry pi to create a hub of audio processing for many different ESP32 connected in different places, but this will take a long time.

[GrumpyMeow]

Hi,

Audio sampling is not done through "PWM". Pulse Width Modulation is a technique of quickly switching a signal on and off (square signal).
Maybe you meant "PCM"? Pulse Code Modulation is the technique of measuring/sending a varying voltage.
The Wled-fork of Atuline (Andrew Tuline) uses a microphone to sample audio. And interpret this signal and turns it into magic with the leds.

Cheers,
Sander