Godot-proposals: Positional Audio Relative to Character / Listener2D node

[nhydock]

Describe the project you are working on:
N/A

Describe how this feature / enhancement will help your project:
Not every game has characters that are centered on the camera. Spatial audio should work not just based on what the player sees, but where the player is technically in the world. This is particularly of importance in situations where cameras maybe be locked or pan over a location but the character moves within the view and the player needs to be aware of audio cues.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

Describe implementation detail for your proposal (in code), if possible:
Currently position audio for 2D, is based on the camera/viewport location, while 3D does have a Listener node which matches the desired behavior. Audio listening should be split apart from the camera and offered as a separate node and concept with ability to toggle if it's an active listener. This node can be attached to whatever target the developer wants, whether it be a camera to replicate the existing functionality, or to a specific character object the wish the player to control in the world.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
No, calculations for audio falloff would need to be reimplemented according to what is already in the engine and for a new node type that does nearly the same functionality. This amounts to unneeded redundancy and a worse developer experience.

Is there a reason why this should be core and not an add-on in the asset library?:
This is an improvement/replacement to existing core functionality.

[CptPotato]

I think the node solution of your mockup is great. It also fits the node-based architecture of Godot.

[golddotasksquestions]

I haven't come around to test this yet, but I was planning to archive this simply with Area2D nodes, on entry load and start the sound with 0 volume, and then measure the distance and position of the player to the Area2D center to adjust volume and audio channel. Is this not a viable solution?

[CptPotato]

I haven't come around to test this yet, but I was planning to archive this simply with Area2D nodes, on entry load and start the sound with 0 volume, and then measure the distance of the player to the Area2D center to adjust volume and audio channel. Is this not a viable solution?

I'd say that's more of a workaround than a solution. You would have to reimplement all existing aspects of spatial audio yourself with this method. In contrast to that, plugging a different orientation into the audio calculation is likely not a big change in the engine.

Though, I'm not sure how many would really benefit from a feature like this.

[Calinou]

There's a Listener node available in 3D, but there doesn't seem to be a 2D equivalent:

[golddotasksquestions]

Though, I'm not sure how many would really benefit from a feature like this.

Who would not benefit from it?
Unless you are making a pure first person game or have a cutscene, sound FX have nothing to do with the camera. It's always the players position in the world that is relevant for sound FX, not the camera. The further the camera is from the player the more obvious this is.

[nhydock]

Thanks @Calinou I forgot about that. Updated the proposal description to specifically point out the need for this in 2D

[ndarilek]

I'd like to see this as well. I assume the 3-D audio node/listener uses something like OpenAL? If so, might be nice to copy the code directly and just use the X and Y coordinates in the 2-D versions, also setting rotation, and setting Z to 0. I'd like other 3-D features in 2-D, dopper for instance.

On a related note, I'd also hope this proposal could eliminate tying 2-D audio to screen position, or even presence on-screen at all. Would be nice if off-screen objects could be heard in the distance--monsters in other rooms not currently rendered, for instance. If you only want on-screen objects to render, that seems better handled by watching visibility and starting/stopping players accordingly.

[ndarilek]

So as a temporary solution, I've created this script but it doesn't work. Any thoughts as to why not? Basically, I add any AudioStreamPlayer3D and Listener nodes to groups, then I iterate through all group members. If the audio node has a Node2D parent, I set its X and Z to the X and Y of the parent node. Unfortunately, changing my AudioStreamPlayer2D node to an AudioStreamPlayer3D silences it. I do have a Listener so that shouldn't be the issue.

Would really appreciate thoughts on this. I'm leaving it here as a workaround, but if this or something like it can't be made to work then I'll have to use another engine. Hoping to avoid that.

extends Node

func add_audio_node(node):
  if node is AudioStreamPlayer3D:
    node.add_to_group("3d_streams")
  elif node is Listener:
    node.add_to_group("3d_listeners")

func add_audio_nodes(node: Node):
  add_audio_node(node)
  for child in node.get_children():
    add_audio_nodes(child)

func _ready():
  get_tree().connect("node_added", self, "add_audio_node")
  add_audio_nodes(get_tree().root)

func sync_audio_to_node2d(audio, node2d: Node2D):
  print(audio)
  if audio is AudioStreamPlayer3D:
    print(audio.playing) # true in my case
  audio.global_transform.origin.x = node2d.position.x
  audio.global_transform.origin.y = 0
  audio.global_transform.origin.z = node2d.position.y

func _physics_process(delta):
  for node in get_tree().get_nodes_in_group("3d_streams"):
    var parent = node.get_parent()
    if parent is Node2D:
      sync_audio_to_node2d(node, parent)
  for node in get_tree().get_nodes_in_group("3d_listeners"):
    var parent = node.get_parent()
    if parent is Node2D:
      sync_audio_to_node2d(node, parent)

[ndarilek]

BTW, I have a workaround, though it took days of me pestering on multiple channels to find a solution and it could probably stand to be made easier in a whole bunch of ways. :) It may also have weird side-effects. But I can confirm that my Node2Ds are playing AudioStream3D nodes that move around and rotate. I can't confirm that this doesn't perform any strange rendering passes or anything. Here's roughly what I did, assuming an empty scene tree.

• Create a Node, Call it Main.
• Under Main, create a Viewport. Call it Audio.
• On Audio, enable the 3-D listener property.
• Under Audio, create a Camera.
• Under the Camera, create a Listener.
• DON'T DO ANYTHING WITH THIS AUDIO VIEWPORT ANYMORE. :) It only exists because apparently to have a Listener we need a Camera, and because we're 2-D only, I'd rather this camera not render anything.
• Under Main, create a Node. Call it AudioSync.
• Under Main, create a Node2D. Call it Player.
• Add Player to a group called "listener" to mark it as the node with which the Listener should sync.
• Under Player, create an AudioStreamPlayer3D. Set it up like normal, attaching your stream and such.
• Attach the below script to AudioSync.

I've gotten reports that this doesn't work if you move a node with a sound around the editor, but that seems intuitive, since the below script needs to run and only works in-game.

Anyhow, I'm not super confident in this solution, so FWIW I support this proposal and would be happy to help move it forward. Would be nice if AudioStreamPlayer2D was mostly identical to its 3-D sibling, as the way it works now isn't very intuitive.

Anyhow, here's the revised script:

extends Node

func add_audio_node(node):
  if node is AudioStreamPlayer3D:
    node.add_to_group("3d_streams")
  elif node is Listener:
    node.add_to_group("3d_listeners")

func add_audio_nodes(node: Node):
  add_audio_node(node)
  for child in node.get_children():
    add_audio_nodes(child)

func _ready():
  get_tree().connect("node_added", self, "add_audio_node")
  add_audio_nodes(get_tree().root)

func sync_audio_to_node2d(audio, node2d: Node2D):
  audio.global_transform.origin.x = node2d.position.x
  audio.global_transform.origin.y = 0
  audio.global_transform.origin.z = node2d.position.y
  audio.global_transform.basis = Basis()    
  audio.rotate_y(node2d.global_rotation)

func _physics_process(delta):
  for node in get_tree().get_nodes_in_group("3d_streams"):
    var parent = node.get_parent()
    if parent is Node2D:
      sync_audio_to_node2d(node, parent)
  var listeners = get_tree().get_nodes_in_group("listener")
  if listeners and len(listeners) == 1:
    var listener = listeners[0]
    for node in get_tree().get_nodes_in_group("3d_listeners"):
      sync_audio_to_node2d(node, listener)

[nhydock]

One other problem you'll encounter with your work around is it doesn't work with physics like a pure node2d system would. Positional audio is godot is capable of allowing walls to block off/dampen sound, but since 3D and 2D have completely different physics systems, the only way it could work in 2D is if you have a hidden 3D recreation of your map.

[ndarilek]

Thanks for the heads-up. Fortunately I won't be using those features in
my initial planned games, but I would like them at some point.

So it looks like there's significant interest in this proposal from a
few folks. What's our next step?

For my part, I'd happily work with others to implement/test this, but I
can't do it alone. My hands are fairly full with my Godot accessibility
work ATM, and I'm blind, so am a bit limited in what I can do with the
engine due to it not giving me enough feedback on things currently.
However, I'd very much like to see this work happen, so if a few folks
wanted to take it up, I'm happy to help and write whatever code I can.

As a start, I wonder how hard it would be to rip out the 2-D streaming
audio support and replace it with what's there in 3-D? Or could we make
the existing 3-D support adapt itself to running in 2-D, similar to what
I've done with this workaround? Then the 2-D listener/stream could be
deprecated and removed in 4.0/5.0?

FWIW, I'm not immediately sure why audio needs 2-D and 3-D branches.
I've used OpenAL and other spatial audio systems in 2-D games without
issues, so from a technical perspective it seems like it should work.
I'm just not sure how many code paths in the current system take 3-D for
granted, and how feasible it is to change them to detect if they're in a
3-D or 2-D context.

Thanks.

[nhydock]

I don't think 2D and 3D need different technical flows, but it at least needs to be kept separate in terms of nodes and units. 2D does everything in terms of pixels while 3D has world units. They could both be backed logically by OpenAL, but the abstraction for the two systems needs to be maintained.

[ndarilek]

Hmm, so what's the process for making significant breaking changes like
these?

I'm glancing through the 2D stream/listener code, and a reasonable
short-term win might be making the listener/2-D stream somewhat spatial
in that it takes position/rotation into account. If someone wants the
current behavior, I imagine they can emulate it much more easily than
I'm emulating spatial audio in 2-D (I.e. just plant the listener at
screen center and they're good, no separate
viewport/camera/listener/script just to get 3-D in 2-D.) But this would
change the node's behavior in a backwards-incompatible way.

On one hand, I'd understand kicking it down the road a ways. On the
other, I actually just ran into someone
else
trying to get more sophisticated audio in what will likely be 2-D, so
I'd be interested in how we'd introduce an incompatible change to a part
of the API that isn't meeting many folks' needs.

Sorry, don't mean for that to sound harsh. I might be willing to take a
stab at making 2-D audio positional, but I just want to know whether
that's aiming too high. :) Making 2-D audio positional seems more
attainable than biting off everything at once.

[nhydock]

I think it'd be good to focus on having the listener logic in another node like in 3D instead of hacking the AudioStream2D node, that way the approach both in the code and how to use the feature between 2D and 3D are the same. As I noted in the proposal at the top of this thread, emulating existing behavior should be no more complicated than just adding a listener node centered and usually a child of the active 2D camera.

Right now it's a bit of a mess that the AudioStream2D node is fully aware of viewports instead of just simple positional nodes. I feel this makes things even worse with the forthcoming multiple window support, so refactoring sooner rather than later could be for the best.

[ndarilek]

Oh, I forgot, there isn't even a Listener2D. FOr some reason I thought
there was and it just didn't work. Working too hard recently...

Why is it, though, that Listener3D has to be a Camera child to work?
I just hit that recently and it seems like surprising behavior,
particularly since it doesn't warn when that condition isn't met. If I'm
considering adding a Listener2D, I'd like to make it not be a Camera
child, and would like to change that requirement in Listener3D while
I'm at it.

[golddotasksquestions]

I'm sorry for repeating myself, but the above workaround seem so hacky that I have to ask again, why not use raycasts and Area2Ds in the meantime for this?
http://kidscancode.org/blog/2018/03/godot3_visibility_raycasts/
As you can read here you can even detect obstacles like walls or even moving enemies and trigger/adjust volume and channel pan accordingly.
In the KidsCanCode tutorial the node casting the rays is stationary, but it could just as well be the moving player. For sound dampening you could have two rays in each direction, one that detects the sounds source, and one that detects the obstacle. The distance between source/obstacle/player regulates dampening and volume.

[ndarilek]

I'm not convinced that raycasting and doing my own math for
panning/direction/attenuation/unit size is any less hacky than my
current workaround. It sounds like I'd basically have to build my own
2-D spatial audio system in GDScript, and at that point I might as well
just use OpenAL and roll my own. Also, I'd basically have to create a
parallel tree of nodes that transforms every set of coordinates relative
to screen center, since that's where the 2-D viewport assumes the
listener is located. I'm familiar with Godot's raycasting, but using
that would be hackier than my workaround IMO.

[ndarilek]

I'm considering taking a crack at this, but I'd like to do so as a foundation for a larger 3-D refactor which I may or may not attempt.

In particular, I'm discovering that Godot's 3-D audio support has no apparent equivalent of OpenAL's rolloff factor--that is, no way of changing the factor at which a sound diminishes over distance. I'd like to attempt bringing that to the 2-D audio engine, but that will make it more incompatible with the 3-D engine. Though, ideally, the design choices made in the 2-D engine might eventually be ported over to 3-D.

How would I start? I don't have enough experience to know how other game engines handle this. Would anyone be willing to work with me on this?

[ndarilek]

Taking a crack at this, but what is the purpose of methods like:

void Viewport::_update_listener_2d() {

    /*
    if (is_inside_tree() && audio_listener && (!get_parent() || (Object::cast_to<Control>(get_parent()) && Object::cast_to<Control>(get_parent())->is_visible_in_tree())))
        SpatialSound2DServer::get_singleton()->listener_set_space(internal_listener_2d, find_world_2d()->get_sound_space());
    else
        SpatialSound2DServer::get_singleton()->listener_set_space(internal_listener_2d, RID());
*/
}

...

void Viewport::_update_listener() {
}

I'd like to simplify the current implementation before starting out, which in part involves removing these empty methods and their callsites. But I don't know if they serve some odd purpose, or if they're just left over from previous implementation work.

[textrivers]

Just curious if this was still being worked on, or if it's on the horizon for v4.0. I'd love to be able to use it in my project.

[Calinou]

@textrivers As far as I know, nobody is currently working on this feature. It's not planned for a specific version either.