Scrcpy: Android 10 audio forwarding

[fenste]

Are you considering adding audio forwarding for Android 10 when they added https://developer.android.com/guide/topics/media/playback-capture ?

[rom1v]

Absolutely: https://github.com/Genymobile/scrcpy/issues/14#issuecomment-523071057

[AndroidDeveloperLB]

Will it be possible officially, then? Even for Windows?

[rom1v]

If it works, yes. That would be great.

The problem is finding time to work on it (especially to investigate how to receive audio packets using private APIs without Android context). :disappointed:

[AndroidDeveloperLB]

Why doesn't Google offer an official API for this? They have an adb command for recording, but not including audio...
How do you use adb for the mirroring anyway?

[rom1v]

How do you use adb for the mirroring anyway?

I explained here and here.

[AndroidDeveloperLB]

Wait, you somehow run code on the device, without installing anything?
How is it even possible?
I don't get it...

[LBensman]

Wait, you somehow run code on the device, without installing anything?
How is it even possible?
I don't get it...

Installing an "app" via an app store is not the only way to get something running on an Android device. You can push files to device via adb push and you can execute those files using app_process command on the device itself (usually via adb shell command executed on the host).

[AndroidDeveloperLB]

I said "without installing anything".
You can push a file and run something in it, without installing anything?

[LBensman]

I said "without installing anything".
You can push a file and run something in it, without installing anything?

Sure, why not? There are limitations, I think, as to what a process can do that is launched that way, so you can still be limited in what APIs you can call from such a process, but at least for grabbing screen surface and video encoding, it has sufficient means to do so.

[AndroidDeveloperLB]

This is incredible. What are the limitations, then?
It can get all that's on the screen. That's a lot, no?
How could it get the screen content? I thought that maybe you will be restricted to pure Java stuff, but this is even beyond what Android's framework offers without user's intervention.

[LBensman]

I'm not exactly sure of the limitations or what would constitute being a lot or a little, but take a look at SurfaceControl wrapper to see what it calls to get the screen data, as well how it is wired as input to encoder.

[AndroidDeveloperLB]

Amazing. Seems you used a lot of reflection though.
How did you find about this? How did you know it's even possible?

[LBensman]

Oh, I'm not the author of it - you'd have to ask @rom1v on how he did the research and the level of experimentation that it took :)

[AndroidDeveloperLB]

OK just know that I had this expression like you have on your avatar...
haha

[rom1v]

https://developer.android.com/guide/topics/media/playback-capture

The app must have the RECORD_AUDIO permission.

The shell does not have this permission :cry:

So it will require to install an apk. :disappointed:

[agent47nh]

So why not keep an _optional APK_ just for audio?

[arex0]

In my phone android 10 and desktop debain testing, https://android.stackexchange.com/questions/58026/how-to-stream-audio-from-android-to-pulseaudio-server working

[AndroidDeveloperLB]

I don't mind having an additional APK for this, as long as it doesn't ruin other things.
What happens if multiple apps try to use the mic?
Maybe you could add an option in the window of scrcpy to turn it on/off?

[rom1v]

• https://blog.rom1v.com/2020/06/audio-forwarding-on-android-10/
• https://github.com/rom1v/sndcpy

[AndroidDeveloperLB]

@rom1v Either I don't get how to make it work, or it doesn't work.

I tried this:

1. Install the apk from there.
2. Run scrcpy.
3. Run sndcpy.
4. Grant permission on the device to capture video.
5. Press Enter on the console window.
6. Play some video with audio on YouTube app on the device.

Nothing occurs .

It shows a notification of "sndcpy" with "Waiting for connection..." and yet it shows as if there is a video capture (the red icon on the status bar).

[rom1v]

It shows a notification of "sndcpy" with "Waiting for connection..."

Did you press "Enter" in the sndcpy terminal?

What's the output in your terminal?

Also, is it Android 10? Or Android 11?

[AndroidDeveloperLB]

Yes, that's step 5 that I wrote about.
BTW, Pixel 4 with Android 10.

There is no output on the terminal , because it was closed when I pressed Enter...

[rom1v]

OK, is VLC installed on your computer?

Try to run it from cmd.exe instead of double-clicking on the .bat.

[AndroidDeveloperLB]

VLC is installed on the PC, but do I need to start it?
Other than seeing that it should be installed, I don't see any other mention of it.

What exactly to do with VLC?

[rom1v]

Other than seeing that it should be installed, I don't see any other mention of it.

Read sndcpy.bat :wink:

[AndroidDeveloperLB]

That's step 3. Already done this.

And I tried to run VLC on my own too.

[AndroidDeveloperLB]

Wait, you wrote "Read " ? Why read it? You meant "run" ?

[rom1v]

No, I meant read. You will understand how VLC is used :)

[AndroidDeveloperLB]

Read? But it doesn't say anything when running it, about VLC.
If you mean read in source code, why read in source code instead of instructions of how to use the new tool?
Can you please tell me what to do?

[rom1v]

But it doesn't say anything when running it, about VLC.

What is the output of the terminal when you run sndcpy? (if you run it manually from cmd.exe so that the terminal does not close automatically)

If you mean read in source code, why read in source code instead of instructions of how to use the new tool?

As a user, you don't really need to know. Running sndcpy should be sufficient. That's why you "don't see any other mention of it" (except in the script source code).

[AndroidDeveloperLB]

As I wrote, when pressing Enter, it closes.
If you mean that I should run sndcpy from console and then press Enter, this is what I see:

Waiting for device...
Performing Streamed Install
Success
Starting: Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=com.rom1v.sndcpy/.MainActivity }
Press Enter once audio capture is authorized on the device to start playing...
Playing audio...
The system cannot find the path specified.

The weird thing is that when I tried on another PC (a laptop actually), it worked fine.
The other PC has Intel, and mine has AMD. Is it related?

[rom1v]

The system cannot find the path specified.

This is the relevant message.

Is your vlc installed in C:\Program Files\VideoLAN\VLC\vlc.exe? If not, pass an environment variable: https://github.com/rom1v/sndcpy/blob/master/README.md#environment-variables

[AndroidDeveloperLB]

For some reason it decided to be installed on "C:\Program Files (x86)\VideoLAN\VLC\vlc.exe" instead.

I decided to uninstall it, and re-install it, because maybe it installed as 32 bit or something. Now it's on the path you wrote.
But now it doesn't close the console window, and yet I still can't hear from the PC.

This is what I see on the console this time:

Waiting for device...
Performing Streamed Install
Success
Starting: Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=com.rom1v.sndcpy/.MainActivity }
Press Enter once audio capture is authorized on the device to start playing...
Playing audio...

[rom1v]

But now it doesn't close the console window

:+1:

and yet I still can't hear from the PC.

What app do you test on your device? Try youtube or VLC android for example.

I think we could continue on https://github.com/rom1v/sndcpy/issues/1#issuecomment-641864289.

[AndroidDeveloperLB]

I used YouTube

[rom1v]

The weird thing is that when I tried on another PC (a laptop actually), it worked fine.

The same device works on another laptop?

yet I still can't hear from the PC.

But on this laptop, no sound?

[AndroidDeveloperLB]

I tried on YouTube.
And it's the same device. It works via the laptop but not on the desktop.

[AndroidDeveloperLB]

OK I tested again today.
This time on Android R.
I've noticed something weird:
It plays on both the PC and device, so I hear twice, with a small delay between them
On the device the volume can be 0 though, so I won't hear anything from it, while hearing from PC.

[rom1v]

:+1:

It plays on both the PC and device

Yes, that's the expected behavior (the playback capture API works this way).

[AndroidDeveloperLB]

Shouldn't it at least set the volume to minimal?
It's weird to hear the same thing with a delay on both PC and device...
Is there perhaps a parameter for it, so that it will try to restore volume when it's done, or something?

[huanhor]

Shouldn't it at least set the volume to minimal?
It's weird to hear the same thing with a delay on both PC and device...
Is there perhaps a parameter for it, so that it will try to restore volume when it's done, or something?

Hey friend,
I know your problem and i solve it. You try to install the version of VLC are the Windows 64bit and try again.
I have same problem like you and i suggest you try this.