Jamulus: Android support

[corrados]

_This issue has been migrated from Sourceforge. Created: 2015-12-13
Creator: Volker Fischer_

Add support for the Android operating system. The software can already be compiled for Android but the audio interface does not work right now. So the first step is to finish the audio interface implementation. As a second step, all GUI functionality must be checked that it is possible to use it with a touch screen only.

[corrados]

Some progress reported here: https://sourceforge.net/p/llcon/discussion/533517/thread/cfd165e0/#452b/6e3b

[corrados]

See here: https://www.facebook.com/groups/507047599870191/permalink/521878088387142/

[j-santander]

Is there any instructions on how to build this? or any specific branch/forked repository I could refer to? From the previous links there seems to be some implementation working to some extent. I might be able to give a hand, at least testing
Thanks

[corrados]

The code on master is from Simon. But it does not work correctly. Recently, I have started a branch where I do some testing. But also, nothing working yet.

[j-santander]

Ok, thanks. I got it to compile and started testing.

There's an mistake on android/sound.cpp: on CSound::onAudioRead() when converting from incoming float buffer to int16_t:

                pSound->vecsTmpAudioSndCrdStereo[frmNum * oboeStream->getChannelCount() + channelNum] =
                    (short) floatData[frmNum * oboeStream->getChannelCount() + channelNum] * _MAXSHORT;

should be

                pSound->vecsTmpAudioSndCrdStereo[frmNum * oboeStream->getChannelCount() + channelNum] =
                    (int16_t)(floatData[frmNum * oboeStream->getChannelCount() + channelNum] * _MAXSHORT);

otherwise the precedence of the casting operator will make the input always 0.

Changing that if I write to a file the content pSound->vecsTmpAudioSndCrdStereo vector I got a correct signal.

To continue progressing I might need to understand better how the whole things work (I'm not at all familiar with neither your code or even the oboe library). A few doubts:

• Is it assumed that the sound card is always stereo? It seems so, but the oboe library seems to be configured in Mono. Changing that I seemed to started receiving a more-or-less-ok signal on another client. What other assumptions should be made for the signal coming from CSound -> CClient (sampling rate, frame size....)?
• You got oboe setup for float samples. Is there any reason to not use int16_t samples? (we're doing the conversion by hand).
• I'm not sure how it works on the playing side. What I believe I see is that at CClient::ProcessAudioDataIntern after the recording data is processed, the same vector used for the recording data is filled with the playing data received from the network. However, my expectation is that I would be receiving an independent call to CSound::onAudioRead() for the playing channel, which might even indicate that the number of frames available for writing is not the same as the number of frames that we've written (however there seem to be an oboe builder call: setFramesPerCallback() that might help in synchronizing).
• What are the expectations for the CClient -> CSound data?
• How the expected buffer sizes work? (both for playing and recording)

In this last case, I'd go probably for a temporary circular buffer where the onAudioRead() processing of input data will write the recording data left in the buffer and the call of onAudioRead() for the output data will read. It probably would need to be mutex protected (unless we can be sure that the oboe callbacks are always run on the same thread)

[corrados]

There's an mistake on android/sound.cpp: on CSound::onAudioRead() when converting from incoming float buffer to int16_t

I'll merge from a branch soon, where the int 16 type is replaced by float.

Is it assumed that the sound card is always stereo?

Yes, the input/output audio vector for the audio interfaces is assumed to be always stereo. If you have only a mono stream available, just write the same audio data in both stereo channels.

On my branch https://github.com/corrados/jamulus/tree/android_testing I tried to use the full duplex oboe example class. But it is not yet finished. There is function onBothStreamsReady which should have the same size for input and output (which is what Jamulus expects).

[j-santander]

I've got some working code at https://github.com/j-santander/jamulus/commit/a3558aa612e4793d7597620293296964abcc5bc1
(Unfortunately I was mistakenly working from the master code, not from the android_testing branch.. and only when I was ready to submit I realized of my mistake).

In my tests I can hear other clients and they can hear me. Not sure of the overall quality of sound.

Key modification was to setup the callback only on the input stream and do explicit write to the output stream after processing the incoming data.

I'll try to have a look at the code in the android_testing branch and incorporate it.

Let me know what you think or how should I proceed.

Cheers

[corrados]

My code on the android_testing branch is not working. I was just experimenting a bit. You do not need to use any of that code. If you have something working, then you could create a pull request. I can then test it on my Android phone. But be aware that some minutes ago I applied a big change to the code on master.

[j-santander]

Created PR #661.

Thanks

[ann0see]

How’s the quality? I assume it would be best to try it with a LAN adapter

[j-santander]

How’s the quality? I assume it would be best to try it with a LAN adapter

I might be able to test with more people today.

[dzpex]

i'm interested to test the android code but i do not have experience to compile qt5 project for andoid can you help me?

[corrados]

@j-santander I tested your code and it crashed with "_AudioStreamRecord: maybeConvertDeviceData() conversion size 256 too large for buffer 128_". I had the same error message on my Android testing branch. This is a known bug in the Oboe library. We have to use a newer version. Just to let you know: I'll now update the Oboe Git submodule to the newest Oboe code now to avoid this issue.

[corrados]

I just updated the Oboe library version but unfortunately, it still crashes, now with a different error message: " _InputEventReceiver: Attempted to finish an input event but the input event receiver has already been disposed._"

[j-santander]

Oh, I'll test it more on my side. I didn't experience neither of the errors. I'm on Android 11, not sure if it has anything to do.

[j-santander]

i'm interested to test the android code but i do not have experience to compile qt5 project for andoid can you help me?

Well, I didn't even have experience on Qt. I used the Qt installer (https://www.qt.io/download-qt-installer) and made sure the Android support was marked. Using the Qt Creator, I had to set up paths to SDKs and NDKs and select the target for android.

[dzpex]

i installed all the requirements for the QT and i'm able to set the Andorid Target. But the compiler return error.....oboe/Oboe.h not found any idea?

[j-santander]

i installed all the requirements for the QT and i'm able to set the Andorid Target. But the compiler return error.....oboe/Oboe.h not found any idea?

The oboe library is stored as a git submodule, I believe you need to do:

git submodule update --init

[dzpex]

i updated the submodule but the error is the same i try to copy oboe include folder into jamulus main path or android folder but nothin

[j-santander]

I just updated the Oboe library version but unfortunately, it still crashes, now with a different error message: " _InputEventReceiver: Attempted to finish an input event but the input event receiver has already been disposed._"

I've been trying with different devices, and although there are problems with the sound quality, I haven't experienced any crashes like the ones you report. What Android version are you in? what terminal?

What I've tried so far:

• Google Pixel 3a with Android 11 (arm64-v8a)
• Google Pixel 3XL with Android 10 (arm64-v8a)
• BQ Aquaris X5 Plus with Android 7.1 (armeabi-v7a)
• Android Emulator on Android 7.1 (x86)

Regarding quality:

• In the BQ and the Emulator it seems I cannot play. Either because they are less powerful hardware, or the Android version. Checking into that.
• Overall architecture: What I have so far is we have the oboe stream callback on the input the stream, this calls into the CClient callback that does its stuff and replaces the input samples with the output samples in the same vector we're using as argument. After the CClient callback returns we do a non-blocking write on oboe's output stream. The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread (which is then the CClient callback thread that I guess allocates memory, do network operations, etc...).

[j-santander]

i updated the submodule but the error is the same i try to copy oboe include folder into jamulus main path or android folder but nothin

Ok, let's recap. What I did was:

• Install the Qt, including the Android support from the Qt installer.
• Configure the Android settings in the Qt Creator (Tools -> Options -> Devices -> Android). Solve all warnings/errors in that dialog. Tick on the "Automatically create kits for Android Toolk chains").
• Make sure the Android Kit is detected on Tools -> Options -> Kits -> Kits (I see Android Qt 5.15.1 Clang Multi-Abi). It should also show in the Qt Versions tab (Qt 5.15.1 for Android).
• Open the Jamulus.pro
• In the Projects tab -> Build & RUn It should appear the Android Qt, might be disabled. Enabled it if it is not enabled.
  Then you can connect your device and build or run the project.

Regarding the actual source. oboe is a git submodule which is located at libs/oboe. Make sure it has been checkout properly.

Hope this helps

[corrados]

I've been trying with different devices, and although there are problems with the sound quality

Have you been able to jam with other on any of these devices yet? Could you measure the overall latency you got?

What Android version are you in? what terminal?

I have a special setup: LineagesOS 16.0 (Android 9) on a Samsung Galaxy S5 (SM-G901F)

The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread

We actually do this for all OSs. It is not the preferred way of doing it but it works fine.

[j-santander]

I've been trying with different devices, and although there are problems with the sound quality

Have you been able to jam with other on any of these devices yet? Could you measure the overall latency you got?

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

Regarding the latency. Right now, I'm connected to a local server in my LAN:

From the Windows Client (desktop - ethernet connected) the "Configuration" reports Overall Delay 39 ms. (Ping Time 1 ms).

From my Google Pixel 3a, over WiFi, Overall Delay varies quite a lot from 55 to 80 and 90 ms. I guess it is due to the great variation on Ping Time from 2 to 33 ms.

Not sure if that's the measurement you're after.

What Android version are you in? what terminal?

I have a special setup: LineagesOS 16.0 (Android 9) on a Samsung Galaxy S5 (SM-G901F)

ok, I'm not sure if I could find a way of emulating that.

The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread

We actually do this for all OSs. It is not the preferred way of doing it but it works fine.

Ok, I look into that. I might need to implement a circular buffer because in some of the models it seems that I cannot write the whole batch of frames of output frames in one go.

I'll keep testing and see how far I can go.

[corrados]

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

So you intention is to bring your choir to jamulus using their Android phones? Very interesting. Please post your results here since I guess a lot of other choir members would love to use their phones for Jamulus, too.

ok, I'm not sure if I could find a way of emulating that.

No problem if not. My device is very old and I do not expect a good latency coming out of it. So just focus on the newer phones, that's fine.

[j-santander]

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

So you intention is to bring your choir to jamulus using their Android phones? Very interesting. Please post your results here since I guess a lot of other choir members would love to use their phones for Jamulus, too.

Well, my objective is enabling the choir to be able to rehearse under Jamulus.
However, I'm finding that for people outside of the computer world, even owning a decently modern computer could be an unsurpassable barrier (not to talk of installing or tinkering with the asio4all), and this cannot be the tool of a few, need to reach to the whole choir.
The possibility of using an Android phone will do marvels to lower the entry barriers and I believe we can pay the cost in quality if that means that everyone can use it.

ok, I'm not sure if I could find a way of emulating that.

No problem if not. My device is very old and I do not expect a good latency coming out of it. So just focus on the newer phones, that's fine.

Submitted a new pull request. I've written the details in the PR, but in summary I changed the tackle moving back to two callbacks and placing a ring buffer to hold the output samples until they're retrieved from the output callback. I found that on devices using the OpenSL backend (or perhaps less powerful devices?) the output stream was not always ready to pull the samples as they were received.

This I tried with an old device (BQ Aquaris X5 Plus on Android 7.1.1) [using OpenSL] and two more moderns Google Pixel 3XL on Android 10 and Google Pixel 3a on Android 11.

I tested with some people outside home, although we didn't sing, I could hear people conversation correctly and they reported that they could hear me fine. All this over WiFi. I did find that that if you were not close to the WiFi AP quality will degrade very quickly.

From my previous test I found that the variation in the ping values on the WiFi connection will be quite large.

I'll try to deploy the application to some additional devices and perhaps do some additional testing and gathering stats to see if there are some additional settings that I can tweak.

[dzpex]

I reinstalled all necessary package and when i try to build and execute the code the compiler returned me this error:
Android resource linking failed
Output: /home/build-Jamulus-Android_Qt_5_12_9_Clang_armeabi_v7a-Debug/android-build/build/intermediates/merged_manifests/debug/processDebugManifest/merged/AndroidManifest.xml:92: error: resource array/load_local_libs (aka com.github.corrados.jamulus:array/load_local_libs) not found.
error: failed processing manifest.

Any idea?? Thanks...

[danryu]

This all sounds very encouraging for the prospect of mobile builds - and being able to support choirs
and similar scenarios as @j-santander described!

I would love to give an Android build a test - is master the best branch to attempt a build from currently?

Also PS - should perhaps open a new issue for this: is there any particular reason that an iOS build has not been attempted yet? I've been scoping out the C++ iOS sound libs available which I believe would be able to integrate with a Qt-iOS build.

[ann0see]

I think there was an attempt for iOS, but the repo disappeared. See iOS version thread on sourceforge.

[j-santander]

This all sounds very encouraging for the prospect of mobile builds - and being able to support choirs
and similar scenarios as @j-santander described!

I would love to give an Android build a test - is master the best branch to attempt a build from currently?

Sorry folks, I believe my disappearance requires a bit of explanation. As I described above, what I did over the existing code for Android that Volker and Simon had in place was just a bit of bug fixing and small adjustments. However, as I was thinking about further involvement I realized that this was uncomfortably close to my day job (not really a direct overlap, but there might be technologies and techniques in common), so before involving myself any further, I decided to clear it up with my management. They sent a request to legal department and we're still waiting for an answer.

Regarding building. All my modifications were accepted to master, but I haven't tried to build it recently. Not sure if recent changes might have broken the build.

[danryu]

Thanks @j-santander for the update.
FWIW - I don't know your exact situation, but from my experience there should be zero conflict with working on open-source projects, unless you are using company time to do so. Most progressive companies welcome their employees working on diverse projects in their own time, as it inevitably leads to an enrichment of experience and ability within the workforce.

[ann0see]

How does the UI look like on Android? I compiled Jamulus for iOS and it doesn't show anything for me. I assume, we might need QML for both operating systems?

[corrados]

The UI on Android looks the same as on the PC. No need for QML.

[ann0see]

Ok.
Also, there was no need to change the code, I assume?
On iOS is that I don't see anything but a black screen, so it's probably an iOS only problem.

I hope, my problem can be fixed in some way without having to rewrite the UI for iOS

[corrados]

Qt should give you the same GUI on all platforms.