Sdl_evolution: [Accepted with Revisions] SDL 0280 - Adding new parameter of requiresAudioSupport and BluetoothDeviceAddress

[theresalech]

Hello SDL community,

The review of the revised proposal "SDL 0280 - Adding new parameter of requiresAudioSupport and BluetoothDeviceAddress" begins now and runs through September 29, 2020. Previous reviews of this proposal took place February 19 - March 17, 2020, June 2 - 23, 2020, and July 22 - August 18, 2020. The proposal is available here:

https://github.com/smartdevicelink/sdl_evolution/blob/master/proposals/0280-Adding-new-parameter-of-requiresAudioSupport-and-BluetoothDeviceAddress.md

Reviews are an important part of the SDL evolution process. All reviews should be sent to the associated Github issue at:

https://github.com/smartdevicelink/sdl_evolution/issues/941

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of SDL. When writing your review, here are some questions you might want to answer in your review:

• Is the problem being addressed significant enough to warrant a change to SDL?
• Does this proposal fit well with the feel and direction of SDL?
• If you have used competitors with a similar feature, how do you feel that this proposal compares to those?
• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
  Please state explicitly whether you believe that the proposal should be accepted into SDL.

More information about the SDL evolution process is available at

https://github.com/smartdevicelink/sdl_evolution/blob/master/process.md

Thank you,
Theresa Lech

Program Manager - Livio
[email protected]

[joeygrover]

1. For clarification, Google deprecated audio streaming through AOA 2.0 in Android 8.0. See here. I'm honestly not sure any OEM has implemented the audio profiles for AOA anyways.

2. bluetoothDeviceAddress is mandatory in the DeviceInfo struct, this is not necessary and causes a breaking change. In the case of iOS devices, there is no need to provide this information anyways and therefore adds extra data. This should be changed to `mandatory="false". Any place in Core/HMI that need this data should assume false if no data is provided.

3.

The current SDL Java Suite library cancels SDL launch if the requiresAudioSupport setting is TRUE and BT A2DP is not connected.
However, with this proposal, the SDL app is always launched without depending on the connection status of BT A2DP.

This completely changes how apps work today. There is no mention of this beyond "apps will launch regardless of audio state". If you go back and review the issue and PR you will see a lot of information about the issue and why the fix was put in place. I don't think this solution adequately addresses how older systems that can't be updated to support this logic will be handled. The author must take this into account as it will break OEMs implementations and create worse user experiences, ie apps showing up but playing audio through the phone speaker is worse than the app not showing up.

Issue for reference: Audio over AOA Issues
Pull request for solution: Add audio requirement ability

@stefanek22 Please pay attention to this proposal. This will break Ford's implementation, you need to review this.

4. What if bluetooth fails to connect? How will the app know? What should happen if the device is currently connected to another bluetooth device, will it disconnect from that device to await the connection from the head unit? Will the app stay registered even though there is no audio support?

5. What happens when the bluetooth adapter is off on the phone? Should the app automatically turn it on?

6. What if the device has never been paired with the IVI system before? Will the user be required to interact with their phone to accept the pair request? Is this an acceptable UX for driver distraction?

7. If a user selects the app before A2DP is connected the audio will start from the phone and transition to the IVI once connected. Does the app need to wait until the bluetooth connection is made before it starts streaming audio?

Overall I think we do need to do something better than what is there today but I don't think this is the right way to go about it.

[Kazuki-Sugimoto]

@joeygrover -san
Thank you for your opinion.
I will reply on behalf of @Shohei-Kawano.

1. bluetoothDeviceAddress is mandatory in the DeviceInfo struct, this is not necessary and causes a breaking change. In the case of iOS devices, there is no need to provide this information anyways and therefore adds extra data. This should be changed to `mandatory="false". Any place in Core/HMI that need this data should assume false if no data is provided.

I agree with you.

1. This completely changes how apps work today. There is no mention of this beyond "apps will launch regardless of audio state". If you go back and review the issue and PR you will see a lot of information about the issue and why the fix was put in place. I don't think this solution adequately addresses how older systems that can't be updated to support this logic will be handled. The author must take this into account as it will break OEMs implementations and create worse user experiences, ie apps showing up but playing audio through the phone speaker is worse than the app not showing up.

After launching the app, display a message prompting automatic connection or BT connection.
I think that UX is much better than before.

1. What if bluetooth fails to connect? How will the app know? What should happen if the device is currently connected to another bluetooth device, will it disconnect from that device to await the connection from the head unit? Will the app stay registered even though there is no audio support?

For these, a message is displayed to the user.

1. What happens when the bluetooth adapter is off on the phone? Should the app automatically turn it on?

If the bluetooth adapter is off, the bluetooth address cannot be obtained, so just display a message.

1. What if the device has never been paired with the IVI system before? Will the user be required to interact with their phone to accept the pair request? Is this an acceptable UX for driver distraction?

The implementation depends on the HMI, but I think it prompt the user with a message or voice requesting pairing later, not immediately while driving.

1. If a user selects the app before A2DP is connected the audio will start from the phone and transition to the IVI once connected. Does the app need to wait until the bluetooth connection is made before it starts streaming audio?

I think the app might be better to wait.

Overall I think we do need to do something better than what is there today but I don't think this is the right way to go about it.

I don't think the current UX that silently rejects connection at least if BT is not connected is not good.
The user should be prompted for a BT connection or, if possible, automatically.
What if you have a better idea?

[joeygrover]

@Kazuki-Sugimoto

3. I am going to have to ask for more explanation to this and some other answers. For example, just stating a message will be displayed to the user is not a satisfactory response. What does the message say? Where will be the message be displayed? What triggers the display of this message specifically? You also never addressed what happens to current systems.

4. See 3.

5. See 3.

6. So the app will be displayed on the HMI, and the user can activate the app but the sound will not be streamed to the IVI. How will the app ever know if the IVI system is still trying to connect bluetooth or if the feature has failed?

7. This is asking a lot from the developer and we would have to define specific requirements for SDL to make sure all apps follow the same behavior. Would the library have this ability or sample code for them? What does that look like?

Currently app developers can change this flag in their settings and display their own messages on the IVI that would tell the user to connect bluetooth and that requires no change to SDL.
For other solutions, why not just ask the user to connect bluetooth before displaying their apps on screen? Make it a requirement for AOA to work that a bluetooth connection must also be present. Or the HMI can display a message that includes info "For media/music applications to use SDL you must connect bluetooth" after an AOA connection? Have you looked into the phone requesting to pair and connect instead of the IVI system?

[theresalech]

The Steering Committee voted to keep this proposal in review, to allow the author time to respond to comments and for additional conversation on the review issue.

[Kazuki-Sugimoto]

@joeygrover -san

1. I am going to have to ask for more explanation to this and some other answers. For example, just stating a message will be displayed to the user is not a satisfactory response. What does the message say?

"BT connection required to launch Media app."

Where will be the message be displayed?

Display on the HU.

What triggers the display of this message specifically?

When an app is selected in the HMI app list (when BT is not connected and BT address is unknown).

You also never addressed what happens to current systems.

I think that there is no effect on cases other than media applications.

1. So the app will be displayed on the HMI, and the user can activate the app but the sound will not be streamed to the IVI. How will the app ever know if the IVI system is still trying to connect bluetooth or if the feature has failed?

It is assumed that the app will not be launched unless a BT connection is established.

This is asking a lot from the developer and we would have to define specific requirements for SDL to make sure all apps follow the same behavior. Would the library have this ability or sample code for them? What does that look like?

The expression "the app might be better to wait" in my answer was incorrect.
It does not do anything on the app side. I wanted to explain that the app does nothing until launched.

The BT connection is determined by the HMI when the app is selected from the app list.
Then the app is launched or the message is displayed.

Currently app developers can change this flag in their settings and display their own messages on the IVI that would tell the user to connect bluetooth and that requires no change to SDL.

I do not know this feature. Is it listed somewhere?

For other solutions, why not just ask the user to connect bluetooth before displaying their apps on screen?

Sorry for making you misunderstand. Same assumption.

Make it a requirement for AOA to work that a bluetooth connection must also be present.

Does that mean that USB and BT connections are required?
It might also be good, but it can be bad UX for users who don't need it.

Or the HMI can display a message that includes info "For media/music applications to use SDL you must connect bluetooth" after an AOA connection?

If this is displayed every time, it is also likely to be bad UX for users.

Have you looked into the phone requesting to pair and connect instead of the IVI system?

I don't understand the meaning of the question, how does it relate to this proposal?

[joeygrover]

3.

I think that there is no effect on cases other than media applications.

That's the entire issue though. Media apps need a way to work both with older implementations and this feature. If they only use this feature, the issue will regress that was fixed previously. So older systems, these media apps will connect over AOA with no way to connect bluetooth automatically; then the media would simply play out of the mobile device again. This has to be addressed.

8. I just noticed DeviceInfo was set to mandatory="true" as well. This is not necessary and should be changed back to mandatory="false".

Have you looked into the phone requesting to pair and connect instead of the IVI system?

I don't understand the meaning of the question, how does it relate to this proposal?

I brought this as a point because Android can programmatically pair/connect with the IVI, so if driver distraction is an issue, it might be possible to do this in the background without user interaction. This would make the feature better and less distracting.

---

I think it's important that all these details are provided in the proposal so OEMs know how this feature might be implemented. While the idea of the proposal is simple, the implications it has are much larger. So it would be helpful if you could provide a complete flow chart of what messages are sent when with the previously discussed situations.

From a high level I understand the feature to work as:

1. IVI connects with mobile device over AOA.
2. Application registers with requiresAudioSupport param set to true and bluetoothDeviceAddress attached to the DeviceInfo struct in the RAI.
3. If phone has previously paired with the IVI, IVI will attempt to automatically connect bluetooth. If it has not, the HMI will prompt the user to pair the mobile device when it is safe to do so.
4. If the device is not paired or connected over bluetooth to the IVI at the time of activating an app that had the requiresAudioSupport set true, the IVI will not put the app into HMI_FULL until that happens (paired and connected).

[Kazuki-Sugimoto]

@joeygrover -san

That's the entire issue though. Media apps need a way to work both with older implementations and this feature. If they only use this feature, the issue will regress that was fixed previously. So older systems, these media apps will connect over AOA with no way to connect bluetooth automatically; then the media would simply play out of the mobile device again. This has to be addressed.

Can we address this problem by adding capability?
If there is no capability that indicates whether there is a BT automatic connection function, perform the conventional movement (do not send RAI).

1. I just noticed DeviceInfo was set to mandatory="true" as well. This is not necessary and should be changed back to mandatory="false".

You're right.

I brought this as a point because Android can programmatically pair/connect with the IVI, so if driver distraction is an issue, it might be possible to do this in the background without user interaction. This would make the feature better and less distracting.

Does the pairing / connection include the first time (never connected)?
Is it an app that does it? OS?
I looked into it and found no information that seemed relevant. Is there a URL that can be helpful?

From a high level I understand the feature to work as:

There is no difference in your understanding of this feature.
As you suggested, I want to add a message flowchart (sequence diagram).

[mrapitis]

From the Ford perspective we have concerns regarding breaking fixes put in place for the resolved issue referenced below:
https://github.com/smartdevicelink/sdl_java_suite/issues/1056

In general, audio over AOA is not well supported across handsets with many implementations behaving differently.

The statement below would break backwards compatibility with existing production head units in the field:

However, with this proposal, the SDL app is always launched without depending on the connection status of BT A2DP

[joeygrover]

3. The issue is that the feature currently ignores the transport connection notification if there is no audio streaming (A2DP, Audio over APA, etc) available. This means the app does not register at all on the IVI system and no messages are sent. This works best for current production head units given the current specs.

The problem with adding a capability is that the app won't be notified of the IVI's capabilities until after the app has registered. This is due to the face the IVI system would have to know the app is a MEDIA app and then include the capability into the RAI response. This means on an a current production system the app will appear connected and then disappear without informing the user of why. App developers currently have the option to allow the MEDIA app to connect through AOA with no audio streaming available and prompt the user to connect such methods through their UI (Show, Alert, etc). I agree this solution is not great as it puts the burden on app developers and there is no standard messaging put in place for all MEDIA apps in this situation.

I believe current production IVI system owners will have to chime in on how they want to handle this feature and what they find acceptable in terms of UX.

9. (Numbering this for reference)
As of API level 19 the createBond() method is available on a BluetoothDevice object. See the Android documentation here. That should allow the phone to at least prompt the user to pair with the IVI programmatically. I might have been mistaken on this point that it could happen without user interaction, but I do believe now that it is still required so it likely does not make the situation much better as the feature currently stands.

10. (Numbering this for reference)
Ok, yes I think having the flow chart will help. Also, in the case a MEDIA app does not register with a bluetooth MAC address or the requiresAudioSupport flag set to true, would you expect Core to reject the RAI request?

[theresalech]

During the Steering Committee meeting, the Ford team expressed their concern about this proposal breaking their current implementation and suggested continuing the behavior currently in place. The author requested that the proposal be kept in review for another week, so they could review the last few comments on the issue and respond. The Steering Committee agreed to keep this proposal in review until the 2020-03-10 meeting.

[Kazuki-Sugimoto]

@mrapitis -san

However, with this proposal, the SDL app is always launched without depending on the connection status of BT A2DP

Sorry. The sentence we wrote was wrong, therefore misleading. It should be, "the SDL app is always registered", not launched.

By adding the capability that indicates the BT automatic connection function, the existing operation is maintained. Therefore, we assume that there will be no problem regarding that matter.

---

@joeygrover -san

1. The issue is that the feature currently ignores the transport connection notification if there is no audio streaming (A2DP, Audio over APA, etc) available. This means the app does not register at all on the IVI system and no messages are sent. This works best for current production head units given the current specs.

If the transport connection is ignored, users will not know which apps and for what reason those apps were not registered. We want to improve this as we have mentioned in our proposal.

The problem with adding a capability is that the app won't be notified of the IVI's capabilities until after the app has registered. This is due to the face the IVI system would have to know the app is a MEDIA app and then include the capability into the RAI response. This means on an a current production system the app will appear connected and then disappear without informing the user of why.

The core will refer to these following three items. First, the requiresAudioSupport that is to be added to the RAI request. Second, the BT connection status. And third, the capability that indicates BT automatic connection function.

So, when requiresAudioSupport is true, but there is no BT connection and the capability is false, the RAI request will be rejected. What do you think about this kind of behavior?

I believe current production IVI system owners will have to chime in on how they want to handle this feature and what they find acceptable in terms of UX

Does this mean that users can choose whether or not to automatically connect to BT?

1. As of API level 19 the createBond() method is available on a BluetoothDevice object. See the Android documentation here. That should allow the phone to at least prompt the user to pair with the IVI programmatically. I might have been mistaken on this point that it could happen without user interaction, but I do believe now that it is still required so it likely does not make the situation much better as the feature currently stands.

Thank you for providing information. I would like to refer to it as needed.

1. Ok, yes I think having the flow chart will help. Also, in the case a MEDIA app does not register with a bluetooth MAC address or the requiresAudioSupport flag set to true, would you expect Core to reject the RAI request?

In order to protect the existing operation, when the capability of BT automatic connection function (for now, we'll refer to it as AutoBTCapability) is false, under applicable conditions, it is expected to reject the RAI request, just like in the existing one.

The following cases are assumed:

• In case of RAI (Core's behavior)
• When requiresAudioSupport is true and there is a BT connection, RAI is successful and the app is registered.
• When both requiresAudioSupport and AutoBTCapability are true, but there is no BT connection, RAI is successful and the app is registered.
• When requiresAudioSupport is true, but AutoBTCapability is false and there is no BTconnection, RAI is rejected.

• When requiresAudioSupport is false, RAI is successful and the app is registered.

• In case of MEDIA app launch (HMI's behavior)

• When there is BT connection, the app is launched.
• When there is no BT connection but BT address is registered, automatically connect BT.
  2-1. When the BT automatic connection is successful, the app is launched.
  2-2. When the BT automatic connection fails, a message that prompts BT connection is displayed.
• When there is no BT connection and BT address is not registered, a message that prompts BT connection is displayed.

[Shohei-Kawano]

@mrapitis -san
We believe it very important to consider the impact on already released HUs.
Nexty team @Kazuki-Sugimoto has responded with comments on how to mitigate the impact.
We think this is a very important discussion, so please answer as soon as possible.
If we have an impact that we are unaware of, then giving us a specific impact will help us figure out how to avoid it.

Thank you.

[mrapitis]

@Shohei-Kawano @Kazuki-Sugimoto one concern around the AutoBTCapability flag would be that the app would need to set / maintain the value based on the head unit that its connected to. An existing head unit implementation would require AutoBTCapability set to false while potential future implementations could accept true. If this work could automatically be managed at the proxy level it may take some burden away from the app developer.

[joeygrover]

If the transport connection is ignored, users will not know which apps and for what reason those apps were not registered. We want to improve this as we have mentioned in our proposal.

For older IVI systems, if the apps connect over USB with no BT connection, they will register and then disappear, users will not know for what reason those apps were registered but then disappeared.

For older IVI systems, if the apps connect over USB, register, stay connected and then the user selects the app to be launched, the app will then play music out of the mobile phone speaker, users will not know for what reason SDL is not handling this.

These items MUST be addressed in this proposal. There needs to be a specific section of what the use case is when the new version of the proxy that includes this feature connects to an older version of Core that does not support this feature.

... one concern around the AutoBTCapability flag would be that the app would need to set / maintain the value based on the head unit that its connected to. An existing head unit implementation would require AutoBTCapability set to false while potential future implementations could accept true. If this work could automatically be managed at the proxy level it may take some burden away from the app developer.

The only way an app knows what head unit it is connected to is after the RAI response. This will cause the app to flash on the screen before being removed. This is worse than not showing the app in my opinion.

[theresalech]

Per the author's request, the Steering Committee voted to keep this proposal in review for an additional week, to allow the author time to respond to the questions posted to the review issue. It was emphasized that the author will need to respond to the major concerns of this feature breaking current implementations (see: this comment and this comment) before the next Steering Committee meeting. It was noted that it will be important to take action on this proposal next week, and if the reviewers' concerns cannot be addressed by the author, we may need to move to reject this proposal.

[Kazuki-Sugimoto]

For older IVI systems, if the apps connect over USB with no BT connection, they will register and then disappear, users will not know for what reason those apps were registered but then disappeared.
For older IVI systems, if the apps connect over USB, register, stay connected and then the user selects the app to be launched, the app will then play music out of the mobile phone speaker, users will not know for what reason SDL is not handling this.

The core version is determined based on the protocol version obtained in StartSession response. The audio requirement judgment that was performed before StartSession will be moved to before RAI, and the core that do not support this feature will not send RAI instead of ignoring the transport connection. I suggest this, what do you think?

These items MUST be addressed in this proposal. There needs to be a specific section of what the use case is when the new version of the proxy that includes this feature connects to an older version of Core that does not support this feature.

I will add the use case and flowcharts for this case as well.

[JackLivio]

The core version is determined based on the protocol version obtained in StartSession response.

Note that the version sent in the StartServiceAck is the negotiated protocol version, not the version of Core or the RPC spec. I believe tying this use case to a specific protocol version would be a major version change. This approach could work, but it means all headunits implementing SDL Core would have to support this feature for apps negotiating with the new protocol version.

[joeygrover]

Well Core version is directly tied to protocol version nor the other way around. While it can be mostly assumed, since they are individual projects with their own specific versioning using one to determine the other is not something that should be done. Instead, features themselves can be tied directly to the protocol version.

With that in mind, moving the check into the StartService protocol message for the RPC service is a potential path forward but does result in a very complicated flow.

Prerequisites:

• The protocol version will need to be incremented for a major change.
• Using the current library apps will not register at all without audio support. So it is assumed in this flow the app will only be using a compatible library.

1. The developer will set their requiresAudioSupport to true.
2. The library will send a StartService for the RPC service with a new param in the payload, requiresAudioSupport.
3. Core will receive the StartService for the RPC service:
   
   
   
   1. Core will check it's current audio connects (BT A2DP), if it has a supported audio connect it will continue in flow.
   
   
   2. If there is no Audio support, core will check if it supports the auto connect BT feature. If it does it will continue in flow.
   
   
   3. If there is no supported audio methods and Core does not support the auto connect BT feature, it will send a StartServiceNAK with the reason No audio support available.
   
   
4. If Core has continued, it will send a StartServiceACK with a new param autoBTCapability set to true.
5. The app receives the response to its StartService for the RPC service:
   1.If the response was a StartServiceNAK the app will shutdown.
   
   
   
   1. If the response was a StartServiceACK, requiresAudioSupport was set to true, but the protocol version of the ACK is less than the major version of this feature, the app will shutdown.
   
   
   2. If the response was a StartServiceACK, requiresAudioSupport was set to true, the protocol version of the ACK is equal or greater than the major version of this feature, and the autoBTCapability flag is set to false, the app will check if audio support is available using the MediaStreamingStatus class. If it is available it will continue, if not it will shutdown.
   
   
   3. If the response was a StartServiceACK, requiresAudioSupport was set to true, the protocol version of the ACK is equal or greater than the major version of this feature, and the autoBTCapability flag is set to true, the app will continue.
   
   
6. The app will send its RegisterAppInterface which will include the hmiTypes and isMediaApplication flag.
7. Core will receive the RegisterAppInterface:
   
   
   
   1. If the requiresAudioSupport flag was not included in the StartService, ~thehmiType of MEDIA~ isMediaApplication is set to true in the RAI and the protocol version for the session is less than the major version of the version that included this feature it will send a RegisterAppInterface response with success=false and deny the app's registration.
   
   
   2. If the requiresAudioSupport flag was not included in the StartService, ~the hmiType of MEDIA is included~ isMediaApplication is set to true in the RAI and the protocol version for the session is equal to or greater than the major version of the version that included this feature it will send a RegisterAppInterface response with success=true but not move forward with this proposals feature.
   
   
   3. If the requiresAudioSupport flag was set to true in the StartService, ~the hmiType of MEDIA is included~ isMediaApplication is set to true in the RAI and the protocol version for the session is equal to or greater than the major version of the version that included this feature it will send a RegisterAppInterface response with success=true and move forward with this proposals feature.
   
   

I might have made an error or two in the flow, but I believe that would be the requirements for a check that ensured old IVI systems still worked as well as preventing flashing of the app.

EDIT: Changed from using hmiType to using isMediaApplication flag.

[JackLivio]

@joeygrover Is there an advantage to requiring the MEDIA appHMIType instead of using the isMediaApplication in the RAI request?

[joeygrover]

@JackLivio actually no. that's a good point. I will update.

[Kazuki-Sugimoto]

@joeygrover -san
Thanks for the supplement.
Will the proposal be accepted if it is revised based on that plan?

[joeygrover]

@Kazuki-Sugimoto I can't guarantee it will be accepted. Other members will need to review the updated proposal to make sure it works for them. I would ask that @mrapitis review my comment and hopefully we can vote to "Return for Revisions". If that is the result of the vote and once the proposal is updated, it will be brought back into review again to make sure the feature as a whole works for all members and is agreed upon.

[mrapitis]

I have also reviewed the comment with the suggested flow, it does make sense and may be an adequate solution for the scenarios that were previously discussed. The flow seem fairly complicated, however I do not believe their is much of a way to simply. It would be beneficial if the proposal was updated to include a state diagram with the suggested flow as it may make it easier for reviewers to follow.

[theresalech]

The Steering Committee voted to return this proposal for revisions. The author will update the proposal to reflect the flow in this comment, and also include a state diagram of the flow.

[theresalech]

The author has updated this proposal to reflect the agreed upon revisions, and the revised proposal is now in review until June 9, 2020. To see the exact revisions made, please see this merged PR: https://github.com/smartdevicelink/sdl_evolution/pull/997.

[Kazuki-Sugimoto]

We recommend Alternatives considered in the updated proposal. Please consider it.

[JackLivio]

1. We see issues with the alternate proposal as it does not consider older libraries and their supported spec/protocol versions.

2. Is the HMI responsible for the auto bluetooth connection process instead of Core's bluetooth transport adapter? Are there any additional HMI RPCs that are required for this part of the proposal to work?

We would appreciate that part of the connection flow to be highlighted in the proposal and applicable flow charts.

1. If bluetooth is auto connected by the HMI/Core after the initial register app interface request, what happens to the apps connection using the initial transport?

[Kazuki-Sugimoto]

1. We see issues with the alternate proposal as it does not consider older libraries and their supported spec/protocol versions.

Support for older libraries and specs/protocol versions of the alternate proposal is equivalent to the main proposal. It is determined whether the version is supported by the presence or absence of parameters.

1. Is the HMI responsible for the auto bluetooth connection process over Core's bluetooth transport adapter? Are there any additional HMI RPCs that are required for this part of the proposal to work?

The auto bluetooth connection is a function of the HMI. Therefore, it is not necessary to add new RPCs.

We would appreciate that part of the connection flow to be highlighted in the proposal and applicable flow charts.

What exactly does that part mean? Is this a difference between the current specifications and this proposal? If not, is this the difference between the main and the alternative?

1. If bluetooth is auto connected by the HMI/Core after the initial register app interface request, what happens to the apps connection using the initial transport?

In this case the initial transport (USB) connection is maintained. The bluetooth connection is an A2DP connection intended for audio output.

[JackLivio]

1. Core might need to be notified by the HMI that the app was connected over A2DP. If the user attempts to activate the app before the HMI was able to connect to the mobile device's BT then audio will start playing out of the phone. Enforcing the BT connection before activation could be done in Core or by the HMI. I propose we add this rule to SDL Cores app activation process.

2. Ok thank you.

[joeygrover]

1. This is unfortunately not true. Every parameter in a control packet message is optional. Therefore, if an app does not send the parameter it is not enough information to determine the app should not be registered. If the parameter is not included it can be temporarily assumed the app does not use this feature up until the RAI is received. The protocol and RPC specs are defined to operate as standalone specifications so we must ensure we are true to them as well as how our implementation of those specs behaves. Because of this is, we must continue to use the RAI logic that is not present in the alternative solution to ensure the older versions of the spec and rules of the spec are honored correctly.

[Kazuki-Sugimoto]

1. Core might need to be notified by the HMI that the app was connected over A2DP. If the user attempts to activate the app before the HMI was able to connect to the mobile device's BT then audio will start playing out of the phone. Enforcing the BT connection before activation could be done in Core or by the HMI. I propose we add this rule to SDL Cores app activation process.

I recommend displaying a message on the HMI that a BT connection is required.

1. This is unfortunately not true. Every parameter in a control packet message is optional. Therefore, if an app does not send the parameter it is not enough information to determine the app should not be registered. If the parameter is not included it can be temporarily assumed the app does not use this feature up until the RAI is received. The protocol and RPC specs are defined to operate as standalone specifications so we must ensure we are true to them as well as how our implementation of those specs behaves. Because of this is, we must continue to use the RAI logic that is not present in the alternative solution to ensure the older versions of the spec and rules of the spec are honored correctly.

I'm not sure if it matches your intention, if requireAudioSuport is not explicitly set, it is true for Media apps, false otherwise (same as existing). See StartService item in Add parameters paragraph.

[theresalech]

During the 2020-06-09 meeting, the Steering Committee voted to keep this proposal in review, so the author could review and respond to the comments from the Project Maintainer.

[joeygrover]

1. I have read the proposal and understand where you are trying to go with the alternate solution, but as previously stated, the BSON payload params are all optional and can't be used specifically determine if something is supported or not only by their inclusion. In this specific instance we add a new parameter that will cause a breaking change. Therefore, previous versions of the protocol must be accounted for. Because we agreed we need to avoid the case where media apps connect when there is no audio we assume that in previous versions of the protocol that this param would be set to true. However, because we don't want to automatically deny all applications if this feature is not present on the IVI, we use the RAI and more specifically isMedia to determine the logic of what to do next. The IVI system will deny registration to media apps connected with protocol versions less than the one where this new feature and param were introduced. The only way to properly do this is with the original flow. The alternate flow does not take this into account and therefore can't be used as the primary solution.

4. The proposal should also mention how the MediaStreamingStatus should be used in the Java Suite library. If the the library supports the newer version of the protocol spec, but the IVI system is using a lower version, the app should still use this class before attempting to register. This will require some refactoring of the library to move the logic for the MediaStreamingStatus class. If this isn't provided and the class just removed, in production IVI systems will have media apps that register without audio connections again and regress to the problem previously described through the reviews of this proposal.

[stefanek22]

Echoing @joeygrover in that we need to see a flow/solution that takes the legacy systems (i.e. older protocol versions that do not have this explicitly defined) into account and will not introduce issues for them...

[JackLivio]

1. For clarity in the proposal, can a note be added that notes the HMI will be responsible for preventing app activation while the HMI is in the process of connecting to the mobile devices bluetooth for an app that has requiresAudioSupport=true. For example the HMI could show a loading icon for that app, or display an error message to the user upon activation.

[theresalech]

The Project Maintainer and Ford agreed that the Proposed solution described in the proposal is preferable, and expressed concerns with the solution described in the Alternatives considered section, as it does not account for previous protocol versions. The Project Maintainer also requested the following revisions: describe MediaStreamingStatus as requested in item 4 of this comment, and adding clarifying language that the HMI will be responsible for preventing app activation as described in item 2 of this comment. Per the author's request, the Steering Committee voted to keep this proposal in review for an additional week, so the author could discuss the feedback internally and respond on the review issue.

[Kazuki-Sugimoto]

1.
I was aware that if the version of App or Core does not support this function, it is expected to register (and not use this function) if it is a Media app according to the conventional flow. But isn't it expected behavior in this case? If so, the alternative flow is certainly incorrect.

4.
Add a description about how to use MediaStreamingStatus.

2.
Make it clear that the HMI is responsible for preventing app activation.

[theresalech]

It was acknowledged that the proposed solution is preferred over that presented in the Alternatives considered section. The Steering Committee voted to return this proposal for the following revisions: add a description about how to use MediaStreamingStatus as requested in this comment, and make it clear that the HMI is responsible for preventing app activation as described in this comment.

[theresalech]

The author has updated this proposal to reflect the agreed upon revisions, and the revised proposal is now in review until July 28, 2020. To see the exact revisions made, please see this merged PR: https://github.com/smartdevicelink/sdl_evolution/pull/1043.

[JackLivio]

I believe there are some behavior details that need clarification in the proposal.

5.

This image suggests that if the protocol version is older than this feature, the app should terminate registration.

To the details in review item #4, I believe the app needs to check the MediaStreamingStatus. Can the flow chart be updated to reflect this requirement?

1. If the requiresAudioSupport flag was not included in the StartService, isMediaApplication is set to true in the RegisterAppInterface and the protocol version for the session is less than the major version of the version that included this feature, it will send a RegisterAppInterface response with success=false and deny the app's registration.

This seems to suggest that all legacy media apps will be denied registration by Core. Should this state that legacy media apps without an audio connection should be denied registration?

[theresalech]

7.
Since this proposal includes RPC and Protocol changes, JavaScript Suite should also be added as an impacted platform.

[Akihiro-Miyazaki]

@JackLivio -san
5.

To the details in review item #4, I believe the app needs to check the MediaStreamingStatus. Can the flow chart be updated to reflect this requirement?

I modified the flow like below.
Does this modification match with your suggestion?

6.

This seems to suggest that all legacy media apps will be denied registration by Core. Should this state that legacy media apps without an audio connection should be denied registration?

OK.
I will modify the sentence to "it will send a RegisterAppInterface response with success=false and legacy media apps without an audio connection should be denied registration".
Does this modification match with your suggestion?

@theresalech -san
7.

Since this proposal includes RPC and Protocol changes, JavaScript Suite should also be added as an impacted platform.

OK.
I will add "JavaScript Suite" to "Impact on existing code" and "Impacted Platforms".

[JackLivio]

1. Thank you the update looks correct. Could you also preface that the flowchart is only for Android devices?

2. Similar to 5. The enforcement for this rule should only take place for Android devices. We could either check the os parameter in the device info struct, or we could check Core's transportType enum that the device is most likely an Android device (SPP_BLUETOOTH, AOA_USB).

[JackLivio]

1. 👍

2. Because the proposal solves an issue that only applies to Android devices and there are RPC and protocol changes required, the proposal should specify how IOS and Javascript suite should behave. Should these platforms omit the requiresAudioSupport parameter? Should IOS always set this parameter to true since IAP always supports audio?

[Akihiro-Miyazaki]

@JackLivio -san
5.

Could you also preface that the flowchart is only for Android devices?

Yes, that's right.
Because this proposal is basically for Android OS's correspondence against not to support AOA2.0.

6.
Thank you for your advice.
As you say, other system excepted Android is performed in the current sequence.
Therefore, the SDL Core gets the transportType during Protocol spec, then, if transportType is SPP_BLUETOOTH or AOA_USB, the system can perform the proposal's sequence.
I modified the flow like below.
Could you please confirm it.

8.

Should these platforms omit the requiresAudioSupport parameter? Should IOS always set this parameter to true since IAP always supports audio?

In the case of other platform, the system can output the audio via USB.
Therefore, other platform can omit the requiresAudioSupport parameter.

[theresalech]

The Steering Committee voted to keep this proposal in review, to allow the author time to respond to the comments on the review issue.

[JackLivio]

1. 👍

2. Agree to the chart updates. One more question, how does SDL Core verify that the BT a2dp profile is connected?

[JackLivio]

1. 👍

[Akihiro-Miyazaki]

@JackLivio -san,
6.
Since the SDL Core does not have bluetooth feature, the SDL Core will confirm the BT A2DP connection by the notification from the HMI.

[JackLivio]

1. OK we will have to define a HMI API notification to use for this behavior. Looking at what is existing in the HMI_API there is this notification that might be worth expanding.

    <function name="OnDeviceStateChanged" messagetype="notification" scope="internal">
      <param name="deviceState" type="Common.DeviceState" mandatory="true" />
      <param name="deviceInternalId" type="String" mandatory="true" minlength="0" maxlength="500" />
      <param name="deviceId" type="Common.DeviceInfo" mandatory="false"/>
    </function>

I am not 100% sure about how this notification is used currently but it looks to be a way for the HMI to say a device was unpaired? (UNKNOWN and UNPAIRED are the only defined DeviceState values).

[theresalech]

The Steering Committee voted to keep this proposal in review, to allow the author time to review and respond to the last comment from the Project Maintainer.

[Akihiro-Miyazaki]

@JackLivio -san,
Thank you for your advice.
We confirmed that this RPC(OnDeviceStateChanged) is sent for the notification which the Bluetooth connection is unavailable during the system connects via Bluetooth.
Therefore we would like to propose to modify like below:
(1) This RPC will send not only when the bluetooth connection is unavailable, but also when it will use.
(2) Add the parameters like below:

  <struct name="DeviceInfo">
      <param name="name" type="String" mandatory="true">
          <description>The name of the device connected.</description>
      </param>
      <param name="id" type="String" mandatory="true">
          <description>The ID of the device connectedi: either hash of device's USB serial number(in case of USB connection) or has of device's MAC address(in case of BlueTooth or WIFI connection</description>
      </param>
      <param name="transportType" type="Common.TransportType" mandatory="false">
         <description>The transport type the named-app's-device is connected over HU(BlueTooth, USB or WiFi). It must be provided in OnAppRegistered and in UpdateDeviceList</description>
      </param>
      <param name="isSDLAllowed" type="Boolean" mandatory="false">
          <description>Sent by SDL in UpdateDeviceList. 'true' - if device is allowed for PolicyTable Exchange; 'false' - if device is NOT allowed for PolicyTable Exchange </description>
      </param>
+     <param name="bluetoothDeviceAddress" type="String" mandatory="false">
+         <description>Device BT Address</description>
+     </param>
+     <param name="a2dpConnectionState" type="Boolean" mandatory="false">
+         <description>Notify the state of A2DP connection. 'true' - Connected at BT A2DP; 'false' - Connected at BT</description>
+     </param>
  </struct>

  <enum name="DeviceState">
      <element name="UNKNOWN"/>
      <element name="UNPAIRED"/>
+     <element name="PAIRED"/>
  </enum>

[JackLivio]

Thank you for the suggestion. Because DeviceInfo is widely used by other rpcs i think the new bluetooth parameters should be contained in a new struct, maybe named "BluetoothInfo"?

  <struct name="BluetoothInfo">
     <param name="bluetoothDeviceAddress" type="String" mandatory="false">
         <description>Device BT Address</description>
     </param>
     <param name="a2dpConnectionState" type="Boolean" mandatory="false">
         <description>Notify the state of A2DP connection. 'true' - Connected at BT A2DP; 'false' - Connected at BT</description>
     </param>
  </struct>

    <function name="OnDeviceStateChanged" messagetype="notification" scope="internal">
      <param name="deviceState" type="Common.DeviceState" mandatory="true" />
      <param name="deviceInternalId" type="String" mandatory="true" minlength="0" maxlength="500" />
      <param name="deviceId" type="Common.DeviceInfo" mandatory="false"/>
+    <param name="bluetoothInfo" type="Common.BluetoothInfo" mandatory="false"/>
    </function>

[Akihiro-Miyazaki]

@JackLivio -san,
Thank you for your response.
We agree your suggestion.
By the way, we would like to confirm to add element "PAIRED" in the deviceState enum and to send the RPC "OnDeviceStateChanged" not only when the bluetooth connection is unavailable, but also when it will use.
Threfore, we will modifiy the below below;

  <function name="OnDeviceStateChanged" messagetype="notification" scope="internal">
      <param name="deviceState" type="Common.DeviceState" mandatory="true" />
      <param name="deviceInternalId" type="String" mandatory="true" minlength="0" maxlength="500" />
      <param name="deviceId" type="Common.DeviceInfo" mandatory="false"/>
+     <param name="bluetoothInfo" type="Common.BluetoothInfo" mandatory="false"/>
  </function>

  <enum name="DeviceState">
      <element name="UNKNOWN"/>
      <element name="UNPAIRED"/>
+     <element name="PAIRED"/>
  </enum>

+ <struct name="BluetoothInfo">
+     <param name="bluetoothDeviceAddress" type="String" mandatory="false">
+         <description>Device BT Address</description>
+     </param>
+     <param name="a2dpConnectionState" type="Boolean" mandatory="false">
+         <description>Notify the state of A2DP connection. 'true' - Connected at BT A2DP; 'false' - Connected at BT</description>
+     </param>
+ </struct>

Do you agree them?

[JackLivio]

Yes that makes sense to me. We should define some behaviors if Core receives the PAIRED state.

If PAIRED state is received by SDL Core, SDL Core should:

• Call PolicyHandler::AddDevice to add the device to the policy table
• Trigger a bluetooth SDP scan of SDL Apps available on that device (transport level)

I am open to other suggestions.

[theresalech]

Here's a summary of the revisions that have been agreed upon by the author and PM:

5. Update to use flow chart in this comment with preface that this is only for Android devices

6. Update to use flow chart in this comment. Also, add BluetoothInfo struct to OnDeviceStateChanged, with bluetoothDeviceAddress and a2dpConnectionState params as described in this comment, and add PAIRED element to DeviceState enum. The proposal should define the following behaviors if Core receives the PAIRED state:

• Call PolicyHandler::AddDevice to add the device to the policy table
• Trigger a bluetooth SDP scan of SDL Apps available on that device (transport level)

7. Add "JavaScript Suite" to Impacted Platforms section.

8. Update Proposed Solution section to specify that iOS and JavaScript Suite should omit the requiresAudioSupport parameter.

[theresalech]

The Steering Committee voted to return this proposal for the revisions included in this comment.

[theresalech]

The author has updated this proposal based on the Steering Committee's agreed upon revisions, and the revised proposal is now in review until September 29, 2020.

The specific items that were updated since the last review can be found in this merged pull request: https://github.com/smartdevicelink/sdl_evolution/pull/1075.

[joeygrover]

_Note: Starting numbering over for the revised proposal._

1.

If the response was a StartServiceACK, requiresAudioSupport was set to true, but the protocol version of the ACK is less than the major version of this feature, the app will shut down.

This needs to include the note about the MediaStreamingStatus class and its usage. For example, if the protocol version is lower, the library could and should check if audio is already connected. Otherwise, apps that require audio support wouldn't connect to anything currently in production. The flow chart will likely also have to be updated to reflect this change.

2.

By using the value of isAudioOutputAvailable() method on MediaStreamingStatus class, the Java Suite library can make a decision whether BT A2DP is connected like below:

The MediaStreamingStatus class actually checks for multiple Audio Output options. I just want to make sure that's clear.

3.

With the changes of the flow, it is necessary to do refactoring of the Java Suite library to move the logic of the MediaStreamingStatus class from the preprocessing of the StartService to the preprocessing of the RegisterAppInterface.

I think this is better stated as post-processing of StartService ACK or NAK.

4. I don't see any description of how the code will be refactored as mentioned, but with the upcoming release a lot of these classes are not accessible to developers so it can be considered implementation details that the project maintainer will have have discretion over unless any other members would like to weigh in beforehand.

All these points are fairly minor and I think overall we could move to accept with those revisions if everything is agreed upon here.

[Akihiro-Miyazaki]

@joeygrover -san,
1.
We understood.
We will modify the sequence like below.
Does this modification match your advice?

2.
That's right.
In this proposal, we will use the value of isAudioOutputAvailable() method which is the one of functions on MediaStreamingStatus class.

3.
OK.
We will modify to replace the sentense from "from the preprocessing of the StartService to the preprocessing of the RegisterAppInterface" to "to the post-processing of StartService ACK or NAK".
Does this modification match your advice?

[theresalech]

@Akihiro-Miyazaki, thanks for your responses! It appears we are in agreement on items 1 - 3. If you could please confirm your agreement with item 4 as well, we can ask the Steering Committee to vote on accepting this proposal with these 4 revisions during tomorrow's meeting.

[Akihiro-Miyazaki]

@joeygrover -san, @theresalech -san,
4.
Since there is no requirement about the implementation, we agree your suggestion.
We would like to depend on the project maintainer about the implementation.
Thank you in advance for your cooperation.

[theresalech]

The Steering Committee voted to accept this proposal with the following revisions:

1. Modify flow chart to incorporate sequence in this comment and revise the section below to include the note about the MediaStreamingStatus class and its usage. For example, if the protocol version is lower, the library could and should check if audio is already connected. Otherwise, apps that require audio support wouldn't connect to anything currently in production.
   >If the response was a StartServiceACK, requiresAudioSupport was set to true, but the protocol version of the ACK is less than the major version of this feature, the app will shut down.
2. Specify in MediaStreamingStatus class section of proposal, that feature will use value of isAudioOutputAvailable() method, since the MediaStreamingStatus class actually checks for multiple Audio Output options.
3. In MediaStreamingStatus class section, change “…to the preprocessing of the RegisterAppInterface" to "…to the post-processing of StartService ACK or NAK".
4. Specify in proposal that since there aren't any public code changes listed in the proposal, the SDLC Project Maintainer will have discretion over implementation details, including changes to classes that are not accessible to developers, especially given changes to Java Suite library in 5.0 release.

[theresalech]

@Akihiro-Miyazaki , please advise when a new PR has been entered to update the proposal to reflect the agreed upon revisions. I'll then merge the PR so the proposal is up to date, and enter issues in impacted repositories for implementation. Thanks!

[theresalech]

Proposal has been updated and implementation issues have been entered:

• iOS
• Java Suite
• JavaScript Suite
• Core
• SDL HMI
• Generic HMI
• RPC
• Protocol