Hello SDL community,
The review of the revised proposal "SDL 0250 - Next RPC Indication for the HMI" begins now and runs through January 14, 2020. Previous reviews of the proposal took place October 2 - 15, 2019 and October 30 - November 19, 2019. The original review of this proposal took place October 2 - 15, 2019. The proposal is available here:
https://github.com/smartdevicelink/sdl_evolution/blob/master/proposals/0250-NextRpcIndication.md
Reviews are an important part of the SDL evolution process. All reviews should be sent to the associated Github issue at:
https://github.com/smartdevicelink/sdl_evolution/issues/827
What goes into a review?
The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of SDL. When writing your review, here are some questions you might want to answer in your review:
More information about the SDL evolution process is available at
https://github.com/smartdevicelink/sdl_evolution/blob/master/process.md
Thank you,
Theresa Lech
Program Manager - Livio
[email protected]
@MichaelCrimando Are you familiar with this accepted but not yet implemented proposal: Template Improvements: Additional SubMenus?
The additional submenus proposal allows for deeper submenus so the developer doesn't have to simulate menu depth by using multiple perform interactions.
Also there is a proposal that is implemented in SDLCore 6.0.0 that allows for in app menu tiles views.
I believe the pictured flow in the proposal could be accomplished using additional submenu depth and menu tiles without the need for the HMI anticipating future RPCs. Also the performance of using submenus over perform interactions would be much better and not require any loading screen.
Hey @JackLivio
That is a great proposal and very similar but executed in a different way. That is great for apps that want to PRE load all their choice sets. The proposal that I've created is great for apps that want to POST load all their choice sets. Plus the proposal here also applies to things like softbuttons given from a Show RPC, ScrollableMessage or PerformInteraction
So reading the proposal the only use case that is really given is for a "Loading" screen to be displayed while waiting to receive an RPC of a certain function ID.
First to Jack's point, submenus do not have to be PRE loaded. They can be added at any time and I would expect that some of them are added POST load because of the new proposal. PRE loading all those choices will take forever if VR synonyms are attached, so it is likely that the sub menus are added as a skeleton and apps add to the menus as they are selected which mimics the chaining of choice set functionality. The choice set chaning that happens was an apps way to get around a lot of initial driver distraction requirements and the flow itself is very clunky. Obviously this doesn't address the soft buttons or ScrollableMessage use case though.
Regarding the solution itself, I think it still needs some work. The next function ID check works in best case scenarios, but what if multiple Shows are sent. For example, the manager layer will check if an image will need to be uploaded to Core before a Show is sent with that image name. If it does, it sends a Show without the image field to start, then once the file is uploaded, sends another show. Is it acceptable for the loading screen or wait dialog to be dismissed on the first show? Has the author through about other information that could be included for a more precise check? For example, message ID, correlation ID, etc?
For alternatives, what about being more accurate with the use case given of showing a loading screen? Possibly a new RPC that triggers a loading screen, and then uses the dismiss ability recently introduced. I really want to make sure we understand the use case to craft the right solution. Because another alternative could be to include the full RPC to process in sort of a SelectionListener if we want to speed things up, but again if it's only to show a loading screen that's overkill.
Furthermore, has the author looked into the implications this will have on the manager layer for the mobile libraries? Changes should be included in this proposal so it can be agreed upon if the proposal is accepted.
In addition to what joey said, there are multiple places in the managers (file, soft button, text and graphic, menu, and especially choice set) that have asynchronous calls and that will pre-load menus for example without images, then automatically resend when the images are uploaded. Just something to keep in mind that RPCs may be sent asynchronously and automatically without the developer ever knowing.
Hey @JoeyGrover,
Those are some great points.
Going along with that, if you post load everything, you dont get the indicators (usually arrows) that something is next. Now if I understand what @BrettyWhite is saying, that could maybe be handled by Core where it knows something is coming next show it shows the indicator but doesn't bother to load anything yet.
Plus this functionality enables specfic loading screens that can be popupluated as the items load in. If Core wanted to get fancy, it could potentially load in only the first 10 items really fast, and work on the next items in the background. This is similar to how iPhone apps load long lists while keeping performance great.
Further, we have a need for the HMI to start pre-exposing things in the HMI before the user actually interacts with anything. If the HMI knows what comes next, and it's the type of RPC it's looking for, it can know what to expect and simulate the user pressing a certain button on the screen in order to get the required info from the app.
Here's one example from Tidal. The first item in the in-app menu for Tidal is "My Mixes"
Now a benefit that the additional Submenus proposal can provide is that it can improve performance in going back and forth in menu structures. However, apps would have to take the time to reorganize their menu structure from PerformInteraction style to Menu style. In addition, addCommand items can only have 1 text field and 1 image. Choice items can have 3 text fields and 2 images, so apps might not want to move from PerformInteraction styles. Also I don't know if this is handled on the Core side, but submenus are completely locked out while driving on SYNC3 for MY16-19 vehicles. Apps would then have to maintain 2 different styles of menu structure depending on what version of SYNC it's on.
The motivation of this proposal is trying to address three issues that exist today:
Better HMI possibilities: can show forward and backward arrows ... which is a feature of sub menus by the nature of sub menus. Choices could benefit from the feature but actually SDL-0148 made sub menus compete with choice sets... Seems like the proposal prioritizes sub menus as it says that on older head units it can automatically translate those sub-submenus into Choice SetsSmart Driver Distraction: if a softbutton or choice would lead to a screen that's normally locked ... this feature doesn't exist yet for choices and the proposal would enable us to do so. It enables the system to detect use cases with many steps. For sub menus it also exist today assuming pre-loading by the app.The HMI can have a much smoother experience with native HMI loading screens which is true for choice sets when initializing the app. Also it helps the "loading"-screen practiceI think the motivation of this proposal is that head units are slow especially with voice and that user interaction can only start once all the content is loaded. Knowing that the use case continues with a choice/menu conmand is very helpful to improve the experience. The proposal says that the HMI could show some loading indication. It's very helpful for the proposal understanding but I want to note that it'll be the SDL integrator responsibility on how the HMI should do this and that it should be out of scope. For instance when I tap on a choice which has a next RPC indication I could think of the HMI to make the choice change to show a "loading circle" as the choice image... again this is the decision of the HMI.
@MichaelCrimando already mentioned background loading which I believe is where we should have some focus. As of right now apps don't know what menu level or what menu entries are visible (except the root level with OnHMIStatus.systemContext = MENU). I think the method of table views in iOS would be a really big benefit to SDL app menus. AddSubMenu should have a parameter to tell how many entries this sub menu will contain but they entries don't need to be created. The HMI will do it's UI and tell the app if the menu is visible and what menu entries the app needs to create (e.g. extending OnHMIStatus with menu id).
The same could be done for PerformInteraction which would already start and show only choices and choice images that are visible. I believe this would require a larger API change.
Regarding the parameter nextFunctionInfo I am wondering about what should happen if the app fails to perform the next function. I'm thinking of specifying a timeout as an HMI requirement. Also I think it might be sufficient if choices would just provide an indication as a flag/bool. Then the HMI would wait for any kind of overlay RPC from the app (another PerformInteraction or Alert, ScrollableMessage, Slider etc.).
PRE loading all those choices will take forever if VR synonyms are attached, so it is likely that the sub menus are added as a skeleton and apps add to the menus as they are selected which mimics the chaining of choice set functionality.
Currently this is not possible. There is no callback when a submenu is pressed; therefore, the entire menu structure has to be preloaded IIRC. This is a major deficiency of the current menu setup.
Has the author through about other information that could be included for a more precise check? For example, message ID, correlation ID, etc?
I don't think that either messageID nor correlationID would work very well in this scenario. The manager layer handles these ids, so a complicated system of "reserving" an ID for a particular message and then the developer indicating that some screen manager call is meant for a "reserved" ID would have to take place, and this would be extremely complicated.
For alternatives, what about being more accurate with the use case given of showing a loading screen? Possibly a new RPC that triggers a loading screen, and then uses the dismiss ability recently introduced.
This already exists within Alert (the progressIndicator boolean parameter), and is probably a better solution, but doesn't handle showing the "arrow" functionality.
Going along with that, if you post load everything, you dont get the indicators (usually arrows) that something is next.
This could be added to Choice and SoftButton through a boolean parameter, but Choices should probably move over to menus for cases where they present another PerformInteraction if possible.
Plus this functionality enables specfic loading screens that can be popupluated as the items load in. If Core wanted to get fancy, it could potentially load in only the first 10 items really fast, and work on the next items in the background. This is similar to how iPhone apps load long lists while keeping performance great.
This is not currently possible given the scheme of PerformInteraction. Everything must be pre-loaded, ironically. This would be more possible if we created additional RPCs to enable it, but it's currently not possible.
Further, we have a need for the HMI to start pre-exposing things in the HMI before the user actually interacts with anything. If the HMI knows what comes next, and it's the type of RPC it's looking for, it can know what to expect and simulate the user pressing a certain button on the screen in order to get the required info from the app.
This is unclear to me, could you elaborate? The diagram is unclear.
In addition, addCommand items can only have 1 text field and 1 image. Choice items can have 3 text fields and 2 images, so apps might not want to move from PerformInteraction styles.
This is true. We should expand AddCommand to enable further functionality if possible.
Also I don't know if this is handled on the Core side, but submenus are completely locked out while driving on SYNC3 for MY16-19 vehicles. Apps would then have to maintain 2 different styles of menu structure depending on what version of SYNC it's on.
I think you're missing the point here. The fact that submenus are locked out while driving doesn't mean that developers should circumvent that lock out by using PerformInteraction. In fact, PerformInteraction depth should be similarly locked out. However, your point is good in terms of the app would have to maintain both if they were using beyond two menu depth (unless we worked around it in the manager layer). But, that doesn't help this proposal at all either.
As of right now apps don't know what menu level or what menu entries are visible (except the root level with OnHMIStatus.systemContext = MENU). I think the method of table views in iOS would be a really big benefit to SDL app menus.
Agreed, I've had a task sitting in my backlog for a few months now to think about this. Your ideas here are pretty good. I think that this task (of making menus load more dynamically) is more worthwhile than this proposal in general.
I think that the only functionality this proposal offers is two-fold:
First, to "disable" some soft buttons when driver distraction says that those buttons' effects will be locked out. This is a decent improvement, but I'm not sure that it's worth the effort.
Second, to show "better" loading screens, i.e. instead of an Alert with progressIndicator=true, they could show the PerformInteraction or ScrollableMessage without the data and a loading indicator, then load it in. I think that this could be better accomplished by the dynamic loading of PerformInteraction and AddCommand / AddSubmenu.
@joeljfischer just to elaborate on a couple of points
Currently with Tidal, the in-app menu lists "My Mix, My Albums, My Playlists, My Tracks, and Explore"
So on 8" SYNC it goes something like this - Home screen -> Menu -> Perform Interaction



So what the HMI on a future headunit with a larger screen size could do with this proposal is look at the in-app menu and see that the first menu item will lead to a performInteraction (or submenu... but that part isn't new). It can send an onCommand to the app, and have the upcoming performInteraction hang out in the side bar like so: (ignoring performInteraction timeout)

Further, the HMI could do something effectively like "Hey the first item in the menu will launch a scrollableMessage, so skip that one. However the second item in the menu will launch a performInteraction so send OnCommand for that"
Just a side note I have: you could use a bool flag for this functionality, however the proposal give you that same information (is a RPC next?) and more (WHICH RPC is next?).
Submenu is locked out completely on MY16-19 on SYNC vehicles which makes it pretty useless in the car vs PerformInteraction is page limited. I actually did the proper work to make sure that submenu is page limited like PerformInteraction is so apps can actually use it.
A major point of discussion during the Steering Committee meeting with regards to this proposal was that it suggests using PerformInteraction in a way that was not intended. It was clarified that the original intention of PerformInteraction was for very dynamic pieces of data that need to be loaded at run time, like POI searches. The author noted that application developers are currently using PerformInteraction to simulate menu depth, and therefore this proposal improves that experience, but it was then questioned by SDLC Members and the Project Maintainer if we should instead focus on improving menus so developers can use those as intended. While there was discussion about this proposal as a short term solution, it was noted that this would likely instead result in two different solutions that both need to be maintained, and could be confusing to developers.
Ultimately, the Steering Committee voted to defer this proposal, and keep it in review until our next meeting. SDLC Members and the Project Maintainer can continue discussion on the review issue, focusing on the motivations for the proposal outside of menu replacement items.
I believe I addressed all the main comments with PR #843
@MichaelCrimando thanks! As this proposal is currently [In Review], we'll need to let the Steering Committee continue to review until our meeting next week. If during the meeting, they agree to return for revisions, we will document what those revisions need to be, and can then bring your revised proposal (PR 843) in for review the following week once we've ensured those revisions are incorporated.
@theresalech ah ok!
If it helps at all, I didn't update any functional pieces of it, just the motivation around it for future documentation
The Steering Committee voted to return this proposal for revisions. The author will update the proposal to update the documentation and motivation specifying that add command/sub menu structure is the main structure for apps.
The author has revised this proposal based on the Steering Committee's feedback, and the revised proposal is now in review until 2019-11-05.
@MichaelCrimando I think the revision puts the use cases for the feature in a much better context.
I have a comment regarding the HMI API implementation. The HMI does not have or use a record of mobile function IDs. When an HMI receives an RPC it receives it as "InterfaceName.RPCType" ie (BasicCommunication.UpdateAppList)
I think nextFunctionID should be renamed to nextRPC and changed to type string in the HMI API.
HMI_API.xml:
<struct name="NextFunctionInfo" since="x.x">
<description>
Outlines information about the next RPC that will be triggered.
</description>
<param name="nextRPC" type="String" mandatory="true"/>
<description>The next function (RPC) that will be triggered by selecting the current option/command/choice etc.</description>
</param>
...
Core will be able to take an integer function ID and convert it into the correct HMI RPC String.
Thanks for the info @JackLivio
Now should we add a bit of that info into the description of the param?
Like
<description>The next function (RPC) that was translated from InterfaceName.RPCType that will be triggered by selecting the current option/command/choice etc.</description>
</param>
The updates to the motivation are very helpful and I agree with Jack that they put the elements and purposes of the proposal in a much better context.
I don't like the parameter name nextInitialText. I think instead it should be title or nextTitle if we're going to just tell the system what the next "initialText" or "title" is. Most times that may map to initialText, but I don't think that name is descriptive, and I don't want us to be required to use that nomenclature in the future.
We could also name it loadingText or something like that and document that it should be the title of the next screen if appropriate otherwise it would be generic loading text.
I agree, the use cases are now better defined and can defend the position of adding this feature.
3. Second the param name change. It is misleading, and I vote for loadingText as it aligns more with the action being taken.
4. It's a new parameter so this would not affect older SDL versions.
This would have an impact on existing code even though it's a new parameter. For example there are currently multiple managers on the app side that handle things like soft buttons, add commands, etc that would have to be updated to handle this additional functionality. It is not initially clear how that would be handled and should be defined in the proposal.
@joeygrover can you elaborate a bit? As i understand it, the mobile library would be updated eventually to enable an app to use this. At that point the mobile library sends JSON to Core and this would be just an additional parameter in JSON that older Core wouldn't be looking for so it'd be ignored. Or do you need me to list the specific managers in the mobile library that would need to be updated?
I kind of mentioned the specific things that would NEED to be updated Choice, AddCommand and SoftButton so we're not being too specific and you can update the associated managers as you see fit
Yea I'm not so much talking about the JSON modifications, but the actual API layer that developers use will have to be modified to support this. Take a look at this guide as you can see this is how developers use the AddCommand at the manager layer. It is abstracted into a more digestible structure. However, in order for developers to use this new feature, they will need a way to do so.
We don't want to get into a situation where the PM has to make a choice about how to do this because it wasn't defined in the proposal and then the author or any other member decides the way in which we implemented it was wrong or they're unhappy with it. The more defined we are at this stage prevents us from those types of conflicts.
The Steering Committee voted to keep this proposal in review until our next meeting, to allow the author and Project Maintainer to continue the conversation on the review issue, especially with regards to the impact on proxy code at the manager layer.
Ok we're going to workshop this with Kujtim when he's in Dearborn on tuesday
We're thinking something like ChoiceCell would have something like hasSubChoices
Well it has to affect more than ChoiceCell and there are multiple options, so a simple boolean probably won't work well / be scalable. I'd recommend something like nextScreen with an enum like choices, scrollable message, alert, main template (assume Show), etc.
Ok so here's roughly what me and @kshala-ford are thinking:
First, Add a new enum thats literally a translation of the FunctionID enum but with more user friendly names
<enum name="Function" since="X.X">
<description>A list of possible Functions to be used with something like </description>
<element name="Menu" value="5" hexvalue="5" since="1.0" />
<element name="SubMenu" value="7" hexvalue="7" since="1.0" />
<element name="PerformInteraction" value="10" hexvalue="A" since="1.0" />
<element name="Alert" value="12" hexvalue="C" since="1.0" />
<element name="Show" value="13" hexvalue="D" since="1.0" />
<element name="Speak" value="14" hexvalue="E" since="1.0" />
<element name="SetMediaClockTimer" value="15" hexvalue="F" since="1.0" />
<element name="PerformAudioPassThru" value="16" hexvalue="10" since="2.0" />
<element name="ScrollableMessage" value="25" hexvalue="19" since="2.0" />
<element name="Slider" value="26" hexvalue="1A" since="2.0" />
<element name="SendLocation" value="39" hexvalue="27" since="3.0" />
<element name="DialNumber" value="40" hexvalue="28" since="3.0" />
<element name="CancelInteraction" value="57" hexvalue="39" since="6.0" />
<element name="ShowAppMenu" value="59" hexvalue="3B" since="6.0" />
</enum>
This is all stuff where a SDL programmer would feel at home with
Next, add an optional property to SDLMenuCell, SDLSoftButtonObject, and SDLChoiceCell called nextFunction of type Function
Where the programmer lists the function that's coming up next
I don't think that adding a new SDL enum is the best way to go. I would prefer to see a manager-level internal enum (e.g. an iOS / Java / JavaScript native enum that is not reflected in the RPC API) be used that the managers then automatically translate to the function ID.
The Steering Committee voted to keep this proposal in review until our next meeting on 2019-11-19 to allow for more discussion on impact on the manager layer. The PM may be able to provide some support to the author offline regarding the manager-level internal enum.
Ok sat down with @joeljfischer on this
We got at least the iOS side down with a 3 step plan. We still need to work out hte java library side
enum NextFunction {
case Default
case PerformChoiceSet
case Alert
case ScreenUpdate
case Speak
case AccessMicrophone
case ScrollableMessage
case Slider
case SendLocation
case DialNumber
case OpenMenu
}
NextFunctionInfo with an initializer class NextFunctionInfo {
var nextFunction = NextFunction.Default
var loadingText : String!
}
init(nextFunction : NextFunction, loadingText : String) {
self.nextFunction = nextFunction
self.loadingText = loadingText
}
nextFunctionInfo parameter to SDLChoiceCell, SDLMenuCell, and SDLSoftButtonObject and their respective initilaizers. Rough Java version
@bilal-alsharifi needs to back me up a bit though
public enum NextFunction{
Default,
PerformChoiceSet,
Alert,
ScreenUpdate,
Speak,
AccessMicrophone,
ScrollableMessage,
Slider,
SendLocation,
DialNumber,
OpenMenu
}
NextFunctionInfo with an initializer public class NextFunctionInfo {
private NextFunction nextFunction = NextFunction.Default;
private String loadingText;
}
public NextFunctionInfo(NextFunction nextFunction, String loadingText) {
this.nextFunction = nextFunction;
this.loadingText = loadingText;
}
nextFunctionInfo parameter to ChoiceCell, MenuCell, and SoftButtonObject and their respective initializers. That looks easy for the developers to use. I just have two questions:
1- Should the first param for the NextFunctionInfo class initializer be of type NextFunction or NextFunctionInfo?
2- The NextFunctionInfo class will also have a variable of type FunctionID as well to store the corresponding functionId for the passed NextFunction enum value right?
@bilal-alsharifi thanks, I cleaned up the code a bit - check it out in the previous comment
I just labeled things wrong so it was confusing
Thanks, @MichaelCrimando, yes it makes sense now. Just the init should have the same name as the class name in Java. And self should be switched to this.
@bilal-alsharifi fixed, thanks!
Ok summary so far, with the manager-level implementation worked out, I don't think anybody else had any other concerns
@joeljfischer @joeygrover
@MichaelCrimando I commented on the PR, but for visibility here, I'm strongly opposed to another "manager-level" class. It's just duplicating the RPC class.
The Steering Committee voted to return this proposal for revisions. The author will revise the proposal to include the manager-level implementation as discussed in the comments on this review issue with the Project Maintainer.
The author has revised this proposal to incorporate the requested revisions from the Steering Committee. The revised proposal is now in review and will be voted upon during the 2020-01-14 meeting.
Apologies if i missed this in the conversation but the enum type "FunctionID" does not exist in the HMI API. We could add FunctionID to the HMI to fix this issue but that needs to be mentioned in the proposal.
Alternatively if we do not add that enum to the hmi api, NextFunctionInfo would require a slightly different param definition for nextFunctionID. In the HMI API nextFunctionID could be a string with values similar to "BasicCommunication.PerformInteraction".
If it is not those Functions that are assigned to the menu but screen transitions, how should NextFunction be set?
Also, how do you determine whether the screen transition destination cannot be operated by driver distraction?
@Shohei-Kawano I'm going to number your questions.
1. NextFunction is set based on the developer's knowledge of what the button press will do when pressed.
2. The HMI determines that and then uses the NextFunction info to show its own UI based on that.
Hey @JackLivio
Good point. Logic-wise adding an enum to the HMI api is more straight forward to me. It just avoids some developer errors that can come up when you use a string instead. When this gets accepted, I can just add in that the enum will be added to the HMI API
@MichaelCrimando Since on the HMI side since there isnt really a concept of function IDs, I believe the alternate I suggested to be a better option (using strings). All rpcs communicated between core and HMI already use strings for the rpc message type.
Here is an example rpc GetVehicleData Request, see method:
{
id: 19,
jsonrpc: "2.0",
method: "VehicleInfo.GetVehicleData",
params: {
vin: true
}
}
@JackLivio
Gotcha, so what would we add to the proposal?
Just something like
The HMI API doesn't use function IDs so SDL Core would communicate the next function through strings:
NextFunction.Default
NextFunction.PerformChoiceSet
NextFunction.Alert
NextFunction.ScreenUpdate
NextFunction.Speak
NextFunction.AccessMicrophone
NextFunction.ScrollableMessage
NextFunction.Slider
NextFunction.SendLocation
NextFunction.DialNumber
NextFunction.OpenMenu
From today's call @JackLivio how about I add
The HMI API doesn't use function IDs so SDL Core would communicate the next function ID through the existing RPC interface strings in the format of `interfaceName.RPCMessage".
I’m sorry that I couldn't respond at the Steering Committee due to my lack of English.
@joeljfischer -san
ScreenUpdate to NextFunction?@Shohei-Kawano -san
No problem, I got certified in the N5 Japanese Language Proficiency Test (JLPT) which took years to get that super basic-level of understanding. I can't imagine learning English is much easier
I can answer for Joel:
The Steering Committee voted to accept this proposal with revisions. The author will revise the proposal to specify that the HMI API doesn't use function IDs so SDL Core would communicate the next function ID through the existing RPC interface strings in the format of interfaceName.RPCMessage.
@MichaelCrimando Please advise when a new PR has been entered to update the proposal to reflect the agreed upon revisions. I'll then merge the PR so the proposal is up to date, and enter issues in the respective repositories for implementation. Thanks!
The author has updated this proposal to reflect the agreed upon revisions, and implementation issues have been entered: