Azure-iot-sdk-c: Message properties are not received with module to module communication

[corollaries]

Development Machine, OS, Compiler (and Other Relevant Toolchain Info)
x86_64 Intel Industrial PC, Custom OS built with Yocto.
Compiled IoT Edge module on Ubuntu 18.04 with gcc 5.4.0.

SDK Version
0.2.0.0-23xenial (downloaded latest via ppa:aziotsdklinux/ppa-azureiot)

Protocol
MQTT

Description

When communicating between IoT Edge modules using messages, a property on a message gets lost on the receiving end. Take the following example:

1. A module sends a message with a property: messageType. The property was added using IoTHubMessage_SetProperty()
2. An other module receives the message and tries to read the property using IoTHubMessage_GetProperty().

Expected: The property is successfully read by the receiving module.
Actual: The IoTHubMessage_GetProperty() returns NULL.

Reproducing

1. Build an IoT Edge module called EventReceiver using sample: iothub_client_sample_module_filter. With the only addition being (void)IoTHubMessage_SetProperty(messages[iterator].messageHandle, "messageType", "event"); call just before sending the message with: IoTHub_ModuleClient_LL_SendEventToOutputAsync().

2. Build an IoT module called EventSender using sample iothub_client_sample_module_sender with the following addition to PrintMessageInformation():

if ((messageType = IoTHubMessage_GetProperty(message, "messageType")) == NULL)
{
        messageType = "<null>";
}

1. Use the following routes in the deployment manifest:

"routes": {
          "EventSender": "FROM /messages/modules/EventSender/outputs/* INTO BrokeredEndpoint(\"/modules/EventReceiver/inputs/input1\")",
          "EventReceiver": "FROM /messages/modules/EventReceiver/outputs/* INTO $upstream"
},

Console Logs

EventSender

Confirmation[497] received for message tracking id = 4464461 with result = IOTHUB_CLIENT_CONFIRMATION_OK
Sending message to output!IoTHubModuleClient_LL_SendEventAsync accepted message [499] for transmission to IoT Hub.
-> 07:01:54 PUBLISH | IS_DUP: false | RETAIN: 0 | QOS: DELIVER_AT_LEAST_ONCE | TOPIC_NAME: devices/LR201807019009/modules/EventSender/messages/events/messageType=event&%24.cid=CORE_ID&%24.mid=MSG_ID&%24.on=temperatureOutput/ | PACKET_ID: 501 | PAYLOAD_LEN: 83
<- 07:01:54 PUBACK | PACKET_ID: 500
Confirmation[498] received for message tracking id = 0 with result = IOTHUB_CLIENT_CONFIRMATION_OK
<- 07:01:55 PUBACK | PACKET_ID: 501
Confirmation[499] received for message tracking id = 4245888 with result = IOTHUB_CLIENT_CONFIRMATION_OK

EventReceiver

Received Message [404]                                                                                                                                                                                    
Message ID: [MSG_ID]                                                                                                                                                                                     
Correlation ID: [CORE_ID]                                                                                                                                                                                
Data: [<null>]                                                                                                                                                                                           
InputQueueName: [input1]                                                                                                                                                                                   
connectionModuleId: [EventSender]                                                                                                                                                                        
connectionDeviceId: [device] 
messageType: [<null>]           

Other Tests

I've built a module that can receive messages in NodeJS as well. The module receives the messages with the properties included. So I'm guessing something must not be going entirely right in the receiving end of the azure-iot-sdk.

[JesvinJoseph]

@corollaries @gregman-msft
This is true. I am also facing the same issue. The problem is with the first property.
If we are setting 2 properties, the receiving module will receive only the second one. But if we send the same to the IoT hub using $upstream, the built-in endpoint will receive both the properties.
So in inter-module communication, the first property is lost.

[alaendle]

More than a year later the problem still persists 😢
I was able to reproduce it with latest SDK - 0.2.0.0-23bionic

[corollaries]

@alaendle: What @JesvinJoseph mentioned is true. It's far from ideal but we found a way around this problem by simply adding a property that we do not need as the first property. By doing this we are receiving the properties correctly that we actually do need.

[alaendle]

@corollaries - Thanks, I also can confirm that this workaround indeed works, however it is still a pretty annoying bug. It needs time to spot the problem and find this issue that mentions the "solution".

[danewalton]

@jspaith can confirm but I believe his PR here:
https://github.com/Azure/azure-iot-sdk-c/pull/1756
should help with this. It's a large PR so we are working to get it reviewed. Thanks for the patience here.

[alaendle]

@ericwol-msft @ewertons The problem persists; and the workaround with a first dummy property only works in an environment where you could control the order of the properties - so the workaround is able to work e.g. for communication from a C-module to a C-module; but what if my sender is a Java-module? Here the message properties are represented by a HashMap, so the client has no (or at least limited) control about the specific order of the properties - so no workaround is available to communicate properties from a java to a c module.