Thingsboard: Support of 'complex' telemetry ?

[myfreescalewebpage]

Hi,
I have just set up a fresh Thingsboard installation and send data from one device.
I notice that 'complex' telemetry data are not handled by the MQTT broker.
I mean, sending:

{"all_gw":[{"lsnr":9.8,"mac":"CC79CFFFFF21A2C1","rssi":-30,"time":"2018-07-08T20:47:04.427057Z"}],"altitude":0,"app":"Cayenne","appargs":"W8IHUL0w7rgup9FZh2bZ","battery":254,"best_gw":{"lsnr":9.8,"mac":"CC79CFFFFF21A2C1","rssi":-30,"time":"2018-07-08T20:47:04.427057Z"},"codr":"4/5","data":"00000000000000FE3E090D0503AB0000","datetime":"2018-07-08T20:47:04Z","datr":"SF12BW125","devaddr":"016C103F","deveui":"343137325636740D","fcnt":0,"freq":868.5,"humidity":0.0,"latitude":4065549,"led":0,"longitude":328619,"lsnr":9.8,"mac":"CC79CFFFFF21A2C1","netid":"000000","port":2,"pressure":0,"rssi":-30,"temperature":0.0}

is not handled (there is an array, some objects...)
Whereas:

{"altitude":0,"battery":254,"humidity":0.0,"latitude":4065549,"led":0,"longitude":328619,"pressure":0,"temperature":0.0}

is working.
Any support of 'complex' telemetry object planned ?
Thanks
Joel

[eykamp]

I've solved this myself by sending complex values as a string, and managing the quoting/unquoting on the client(s).

[myfreescalewebpage]

Well @eykamp that's good when you can modify the client but I can't ! The only solution I have today is to not transmit the expected fields at all, but I would like to send them to make maintenance/admin dashboard and rules...

[eykamp]

True. I just wanted to offer one possible solution; even if it doesn't help you, maybe it will help someone else. I too would prefer a richer data model.

[mfalkvidd]

Couldn't you just use jsonpath expressions? https://thingsboard.io/docs/iot-gateway/getting-started/#step-834-advanced-mapping-syntax-and-filtering
Otherwise, implement this interface: https://github.com/thingsboard/thingsboard-gateway/blob/release-1.0/src/main/java/org/thingsboard/gateway/extensions/mqtt/client/conf/mapping/MqttDataConverter.java

[vparomskiy]

You can also use data converters with 'mqtt' integration. They allow to convert inbound and outbound messages into any format (json, hex, binary ). But they only available in the Thingsboard PE.

[myfreescalewebpage]

@mfalkvidd and @vparomskiy thnaks for your answers, I see it's about ThingsBoard Gateway, so do you mean the solution is to use it to handle the specific fields ? I'm not using it at the moment (I have tried but not really understood how to easilly integrate ThingsBoard and ThingsBoard Gateway... At the moment, my sensors send data with MQTT at Thingboard immediatly.

[mp-loki]

Hi @myfreescalewebpage !
@vparomskiy was talking about ThingsBoard Integrations which is a Professional Edition feature.
For the Gateway it might be possible to configure the mapping to flatten your data structure. I'm telling I'm not sure because I don't remember such use case. I can try to come up with the mapping configuration and see if it's possible or not.

[vparomskiy]

@myfreescalewebpage does this issue still valid for you?

[myfreescalewebpage]

@vparomskiy I'm still using 2.1.0 version (can't upgrade actually) but I haven't seen any enhancement regarding to this on the newest releases... (https://github.com/thingsboard/thingsboard/releases), so yes that's still valid for me.

[vparomskiy]

You are right. Support of complex telemetry is not planned in the community edition. The only way to handle it is to use Data converters that are available in the Professional Edition.

[myfreescalewebpage]

I don't know the data converters, do they allow to "convert" some arrays, list of element etc ? i.e. even if the size of the list can change at each transmission of the telemetry by the distant IoT device ? I'm thinking to some context data associated to events, that can be plotted etc and I do not understand how it is possible to handle this kind of things except using an array of objects...

[vparomskiy]

In Data Converter you create a javascript function that takes original payload transmitted from your device (in any format - JSON, binary, string) and returns flat structure of key-value pairs
https://thingsboard.io/docs/user-guide/integrations/#data-converters

For example, you can save an array of objects as a JSON string. And then load this JSON string to UI widget, parse it and plot it.

[myfreescalewebpage]

I'm afraid it is absolutely not convenient.... JSON object are so common today, I do not understand why the telemetry cannot propose it. It is still a key/value pair, except that the value is an object, not only an integer, string or anything else.

[vparomskiy]

It is technically possible, but it is not planned for the nearest releases. Contributions are welcome.
Entry point is JsonConverter class on the backend. Ot is responsible for parsing incoming payload into object

[myfreescalewebpage]

Ok, will see if I can find time to contribute :) Thanks !

[Jalili-73]

change in source code
https://github.com/Jalili-73/thingsboard

[myfreescalewebpage]

@Jalili-73 interesting, probably you should make a pull request to the thingsboard repo to get the support in the mainline, because if the original file change, then your modification will not work anymore!

[datenheim]

Hm, maybe also a rule chain could be used for that. At least I use it to modify incoming JSON payload before it is stored in the time series.

and then

Basically I modify the payload depending on the deviceType.
Then in the "gen3-calibration" script the mod:
switch (metadata.deviceName) { case "uvi-007": msg.uvir = msg.uvi; msg.uvi = msg.uvir * 9.12; break; default: break; } return { msg: msg, metadata: metadata, msgType: msgType };
All that is in the free version. Dont know if that would help you. But the script opens a lot of possibilities....

[BarodiaanjaliDS]

It is possible to handle complex telemetry data in the community edition.
the existing python gateway use jsonpath-rw library which doesnot support many filter fetaures.
Try using jsonpath-rw-ext library which handle complex filters. It can be modify in tb_utility.py file.

[Karabiy]

the complex jsons are being supported for the latest version, so that you can have an upgrade for those. Also, you can use the data converters in case if you are using the PE version and the Python Gateway customization in case if you are using the CE in advance. refer to the links below on those.
ThingsBoard IoT Gateway: https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
ThingsBoard Integration: https://thingsboard.io/docs/user-guide/integrations/
Rule engine capabilities: https://thingsboard.io/docs/user-guide/rule-engine-2-0/re-getting-started/

[Purvi0499]

how to send the data from excel file to device on community edition of things board?
The code below isn't working.
This code updates the telemetry in demo account but not on the tenant account on the server installed on my system.
(broker and port is changed while using demo)

import paho.mqtt.client as paho      #mqtt library
import os
import json
import random
import time
import pandas as pd
import xlrd
import psycopg2
from datetime import datetime

ACCESS_TOKEN='DEMO_DEVICE'                 #Token of your device(Thermometer)
broker="52.183.160.251"                        #host name
port=8090                                           #data listening port
loc = ("E:\Solar Internship 2020\steam monitoring\data.xlsx")
wb = xlrd.open_workbook(loc) 
sheet = wb.sheet_by_index(0) 
sheet.cell_value(0, 0) 
n = 0


#for i in range(sheet.nrows):
    #print(sheet.cell_value(i, 0))
#   n = sheet.cell_value(i,0)
#   print(n)
#   time.sleep(2)

def on_publish(client,userdata,result):             #create function for callback
    pass
client1= paho.Client("control1")                    #create client object
client1.on_publish = on_publish                     #assign function to callback
client1.username_pw_set(ACCESS_TOKEN)               #access token from thingsboard device
client1.connect(broker,port,keepalive=60)           #establish connection

for j in range(sheet.nrows):
    for i in range(1,sheet.nrows):
        t = sheet.cell_value(i,0)
        print(t)
        h = sheet.cell_value(i,1)
        print(h)
        p = sheet.cell_value(i,2)
        print(p)
        a = sheet.cell_value(i,3)
        print(a)
        s = sheet.cell_value(i,4)
        print(s)
        c = sheet.cell_value(i,5)
        print(c)
        f = sheet.cell_value(i,6)
        print(f)
        o = sheet.cell_value(i,7)
        print(o)
        print(i)
        time.sleep(2)
        m="{"
        m+="\"Temperature1\":{t},".format(t=t)
        m+="\"Humidity\":{h},".format(h=h)
        m+="\"Pressure\":{p},".format(p=p)
        m+="\"Air Flow Rate\":{a},".format(a=a)
        m+="\"Steam Flow Rate\":{s},".format(s=s)
        m+="\"Temperature2\":{c},".format(c=c)
        m+="\"Flue Gas Temperature\":{f},".format(f=f)
        m+="\"CO content\":{o}".format(o=o)
        m+="}"
        ret= client1.publish("v1/devices/me/telemetry",m) #topic-v1/devices/me/telemetry
        print(m);
        print(type(m))
        time.sleep(2)
        break;

[Karabiy]

you can use the ThingsBoard IoT Gateway to convert the data to the ThingsBoard appropriate format, and then this will be ok. Convert from XML to JSON. In case if the code of yours is not working, there should be an error, or maybe those failed on the particular raw? Make sure that you are sending JSON.