Thingsboard: Thingsboard is unable to load any data

[fotis400]

Some times the web UI hangs and fails to load any data. This is a very common occurrence for me. All fields remain in a "loading" state forever.

Example:
I try to access a dashboard. All widgets show an orange icon cycling, indicating that it is loading. All widgets remain in this state indefinitely, and the page never loads completely.

I haven't found yet how to reproduce this issue reliably. It seems like it is appearing randomly. A fix for this is usually to reload the page. Note however that some times multiple refreshes may be needed.

[vparomskiy]

1. What database is used?
2. How many devices are connected to the Thingsbaord and how often they send data to Thingsboard?
3. What instance is used for Thingsboard (RAM, CPU)
4. Are there any other system processes that can consume a lot of CPU resources at that time?
5. What is CPU load during those 'hangs'?
6. Are there any widgets on the page that loads data for long time range (months) AND configured to use very small aggregation interval (1-10 minutes)?
7. Are there any errors in the log file?

[fotis400]

1. PostgreSQL
2. Only one device. It is sending data every 30 secs. Usually this device is disconnected though, so totally there should be only a few hours of data logged.
3. I use a VPS, with 2 CPU cores, 6GB of RAM, and an SSD of 500GB.
4. This server uses a vanilla CentOS distribution, and run exclusively Thingsboard, nothing else.
5. I don't have this information, I will try to check next time it happens.
6. Definitely not. In some cases only latest values are displayed. In other cases, the worst scenario is 1 month with 2 hours aggregation.
7. I will attach the log on the next occurrence of this problem.

[fotis400]

I have an occurrence of this issue right now. Thingsboard in general responds very well, but the widgets are not loading any data.

CPU utilization is completelly idle.
Memory usage is at about 1GB, out of the 6GB available.
Server uptime is app 6 days.

(I will try to upload the log too, but I have difficulties for the moment, as it is a few megabytes...)

[fotis400]

thingsboard.log

[vparomskiy]

Could you please also open Browser console and track:

1. console messages
2. network requests

Please write what REST requests are slow or maybe errors are returned.

PS: Data for widgets are fetched via websockets, so please monitor those requests also.

[fotis400]

Currently I am experiencing this on my mobile phone, so I do not have access to such features.

I am using Firefox 59.0.2 on Android 7.1.1.

I will try to reproduce on my desktop, so I can log the traffic.

[vparomskiy]

Also, I see many errors in the log:
ERROR o.s.w.s.s.s.DefaultHandshakeHandler - Handshake failed due to invalid Upgrade header: null
This error tells that websocket could not be established correctly. IT could be a reason of your problem.

If you use a proxy for accessing UI, are websockets are also proxied?
You can take a look at this thread for more details:
https://stackoverflow.com/questions/31211919/spring-websocket-handshake-failed-due-to-invalid-upgrade-header-null

[fotis400]

For the moment I do not use SSL, so I am not sure whether the provided link can help me.

I have some new information on the matter though:

• It seems that the problem appears only on 3G/4G connections. Through Ethernet or any WiFi all computers / phones seem to work OK.

• Thingsboard runs on port 8080. However I have configured Centos to redirect traffic from port 80 to port 8080. This way you can normally reach my instance through mydomain.com without specifying a port on your browser.

*nat
:PREROUTING ACCEPT [2:402]
:INPUT ACCEPT [2:402]
:OUTPUT ACCEPT [1:112]
:POSTROUTING ACCEPT [1:112]
-A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
COMMIT

• Seems like websockets do not like the above setting. If I try to access Thingsboard via mydomain.com:8080 it works every time. If I try to access it through port 80 and 3G then it fails.

Any ideas on this?

[NopMicrowave]

Hi !

I was using Apache2 as reverse Proxy to route https://www.mydomain.com to 127.0.0.0:8080. I remember that I had to make some extra conf on Apache2 to allow WebSocket, otherwise, Widgets never update.
Hope it helps,

[fotis400]

@NopMicrowave Can you please provide some more info on your configuration?

[NopMicrowave]

Take a look at this, but not sure about this :

https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html

[fotis400]

That didn't help me, unfortunately. I do not use a proxy.

Any other ideas? Anyone?

[fotis400]

I don't see how changing the database can solve this issue.

Seems more like network related.

[jayhding]

I have seen similiar issue with PostgreSql, and changing to access 8080 directly does solve the issue.

However, I can’t think of a reasonable explanation as traffic from 80 to 8080 is routed by AWS ALB, while same traffic route also exists from 8080 on ALB to 8080 on ec2 instance.

[babupt]

Adding the below to Apache Virtual Host configuration should resolve the issue
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://localhost:8080%{REQUEST_URI} [P]

[lcx]

had this issue as well, problem was websocket in nginx needing some extra config. If someone comes across this issue with nginx:

    location /api/ws {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header CLIENT_IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
        proxy_pass http://thingsboard/api/ws;
    }

[friendlyemu]

@lcx

had this issue as well, problem was websocket in nginx needing some extra config. If someone comes across this issue with nginx:

    location /api/ws {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header CLIENT_IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
        proxy_pass http://thingsboard/api/ws;
    }

Thanks, this actually solved my issue.

[Eckehard]

Adding the below to Apache Virtual Host configuration should resolve the issue
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://localhost:8080%{REQUEST_URI} [P]

I have the same issue (No data, Handshake failed... in the logs), but Apache configurations was already set correctly. The system had been running for some weeks, but then stopped. Does not show any telemetry data anymore. I have no clue how to debug the system.

Any advise welcome.
Eckehard

PostScript: Strange things going on. Today, everything ist ok?!?!?!? Yesterday, all dashboards have been empty, Devices did not show any telemetry. Today things are ok, even data from yesterday have been received correctly. Anyway, O´I have no idea how to debug thingsboard.