Zwavejs2mqtt: [feat] Improved Z-Wave Network Visualization

[ahochsteger]

Current Visualization

The current visualization of the Z-Wave mesh network gives a good overview of the network but does not work well for bigger networks (shows big ball of wool ;-) ).
Additional information or improved visualization is missing that can help analyze problems of the network. In addition the network visualization could be used to directly interact with the nodes.

Screenshot:

Proposed Features

It has the potential of doing much more.
Here are some ideas, as a starting point for discussion:

• Show information:
  
  
  
  • link quality between nodes (bi-di vs. single-direction)
  
  
  • node status
  
  
  • battery or mains powered
  
  
  • battery level
  
  
  • number of hops to controller
  
  
  • node queue length
  
  
  • node errors
  
  
  • response times (support in zwave-js coming soon)
  
  
• Grouping of nodes:
  
  
  
  • no grouping
  
  
  • by location, device type, ... (similar to nodes table goruping)
  
  
• Filtering of nodes:
  
  
  
  • show all or only relevant nodes for routing
  
  
  • filter by different criteria (similar to nodes table filter)
  
  
• Interaction with nodes:
  
  
  
  • highlight neighbors on click
  
  
  • trigger actions (heal, refresh, replace, ...)
  
  
  • direct link to node in the control panel and vice versa (quickly switch between both views)
  
  
• User experience
  
  
  
  • better visualization of large networks (see filtering above)
  
  
  • fix canvas height problems in current implementation
  
  
  • don't triggering a redraw with zoom out/in every few seconds if devices are updated
  
  
• ... (more to be discussed)

Inspiration of existing Visualizations

This is a quick research about what kind of visualizations are already available for Z-Wave networks for inspiration:

• ZWaveGraphHA:
  
  
  
  • Source+Code: https://github.com/AdamNaj/ZWaveGraphHA
  
  
  • Screenshot:
    
    
  
  
• Home Assistant Z-Wave Graph:
  
  
  
  • Source+Code: https://github.com/OmenWild/home-assistant-z-wave-graph
  
  
  • Screenshot:
    
    
  
  
• Habmin Network Viewer
  
  
  
  • Source: https://community.openhab.org/t/z-wave-routing-basics/92408/2
  
  
  • Code: https://github.com/openhab/org.openhab.ui.habmin/blob/master/src/web/app/tools/zwave/zwaveNetwork.js
  
  
  • Screenshot:
    
    
  
  
• Z-Wave PC Controller Topology Map:
  
  
  
  • Source: https://community.openhab.org/t/z-wave-routing-basics/92408/3
  
  
  • Screenshot:
    
    
  
  
• ... (please add more you may know of)

Request for Discussion

Let's discuss the various requirements and use cases people have and based on that implement an improved (or additional) visualization(s) of the Z-Wave mesh network.
What are your thoughts on this?

Possible Visualization Libraries

• vue-d3-network (current solution):
  
  
  
  • Code: https://github.com/emiliorizzo/vue-d3-network
  
  
  • Demo: https://emiliorizzo.github.io/vue-d3-network/
  
  
  • Screenshot:
    
    
  
  
• vue-org-chart:
  
  
  
  • Code: https://github.com/Hoogkamer/vue-org-chart
  
  
  • Demo: https://hoogkamer.github.io/vue-org-chart/
  
  
  • Website: https://freeorgchart.netlify.app/
  
  
  • Screenshot:
    
    
  
  
• D3 Chord Dependency Graph
  
  
  
  • Demo + Source: https://observablehq.com/@d3/chord-dependency-diagram
  
  
  • Screenshot:
    
    
  
  
• ioBroker Network Map:
  
  
  
  • Code: https://github.com/AlCalzone/ioBroker.zwave2/blob/master/admin/src/lib/networkMap.ts
  
  

[robertsLando]

@ahochsteger I think that zwave mesh visualization can be improved a lot. IMO the steps are firstly to identify what we need to visualize (like disintic if it's a battery powered device or not, the battery level, highilight neighborns on click), then identify the library to use, maybe vue-org-chart could be better for us.

[ahochsteger]

@robertsLando yes, that's what I'm trying to start with this issue to have a discussion about what we want to visualize and then decide which visualization works best for which kind of information.
Maybe we can provide more than one view if not everything can be put into a single graphical representation.
I'll put a section into the description to collect possible visualization libraries and add vue-org-chart.

[robertsLando]

IMO what's important to show is:

• See the actual node status (dead, alive, sleep etc..)
• See if a node is a battery powered device or not, if battery show the battery % (I like the way used by ZWaveGraphHA)
• Try to group devices by location (this could be tricky to implement as I dunno if org chart allows to draw overlays)
• when pressing on a node:
  
  
  
  • Highlight node neighborns
  
  
  • Show a window (like the one already existing) with the node details
  
  
  • Allow to send action (like healing etc) ?
  
  

[robertsLando]

From what I can see seems that https://github.com/AdamNaj/ZWaveGraphHA provides the js script to render that graph. We could reuse that for our needs. I think it has almost all the needs. It also allow us to prevent using some 3rd party library

Edit: it seems using d3 library, just saw the imports

[AlCalzone]

Thinking a bit out of the box (also with d3.js):

Seeing the number of hops between two targets could be interesting for debugging purposes though. As well as response times (support in zwave-js coming soon).

[cbho]

I was just about to reply with examples just like the one showed by @AlCalzone above...
It was like that on Homey in the beginning and I really found it usefull and intuitive (and aestethical ;-)).

Examples from Homey:

[maivorbim]

Leaving my two cents here. I think that any visualisation should include two modes: 1. all connected nodes (similar to vue-3d-network) and 2. most relevant node which shows hierarchy. (similar to vue-org-chart).

ZWaveGraphHA actually has a toggle like this, you can toggle between all neighbours or relevant ones. Both types of visualisations have their use cases and combining them into one IMHO would make a mess.

As an user, I would like to both being able to see which is the path a device will most likely take to the controller and number of hops, and which are all the neighbours of a node in order to assess network coverage.

In addition to above propositions on what information should be displayed, I'm thinking that node queue length and node errors (probably setting a node color based on certain error/queue length values) would be an easy way to spot node problems.

[ahochsteger]

@AlCalzone, @cbho, @maivorbim thanks for your feedback, I've restructured the feature list in the issue description and included your information.

[AlCalzone]

@ahochsteger FYI the implementation I screenshotted above is this:
https://github.com/AlCalzone/ioBroker.zwave2/blob/master/admin/src/lib/networkMap.ts

[Lopton]

I also think there should be a way to get the underlying data. As far as I can find there is no-where in the UI that lets me simply see the neighbors of a node. Either a table with all nodes and all neighbors, or having it when you're viewing a node would be really helpful in addition to the visualization.

[ahochsteger]

The Data Visualization Catalogue may be a helpful resource to find the best way to visualize different kinds of information:

[ahochsteger]

I've played around on Observable to see if it would be possible to use a Z-Wave node list for visualization experiments and it turns out that it is indeed possible and a lot of fun!

Here's a notebook that can be used as the basis for your own visualization experiments by importing the data and functions:
https://observablehq.com/@ahochsteger/zwavejs2mqtt-base
It provides an export of a real-world Z-Wave network with 62 nodes.

This is a simple example that represents more-or-less the currently available visualization of zwavejs2mqtt by using a D3 force directed graph network:
https://observablehq.com/@ahochsteger/zwavejs2mqtt-simple-force-directed-graph-network
It demonstrates how the Z-Wave node data can be used for visualization.

There's a notebook collection to collect all notebooks for zwavejs2mqtt:
https://observablehq.com/collection/@ahochsteger/zwavejs2mqtt

I'd like to invite everyone to join the visualization experiments :-).
Create your own notebooks or fork an existing one and improve them.

Disclaimer:
It's the first time I'm using D3 and Observable so I'm very interested in feedback to improve the notebooks :-)

FYI: @robertsLando @AlCalzone

[robertsLando]

@ahochsteger I still think that the most complete solution is the one provided by ZWaveGraphHA. I have looked at your notebooks, I really like them and I think it's a good tool for testing and proposal :) Didn't know about that.

BTW every solution has it's pro/cons, we should firstly focus on what we want to be visible "at first sight" and what/how to show the details about a node

[robertsLando]

@ahochsteger I have an idea about this. We could keep using the d3 network plugin we are using now, add a dropdown (with search) list where you can select the nodes to show in the network (by default all are selected) and add some checkboxes to hide/show ionformations about those nodes. For example we could add Show Neightbors (to show/hide all nodes that are neightbors of the selected ones) Same Location (show nodes that are in the same location)

Then we coould also add some custom svg or else to better identify the devices, like locks if one is added securely, a battey level if them are battery devices etc...

[jarrettv]

so major needs are:

• identify how many hops from controller
• identify link quality (nodes with few/weak connections)

I feel like a social network graph that shows hops from controller would be best. Here is how I picture it:

[jarrettv]

Another interesting idea is to let the user move individual nodes on a canvas and store the x/y positions. Then they can group or move them around how they like. For example, i'd likely have controller in the middle and put second floor above and first floor below roughly in the shape of my floorplan.

[robertsLando]

@jarrettv Yep but someone would also know if a device is secure or not? We could create multiple visualization types like 'Hops' 'Node types' etc based on what the graph should show at first sight?

[robertsLando]

Another interesting idea is to let the user move individual nodes on a canvas and store the x/y positions. Then they can group or move them around how they like. For example, i'd likely have controller in the middle and put second floor above and first floor below roughly in the shape of my floorplan.

Sure that's nice, the problem is also to find a js library for this, because building it by your own is huge, D3 offers a large set of choices and maybe has something that fits our needs

[robertsLando]

BTW I'm still conviced that converting this into a vue template would be the best choice for us

[robertsLando]

Just drafted a PR @jarrettv @ahochsteger

[ahochsteger]

@robertsLando thanks for starting the PR!

On my test setup there were too many failed nodes so I didn't see the relevant ones:

I've added a simple filter to remove failed nodes to improve the relevant ones:

We will have to improve it to better work with a larger set of nodes.

What I'd really like to see is to quickly switch between nodes in the table and the graph representation but I guess we'll have to introduce deep-linking to make that easily possible. Others would benefit from it as well (e.g. deep link from a HA device directly to zwavejs2mqtt). Sounds like a separate, independent PR though ...

[robertsLando]

Thanks for looking at this @ahochsteger ! Will keep work on this tomorrow 💪

[robertsLando]

I don't understand why the nodelist is always undefined in the handlers :thinking:

[robertsLando]

For anyone interested, give a try to: https://github.com/zwave-js/zwavejs2mqtt/pull/458