How is the Z-wave database regenerated

[mulu]

I have a HS SmartStick+ and experience some issue. So I shut down HS3 and deleted the z-wave database in C:\Program Files (x86)\HomeSeer HS3\Data\Z-Wave. When I then restart HS3 it recreates the database. How does it do that? Where does it get the information from?

I first suspected it goes and reads all the devices and with that information scans all the z-wave nodes. However, when I delete a device that was created by a z-wave node then HS3 still puts that node back. So it must be getting that information from somewhere else.

As background information, I have some phantom z-wave nodes and phantom devices. I suspect that the phantom z-wave nodes create those phantom devices. So I am trying to delete those phantom z-wave nodes but there is no such function in the z-wave interface, at least from what I can tell. I don't want to hack into the database as I probably will mess something up. So the goal is to regenerate the database but make HS3 not re-add those phantom z-wave nodes. To do that I need to know how HS3 finds all the z-wave nodes.

[mikee123]

You have to create a backup, from plugin zwave controller create backup. Then make note of the file name, find it and edit it in notepad. delete all your phantom nodes, and save the backup (you have to use a different name). The go back to your zwave controller and restore the edited backup to the controller. After that you will have to optimize your network again.
When you delete the zwave database, HS3 recreates it from the information stored in the controller. Thats why all your phantom nodes are recreated.

[mulu]

Thanks a lot! I will give this a try. I saw those type of files before. I had a lot of them all with the name ZNet_[HomeID]_[Date]_[Time].ZWave. There were usually a bunch of them for a particular date but then none for a long time spans. No idea what triggered the creating of those.

So the stick does actually store some data and then the database stores additional data that I guess is being retrieved from the device. Does anybody know what the numbers mean in those .ZWave files? Not that it's important for my purpose but I am just curious.

[randy]

Some of what I am going to write is based upon what has been disclosed and some is based on what I have discovered.

The Z-Wave controller stores Node information. This is the electronic address of a node. If you look at the backup of an interface the file looks like this one from one of my controllers

EF5ED36D
1
1,211,22,1,2,2,1
2,19,150,1,1,1,6
4,211,156,1,4,16,1
5,19,22,1,1,1,1

The first line is the HomeID of the controller. This must be unique to each controller in your system, unless you have created secondary inclusion controllers. The second line is the node number of the controller. The last 4 lines are node addresses, starting with the controller. This address and separate routing information are all that is stored in the controller. The routing information is NOT in the backup. The controller does not store any other information. It does not know (or need to know) the brand, type number of child devices, etc. It doesn't need to know if it is a dimmer or a thermostat. It only needs to know the address of the device and any routing information to reach it. This routing information is much more important for legacy devices than for Z-Wave plus devices.

This corresponds to the Nodes in HomeSeer



When mikee123 suggest that you an take a backup of the controller, edit out a line and restore the controller, this is absolutely correct. This will remove a node from the controller. This should only be done as a last resort as this also wipes out all stored routing information. I'll get to the methods you should try first further on.

When a node is included in the controller, its address is stored. Then HomeSeer interrogates the node. It finds out the brand and model and all of its capabilities, then builds the root device and all child devices. ON legacy switches and dimmers it is not unusual for the root device to also be the control for the switch or dimmer. Because this interrogation takes place after inclusion, it is important for there to be good communication between the controller and the device. On legacy devices, this inclusion and initial interrogation takes place at a reduced radio power making proximity more important for communication. After inclusion and interrogation the root and any child devices in HomeSeer contain all the information about the node. The root device is tied to the Node ID in the controller.

The Z-Wave database is created and maintained by the information in the HomeSeer devices. If you delete the database it will be rebuilt by interrogating all of the HomeSeer Z-Wave root and child devices.

here are two kinds of "ghosts": 1) A HomeSeer device that has no connection to a Z-Wave node in the controller and 2) a Z-Wave node in the controller that has no associated HomeSeer devices.

These are my recommendations based on my understanding of the system.

1. If you have phantom or ghost child nodes that are duplicates, say a multi-sensor that has two Luminance devices where one is updated just now and another that was updated in the past or not at all, delete the HomeSeer device for that child only
2. IF you have a HomeSeer root device that is no longer associated with a Z-Wave node (look at the Z-Wave tab Node number) and you are sure the device is no longer in your system, delete the root. When you delete a root device all child devices are also deleted. If this device that no longer has an association with a node in your controller, but it should have, do not delete it. Go to the next step
3. There may be a node in the controller that is no longer associated with a HomeSeer device. Go to the controller menu and under controller actions "Import Node Info from Controller and Scan Devices. If there is an unaffiliated node in the controller, this process will create new HomeSeer device(s). If the node exists in your network, it will be interrogated and new devices will be build. You will see node numbers if there were errors, etc while this process is completed. If the interrogation fails, the device created will likely be a Sigma Generic device. If that is the case you probably have a ghost in the controller. You should be able to go to the root device's Z-Wave tab and "Remove Bad Node" If this is successful, the node address will be removed from the controller and the HomeSeer root and child devices will be deleted. If the interrogation found a device it will create a root and child devices that have the actual brand and model of the device and it will control a device in your network. If you had an unaffiliated device in step 2 above, you may have found the node number. You may be able to take the node number from this new device and assign it to the orphaned HomeSeer root device, then delete this new root device (deleting all child devices as well).
4. If you are unable to use "Remove Bad Node" to get rid of the phantom in the controller and you are absolutely sure that it shouldn't be there,then you can backup the controller, edit the backup to remove the bad node, then restore from the edited backup.

[mulu]

Randy, thanks a LOT for this detailed answer. If anybody is interested, here is a good summary of how z-wave networks are composed https://www.vesternet.com/resources/...wave-networks/

So if the routing information is not in the backup then what are the last 6 numbers in line “2,19,150,1,1,1,6”?

In any case I did #3, i.e. I did do a “Import Node Info from Controller and Scan Devices”. I get a lot of warnings saying “Warning: Failed to retrieve Z-Wave class information for node xx, device synchronization/creation will not continue for this node.” Eventually it times out and says “Warning: Device synchronization/creation aborted by timeout or by the user.” Some of those nodes are what I believe phantom nodes but others are real ones. So I think I cannot really go by that.

I did go to PluginsZ-WaveNode Information to get a list of nodes. Here I see some interesting cases of nodes that I believe are phantom nodes:

• Nodes with no neighbors and “Last Working Route: None”.
• Nodes with some neighbors but those neighbors are 1 and some other ones that I believe are also phantom nodes. “Last Working Route: None”
• Nodes with up to a dozen neighbors, some of them are good nodes. Note that good nodes usually have 30-60 neighbors. “Last Working Route: None”.

In all those cases the full name is listed as “Node xx Z-Wave Sigma Switch” with an isolated few Sigma Remote Switch or Sigma Static Controller. So it appears to me that the list of neighbor nodes is old information stored on the Z-Wave SmartStick. The ones with no neighbors are true phantom nodes. The ones with some neighbors might be nodes that once existed but were not properly excluded (on the Z-Wave Stick) when they were removed but the information was erased from the database, hence they end up with the generic name “Z-Wave Sigma Switch”. So the best criteria to find out if a node is still valid or not is by looking at the “Last Working Route:”. If this is “None” then that node doesn’t exist anymore and needs to be removed. Does that sound right?

I was looking for a way to remove nodes before and wanted to do that through the Z-Wave interface. I didn't realize that there is a "Remove Bad Node" in the Z-wave Tab of the HomeSeer device. That indeed seems much more appropriate than modifying the backup.

[Bernold]

Interesting. I am having trouble removing 2 nodes. Actually just one, which somehow was added twice, What happened is that it was not working anymore. So I removed/excluded it. Then added it again. After a while it stopped working again. So I tried removing/excluding it again. Then found out it exists twice. And does not let me remove it. Tried several ways, including removing it from the controller and then Import Node Info from Controller and Scan Devices. Still the same.

So I thought I would try this backup, edit and restore method. The problem with this method is that I don't see the restore option. Is this because my stick (Aeon Labs Aeotec Z-Stick GEN5) does not support it?

Also I don't see the Remove Bad Node option in the Z-Wave tab.

Am I looking in the wrong places? Could somebody add screenshots? Thanks in advance.

[mulu]

I have uploaded two images that show where the restore network and remove bad node are in my case. If you don't see them there then maybe it's indeed because of the Aeotec stick. Maybe somebody with that stick can comment.

[mulu]

Ok, so that "None" is a bit tricky. After you delete the database HS scans all the nodes. I was assuming that also updates the "Last Working Route". So after restoring the DB and maybe 12h passing I had 49 None. Then I did an "Optimize a Network, No Return Route Changes". Interestingly, after that I had 77. Since it says "No Return Route Changes" I would have expected the number to be the same or if anything going lower. So then I triggered a few events in HS and the number dropped to 65. So it means after you delete the database you pretty much have to trigger each z-wave device so that "Last Working Route" is being updated. Is there some other way to 'touch' each device so that field gets updated? Then if one still says "None" I can assume it's a bad node.

[randy]

Randy, thanks a LOT for this detailed answer. If anybody is interested, here is a good summary of how z-wave networks are composed https://www.vesternet.com/resources/...wave-networks/

So if the routing information is not in the backup then what are the last 6 numbers in line “2,19,150,1,1,1,6”?
...

You can prove it is not routing or neighbors by looking at a backup of my HomeID with the largest number of nodes

XXXXXXXX
1
1,211,150,1,2,2,1
2,211,156,1,4,16,1
3,211,156,1,4,17,1
4,111,69,157,3,69,0
7,211,156,1,4,16,1
10,211,156,1,4,17,1
11,146,150,0,2,17,0
13,209,140,0,4,18,1
14,211,156,1,4,16,1
19,211,156,1,4,17,1
21,211,156,1,4,17,1
23,211,156,1,4,16,1
26,211,156,0,4,16,1
27,211,156,1,4,16,1
31,211,156,1,4,16,1
33,211,156,1,4,16,1
34,211,156,0,4,16,1
35,211,156,1,4,17,1
36,211,156,1,4,17,1
37,211,156,1,4,16,1
38,211,156,1,4,17,1
41,83,22,1,1,2,3
44,211,156,1,4,17,1
48,211,156,1,4,17,1
49,211,156,1,4,16,1
53,211,156,0,4,16,1
56,211,156,0,4,16,1
57,211,156,1,4,16,1
58,211,156,1,4,16,1
60,211,156,1,4,17,8
61,211,156,1,4,16,1
63,211,156,1,4,17,1
64,211,156,1,4,17,1
65,211,156,1,4,17,1
67,211,156,1,4,17,1
68,211,156,1,4,16,1
69,211,156,1,4,17,1
71,211,156,1,4,17,8
72,211,156,1,4,16,1
73,211,156,0,4,8,6
74,211,156,0,4,16,1
75,211,156,1,4,17,1
76,211,156,1,4,8,6
77,211,156,1,4,8,6
80,211,156,1,4,17,1
81,211,156,0,4,8,6
83,83,156,1,4,7,1
84,209,140,0,4,16,1
85,83,156,1,4,7,1
103,211,156,0,4,16,1
111,211,156,0,4,16,1
134,211,156,0,4,16,1
148,211,156,0,4,16,1
157,211,156,0,4,16,1
222,211,156,1,4,16,1
225,211,156,1,4,17,1
228,211,156,1,4,16,1
229,211,156,1,4,17,1
232,211,156,1,4,16,1

You can see that a lot of those numbers do not correspond to other nodes on the network. I'm thinking it is some other data assigned to the device at inclusion, but I have been unable to figure out what it relates to. There is also a lot of repetition among the groups of 6 numbers. Rich might be able to answer this.

[mulu]

I have pretty similar numbers, i.e. a lot of 211, 156 followed by a lot of 1s, 4s, and 17s. First I thought it could be some compressed routing table but I don't think that's the case. There is a node information frame that nodes send to the controller during the inclusion process. It includes capabilities, etc of the node. I suspect that those numbers represent some of that information. If that's the case, then it would be nice to know when you try to figure out what a particular node is/was. However, I am mostly just curious about it.

My real problem is to figure out which nodes are actually dead/unresponsive. So far I haven't found a reliable way to determine that.

[Bernold]

I have uploaded two images that show where the restore network and remove bad node are in my case. If you don't see them there then maybe it's indeed because of the Aeotec stick. Maybe somebody with that stick can comment.

Thank you, mulu. So I looked in the right places, but the options are just not there. Must be the stick.

Edit: Found Remove Bad Node after all (at the other instance). I don't know if it fixes the problem with the node, because I'm not at home
And if anybody ever comes across the same issue: It seems to be correct that the restore function does not work for the Aeotec stick:. This article says both backup and restore are not supported (but backup is..): https://homeseer.com/compare-z-wave-interfaces/

[Airtog]

Excellent conversation here. I learned a lot. I figured out what the numbers mean after each node from the zwave backup table.
1,211,150,1,2,2,1
9,211,156,1,4,7,1
11,83,156,1,4,32,1
By looking in the backup nodes file.
You can see in the nodes file where these numbers are referenced.
The 2, or 4 in the 5th column can't be found --- looks like it's skipped.

For Node 9 FLS100 Flood light)
nodeId":9,"pendingUpdate":false,"pendingDiscovery":false,"ro uteSlaveSuc":false,"controllerSucUpdateIndex":255,"neighbour s":[],"nodeInfo":{"capability":211,"security":156,"reserved":1,"generic":7,"spe cific":1}

For node 11 (battery operated motion sensor):
"nodeId":11,"pendingUpdate":false,"pendingDiscovery":false," routeSlaveSuc":false,"controllerSucUpdateIndex":255,"neighbo urs":[],"nodeInfo":{"capability":83,"security":156,"reserved":1,"generic":32,"specific":1}

If the 2,7, 32 in column 6 refers to other nodes, then this may be an issue as I excluded Nodes 2, 7 and 32 -- they no longer exist and I subsequently did full optimization on specific devices and on the entire system. They still appear. Is this a Zwave stick issue?

With that said, I'm having an issue with CPU run-away and according to HS, they are saying I have phantom nodes and my Zwave database has to be rebuilt.
I've examined all files and followed all procedures and cannot find any phantom nodes. All files containing node information is correct.
My only issue is in NEIGHBOR reports -- where nodes appear as neighbors but the node does not exist in the system. I think that Zwave only uses 4 nodes for mesh communications anyway. I have no issues with my system functionality. Everything works fine except when adding or deleting a device, Zwave will go into CPU Exhaustion.
I'm on HS4 (4.16), ZW Stick G3 (ver 4) and only have 32 nodes.

Any idea where the routing information is held ?
I'd do the Zwave database re-build but want to be sure there's no other fix before doing so as I believe -- as far as I know -- that the re-build is like starting from scratch with event and device rebuilds.
I noted that this thread is old -- please refer me to other threads if appropriated.
Thanks!

[randy]

Any idea where the routing information is held ?

Thanks!

It is stored in the controller’s memory and is not included in backup/restore. After you restore the routing needs to be regenerated. On Z-Wave plus devices, the routing will fix itself over time. For older devices it must be done manually. Do not use bulk optimization, do it one-by-one on the Z-Wave tab of each device. Optimize from the controller outward, starting with the closest devices.

[Airtog]

Got it -- thanks so much --

[Airtog]

I optimized one at a time and removed some of the biggest offenders. Had some good results, but not totally complete. Still have nodes appearing in optimization paths that are no longer in the system. I've also been seeing how they do drop by themselves except for a few specific ones.
I've come to realize that my main issue is with the Homeseer HS200 devices. While others will remove unused nodes, these devices will not. I'll have to remove the device and re-include plus re-apply the software upgrade again. fyi -- if you have any of these sensors, you may want to check them. They were flaky to begin with and now I'm seeing they are even more flaky than I realized.