HomeSeer Disaster Recovery Test

[Tillsy]

I've read plenty of posts in this forum about nightmares trying to restore HomeSeer.

That there are limited options for Z-Wave restores is separate matter.

So tonight I have assumed the Z-Wave stick itself is okay and simulated a full DR of the rest of the system using new gear, a perfectly clean system, and just restoring the core data.

Preparation:
Most recent backup of HomeSeer done from the GUI (Tools/Setup/Backup Config)
Linux version of HomeSeer
Another Raspberry Pi
Another SD card

System steps:
Prepare SD card with latest Raspbian
Plug SD card into Raspberry Pi
Remove Z-Wave stick from the former HomeSeer device and plug into exact same port on this Raspberry Pi
Configure Raspbian itself (locale, timezone, enabling SSH, expanding file system, etc)
Install necessity such as openssh, nano
Configure the network exactly the same as the (simulated dead) HomeSeer - IP address, WiFi off, Bluetooth off, etc
Install all updates
Install mono-complete and mono-vbnc

HomeSeer steps:
Copy the pre-prepared Linux version of HomeSeer to the Pi
In my case copy across a few customised scripts, images, speech files, etc
Configure for automatic startup
Restart Pi
Open a web browser and connect to HomeSeer
Assign password details
Go to Tools/Setup and activate the same license used on old HomeSeer on this one
Go to Tools/Setup and install the latest update available
Wait for system to restart after the update
Install all the plugins that were on the old HomeSeer - don't do anything with them though
Do a Tools/Setup/Restore Config and point at the most recent backup taken of former HomeSeer
Wait for restore to complete and system to restart

Result? 100% perfectly running system - system licensed, plugins licensed, devices all up in HomeSeer, devices all up in Z-Wave, all devices communicating both ways, events all intact and triggering, HomeSeer Mobile reconnected, ran an optimisation to prove everything is working (the slightest Z-Wave problem in the past has always caused this to go haywire for me), no errors in log... in fact I've left this DR running as my live system as it's perfect.

Hope this helps anyone facing a real restore scenario.

[Krumpy]

Good job!

With Linux it should be easier than with Windows. But still possible with Windows with the right steps. This reminds me that I should do the same.

[RoChess]

With Linux it should be easier than with Windows.

Why would this be more difficult in Windows?

It's comforting to know a disaster recovery on a fresh OS install works, because normally you would recover from a system backup.

Both Linux and Windows make this easy, but often this is not done and can get more difficult if the DR system contains different hardware.

In this case HomeSeer is the key with all the events/names/etc. alongside the Z-Wave controller having the network knowledge within flash memory.

Tilly confirmed that HS PC replacement works well (A), so that leaves two additional DR scenarios left.

B) It might take time to obtain new HS PC, so temporarily relying on say a laptop or your main system to take over while you wait for new HS-PC parts would be comforting. Relying on a VM solution might be easiest then if it supports USB passthrough for controller, such as VirtualBox (not an issue for Z-Nets). That way messing around and change network settings to take over HS-PC IP are not needed. Thinking this scenario out, I'm going to look into automating it where the HS backup to NAS from primary system will be auto restored when VM is turned on. Should even be possible to add extra script logic that it won't claim the same IP if the primary HS-PC is still running.

C) Z-Wave interface controller fails and you have to recover the Z-Wave backup file onto a new controller. Ideally that is one you have a spare on whenever you have the budget to do so.

Finding the time to test and prepare is another matter

[drhtmal]

The most likely failure would be a hard drive. I decided to rebuild my system with RAID 1 disk mirroring. I know it is only software RAID but as I said the most likely failure will be a hard drive. If the mother board dies I have a SATA to USB3 dock that I can plug into a temporary system and boot. Disclosure: I have not tested this yet.

There are 2 types of hard drives. Those that have failed and those that will fail.

[RoChess]

There are 2 types of hard drives. Those that have failed and those that will fail.

Amen!

Must admit I've got a good track-record on SSDs now, with the oldest one still in active service being over 7 years old. Keeping track of SMART data is not a guarantee to be warned of failure, but I combine that with plenty of spare blocks and mostly an SSD will fail on write first, still allowing time to do a last-second read to replicate it to a new drive.

[bmsmithvb]

This is excellent. Thank you for posting this guide.

[randy]

If the architecture is the same (32-bit or 64-bit), Windows is very easy. Backup the entire HomeSeer directory structure to an external drive or NAS, Using the backup utility of your choosing. Copy the entire HS3 folder structure to the new hardware in the same location. If you don’t use any serial or or other hardware dependent interfaces, HomeSeer will immediately run on the new or repaired machine. I posted this a couple of months ago:

Here is my plan and it has served me well. I run HomeSeer under Windows 10 Pro on a Supermicro 1U server. I have an older Supermicro 1U server running Windows 10 as a backup.

• I have BLBackup do a folder backup twice daily to a NAS. I run it at 1:00AM and 1:00PM
• I have Macirum Reflect do a full image backup once a month, then a differential on Mondays and an incremental every day to the NAS. The NAS is a Synology with mirrored 10TB drives
• Once a month I manually bring up the the second server and update it to match the primary. I do any Windows updates and run a batch file that copies the entire HomeSeer folder structure from my primary to the secondary, replacing all files.
• All of my Z-Nets are backed up once a month using the Python utility that zwolfpack posted. Rich changed the behavior of the Z-Wave plug-in such that a backup is automatically created any time a node is included or excluded. Since these backups are saved on the primary machine, they are copied to the NAS when BLBackup runs.
• All of my Z-Nets run the same image so have a couple of imaged MicroSD cards ready. Since I use DHCP reservations for IP addressing, swapping the card is all that is needed in the event one becomes corrupted. I have never had a card corruption since i moved to high quality SanDisk or Samsung cards. I am also running all Pi3 boards with POE hats as described here.

Both of my servers have IPMI interfaces so I can remotely start, stop and have console access through a VPN. I also have remote RDP access.

I have had three situations where I had to take action.

• Several years ago (2014 IIRC), HomeSeer stopped while I was out of town. This one was easy. I was not running HSSentry and HomeSeer crashed. I was able to access the server remotely and restart it. Everything came up working without further actions. I now run HSSentry
• In 2016 HomeSeer stopped while I was out of town. The primary server would not respond even to IPMI. I brought up the secondary (a VM at that time) copied the most recent BLBackup folder structure to the secondary, started HomeSeer
• Three months ago I had a Samsung 850 PRO SSD in my primary go tits up. I was home, so I brought up the secondary as above, while I replaced the SSD. Rebuilding the primary from a Macirum image took just under 1/2 hour across the network and was painless. I have a rescue USB thumb drive at the ready for this purpose. Every Windows PC, laptop and both HomeSeer servers use the same Macirum imaging routine

I have never had a Z-Wave interface fail, though I did have a couple of corrupted SD cards in the early days of Z-Nets. Since late in 2015, I have not had the slightest hiccup in any of my (now 4) Z-nets. Should there be a problem in one I have

• A spare SD card
• Spare Raspberry Pi boards
• Spare POE Hats
• a USB Smartstick+ interface

The drill would be to replace the SD card if it was corrupted, the Pi board if it failed. If the Z-Wave interface fails, I would remove the GPIO Z-Wave card and plug a Smartstick+ into the USB, then restore from the latest backup. Since all of my Z-Nets are POE powered, I also have the ability to force a restart by cycling to POE power to the port.

I have tested all of the above methods and all of them are proven to work. Additionally my heating system that is controlled through HomeSeer will fail over to manual control if HomeSeer goes down.

[RoChess]

thank you as usual for your explanations, and I'm going to have to look into HSSentry now

Is this a tool only included with HomeSeer Pro?

[Krumpy]

You ask why Windows is more difficult? Well, there is this thing called the registry. Once it gets corrupt, well then you're in trouble. It can be really fun (not) trying to go from 32bit Windows to 64 Windows HAL, or different disk controller types. Remember that one always needs compatible disk controller and processor architecture to get Windows to boot off backup.

Randy, you ought to start thinking about active-active clustering next. There you go... A feature request of HS4 that would worthwhile.

[RoChess]

You ask why Windows is more difficult? Well, there is this thing called the registry. Once it gets corrupt, well then you're in trouble. It can be really fun (not) trying to go from 32bit Windows to 64 Windows HAL, or different disk controller types. Remember that one always needs compatible disk controller and processor architecture to get Windows to boot off backup.

I remember those struggles in the days of Windows 95/98, and partly in XP where it was difficult to obtain text-mode drivers sometimes, but I've done 32-bit to 64-bit HAL upgrades, and even 64-bit to 32-bit (they had 16-bit need for BobCAD program and due to real-time RS232 requirements driving a CNC machine it caused VM solution to fail).

Even used Windows Me and Vista without major issues, but happened to luck out by shunning NVidia already at the time for Vista

Relying on WIM backups also makes it possible to restore from an Adaptec RAID controller A to an Adaptec B one, or Intel onboard for that matter.

Registry corruptions are no fun, but Windows does a fine job at keeping internal backups that can be restored easy via script commands during recovery boot. And you can boot that on system itself, or keep a USB boot medium around. Newegg today send me a shellshock alert that they are selling Microsoft Windows 10 installation/recovery optical discs for $7.99, but I haven't bothered with optical install medium for over a decade relying on a wide variety of USB sticks, which a couple years ago I consolidated on an SSD inside: https://www.amazon.com/gp/product/B00S3G12E6

[randy]

thank you as usual for your explanations, and I'm going to have to look into HSSentry now

Is this a tool only included with HomeSeer Pro?

Both versions have it.

[RoChess]

Both versions have it.

Never noticed the file, and of course I paid close attention when I skipped reading the manual

Fired it up, hs_sentry.log created, and first entry: 11/14/2019 7:49:13 PM:HomeSeer HS3 Sentry Service Started, using IP: 192.168.0.5

What would be a way to "test" that it works?

Manually killing HS3.exe process leads to: 11/14/2019 7:50:36 PM:Error accessing web server: Unable to connect to the remote server

But that did not auto-restart HomeSeer. Of course killing process is a lot different than it becoming unresponsive, but I still expected it to recover

Am I missing something?

[randy]

You ask why Windows is more difficult? Well, there is this thing called the registry. Once it gets corrupt, well then you're in trouble. It can be really fun (not) trying to go from 32bit Windows to 64 Windows HAL, or different disk controller types. Remember that one always needs compatible disk controller and processor architecture to get Windows to boot off backup.

Randy, you ought to start thinking about active-active clustering next. There you go... A feature request of HS4 that would worthwhile.

With three failures (one easily preventable) in five years, it is not that important to me. The only redundancy I really need is covered by a whole house generator I have no interest in increasing my energy footprint for hardware redundancy, or depending on cloud based services. I also cannot add redundancy to more than 140 Z-Wave devices, 3 OneWire servers or 4 Arduinos. My only critical automated system is heating and hot water and it reverts to manual control (thermostats and direct boiler control) any time the primary or secondary Arduino loses contact with HS. If automation is dead, most everything in the house still works by standard switches/dimmers/thermostats or an iOS app in the case of Hue bulbs. As far as HomeSeer software goes, it has been very reliable. The real test for me is it such that everything still works when HomeSeer is not running.

[randy]

Never noticed the file, and of course I paid close attention when I skipped reading the manual

Fired it up, hs_sentry.log created, and first entry: 11/14/2019 7:49:13 PM:HomeSeer HS3 Sentry Service Started, using IP: 192.168.0.5

What would be a way to "test" that it works?

Manually killing HS3.exe process leads to: 11/14/2019 7:50:36 PM:Error accessing web server: Unable to connect to the remote server

But that did not auto-restart HomeSeer. Of course killing process is a lot different than it becoming unresponsive, but I still expected it to recover

Am I missing something?

How long did you wait? The default behavior should make HS restart after 2 minutes 20 seconds (7 errors, checked once every 20 seconds) It always works here, such that I have to kill HSSentry when I shut HomeSeer down or it will restart on its own. From the help file regarding the HS config file:

[TankHankerous]

The most likely failure would be a hard drive. I decided to rebuild my system with RAID 1 disk mirroring. I know it is only software RAID but as I said the most likely failure will be a hard drive. If the mother board dies I have a SATA to USB3 dock that I can plug into a temporary system and boot. Disclosure: I have not tested this yet.

There are 2 types of hard drives. Those that have failed and those that will fail.

Do yourself a favor. Make a system image and store it on a thumb drive. Update it, and verify that it works and is not corrupt on a regular basis. There are 2 kinds of RAID users. Those who have had a RAID array let them down, and those who will. It's not a backup system.