Extremely slow backup

[pepsimax]

Your database does not look right to me
I would try this:

1. Disable BLBackup (make sure the HSPI_BLBackup.exe process is not running - if it is then kill it)
2. Delete the BLBackup.s3db
3. Re-install the plugin
4. Enable the plugin

See if that helps

THANKS, the .s3db-file was 1,5 GB.

From 1 day + to less than 5 minutes in HS4.

[TomB]

That did it, it took 1 min, 10 secs to do a full backup after I deleted the database and reinstalled the plugin. That's more like the speed I would expect

Thank you so much to all for the input, glad we got there in the end.

Over the network to SMB/NAS it took 3 mins 50 secs, so very quick indeed.

[Blade]

Here is the log-file you asked for now running with SSD backing, and taking about 40-45 minutes. It's a dropbox link due to size.

https://www.dropbox.com/s/e30f92xws6...ailed.log?dl=0

Edit: It seems it process each file a ton of times, but I am not sure if that's how it should be or not. If you have time to check it out I would be most grateful.

Your database does not look right to me
I would try this:

1. Disable BLBackup (make sure the HSPI_BLBackup.exe process is not running - if it is then kill it)
2. Delete the BLBackup.s3db
3. Re-install the plugin
4. Enable the plugin

See if that helps

[Blade]

Quick question on the BLBackup.s3db file, is that necessary to keep for a long time or can I delete it and let it refresh? I have had it set to daily backup and 60 backups to keep, not sure if that is the reason but that file is now 160MB.

Don't want to remove it if it's vital to the operation of BLBackup though

Mine is only 29MB
The database is used by the plugin when running a backup. It will cause issues if you delete it

[TomB]

Here is the log-file you asked for now running with SSD backing, and taking about 40-45 minutes. It's a dropbox link due to size.

https://www.dropbox.com/s/e30f92xws6...ailed.log?dl=0

Edit: It seems it process each file a ton of times, but I am not sure if that's how it should be or not. If you have time to check it out I would be most grateful.

[TomB]

Quick question on the BLBackup.s3db file, is that necessary to keep for a long time or can I delete it and let it refresh? I have had it set to daily backup and 60 backups to keep, not sure if that is the reason but that file is now 160MB.

Don't want to remove it if it's vital to the operation of BLBackup though

[TomB]

Tom, I use BLBackup to backup my entire HS3 folder to the hard disk on another computer (not SSD) over the local network and it takes a fraction over 2 minutes for 2403 files. In my case I have it set to create a zip file so it is only writing one big file over the network rather than thousands of small files. That is probably the difference.

Steve

If you monitor your ZiP-file while the backup is going I assume you as well see it gradually increasing, its not doing local ZIP and then one file move, it adds to the remote ZIP-file the whole process.

But that's not whats making it slow on my side at least since it's the same speed locally as well.

[TomB]

Yes, as one mentioned, I am also creating just one ZIP-file.

The system is quick as any, it caps the Gbit NIC when doing SMB copying both to and from the system and my NAS (where I run my backups to), so it is just with BLBackup it is this slow. There was very little difference when I do file-structure instead of ZIP, just less CPU on the system itself.

A manual copy of the whole HS3 folder with about 3300 files takes less than 30 seconds so it does not seem to be any network issues.

The only thing common between the old and new system is the HS3 folder itself. I installed Windows 10, did all updates, then installed HS3 from fresh file, renamed the folder and copied in the old HS3 folder and run it.

It is fine though, I can live with the 45 minutes time, it just seems odd if should take so long. It was worse when it took 5+ hours but this is okay. I have like 20 SSDs in the house so why I decided to go with the spin-disk for this machine I don't really know, I won't ever do it again

[SteveMSJ]

Yes, I should read more carefully. However, mine is a 99MB zip file so something is up with Tom's system.

Steve

[randy]

Tom, I use BLBackup to backup my entire HS3 folder to the hard disk on another computer (not SSD) over the local network and it takes a fraction over 2 minutes for 2403 files. In my case I have it set to create a zip file so it is only writing one big file over the network rather than thousands of small files. That is probably the difference.

Steve

[randy]

My backup is folder based for the entire HS3 directory and all subdirectories. It is over 5gb to a Synology NAS over the network. It currently takes just under 15 minutes.

Jan-20 1:29:46 AM BLBackup Info The backup process (Folder Tree - Full) has completed at 1/20/2018 1:29:46 AM
Jan-20 1:15:00 AM BLBackup Info The backup process (Folder Tree - Full) has started at 1/20/2018 1:15:00 AM...

[SteveMSJ]

Ok, so I found the reason for the extremely slow performance. It was apparently the disk, not that the disk itself was slow, but it seems when doing backup in this way of Homeseer, it opens one session for each and every small file, and the result is slower access with spin-disk. I changed to a SSD drive again (cloned system-drive) and now I am back to about 45 minutes backup window, and I can live with it.

However, normal copy in Windows is a LOT faster, so I am not sure how you are doing the backup with your plugin as the performance is so much slower within than using normal copy/zip.

Either way, I am back to where I was before, so I guess that's as good as it gets right now

Tom, I use BLBackup to backup my entire HS3 folder to the hard disk on another computer (not SSD) over the local network and it takes a fraction over 2 minutes for 2403 files. In my case I have it set to create a zip file so it is only writing one big file over the network rather than thousands of small files. That is probably the difference.

Steve

[TomB]

Ok, so I found the reason for the extremely slow performance. It was apparently the disk, not that the disk itself was slow, but it seems when doing backup in this way of Homeseer, it opens one session for each and every small file, and the result is slower access with spin-disk. I changed to a SSD drive again (cloned system-drive) and now I am back to about 45 minutes backup window, and I can live with it.

However, normal copy in Windows is a LOT faster, so I am not sure how you are doing the backup with your plugin as the performance is so much slower within than using normal copy/zip.

Either way, I am back to where I was before, so I guess that's as good as it gets right now

[Blade]

I use a zip backup and I backup many folders from different computers and I have not seen it take an extended amount of time
Try and get me a debug log of the backup so I can see if something is taking longer than it should

[TomB]

Just update, took 5 hours 10 minutes with a local backup even.

The new computer is an Intel I5 with 16GB ram while the old was a Intel Celeron with 8GB ram, but the old one had a 256GB SSD while the new has a 750GB hybrid disk. But it should not be that big a difference, backing up 1,5GB shouldn't really take 45 minutes on the old one either.

What would you say is the normal backup period? As Zip or as folder, what do you usually use? I have always used ZIP, but the CPU is only at 10% or so when running backup.