In my server I use drive pooling software. I've been bitten by RAID, by non-raid, by whatever I have used in the past I've lost hard drives and data.
The best solution I had was when I used Windows Home Server v1. It had drive pooling built in and that worked great! The V1 had it's drawbacks, but it was better than anything I had used. One such drawback was the tombstoning and other pool coordinating information. I was not bitten by this, but there was a single drive point of failure. If that drive went, you had to HOPE you could recover the pool. Not ideal for someone already weary with recovering data.
When I moved to WHSV2 (for unrelated issues) Windows Home Server did not come with any drive pooling any longer. I found some aftermarket stuff. The one I settled on was From StableBit. I think I've posted on here regarding it before.
https://stablebit.com/DrivePool/Features
Anyway, I can add/eject drives at will and that lets me remove bad drives and grow my pool without the need to copy the entire set of data over to a new place (such as in RAID). It also can take ANYTHING that Windows sees as a drive as an input to the pool. I have one pool that is made up of SATA, USB, ISCSI drives. My understanding is that you can also pool SMB drives, but I've not tested that. It would be really neat to setup drive quotas on machines around the house and when they are available use that as a backup location. The physical placement of the files - who cares the software will manage that.
It is also very RAID-like in that when you setup a Pool, the Pool data is stored on each drive itself. This makes it very portable as you can move the drive to ANY PC and as long as you have the Stablebit middleware driver installed you will see that drive as part of the pool.
The other major point is related to the above point. You can actually READ the drives as normal hard drives in ANY PC anywhere. Mind you, it gets complex to do this, but you CAN access your files. I say it gets complex since the files of 1 folder could be on any number of your hard drives. However I've tested this and you just need to access each drive and copy the files to the same location and let Windows "merge" the folders. This will give you all your files back into 1 folder for recovery.
Note: it uses YOUR preferences on how it splits up files. I typically have each folder split all over the place since I tuned it for parallel read/write and network access preferred. If I copy 3-4 files at the same time in parallel, I end up utilizing almost every hard drive at the same time (have 10 hard drives in there now, copy 4 files, turns into 8 writes into the system).
I also realize that you would be saying that I'm now network limited...so why care? I use a really nice low end 24 port commercial switch from Netgear that does port binding. My server has 7 NICS in it with everything virtualized through ESXi. I have the motherboard built in NIC. Then added a 6 port Enterprise Server PCIe NIC that I got on Ebay for a STEAL (I think $75 including shipping - retail was over $400). 3 are for my router (Cable Modem / DSL / LAN) - and are attached to PFSense. One is dedicated for the ISCSI hard drive. Two are for the non-router LAN access and are bound together.
*note, I know I could use ESXi to eliminate the need for the "router-Lan" however this gives me a PHYSICAL connection that I can do things with for...reasons :-)
One spare for future use.
I also have a PCIe wifi card that I use with PFSense to setup a wifi connection.
When you couple this software with the Drive Scanner they make, you get a real system that manages the physical location of your data, amount of and location of duplication of data (for drive failure safety). Scanner also can alert the Pooling software of a pending failure (using S.M.A.R.T.). Then the pooling software can begin the evacuation of data in preparation of that drive being ejected. Really great stuff.
https://stablebit.com/Scanner/Features
Looking at my topology for the SATA setup, I've run out of space for direct connect SATA drives. Currently I have 2 hard drive cases (Vantech USB 3.0/SATA case... http://www.vantecusa.com/en/product/view_detail/488). Those cases support 4 drives as well as SATA Port Multiplying. Using the PCIe SATA card I have (RocketRaid), I actually use that card's 2 ports in SINGLE drive mode and attach the 4 drive cage to each port. They also have a nice cooling feature due to their shape. Looking at Google's compiled data (http://static.googleusercontent.com/...k_failures.pdf) you don't want the drives too cool anyway, so I tend to run these at a low speed. It's nice to keep the noise down anyway (my home office is 4' away from the server rack).
Recently I bought 4 SATA/IDE to USB connectors. The reason is that it was $2 cheaper to buy that and get the power supplies vs buying hard drive power supplies. Now you might be asking, why do you need JUST power supplies? I found an add-on card that does port multiplying. I intend to use an old raid cage that my father built me out of steel to hold my new drives. I use the port multiplier add-on to give me access to the machine and just needed power.
Why not just use the internal SATA ports? That is an issue because of how ESXi works. The drives basically get encoded with a container. So, all my data will be wrapped up into the containers. Not really ideal for portability as any PC will require ESXi for a purchased version of VMWare (as I understand the licensing) to be able to read that container. So, I really do NOT want to do that if I can help it.
Now, if you are still with me, memories...WHAT does any of this have to do with that?
Welp, I have (as I'm sure most do) old hard drives laying about that I need to destroy at some point. Since I needed to test out the hard drive power supply and it takes 5 seconds to plug in the SATA/IDE to USB cable...I was sorting through my pile of drives while I waited for my work machine to finish compiling programs.
I found that I have 3 unrecoverable click of death drives in this pile and 1 that is marginal, 1 that is fantastic but small and 1 that is fantastic but even smaller (100GB). The reasonable drives are being put into a pile to setup a large NAS that I'd like to run over ISCSI and have in a fire resistant box of some kind (however this is for another PeteC type post).
That smallest drive must have been an OS drive from one of my previous servers. I've had 4 machines that I would call actual servers. I can still remember testing out HomeSeer V2.4. That used an OLD Asus gaming motherboard. I bought it because it had Serial ports and one needs serial ports to access Home Automation goodies. I can still remember Rupp giving me advice (as well as many others). That motherboard was NOT designed to stay running 24x7. It eventually died. I got a new one under warranty. That one died and they blamed me for doing something to the boards. Anyway, they stretched out the tech support on that long enough that based on the original purchase date of the original board - my warranty ran out. I still have the board in a box. I should just toss it out.
I then went on to get a Super Micro Board. That thing was stable and worked well for a long time. Eventually I upgraded to a new Super Micro Board. In fact those boards are still in their server cases and sitting under a desk no less than 5' from me. One was a repurposed Dell case (my first PC I bought with my own pennies). I loved the way the air flowed through that case. The other was a monster SuperMicro Case. It stands 3.5' tall. It has 8 fans in it. WHAT WAS I THINKING? It also held 6 hard drives straight from the factory - WITHOUT taking up the 4 front slots (for CDroms / etc.). I now have a purchased Dell. I find that self built PCs only tend to last <5 years. That is good for a Desktop where you are upgrading it or replacing it in that time-frame. My servers I like to keep a LONG time. The Super Micros were really good units. They kept me up and running for almost my entire time I've used HomeSeer. My brother found a great deal on the Dell T110 Power Edge Servers. I ended up buying one and have been happier than anything. This is WAY more stable than ANYTHING I built and it was at least 1/2 the cost (due to the deal). It uses way less power (well that is the technology in it). I think I measured 38W without the hard drives running. However it has processor power (Xeon processor). When that thing ramps up the Server can utilize close to 350W (not including hard drives).
Between the last two servers. I think I found the OS drive. It is IDE, so I think it was the older of the two machines. This drive has HS 2.5.05, 2.5.0.8, as well as backups from 2011-2-27. I think this was the original OS drive that I had when I first started with HomeSeer!
I'm not willing to sort through too much of it since the drive started to make some noises and I'd like to pull the scripts off of it and place then into my GIT server. I'll do that later on when I have more time (I'm on lunch right now, so I need to get back to work). It made me remember all the fun I've had with HS 2 and finally getting to where it was SUPER ULTRA stable - where I could leave it alone and let life happen for the last 2-3 years. Everything just runs and I'm excited to finally start to get HS3 running - which as I understand it is even more stable. I of course have more sensors I need to add into the system / etc.
I suppose at some point, I need to start tossing out some of this OLD computer junk. I still have my first PC (a Packard Bell 386sx16 [didn't splurge for the DX]). I have 7 old computer cases, old video cards, etc. I'd ebay it for vintage reasons, but not sure it is worth my time to go through that effort. At least 1 of the old servers might be useful someday - however with all these RASPI, BeagleBone, and similar coming out to make IoT like purposeful systems, I'm not really certain there is a use for these old machines). Just makes me really think about it. And I've not been doing this anywhere NEAR a long as Pete! I remember a conversation with him regarding one of his first setups using the UltraSonic triggered modules from (I think) Radio Shack. I'm sure those were more based on the original Scottish(?) invention than the later Infomercial-Based X10.com.
Thank you to everyone on the board, I've certainly made some great friends, even 2 locals! Thank you to Rupp (I don't see any recent posts by him!?), he helped me through getting started and really seemed to have the patience needed as I started out with only some basic X10 knowledge. Thank you to PeteC and UncleMike as you both have helped me with my holistic / big picture goals of my system and what it needs to do. Also, thank you to the HS team. I've had a great and positive experience with you, one of the reasons I continue to support you as much as I can - when I can. I am certain I am leaving out others, please forgive me. I know my time on here was better because of this community.
--Dan (drozwood90)
The best solution I had was when I used Windows Home Server v1. It had drive pooling built in and that worked great! The V1 had it's drawbacks, but it was better than anything I had used. One such drawback was the tombstoning and other pool coordinating information. I was not bitten by this, but there was a single drive point of failure. If that drive went, you had to HOPE you could recover the pool. Not ideal for someone already weary with recovering data.
When I moved to WHSV2 (for unrelated issues) Windows Home Server did not come with any drive pooling any longer. I found some aftermarket stuff. The one I settled on was From StableBit. I think I've posted on here regarding it before.
https://stablebit.com/DrivePool/Features
Anyway, I can add/eject drives at will and that lets me remove bad drives and grow my pool without the need to copy the entire set of data over to a new place (such as in RAID). It also can take ANYTHING that Windows sees as a drive as an input to the pool. I have one pool that is made up of SATA, USB, ISCSI drives. My understanding is that you can also pool SMB drives, but I've not tested that. It would be really neat to setup drive quotas on machines around the house and when they are available use that as a backup location. The physical placement of the files - who cares the software will manage that.
It is also very RAID-like in that when you setup a Pool, the Pool data is stored on each drive itself. This makes it very portable as you can move the drive to ANY PC and as long as you have the Stablebit middleware driver installed you will see that drive as part of the pool.
The other major point is related to the above point. You can actually READ the drives as normal hard drives in ANY PC anywhere. Mind you, it gets complex to do this, but you CAN access your files. I say it gets complex since the files of 1 folder could be on any number of your hard drives. However I've tested this and you just need to access each drive and copy the files to the same location and let Windows "merge" the folders. This will give you all your files back into 1 folder for recovery.
Note: it uses YOUR preferences on how it splits up files. I typically have each folder split all over the place since I tuned it for parallel read/write and network access preferred. If I copy 3-4 files at the same time in parallel, I end up utilizing almost every hard drive at the same time (have 10 hard drives in there now, copy 4 files, turns into 8 writes into the system).
I also realize that you would be saying that I'm now network limited...so why care? I use a really nice low end 24 port commercial switch from Netgear that does port binding. My server has 7 NICS in it with everything virtualized through ESXi. I have the motherboard built in NIC. Then added a 6 port Enterprise Server PCIe NIC that I got on Ebay for a STEAL (I think $75 including shipping - retail was over $400). 3 are for my router (Cable Modem / DSL / LAN) - and are attached to PFSense. One is dedicated for the ISCSI hard drive. Two are for the non-router LAN access and are bound together.
*note, I know I could use ESXi to eliminate the need for the "router-Lan" however this gives me a PHYSICAL connection that I can do things with for...reasons :-)
One spare for future use.
I also have a PCIe wifi card that I use with PFSense to setup a wifi connection.
When you couple this software with the Drive Scanner they make, you get a real system that manages the physical location of your data, amount of and location of duplication of data (for drive failure safety). Scanner also can alert the Pooling software of a pending failure (using S.M.A.R.T.). Then the pooling software can begin the evacuation of data in preparation of that drive being ejected. Really great stuff.
https://stablebit.com/Scanner/Features
Looking at my topology for the SATA setup, I've run out of space for direct connect SATA drives. Currently I have 2 hard drive cases (Vantech USB 3.0/SATA case... http://www.vantecusa.com/en/product/view_detail/488). Those cases support 4 drives as well as SATA Port Multiplying. Using the PCIe SATA card I have (RocketRaid), I actually use that card's 2 ports in SINGLE drive mode and attach the 4 drive cage to each port. They also have a nice cooling feature due to their shape. Looking at Google's compiled data (http://static.googleusercontent.com/...k_failures.pdf) you don't want the drives too cool anyway, so I tend to run these at a low speed. It's nice to keep the noise down anyway (my home office is 4' away from the server rack).
Recently I bought 4 SATA/IDE to USB connectors. The reason is that it was $2 cheaper to buy that and get the power supplies vs buying hard drive power supplies. Now you might be asking, why do you need JUST power supplies? I found an add-on card that does port multiplying. I intend to use an old raid cage that my father built me out of steel to hold my new drives. I use the port multiplier add-on to give me access to the machine and just needed power.
Why not just use the internal SATA ports? That is an issue because of how ESXi works. The drives basically get encoded with a container. So, all my data will be wrapped up into the containers. Not really ideal for portability as any PC will require ESXi for a purchased version of VMWare (as I understand the licensing) to be able to read that container. So, I really do NOT want to do that if I can help it.
Now, if you are still with me, memories...WHAT does any of this have to do with that?
Welp, I have (as I'm sure most do) old hard drives laying about that I need to destroy at some point. Since I needed to test out the hard drive power supply and it takes 5 seconds to plug in the SATA/IDE to USB cable...I was sorting through my pile of drives while I waited for my work machine to finish compiling programs.
I found that I have 3 unrecoverable click of death drives in this pile and 1 that is marginal, 1 that is fantastic but small and 1 that is fantastic but even smaller (100GB). The reasonable drives are being put into a pile to setup a large NAS that I'd like to run over ISCSI and have in a fire resistant box of some kind (however this is for another PeteC type post).
That smallest drive must have been an OS drive from one of my previous servers. I've had 4 machines that I would call actual servers. I can still remember testing out HomeSeer V2.4. That used an OLD Asus gaming motherboard. I bought it because it had Serial ports and one needs serial ports to access Home Automation goodies. I can still remember Rupp giving me advice (as well as many others). That motherboard was NOT designed to stay running 24x7. It eventually died. I got a new one under warranty. That one died and they blamed me for doing something to the boards. Anyway, they stretched out the tech support on that long enough that based on the original purchase date of the original board - my warranty ran out. I still have the board in a box. I should just toss it out.
I then went on to get a Super Micro Board. That thing was stable and worked well for a long time. Eventually I upgraded to a new Super Micro Board. In fact those boards are still in their server cases and sitting under a desk no less than 5' from me. One was a repurposed Dell case (my first PC I bought with my own pennies). I loved the way the air flowed through that case. The other was a monster SuperMicro Case. It stands 3.5' tall. It has 8 fans in it. WHAT WAS I THINKING? It also held 6 hard drives straight from the factory - WITHOUT taking up the 4 front slots (for CDroms / etc.). I now have a purchased Dell. I find that self built PCs only tend to last <5 years. That is good for a Desktop where you are upgrading it or replacing it in that time-frame. My servers I like to keep a LONG time. The Super Micros were really good units. They kept me up and running for almost my entire time I've used HomeSeer. My brother found a great deal on the Dell T110 Power Edge Servers. I ended up buying one and have been happier than anything. This is WAY more stable than ANYTHING I built and it was at least 1/2 the cost (due to the deal). It uses way less power (well that is the technology in it). I think I measured 38W without the hard drives running. However it has processor power (Xeon processor). When that thing ramps up the Server can utilize close to 350W (not including hard drives).
Between the last two servers. I think I found the OS drive. It is IDE, so I think it was the older of the two machines. This drive has HS 2.5.05, 2.5.0.8, as well as backups from 2011-2-27. I think this was the original OS drive that I had when I first started with HomeSeer!
I'm not willing to sort through too much of it since the drive started to make some noises and I'd like to pull the scripts off of it and place then into my GIT server. I'll do that later on when I have more time (I'm on lunch right now, so I need to get back to work). It made me remember all the fun I've had with HS 2 and finally getting to where it was SUPER ULTRA stable - where I could leave it alone and let life happen for the last 2-3 years. Everything just runs and I'm excited to finally start to get HS3 running - which as I understand it is even more stable. I of course have more sensors I need to add into the system / etc.
I suppose at some point, I need to start tossing out some of this OLD computer junk. I still have my first PC (a Packard Bell 386sx16 [didn't splurge for the DX]). I have 7 old computer cases, old video cards, etc. I'd ebay it for vintage reasons, but not sure it is worth my time to go through that effort. At least 1 of the old servers might be useful someday - however with all these RASPI, BeagleBone, and similar coming out to make IoT like purposeful systems, I'm not really certain there is a use for these old machines). Just makes me really think about it. And I've not been doing this anywhere NEAR a long as Pete! I remember a conversation with him regarding one of his first setups using the UltraSonic triggered modules from (I think) Radio Shack. I'm sure those were more based on the original Scottish(?) invention than the later Infomercial-Based X10.com.
Thank you to everyone on the board, I've certainly made some great friends, even 2 locals! Thank you to Rupp (I don't see any recent posts by him!?), he helped me through getting started and really seemed to have the patience needed as I started out with only some basic X10 knowledge. Thank you to PeteC and UncleMike as you both have helped me with my holistic / big picture goals of my system and what it needs to do. Also, thank you to the HS team. I've had a great and positive experience with you, one of the reasons I continue to support you as much as I can - when I can. I am certain I am leaving out others, please forgive me. I know my time on here was better because of this community.
--Dan (drozwood90)
Comment