The Disk Drive Array

[jbv]

This is what I am thinking of for the drive-pool configuration file

Code: Select all

# FoxyRoxyLinux drive-pool configuration file

logfile=/var/log/foxydrives.log      # logfile (drive pooling and hot-swap)
mapfile=/var/log/foxypool.map      # current drive-pool mapping
drvpool1=/mediapool               # mount point for drive-pool 1
drvpool2=none                  # mount point for drive-pool 2
drvpool3=none                  # mount point for drive-pool 3
drvpool4=none                  # mount point for drive-pool 4

newdrive=no                     # Add unknown partitions to the Drive-Pool (yes/no)
newmount=drvpool1 rw root         # parameters to use if newdrive is to be added to pool

# Format of mounting parameters:
# 1 = partition label to add to the pool
# 2 = drive-pool to include the partition in
# 3 rw = full read/write access to the partition when pooled
# 3 ro = read only access to the partition when pooled
# 4 = directory to mount into the drive-pool (root) is the entire partition

label=disk_11 drvpool1 rw root
label=disk_12 drvpool1 rw root
label=disk_13 drvpool1 rw root
label=disk_14 drvpool1 rw root
label=disk_15 drvpool1 rw root



[jbv]

Since posting this and being swamped with comments/feedback, I've decided that what I have posted above is solid and robust enough for my requirements, and should do everything that I want and can foresee others finding better than the alternatives.

Therefore it is now "locked-and-loaded"

The disk array sub-system is going to take a bit of a back-seat for a couple of weeks, while I dig a little deeper and make some changes to the various filesystems we support. which basically means increasing, improving, and generally making it better. I am also going to add some disk/filesystem recovery/repair tools. This will take a couple/few weeks and then I'll pick this up where it was left off.

While this will mean a slight delay now, it will pay real dividends when you (or I) have a disk failure (on any machine) as FoxyRoxyLinux will also be the ultimate disk/filesystem recovery tool for almost any machine/filesystem (linux/windows/apple).

It also won't matter if the disk be a real HDD, USB, or SSD. Nor will it matter if the physical disk (HDD/USB/SSD) was create on another machine or operating system ... FoxyRoxyLinux will help you repair it and salvage what you can from the device. Simply boot the machine from a FoxyRoxyLinux USB stick or bring the device to a FoxyRoxyLinux machine. It really won't make any difference.

As my goal is to build a NAS with a lot of Hard Disk Drives and I want the ability to quickly move stuff from other media/mediums onto my NAS and easily recover from faults, this fits with the overall design plan/goal. I hope you agree and understand/recognise that while this may incur a slight delay now, we are much better handling it now instead of waiting until we have a disaster.

As with most of the things I have done to date, the testing and confirmation will probably take longer than the development, but I am genuinely hopeful that we will be back on-track in 2-3 weeks.

At the end of this, we should be able to take a HDD/USB/SSD drive from any Windows/Apple/Linux machine and drop it (or hot-plug it if your physical machine supports it) into FoxyRoxyLinux, at which point it will auto-magically be mounted to let you copy files to/from it at the maximum speed possible (for your machine) and/or recover/repair media from another machine that the other machine won't read/see.

As FoxyRoxyLinux can now read/write to 3TB (and larger) drives, the term SneakerNet now has a whole new meaning.
LAN parties will be a thing of the past.
I think we will need to define a new phrase ... " NAS nights "

... Remember, you heard it here first

Cheers, Brenton

[KazzaMozz]

Hi Brenton

At the end of this, we should be able to take a HDD/USB/SSD drive from any Windows/Apple/Linux machine and drop it (or hot-plug it if your physical machine supports it) into FoxyRoxyLinux, at which point it will auto-magically be mounted to let you copy files to/from it at the maximum speed possible (for your machine) and/or recover/repair media from another machine that the other machine won't read/see.

Foxy sure is going to be an amazing beast when it's all finalised. This gives my little NAS system with 5 snap-in drives a whole new life. I could just have that one for all the copying and setting up for my small client base, that use a licorice all-sorts of drives and operating systems and not have to have anything else.

However,I will add some partition repair stuff and other disk/filesystem test/check/repair tools, that I already have sort-a-kinda working. In all, I think this may end up being a very small injection (it will/should be under 20Mb)

This is very exciting. I've always thought Gparted lacked in this area and could be so much better than it is. If it all works to your plan it will be just awesome.
For me to be able to just snap a drive in repair/copy/format well pretty much by the sound of it whatever I need for repairing when my clients have screwed their drives up. Along with 3TB plus support..................you do my head in

This is like waiting for a Christmas present and not being able to open it till next year. Damn moving damn damn damn. That was my dummy spit for the day.

Hoopla can't wait to use this!!!
Cheers
Kazza

[jbv]

Hi Kazza,

to be able to just snap a drive in repair/copy/format well pretty much by the sound of it whatever I need for repairing when my clients have screwed their drives up. Along with 3TB plus support...

In a lot of scenarios, it could/should be even easier than that. I hope that you will quite probably just be able to pop a FoxyRoxyLinux USB stick into their machine. Boot it into FoxyRoxyLinux, and fix the issue there and then.

Cheers, Brenton

[KazzaMozz]

Hey Brenton

Boot it into FoxyRoxyLinux, and fix the issue there and then.

A dream come true. I only visit a few special people. Most of the time they bring the drives to me to fix these days I don't tend to do onsite fixes too bloody old for house calls.
Off topic. Looked at the apartment. Where I am going to fit all my stuff...........I'm in trouble it's lovely buy so tiny compared to here. Mind you there are 2 very interesting storage cupboards with the study nook. Definitely know what I will squeeze in there. Well some of it anyway. I will have to become very creative with such a small space from that which I have become accustomed to.

Cheers
Kazza

[jbv]

I made a bit of progress on this today. There were a few scenarios where things weren't being auto-mounted properly. This was primarily related to the way the auto-mount script supports the various partition tables and filesystems. Anyway, I think I've got it sorted now. The current test bed has drives with different partition structures and filesystems, all of which are properly being mounted at start-up and also during any hot-swap scenarios. In case I hadn't mentioned it earlier, there are also scripts to dismount and spin-down any drive. I'm going to spend a little ore time testing this before getting back into the drive pool stuff.

Cheers, Brenton

[jbv]

As at Beta-4, the Drive Array (Drive Pooling) is now 100% solid.
The drive pool is managed and controlled by /etc/udev/FoxyDrives.conf
If this file exists, the Drive Pool is built at start-up - it is looked for in /etc/rc.local

If Drive Pooling is not enabled (no FoxyDrives.conf file), then all drive/volume/partition changes are logged to /var/log/hotmount.log

If Drive Pooing is enabled (FoxyDrives.conf exists) then all drive/volume/partition changes are logged to the file pointed to in FoxyDrives.conf
The default drive log file is: /var/log/foxydisk.log
The default DrivePool MAP file is: /var/log/foxypool.map

Changes to the DrivePool can be made dynamically by adding/removing drives and the DrivePool MAP file will be kept up to date and orderly.
Drives/volumes/partitions are automatically added/removed from the drive pool in accordance with FoxyDrives.conf

While I have updated a few of the filesystem drivers, including exFAT which is the latest as at January 2015, an exFAT drive may not mount properly from a system start-up. I can't determine exactly why this is. Hot swapping and exFAT drive works fine, and sometimes it will also be okay from startup, but from startup sometimes it isn't. I have not seen any issues while Hot Swapping an exFAT drive.

All disk auto-mount and drive-pooling is done based on partition labels.
It is important that you do NOT have any spaces in the partition label.

You can have up to 4 independent drive pools.
A drive pool can use any combination of drives and any combination of filesystems.
For example, a single drive pool could contain one drive with ntfs, one drive with hfs+, one drive with xfs, one drive with exFAT, and it will work.

Drives are mounted into /media based on the partition label.
If you add/change/delete a file directly in /media/xxxx and that file is a part of the /mediapool, the change is immediately reflected in the /mediapool

There were some issues with some of the aufs compile options in the standard Debian Live distribution, which wreaked havoc in this area.
For this reason, FoxyRoxy has now been recompiled (using 100% standard Debian Sources - with all patches applied as at January 2015)
The freshly built kernel resolves most if not all of the issues found while getting the drive pool to work properly.
As this then required starting a lot of things from scratch, it was decided to no longer support the 464 and basic 686 kernels.
From this point forward, FoxyRoxy will only be developed using the new kernel (still 2.6.32-5-686-bigmem)

minidlna and other daemons/services that are watching /mediapool for changes will not see the change though. I hope to fix this in a future release.

For more info on the changes, please refer to This Thread