Curent Status

Tips and Techniques on day to day stuff

Re: Curent Status

Postby jbv » Sat Jan 04, 2014 6:07 am

Hi,

Item 1) on my DrivePool "todo" list has been done :)

It is now possible to have the following structure on a Hard Drive
\video ... This directory can be pooled with any other drives or directories
\audio ... This directory can be pooled with any other drives or directories
\image ... This directory can be pooled with any other drives or directories
Of course, you can still put an entire drive (from the root directory) into a DrivePool.

DrivePooling is easily configured using a configuration file named FoxyDrives.conf which must live in /etc/udev
Code: Select all
# FoxyDrives.conf FoxyRoxyLinux drive-pool configuration file
# /etc/udev

logfile=/var/log/foxydisk.log        # logfile (drive pooling and hot-swap)
mapfile=/var/log/foxypool.map         # current drive-pool mapping

pool=drvpool1 /mediapool/video      # mount point for drive-pool 1
pool=drvpool2 /mediapool/audio      # mount point for drive-pool 2
pool=drvpool3 none                     # mount point for drive-pool 3
pool=drvpool4 none                     # mount point for drive-pool 4

newdrive=no                        # Add unknown partitions to the Drive-Pool (yes/no)
newmount=drvpool1 rw root            # parameters to use if newdrive is to be added to pool

# Format of mounting parameters:
# 1 = partition label to add to the pool
# 2 = drive-pool to include the partition in
# 3 rw = full read/write access to the partition when pooled
# 3 ro = read only access to the partition when pooled
# 4 = directory to mount into the drive-pool (root) is the entire partition

label=disc_1 drvpool1 rw /video         # Line Must be Terminated
label=disc_1 drvpool2 rw /audio         # Line Must be Terminated
label=disc_2 drvpool1 rw /video      # Line Must be Terminated
label=disc_3 drvpool1 rw /video      # Line Must be Terminated
label=disc_3 drvpool2 rw /audio      # Line Must be Terminated
label=disk11 drvpool2 rw root      # Line Must be Terminated
label=disk12 drvpool2 rw root      # Line Must be Terminated

I should note the option to add unknown drives to the pool has not been implemented and it probably won't be.
This is something that I thought may have been a good idea when I started thinking about how to do the configuration file, so I made provision for it.
As I started testing, I came to realise that letting an unknown drive be pooled can have some nasty and unintended consequences, so decided to not implement it.
I have left the configuration option there because it doesn't hurt anything. I will probably remove it sometime later.

This is the contents of /var/log/foxydisk.log
Code: Select all
Mounted exFAT: sdf1 to /media/EXFAT
Mounted ntfs: sde1 to /media/disc_1
Mounted hfsplus: sdf2 to /media/HFSPLUS
Mounted Linux: sdd1 to /media/disc_3
Mounted Linux: sdc1 to /media/disk12
Mounted Linux: sdb1 to /media/disk11
Building DrivePool:
DrivePool (start): /mediapool/audio with /media/disk11
DrivePool (append): /mediapool/audio with /media/disk12
DrivePool (create): /mediapool/video with /media/disc_1/video
DrivePool (append): /mediapool/audio with /media/disc_1/audio
DrivePool (append): /mediapool/video with /media/disc_3/video
DrivePool (append): /mediapool/audio with /media/disc_3/audio


This is the contents of /var/log/foxypool.map
Code: Select all
/mediapool/audio rw /media/disk11
/mediapool/audio rw /media/disk12
/mediapool/video rw /media/disc_1/video
/mediapool/audio rw /media/disc_1/audio
/mediapool/video rw /media/disc_3/video
/mediapool/audio rw /media/disc_3/audio


I may need to create or find some images to help explain what DrivePooling is and how it works, but basically, it lets you configure all of your Hard Drives into groups. Then you just read/write to the group and you don't need to worry which physical disc drive things are on. If you were to place 4 x 500Gb hard disks into one DrivePool then you will end up with a single 2Tb drive.

In the stuff above, I have created two DrivePools, one is /mediapool/video and the other is /mediapool/audio

All of the directories from every drive appear in the DrivePool, so I can read/write from/to any file. If I put a new file into a directory, the DrivePool will automatically put the file on the physical disk that has the most space available. If that physical disk drive does not yet contain the right directory, it is automatically created when the new file is added.

Oh, I should add that this has used about 0 Bytes of additional RAM, so basically, it's another "freebie" :)

I won't do any more today. I will have a look at unpooling/removing drives from the pool during the week.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: Curent Status

Postby saintless » Sat Jan 04, 2014 6:37 am

Hi, Brenton,
jbv wrote:DrivePooling is working :P <fx: yay-team>

I did have one dumb little thing in my scripts that would have taken ages to find last night, but I found it first up this morning. :)

Good news. It means you should rest more for a while when a problem can't be solved at one point. :)
I read your explanations how all drive-pooling works. I think I see the general picture of it and I admire your efforts to make it work so good. I also understand how critical is this for FoxyRoxy as a NAS system. I'm sure we can't find some similar drive-pooling solution in any other OS.
If NAS box was something as good I'm sure you would not bother at all to start FoxyRoxy :)

BTW I think taking the squeeze base was the best thing we can do. From what I have as a results with wheezy it seems it was too soon to make it stable. It has a lot of problems and bugs. For example if you use the new full persistent (now called persistence) method and you have more than one squashfs in /live , the overlay order changes backward. This means 01 will overlay 02 squashfs. If you don't use full persistent or without persistent file it works as it should. Also the new version of live-boot and initramfs doesn't have at all Copy on Write option.
The only improvement I see in wheezy is that the kernel supports new and very old hardware and it has xz compression support builtin so it makes around 20% smaller squash files.

Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: Curent Status

Postby jbv » Sat Jan 04, 2014 7:44 am

Hi Toni,

Yeah, but a year long rest is a little over the top :lol:

DrivePooling is sort of like the way we use the Squash files and how they get Overlaid, except it is with live disk drives that you can read/write to. Where a Squash File has all of the various files already inside it and can't grow or be written to, the DrivePool has all of the available space from all of the disk drives available.

You are correct in that there is nothing I know of that does this the way that FoxyRoxy does, and I really do think this is the best way to do it. Before I started building FoxyRoxy, I spent a lot of time researching all of the commercial NAS product and I also reviewed most of the commercial and free (OpenSource) NAS software products. I also carefully reviewed all of the various filesystems that had various forms of RAID and DrivePooling internally (such as ZFS and BTRFS). ZFS came close to being ideal however it has some really messy areas if you need or want to increase a DrivePool (add another disk drive) or change the size. I don't think BTRFS is much better in this regard. While both products have probably changed a bit in the time that I've been playing with this, I am still confidant that where I'm headed is right - at least it is for me :)

If you want or need more drive space, just chuck in another drive and add it to the Pool = Job Done :)
As I've explained before, everything always gets written in full to a single drive, so if a drive crashes, you just lose what is on that drive and nothing more. When I have the snapshot raid working (once Pooling is 100% finished), you will easily be able to recover the data that was on the crashed drive. As an added bonus, you will still be able to take the drive out and put it in another machine to run some low-level diagnostics on it or recover what you can. Chances are that you will even be able to do this from within your FoxyRoxy machine without needing to take the drive out.

Most if not all of the Linux NAS product I have looked at (both free and commercial) have either used mhddfs or similar tools to provide their RAID and/or Drive Pool capabilities. Some now use something called Greyhole, which in turn uses the lower levels of SAMBA networking to provide Drive Pooling within the same machine. While mhddfs works, it just didn't seem elegant. If I was planning to implement a traditional form of RAID, the I would definitely use mhddfs to provide both the RAID and Drive Pool, even though I'm not sure how easy it is to change/modify/add to the Drive Pool compared to how we do it (I forget the details of all of these products, as it was a while ago now). Greyhole does some neat stuff, but it spends a lot of time copying things around which in turn places a pretty high overhead on the system and I can only imagine that it has the potential for things to get pretty ugly when stuff goes wrong. We can stick in any sort of Hard Drive with any partition and of any size. It doesn't matter if it is a USB Flash Drive, a USB HDD, or a 3 or 4 Tb SATA drive. We handle it.

I was pretty sure I could get AUFS to do this when I first started playing with sqf files and working out how to dynamically load and unload them - something I may come back to once everything else is finished, although the way we can now neatly inject stuff I don't see this as being as important now. Anyway, getting AUFS to do this is pretty straight forward once you've got it sussed. Scripting it properly was a tad tricky as you need to know quite a bit about the system and the state of AUFS and make tweaks to the mount process depending on a few things. Being able to have a basic configuration file and doing the whole thing with scripts took me longer than I thought because as you know I'm not a Linux person and every time I need to do something with BASH, awk, sed and the other command-line tools, I need to research it. Not being something I use other than here, I also tend to forget most (all) of it :)

Anyway, I'm really quite pleased with what we have working. The system overhead is zero (on both memory and processor), it is configurable, you can change it on the fly, and best of all, you can make changes without loosing or damaging anything. I"m sure there will be a bit of a memory overhead once I've got 20 4Tb drives plugged into it, but I'll worry about that when it happens :) Once I've got the removal working nicely, I will get the RAID stuff sorted. Then I will review the various services and where to best put things like database files for minidlna, the Logitech Media Server, then I'm going to have a beer :)

Actually, I'm going to have a beer now. I've earned it :)

Cheers, Brenton

PS ... May chat to you about your Wheezy observations in a day or two.
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: Curent Status

Postby saintless » Sat Jan 04, 2014 9:32 am

jbv wrote:Actually, I'm going to have a beer now. I've earned it :

PS ... May chat to you about your Wheezy observations in a day or two.

You deserve a beer , Brenton. I will have one also :P

Wheezy is not something that interesting me personal because of my old hardware. I have what I need from squeezy to be light and fast on my computers. If there was not others to have interest from wheezy it was going to be just boot to desktop + working apt-get core system.
I've got /live/cow back with initrd.img created with old version of live-boot and initramfs-tools. It works well.
If you think we can use something from wheezy in FoxyRoxy it's OK, but otherwise don't waste time to think of this.
Even if wheezy doesn't have full persistent backward overlay problem it will still missing /live/cow option which I need.

Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: Curent Status

Postby KazzaMozz » Fri Jan 24, 2014 11:56 pm

Hi Fellas
I'm not dead just caught up with our business model right now and getting everything right.
All I can say is WOW with what you have been doing and after having to go the commercial NAS systems which abosulutely do suck I truly need to get back into Foxy training again.
Due to the concensus of the combined group 4 of us. 3 pushed for Windows and little old me wanted to go the Linux way. Needless to say I got voted down and out, even though I do run that side of things. They have their way for now......Well their end is Windows but I have a combination of the two and I don't need to tell you which I prefer. Windows 8.1 is an absolute dog and waste of space...Grrr but this is what they all insisted on needing and wanting and could handle. All of them have grown up with Windblows and so love it. Also Exchange Mail (OMG) they can't run simple old Gmail but they love Exchange, please explain........Anyway that's a whole different issue I have with them. It's left up to me to keep it on the straight and narrow and I HATE IT.

I'm very interested in what you have achieved so far and sorry I haven't been around much either but priorities and such. Hopefully as we progress further as a business and things settle down somewhat I can start to spend some time on catching up with all the work you have been doing in here. I really like the sound of the drive pooling as this would work brilliantly with what I'm doing. For the office 3 non savvy gurus we ended up with a DX4000 Windows 2008R2 server for their data......which aint to happy with Windows 8.1 but with workarounds it's OK not ideal but they needed a quick solution. I now have it at home as they couldn't work that out either. They all admit they just can't understand it.....nope they don't want to. So they stick to selling and input of steel crap and documents and well I've now got all the control I want..... :twisted: suits me.

Meanwhile I work mainly from home (thank goodness) as the office is too small and originally there was only to be 3 of us and we have now grown to 4. Anyway that's another story.
I'm glad you have decided to continue with your project as it is definitely needed with the rubbish NAS out there right now. As I have a little 2 x HP Proliants 50L & 40L which I think you already know, this will be gold for those. Currently I run GeeXbox 3.1 on one as a media center the other has Mint 13 and runs XBMC used as a backup off site mainly.

Okay enough said I will endeavour to catchup and see how far behind I am and hopefully soon be able to engage again in what's going on. I also have a number of other little NAS systems which I will definitely be testing your drive pooling on ...............I have some old drives lying around so if this work like it reads that will be brilliant.

Great work guys!!
Cheers

Kazza
User avatar
KazzaMozz
 
Posts: 332
Joined: Tue Aug 21, 2012 9:59 pm
Location: Australia

Re: Curent Status

Postby saintless » Sat Jan 25, 2014 10:13 pm

Hi, Kazza :)
And Happy New Year to you!
Nice to read you are OK and have fun with Windows 8 :lol:
I do some test trials with Debian on different direction the last 2-3 months and hope this keeps me in shape.
Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: Curent Status

Postby KazzaMozz » Sun Nov 30, 2014 9:22 am

As always impressed in what you are trying to achieve
User avatar
KazzaMozz
 
Posts: 332
Joined: Tue Aug 21, 2012 9:59 pm
Location: Australia

Previous

Return to Using FoxyRoxyLinux



Who is online

Users browsing this forum: No registered users and 5 guests

cron