Curent Status

Tips and Techniques on day to day stuff

Curent Status

Postby jbv » Sat Sep 28, 2013 10:13 am

Hi Folks,

We are making progress. While I don't get much time during the week, I am pretty close to having the Drive Pooling fully functional.

I've still got a few glitches to sort out, although it is darn close. As always, it is the tricky stuff I'm trying to do like letting you have up to 4 independent DrivePools and having this all seamlessly work with our hot-mount/dismount so you can just plug stuff in and it auto-magically works, that is causing a bit of grief. I hope to have this sorted soon. Right now, it is so darn close that it's starting to become annoying.

As always, extensive testing is being conducted at every step. I'm currently playing with 3 different filesystems over 5 drives of different sizes with full load balancing on disk writes, so when this is working properly and released it won't be a mamby-pamby half-baked solution.

I'll keep you posted, but just wanted to let you know that right now I think we are about this >< close.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: Curent Status

Postby saintless » Sat Sep 28, 2013 1:35 pm

Hi, Brenton :)

Thanks for posting the FoxyRoxy progress.
I have managed to get back in order my IBM Netvista and it is again my testing machine.
This means I don't need firmware-linux packages anymore.

Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: Curent Status

Postby jbv » Wed Jan 01, 2014 11:16 am

Hi,

Sorry there hasn't been any progress for quite a while. I've been doing 12-14 hour days for the last 6 months and by the time I get home the last thing I've wanted to see has been a computer.

While my Christmas Break is almost over, I have been doing quite a bit of playing with FoxyRoxy. Specifically, I've been working on the Drive-Pooling which is critical to how I want this thing to work. Over the last week, I've been battling with quite a few things and I really don't know if they are just Linux glitches or the way my hardware is behaving. I have the Drive-Pooing working as originally intended but it is not consistent through reboots, nor is it consistent from a power-up. Basically, it does strange things and they are often different. The only time it is consistent is if I manually insert the hard drives one by one. When I do this, it auto-mounts everything perfectly and the Drive-Pool is also built perfectly. When the drive are mounted properly (manually) everything works exactly as I had hoped for including the space balancing. It is just weird how from a system start-up I get different results with no rhyme or reason as to what it does.

It is a bit like the way that my USB stick can either be seen as the first drive or the last drive in the system.
It's just weird and I can't explain it, nor can I work out why (that is the bit I hate the most).

I am starting to think that I may just need to bite the bullet, and create one script to build (start) the Drive-Pool and another to disassemble (stop) it. This may mean that the Drive-Pool won't be dynamically maintained, although there may be a way around that yet.

I have also just realised that at the moment, I can only have a single mount point into a drive pool for each drive. While you can have up to 4 Drive-Pools, I would like to let you split one physical partition into multiple areas, and place a different part of the partition in a different Drive-Pool. There is a reason for this and it relates to the databases used by things like the Logitech Media Server, minidlna and the like. Although it may be better to just have a special HDD/partition that these can live on as in reality they will probably need to be rebuilt at startup anyway. I will give this a little more thought over the next few days as I try to sort out the whole Drive-Pool pooling properly at startup thing.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: Curent Status

Postby SBP » Wed Jan 01, 2014 7:16 pm

Hi Brenton.

Good to hear from you, hopefully things Will cool Down for you in the new year. I often come by the foxyroxy forum, to see if the project is still alive, so it is nice to see you again here.

Steen
SBP
 
Posts: 74
Joined: Thu Aug 16, 2012 5:27 am
Location: Denmark

Re: Curent Status

Postby saintless » Wed Jan 01, 2014 8:25 pm

Hi, Brenton :)

Glad to chat with you here again. In my experience with puppy linux usb mounting is always a problem. I can't even copy large files over 600-700 Mb. The mount point is lost and mounting again changes the mount point. Debian and especially FoxyRoxy is much more stable in this.
If there is something I can research in this direction just name it.

Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: Curent Status

Postby jbv » Fri Jan 03, 2014 1:56 am

Hi Steen, Toni,

I also hope things will go a bit smoother this year. The last 12 months has just been silly.

With regard to the DrivePool, While I have updated the BIOS to the latest, it didn't make any difference. I'm pretty sure the "issue" is somewhere in the BIOS and/or the whole SATA drive thing.

My setup may be a tad more complex because I have port multipliers on the native SATA channels. This means that for each SATA channel on the main-board, I have 5 Hard Drives. The screwy thing is that just like the USB ports, they do not always enumerate exactly the same. I noticed this early on which is why I have made FoxyRoxy mount the drives/partitions by Volume Label rather than the /dev device which always changes.Something strange must be happening during the initialization and it may be something to do with SATA command-queuing, the way the drives respond, or it may be something as simple as threading. It may even be that for the initial testing I am using 5 old SATA drives and I know that 2 of them have problems. I really can't tell. All I can tell for sure is that sometimes the first drive doesn't mount now and sometimes all drives mount yet they don't get added to the DrivePool, even though all of my log files show that they have been added. When I test it all manually or run things after start-up it all works perfectly.

I don't want to put in my faster/newer known good drives for a few reasons, although the primary reason is that once I get this running solid with the current mix, I can be really confidant that the final system will be absolutely rock solid and bullet-proof. To make things even a little more exciting, I also have a mix of file-systems (including NTFS) on the 5 drives.

I am encouraged because I know that once things get mounted they stay mounted and as Toni has confirmed, it is all rock-solid. I have copied Giga-Bytes to the USB sticks and TerraBytes to the Hard Drives (both pooled and un-pooled) without any issue whatsoever. Thanks for the research offer Toni, but as I can't clearly explain what is happening or where, I wouldn't even know where to start pointing you :)

Over the last few days, I've been giving this a little thought and also looking at other options for Drive Pooling. While there are other ways to do what I am trying to achieve with the DrivePool, the more I look at the other options, the more I am convinced that the way I have chosen will be the most reliable, fastest and cleanest/easiest in the long run. Basically, I am just using the AUFS filesystem that we already have in FoxyRoxy. The documentation is a little screwy (in that it is somewhat brief - typical *nix) and the exact format/syntax of the commands to add/delete drives changes based on a few things, but I've got all that sorted and scripted, so in many ways the tough part is done. FWIW, you will even be able to plug in a USB HDD (or stick) and add it to the DrivePool if you wanted to, although I wouldn't recommend doing so.

Anyway, I think I've got a good solution (way around the start-up issue) and I may play with that a little today.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: Curent Status

Postby jbv » Fri Jan 03, 2014 12:44 pm

Hi Folks,

Got quite a bit done today. I totally restructured the DrivePool stuff and while I haven't done The Big Test yet, all of the scripts are working. They are much smoother than before and everything looks very solid. If I have any issues, I am pretty sure it will turn out to be something stupid I have done and that is why I won't give it the final test tonight. I'm a little weary and will probably spend hours chasing something that should only take minutes to find.

You can now have two directories on a single partition pooled into different drive pools and the whole thing looks to be working really nicely.

I will give it The Big Test tomorrow morning, but I am feeling quite confidant.

To make things a little more fun, I have one drive partitioned so that half of the drive is exFAT and the other half of the drive is HFS+
Yep, it works :)

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: Curent Status

Postby saintless » Fri Jan 03, 2014 4:02 pm

Hi, Brenton :)

I'm glad you have this working better and better. I understand drive-pooling is critical for FoxyRoxy as a NAS system.

Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: Curent Status

Postby jbv » Sat Jan 04, 2014 2:09 am

Hi Toni,

DrivePooling is working :P <fx: yay-team>

I did have one dumb little thing in my scripts that would have taken ages to find last night, but I found it first up this morning. :)

This confirms my thoughts that the reason for it not working properly is due to some race conditions that occur at initial start-up when the system is probing the drives.
What I have done is a little screwy but I'll try to explain it. Basically how the whole system (including hot-mounting) works is as follows:

I have hooks into the D-Bus message stream. When D-Bus sends a drive add message, it calls the hotmount script. The hotmount script examines the drive, determines if it is a traditional or GPT partition, then reads the partition table. It examines each partition table to determine the filesystem and then checks to see if it has a valid Volume Label. If it does, it then mounts that partition in /media using the Volume Label. It then looks for a DrivePool map file. If it finds one it then calls a postmount_partition script. The DrivePool map file is created a the end of rc.local, so it only exists after everything else has been started.

This means that when the system is first starting up, D-Bus will fire off the hotmount script for each volume (drive) and all of the partitions will be mounted, but because the (empty) DrivePool map file has not been created, the postmount script won't be called. In short, our drives get mounted properly, but there is no DrivePool yet.

When rc.local get processed, the first thing it does is clean-up the dpkg database and then looks to see if there are any other special scripts laying around. At the moment, there a 3 scripts that a person can have run at start-up (if they want)

1) Attach to and mount any remote NFS servers
2) Set a Webcam microphone level and/or video configuration
3) Build a DrivePool

Building a DrivePool, first creates an empty DrivePool map file (/var/logs/foxypool.map) Then it runs /etc/udev/scripts/BuildDrivePool.sh

The BuildDrivePool script parses the contents of /proc/partitions which shows all of the partitions the system has found. It strips out (ignores) any loopback devices (sqf container files) and only does anything with devices that are reported as being HDD Block type devices. For each device (partition) it finds, it then looks to see if it has been mounted. If it has been mounted, it then calls the postmount_partition script. As the DrivePool map file now exists, this then parses the FoxyDrive.conf file (Drive Pool Configuration file) and works out if there is anything in that partition that needs to be added to the drive pool and if so, which DrivePool (you can have 4) and where. Then it does it and updates the DrivePool map file.

It probably sounds a little messier than it really is. In reality it is quite simple, straightforward and quite elegant. As usual, there is quite a bit of error checking done with key things being logged.

We now have a few extra files.
/etc/udev/FoxyDrive.conf - This is the Drive Pool configuration file
/var/logs/foxydrives.log - Log file of all AutoMagic mounts, dis-mounts, DrivePool creation, addition, deletion
/var/logs/foxypool.map - Current state of the DrivePool(s)

There is a little bit more work to do on this before it is 100% finished.
1) I need to have more than one mount point into a drive pool from a single partition
- Basically, at the moment you can only put one directory from a partition into a Drive Pool.
I want to have my data directories on every Hard Drive laid out something like this:
\audio
\video
\image

Then I want to be able to put the \audio directory on that partition into the audio DrivePool, with \video and \image being similar. At the moment, you can put any single directory (or the entire root directory) into a DrivePool, but you can't have two different directories on the one partition in the same or different DrivePools. This won't be a big deal and I think I can can sort this one pretty easily.

2) I need to remove a volume/partition/mount-point from a DrivePool if the disk is removed or spun-down
This will take a little work as it will mean reading the DrivePool map file and finding everything associated with the drive/partition that has been removed/spun-down and then un-hooking it from the DrivePool. It shouldn't be a real problem as I have built the DrivePool map file in a way that I hope will make it somewhat easy to find stuff. As always, what will bring me undone will be the whole awk/sed thing, so there will be lots of reading and experimentation to get it perfect, but I know I can do it.

I may not get to do any of this this weekend, but I will try to keep things moving again.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: Curent Status

Postby jbv » Sat Jan 04, 2014 2:31 am

Hi Toni,

While reading your threads about Wheezy Lite, I noticed that someone thought it strange that I would waste my time trying to build a NAS when you can buy one for about a hundred bucks. He is right, you can, and you can also spend a whole lot more. I have tested quite a few NAS systems from reputable suppliers and I even have two at home. One is a QNAP with 5 x Very Fast 2Tb Samsung Drives in a RAID-5 configuration, the other is a smaller and much cheaper Netgear Stora with a 1TB Western Digital HDD. Personally, I have quite a few issues with commercial NAS boxes that I wont bother going into here.

However, now that I have the basics up and running, I thought it might be worthwhile doing some quick tests.
Now, keep in mind that The Stora is a cheap little NAS with a somewhat slow drive. The QNAP is what would now be a medium-range NAS but with good high-speed drives, and FoxyRoxy is my home-brew that currently has old and dodgy drives. Anyway, the results were somewhat of a surprise..

All tests were done from my main machine, over a 1Gb Ethernet link (all units are Gb Ethernet)

The Local Drive (6Gb SATA Super Fast Drive)
Local_6Gb-SATA_3Tb-Drive.png
Local_6Gb-SATA_3Tb-Drive.png (31.18 KiB) Viewed 10308 times


The Stora
STORA_1TB.png
STORA_1TB.png (25.75 KiB) Viewed 10308 times


The QNAP
QNAP_RAID-5.png
QNAP_RAID-5.png (25.79 KiB) Viewed 10308 times


FoxyRoxy
FoxyDrivePool.png
FoxyDrivePool.png (24.49 KiB) Viewed 10308 times


Nuff Said :)
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Next

Return to Using FoxyRoxyLinux



Who is online

Users browsing this forum: No registered users and 1 guest

cron