2016 ... The Good, The Bad, The Ugly

Tips and Techniques on day to day stuff

2016 ... The Good, The Bad, The Ugly

Postby jbv » Sat Feb 13, 2016 7:05 am

Hi folks,

Yet another new year, wow how time drags when you're super busy :lol:

I know it has been a very long-time between drinks, however it is almost there.
I am currently preparing for a working release of my end goal - The ulimate Home NAS, however as expected there have been a few speed bumps along the way.

Lets start with: The Good
I have achieved all of my goals for FoxyRoxy to be my ultimate Home NAS
FoxyRoxy is currently in the final stages of testing, which means I am currently copying files onto their final medium.
- This will take a little time, as I currently have about 30 Hard Drives (various capacities from 2Tb to 4Tb) with files scattered all over them and I am trying to sort them as I build the final NAS drives. I am also doing my best to ensure that the final drives are not fragmented, this means I need to copy stuff manually and am checking where things need to be before copying. Basically, this will take time.

Having done a lot of testing and thinking, planning, along the way, I have also changed how quite a few things work.
This includes everything from the DLNA Servers, the SAMBA Servers, the Drive Pooling, the Snap-Shot RAID, basically everything.

To this end, FoxyRoxy now supports:
Multiple DLNA Servers from the same box - My configuration has 5 DLNA Servers, however there is no limit
Multiple Drive Pools within the same box - My configuration has 5 Drive Pools, however there is no limit
Multiple SnapShot-RAID arrangements - My configuration has 5 arrangements, however there is no limit
Provision for 240 Hard Drives in the one server - My configuration has 24 Hard Drives in the one server (plus 1 x USB and 6 x SSD drives)

The Bad
I have run into some serious limitations (restrictions/issues) with using the 2.6.3x kernel from Debian Squeeze.
There are no short-cuts or easy fixes for these that I have been able to find, and I have spent a lot of time looking and trying things.
These restrictions/limitations are:

USB 3 devices will not work properly. They will work as USB 2 devices, but I can't get USB-3 to work reliably or properly. This is a kernel problem and won't be resolved. As my intent is to use this as a NAS with real Hard Drives, this is not a show-stopper for me, as it just means it may take a little longer for files to be copied from a USB3 drive.

Some of the newer sound chips won't work properly. Once again, this is a kernel limitation so don't expect your NAS Server to be able to play audio. Once again, something I can live without so I am ignoring it.

The Ugly
These are not really issues with FoxyRoxy, but more with hardware, however I need to offer some advice based on my experience.
Intel Atom processors do not make good CPUs for FoxyRoxy. While FoxyRoxy was initially built on Gigabyte mainboard with an Intel Atom D525 CPU and 4Gb of RAM, the Atom has some serious issues. Firstly, you can not use SATA port-multipliers with the internal SATA ports. I have also learnt the hard way, that third-party SATA chips and port-expander chips have problems with drives over 4Tb.

To this end, my current configuration for FoxyRoxy is an Intel core i3 processor (i3-4170), a Gigabyte B85M-D3H-A mainboard, a LSI SAS-9211-8i Disk Controller, and Intel RES2SV240 SAS Port-Expander

FoxyRoxy
I can confirm that FoxyRoxy supports 8Tb Hard Drives, although there may be issues if you are using an Intel Atom or sata port expanders. With a SAS Disk Controller and port-expander, you will not have problems. My current hardware can support up to 24 x 8Tb HDD - Tested and confirmed with 8Tb HDDs on each disk channel.

Testing and debugging has taken ages. Using faulty HDDs for testing and certification was very helpful, yet it also took ages to work out if issues were related to the faulty drives or FoxyRoxy. Similarly, the issues I discovered with various logic chips took ages to find and confirm as being IC design issues or within FoxyRoxy itself.

Along the way, I have updated pretty much everything within FoxyRoxy. The latest Intel Network cards and CPU based network devices are supported, and I have done my best to keep everything else at the most recent revision. I actually stopped updating stuff in January of 2016 and I am not likely to update anything now unless there are major issues.

I have setup the basic FoxyRoxy image so that it will work and provide full networking services and DLNA server support with no external Hard Drives.
If you have external Hard Drives, then using a simple naming convention (which I will detail later) will see it auto-magically supported.
Similarly, if you have a mega-drive arrangement then it can automagically detect this and support things right out of the box.

I've got another week or two of copying files and preparing for the final certification (testing) before I make the final ISO for uploading.

Over the next few days/week, I will try to give a little more background as to how I have arrived where I am and and try to give some insight into the whys and wherefores.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: 2016 ... The Good, The Bad, The Ugly

Postby saintless » Sat Feb 13, 2016 8:52 am

Hi Brenton.

Nice to read about new iso update.

The older kernel limitation is not easy to deal with. Changing to newer kernel is easy but since you had to compile the included kernel to fix some things I'm sure changing the kernel will break something important in FoxyRoxy.

After so much work done on FoxyRoxy you should be proud and happy with the result.
Congratulations!

Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: 2016 ... The Good, The Bad, The Ugly

Postby jbv » Sun Feb 14, 2016 6:20 am

Hi Toni,

Thanks. I am happy, although it is a little disappointing that it's not perfect.
I spent about 2 weeks trying to sort out the sound and USB3 for my mainboard, but sadly, it just can't be fixed.

My kernel compilations were really only to ensure that I had the latest Intel Network drivers, as so many new mainboards are now using the Intel embedded Chipset networking and sound. While I could compile the latest network drivers with the 2.x kernel, the latest Intel sound also requires a 3.x kernel, as does the sound chip my new mainboard uses. The only other things I did while compiling the kernel was to increase some buffers and the number of iwatch's you can have out. Changing to a 3.x kernel will just break to much stuff and if I am honest with myself, I can live with the lack of USB3 and sound on my server.

Getting the disk subsystem right was a much higher priority. It took quite a while to work it all out and be sure that it was going to work properly.
I blame a friend of mine who sent me a link to a 24 x HDD bay rack mount enclosure, which in hindsight, I am very glad he did.
When I first started, my vision was to use some SATA port expanders and some 5-in-3 enclosures. I've got a few of these and everything was going fine.
The real cost of a NAS is the cost of the enclosure. You've really got to work out the cost per hard drive. In this regard, when I started SATA port expanders and the 5-in-3 enclosures were the most cost effective way to go. As FoxyRoxy started to take shape, hard drive capacities kept growing, and SAS controllers and suitable enclosures started to drop significantly in price. They are now at the point where it is cheaper to use a SAS setup with SATA drives. While I started sourcing SAS parts, I kept using the SATA equipment for development and testing. When I get my first couple of 8Tb SATA drives, the SATA controllers and expanders started falling over and doing strange things. As you may recall, I have been using quite a few drives that have faults to ensure that everything works and handles faults and errors. It took ages to confirm that most of my issues and problems were the various SATA controllers and expanders, and not the drives or FoxyRoxy. I only got the last few bits of the final setup before Christmas and during the holiday period, I finally built the machine. I then had to test everything again, which took quite a few days. With the SAS disk sub-system, everything worked perfectly, even with the faulty drives. The 8Tb drives are also running without any issues or problems, which is to be expected as they are documented as having been tested and certified by Avago (the SAS controller manufacturer). All up the new machine cost about $1,500 which is only $62 per drive. Not bad for a box that can hold 192 Tb !

Anyway, FoxyRoxy still works perfectly without any hard drives, with only 1 hard drive, or with however many drives you have. It does support SATA port expanders, but as I have learnt, not all chipsets and disc controllers do. Along similar lines, you can get strange things happening with a purely SATA setup, especially if you have port-expanders and very large drives.

I will leave FoxyRoxy as it is now. Later in the year, I may review things and look to rebuild a newer version on a later release.

I still have a bit of certifcation testing to do which is going to take some time.
During this time, I will start to document things here and then I will prepare a final build which I will post.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

The Disk Sub-System

Postby jbv » Sun Feb 14, 2016 7:38 am

The following information is not an acknowledgement or endorsement for any product, nor is it a guarantee that something will work for you or is supported.
FoxyRoxy is available to you free of charge and comes with absolutely no warranties, guarantees, or support. The following is simply what works for me.
The exact same equipment may or may not work for you. You should exercise all care and be prepared to accept all responsibility as to weather something will work or not.
If it does not, then I cannot and will not be able to assist you. Even if it does work for me. - You have been warned.

To the best of my knowledge, FoxyRoxy will work with almost any Disk Controller.
It has been tested with various mainboards and chipsets.

To date, the only issues that have been found are as follows:
USB3 does not work. Audio may not work with Later Generation Mainboards
Issues have been found with the Intel G33 express chipset

It has been tested with various SATA controllers and SATA port-expanders.
To date, the only issues that have been found are with some SATA port-expanders, when under heavy load, or with 8Tb hard drives attached via a port-expander.
These issues are related to the SATA controllers and port-expanders. Note, not all SATA controllers support port-expanders.
To date, no issues have been found with SATA drives connected directly to mainboard SATA controller ports.

One multi-drive arrangement that is working for me is:
Gigabyte B85M-D3H-A mainboard
Avago LSI SAS 9211-8i SAS controller
Intel RES2SV240 SAS expander

In my configuration, I have 24 x 6Gb SAS/SATA ports to support up to 24 x 6Gb SAS/SATA drives
The mainboard SATA ports are currently not used, although this will change the near future, as I intend to connect a SSD drive to the primary mainboard SATA connector.

I have wired/connected the Hard Drives as show in this map:

FoxyDrives.png
FoxyDrives
FoxyDrives.png (42.1 KiB) Viewed 9134 times


Some notes on the hardware chosen.
The Gigabyte mainboard was chosen as it has 1 x PCIe-16 socket required for the SAS controller and 1 x PCIe-4 socket for the SAS expander.
It is/was at the bottom end of the market price wise, allows for a lower cost CPU and yet provides the features required.

The Avago LSI SAS 9211-8i controller can be found on eBay for about US$100 (new in original packaging). It will usually be supplied with raid firmware as the card can provide hardware raid-0 and raid-1. However, for my setup, I just need it to be a Host Bus Adapter and therefore the card as received requires the firmware to be reflashed with "IT" firmware. As at the time of writing, this card is still fully supported by Avago and the firmware is updated to be kept in line with the current newer controllers from Avago. While you could (theoretically - not tested) use a later more capable controller, you would still need to ensure that the controller has "IT" firmware loaded (or is reflashed). Reflashing the firmware on UEFI BIOS mainboards is a real pain the the backside. I can not tell you how to do this, nor can I, or will I offer any assistance. This controller has also been supplied to a few OEM customers including IBM, and Dell, where it was used in some of their older servers. You may be able to purchase an older IBM, Dell, or other card and reflash it with the latest Avago firmware. The latest IT firmware is freely available on the Avago website. As for how to re-flash the firmware, you will need to find a UEFI command interpreter that suits your mainboard. The Avago firmware package does include tools for reflashing from DOS, Linux and UEFI BIOS. On my mainboards, the UEFI BIOS' stopped reflashing from DOS or Linux.

The Intel RES2SV240 SAS expander can be found on eBay for about US$100 (also new in original packaging and with cables).
There are other slightly lower cost options from HP and others, although to the best of my knowledge, these other expanders do not support 6Gb datarates.

SAS drives are typically used in commercial NAS and data-centre storage sub-systems. SAS drives typically have higher spin rates and higher data throughput. They are also quite expensive when compared to SATA drives. SATA is a sub-set of SAS. As such, you can use SATA drives in a SAS disk system. My configuration is currently a mix of 8Tb Seagate SATA drives, 2/3/4 Tb Seagate and Samsung drives. All of my hard drives are SATA. Most support SATA-2 3Gb transfer rates, whereas the 8Tb and 4Tb drives support SATA-3 6Gb transfer rates. The SAS controller and expander as detailed above use SFF-8087 connectors. These connectors send 4 data channels down the 1 multi-core cable. My enclosure also has SFF-8087 connectors. You can get a SAS to SATA fan-out cable that has 1 x SFF-8087 connector and 4 x SATA connectors. These are easily found on eBay and in most discount (mail-order) computer stores.

I will try to explain my logic for the disk arrangement, and channel layout in the next section where I will show my intended drive-pool arrangement.
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Hard Drive - AutoMounting

Postby jbv » Sun Feb 21, 2016 4:45 am

FoxyRoxyLinux will auto-magically mount any hard drive (including USB thumb-drives and external drives).
Drive detection is done by monitoring and hooking into D-Bus messages that are generated when devices are detected.

The script that does this is \etc\udev\scripts\disk-hotmount.sh
The drive can have a standard or GPT partition, and multiple partitions are also supported. Almost every filesystem is supported.
The volume is mounted into /media with the volume (partition) label used as the mountpoint.
I chose to use the volume (partition) label instead of UUID's or any other mechanism as in my view this is the easiest way for a person (aka: me) to know where things will be located and found. It also allows a drive to easily be replaced by simply giving the new drive the same label. There is one caveat here. You should not have any spaces in the volume label.

At the tail of the disk-hotmount.sh script, \etc\udev\scripts\postmount_partition.sh is called.
This is the script that creates and maintains the drive pool(s)
All script based mounting and un-mounting writes to a logfile, /var/log/hotmount.log

While mounting drives and volumes, some parameters are set such as read-ahead buffer size, time of disk in-activity before placing a drive in standby mode.
To keep the number of configuration files to a minimum, these parameters are stored in the Drive Pool configuration file.

The following scripts are in /scripts which is in the path, so they are effectively command line commands you can enter from a Terminal or Command prompt.
watchudev ... Shows udev D-Bus messages
drop_cache ... Flushes internal disk read-ahead buffers/caches
CreateDisk ... Prepares a disk. Creates a GPT partition, configures the entire disk to be a single volume with XFS filesystem, formats the volume to XFS and thennames the volume
spindown ... flushes any buffered writes to a disk, removes the disk from a drive pool (if it was in one) spinsdown the drive and unmounts it, ready for drive removal
remount ... remounts a volume

Calling any of the above scripts without parameters will show the command-line usage.
Or, you could look at the script :)
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Drive Pooling

Postby jbv » Sun Feb 21, 2016 5:05 am

FoxyRoxyLinux supports an unlimited number of "Drive Pools"
Any hard drive volume (partition) can be placed in any drive pool.

A volume can have any supported filesystem, and you can mix and match filesystems within a single drive pool. This means that you could have drives (volumes) with any mix of NTFS, ext3, FAT32 or XFS, all appear as a single volume. Drive pooling is provided using the AUFS filesystem and each pool is setup so that if you write a file to the pool base path, the file will be placed on the drive with the most amount of available space. Hard disks (volumes) within a pool will appear as though the drives are stacked, so when you look at the drive pool base path, you will see all files on all drives.

Drive Pooling is controlled or configured by the file: \etc\udev\FoxyDrives.conf
My FoxyDrives.conf file is as follows: This configuration provides 8 independent drive pools.
Code: Select all
# FoxyDrives.conf FoxyRoxyLinux drive-pool configuration file
# Remove (or rename) this file in 05-FoxyConfig to disable drive pooling
# /etc/udev

standby=241                 # HDD Standby Timeout 241=30 minutes (refer HDPARM docs)
                                        # Standard Linux read-ahead cache :  256 x 512 = 128kb
                                        # ...   128 x 512 = 64kb
                                        # ...   256 x 512 = 128kb
                                        # ...   512 x 512 = 256kb
                                        # ***  1024 x 512 = 512kb
                                        # ...  2048 x 512 = 1Mb
                                        # ...  4096 x 512 = 2Mb
                                        # ...  8192 x 512 = 4Mb
                                        # ... 16384 x 512 = 8Mb
readahead=1024            # ... Set/Change read-ahead cache
logfile=/var/log/foxydisk.log        # logfile (drive pooling and hot-swap)
mapfile=/var/log/foxypool.map         # current drive-pool mapping

pool=drvpool_01 /mediapool/data      # mount point for drive-pool 1
pool=drvpool_02 /mediapool/video   # mount point for drive-pool 2
pool=drvpool_03 /mediapool/audio        # mount point for drive-pool 3
pool=drvpool_04 /mediapool/image        # mount point for drive-pool 4
pool=drvpool_v1 /mediapool/video1   # mount point for drive-pool 5
pool=drvpool_v2 /mediapool/video2   # mount point for drive-pool 6
pool=drvpool_v3 /mediapool/video3   # mount point for drive-pool 7
pool=drvpool_v4 /mediapool/video4   # mount point for drive-pool 8

newdrive=no                        # Add unknown partitions to the Drive-Pool (yes/no)
newmount=drvpool_01 rw root            # parameters to use if newdrive is to be added to pool

# Format of mounting parameters:
# 1 = partition label to add to the pool
# 2 = drive-pool to include the partition in
# 3 rw = full read/write access to the partition when pooled
# 3 ro = read only access to the partition when pooled
# 4 = directory to mount into the drive-pool (root) is the entire partition

label=FoxyNAS_001 drvpool_01 rw /data   # Line Must be Terminated
label=FoxyNAS_002 drvpool_01 rw /data   # Line Must be Terminated
label=FoxyNAS_003 drvpool_01 rw /data   # Line Must be Terminated
label=FoxyNAS_004 drvpool_01 rw /data   # Only 4 Drives Installed

label=FoxyNAS_001 drvpool_02 rw /video   # Line Must be Terminated
label=FoxyNAS_002 drvpool_02 rw /video  # Line Must be Terminated
label=FoxyNAS_003 drvpool_02 rw /video  # Line Must be Terminated
label=FoxyNAS_004 drvpool_02 rw /video  # Only 4 Drives Installed

label=FoxyNAS_001 drvpool_03 rw /audio  # Line Must be Terminated
label=FoxyNAS_002 drvpool_03 rw /audio  # Line Must be Terminated
label=FoxyNAS_003 drvpool_03 rw /audio  # Line Must be Terminated
label=FoxyNAS_004 drvpool_03 rw /audio  # Only 4 Drives Installed

label=FoxyNAS_001 drvpool_04 rw /image  # Line Must be Terminated
label=FoxyNAS_002 drvpool_04 rw /image  # Line Must be Terminated
label=FoxyNAS_003 drvpool_04 rw /image  # Line Must be Terminated
label=FoxyNAS_004 drvpool_04 rw /image  # Only 4 Drives Installed

label=FoxyNAS_011 drvpool_v1 rw /video    # Line Must be Terminated
label=FoxyNAS_012 drvpool_v1 rw /video    # Line Must be Terminated
label=FoxyNAS_013 drvpool_v1 rw /video    # Line Must be Terminated
label=FoxyNAS_014 drvpool_v1 rw /video    # Line Must be Terminated

label=FoxyNAS_021 drvpool_v2 rw /video    # Line Must be Terminated
label=FoxyNAS_022 drvpool_v2 rw /video    # Line Must be Terminated
label=FoxyNAS_023 drvpool_v2 rw /video    # Line Must be Terminated
label=FoxyNAS_024 drvpool_v2 rw /video    # Line Must be Terminated

label=FoxyNAS_031 drvpool_v3 rw /video    # Line Must be Terminated
label=FoxyNAS_032 drvpool_v3 rw /video    # Line Must be Terminated
label=FoxyNAS_033 drvpool_v3 rw /video    # Line Must be Terminated
label=FoxyNAS_034 drvpool_v3 rw /video    # Line Must be Terminated

label=FoxyNAS_041 drvpool_v4 rw /video    # Line Must be Terminated
label=FoxyNAS_042 drvpool_v4 rw /video    # Line Must be Terminated
label=FoxyNAS_043 drvpool_v4 rw /video    # Line Must be Terminated
label=FoxyNAS_044 drvpool_v4 rw /video    # Line Must be Terminated



standby ... The delay time in minutes required for a drive to be placed in StandBy mode.
This is a formulated time which is explained in the HDPARM documentation

readahead ... The size of the disk read ahead buffer
logfile ... The logfile to use for all log entries
mapfile ... The map file created and used for adding and removing a volume from the drive pool

pool= The name of the drive-pool, the mount point of the drive-pool
My standard arrangement has 8 drive pools

newdrive ... if set to yes, a non-specified drive/volume can be added to a pool as specified in the newmount configuration option
newmount ... The drive-pool to mount an unknown volume into, the access mode, and the path to use (root= the entire drive/volume)

The parameters for each volume are clearly explained in the configuration file itself.

If you do not want to use drive-pooling, simply rename the configuration file. I suggest you rename it to: FoxyDrives.conf.none
Alternatively, do not use any of the volume labels I have used in the configuration file :)
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Why the disk channels and drive pools are arranged as they a

Postby jbv » Sun Feb 21, 2016 5:34 am

If you look at how I have connected the hard drives to the disk sub-system and arranged the drive pools, you may see a pattern.
I don't know if this is really going to achieve the desired results or if it will make a difference, although in theory I expect it will.

Basically, I have 6 core channels.
Five Channels are from the Intel SAS port-expander.

Of these channels:
port-6 is driving all of the disks/volumes designated as being FoxyNAS_0x1
port-5 is driving all of the disks/volumes designated as being FoxyNAS_0x2
port-4 is driving all of the disks/volumes designated as being FoxyNAS_0x3
port-3 is driving all of the disks/volumes designated as being FoxyNAS_0x4
port-2 is driving all of the disks/volumes designated as being FoxyNAS_00x
All of these are "data" volumes and contain live data (except FoxyNAS_00P - which I will explain in a moment)

port-2 of the Host Bus Adapter is driving all of the disks/volumes designated as being FoxyNAS_0xP
This port has been designated for use as being the "parity" channel as all of these drives hold the SnapShotRAID parity data

My "theory" is that in operation, each disk will typically have exclusive use of the channel, this should definitely be true while SnapShotRAID operations are occurring. As the RAID operations will only be run on one sub-system at a time, the parity channel should also be available for the exclusive use of the parity data. When a RAID update or check is running, each drive in a RAID array will have it's own channel, which should allow the entire operation to run faster and should also be less prone to bottle-necks or other issues. Or at least, that is my theory and I'm happy with it :)

Within the drive pool. All volumes labeled as being FoxyNAS_01x are within the same pool and will also be assigned to the same media-server (more about this in a moment)
If you have another look at my FoxyDrives.conf above, you will see that all FoxyNAS_01x pool into drvpool_v1 while all FoxyNAS_02x volumes pool into drvpool_v2 (v1 and v2 refer to video1 and video2, which will be explained soon) These are all 5 drive arrangements which provides for 4 x data volumes and 1 x parity volume. These drive sets will be used to hold data that will not be dynamically updated. In other words, these are the long-term storage volumes, where data will typically only be written to once and then read from.

The 4 drives/volumes on FoxyNAS_00x provide a 3 data volume array with a single parity drive. These drives/volumes will hold data that will be moved around a bit and used as temporary storage before things are moved into the long-term storage provided for by the other volumes. The drives to be used on this channel will typically be 2 or 4 Tb drives. I felt this was the best way to achieve the required trade-offs for performance vs optimal use of hte available disk channels.
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

SnapShot RAID

Postby jbv » Sun Feb 21, 2016 5:53 am

FoxyRoxyLinux uses SnapRAID to provide data integrity across the various disk drives.
In my opinion, this type of RAID is well suited to my intended goal. My data will not change often. I want all of my files to be written to a single disk, and I would like the basic safety provided by a RAOD-5 type arrangement.

To this end, I have created 5 SnapRAID datasets/configuration files. One for each disk sub-system as described above.
These configuration files are located in /etc/snapraid
There is a script file named runSnapRaid located in /scripts that you can run at any time to manually update any one of the SnapRAID sets.
To run a manual update, you simply typerunSnapRaid snapsetx where x is the drive set you want to update. With no parameter, you are reminded of the options :)

From a scheduling point of view, there is a script called SnapRaidScheduled.sh that is called every day at 2am
This script will run snapset0 on Monday, snapset1 on Tuesday, snapset2 on Wednesday, snapset3 on Thursday, and snapset4 on Friday
It creates and updates a logfile for each snapset in /var/log/snapraid
In addition to output from SnapRAID, the logfile shows start/stop times, if anything was done, or any problems were encountered.
The scheduled script still requires a few more things to be added in coming weeks, such as archiving the logfiles themselves (perhaps keeping only 2 weeks of logfiles for each dataset) , and running a sanity check (rear only testing) of datasets not being updated. Basically, this will see a full refresh doe on each SnapRAID dataset on the appropriate day and it will read/verify 20% of all other datasets on each day to ensure the drive is still running, the data is good etc. This way, over the course of a week, all drives will be updated (if any data changed) and all data will be validated - in theory :)


To assist with scheduling, Gnome Schedule (a GUI based scheduler) has been added to FoxyRoxyLinux.
This appears on the System Tools menu.

Another neat tool has also been added to this menu Disk Utility is a tool from Red Hat Inc, that provides details on every Drive in the system. It can be used as an easy way to check drives, play with Volume Labels, check status (including SMART status) etc. It shows where drives are plugged in (as in the controller. port etc), the partition type, filesystem, volume label and the mount point. It is quite neat.
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Shared Network Drives - SAMBA (Windows Networking) and FTP

Postby jbv » Sun Feb 21, 2016 6:25 am

By default, FoxyRoxyLinux shares the following paths:

Public ... \home\share\public
Media ... \media
MediaPool ... \mediapool

All 3 shared resources should be easily connected to from any Windows machine.
Your Windows computer should be able to "browse" to find the machine Debian and then browse to see the 3 shared resources.
All resources are shared with full read and write access. Your Windows machine should also see and follow all symlinks and all other wide links.

The SAMBA configuration file is: /etc/samba/smb.conf

Some problems were seen when copying huge amounts of data (over 4Tb) onto a pooled drive through MediaPool
I am almost certain this was due to hardware issues described above when using SATA port-expanders

FoxyRoxyLinux also contains an FTP Server that provides root access using the root password.
ProFTPD is the package/service being used.
If you wish to change this behavior you should refer to the ProFTPD website and the configuration file: /etc/proftpd/proftpd.conf

An FTP client can be found in the Internet menu from the toolbar.
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

The Media Servers - There are 8 of them (DLNA Servers)

Postby jbv » Sun Feb 21, 2016 7:28 am

FoxyRoxyLinux has up to 8 Independent Media Servers. miniDLNA or ReadyDLNA (as it is now called) is used to provide multiple DLNA Servers.

The MediaServers start AutoMagically and do not require any special configuration or settings - unless you want to change the way I have set them to behave.

This may take a little to explain clearly, so this message may not be completed in one session :)

Out of the box, FoxyRoxy provides a basic media server with all datafile embedded within the FoxyRoxy USB image file.
This media server provides 1 video file, 1 audio file, 10 photos. With the sole exception of the small video file, it does this using links to basic wallpapers and the test .wav file.
The Media Server will appear on your DLNA client (TV or Media Player) as FoxyDLNA
You can use this media server to test the basics of FoxyRoxy's media player(s) without having to configure anything.
If these files are all visible and play correctly on your media player(s), then you will not have any issues with the media servers within FoxyRoxy.

If they don't, then I can almost assure you that the problem will be in your TV or MediaPlayer. It does not matter how new your TV or MediaPlayer is, or how much it cost. I can assure you the problem will reside in the client - even if it is DLNA certified - the problem will be in your equipment. If it isn't, then I'm sorry there is still nothing I can or will be able to do to help. DLNA is a can of worms. There are a lot of issues and problems with various clients and servers. I have put together the best Media Server I can. It works for me. I have been able to get it to work exactly the way I want it to and I have no intention of changing anything in this area.

With that out of the way, we can now move onto the fun stuff

At system startup, FoxyRoxy will always start the basic media server in "standard" mode.
All of the files and links for the basic (embedded) media-server live in our image file in \mediapool sub-directories
This occurs as soon as a valid IP address is found, either through command-line assignment or received by DHCP.
If you are using DHCP, FoxyRoxy waits until a valid IP is received because ReadyDLNA won't start properly without a valid IP.
Once we have a valid IP, FoxyRoxy then mounts all of the hard drives.

Now, this creates an issue. We have already started our media-server and now we have mounted the hard drives.
What if our hard drives contain our media files, which they usually will?
Well, FoxyRoxy is smarter than the average Fox, so we stop the initial media-server and we restart it, but this time we look to see if we have a valid media-server living in \mediapool/data/FoxyDLNA
If it finds a valid media-server database in this location it then starts the media-server in "pooled" mode.

It's okay, relax ... I can hear you ask ... "How Do I create the database I need to run in pooled mode, so that the media-server knows my files are on a hard disk"?
It is really quite simple. You just follow these steps.

1) Stop the default media server
Enter the command: foxydlna stop

2) Create a hard-disk volume (partition) label using any of the following labels:
FoxyNAS_001
FoxyNAS_002
FoxyNAS_003
FoxyNAS_004

3) On at least one, or (I suggest) all of these drives, create the following sub-directories (folders)
\data
\audio
\image
\video

I suggest you use the Disk Utility in the System Tools menu for this
Note: you have now just created a drive-pool :)

4) One one drive, use the File Manager (right mouse-click, or press Windows+E keys) and create the following sub-directories in \data
FoxyDLNA
FoxyMusic
FoxyPhoto
FoxyVideo1
FoxyVideo2
FoxyVideo3
FoxyVideo4

Assuming you used volume FoxyNAS_001 as your "data" drive, when viewed from the \media mount point, the full path should look like this:
\media\FoxyNAS_001\data\FoxyDLNA
\media\FoxyNAS_001\data\FoxyMusic
\media\FoxyNAS_001\data\FoxyPhoto
\media\FoxyNAS_001\data\FoxyVideo1
\media\FoxyNAS_001\data\FoxyVideo2
\media\FoxyNAS_001\data\FoxyVideo3
\media\FoxyNAS_001\data\FoxyVideo4/b]

5) Now we just need to remount all of the drives.
You can either shutdown FoxyRoxy and restart, or try using the [b]remount
script, or use the unmount volume/mount volume buttons in the Disk Utility

6) At this point the media-server database for your files will not exist.
FoxyRoxy will never automatically overwrite any configuration or database files, so we will need to create the media-server database.
Before you do the next step you should make sure you have your media files in the appropriate directories.
I hope it is somewhat clear as to what sort of files you should place where :)
For now, only use the \audio, \image, and [/video[/b] sub-directories.
Of course, you can and should create sub-directories below each of these so you can layout the files as you want to see them.
Once you have your media files in the correct place, we can build the media-database.

7) Enter the command: foxydlna rebuild
If everything is correct, FoxyRoxy will then build you a pooled media database in \media\FoxyNAS_001\data\FoxyDLNA (assuming you have your data directory on FoxyNAS_001 :))
If you want to watch the progress to get a feel for how things are progressing, you can use the command debug-rebuild instead of just rebuild If you use this command, you will see a lot or meaningless debug info in the terminal console screen. When the process has finished, if you used the debug-rebuild command you will see that the output has stopped. At this point, you can test the media-server. In debug mode, you will need to stop the debug to get the command prompt back. If you kill the terminal window, you will also kill the media-server process. If you stop the debug mode using control-C or killing the thread, you can restart the media-server by simply entering the command foxydlna start

If you used foxydlna rebuild you don't need to stop anything as the media-server will start running the moment you issue the command, even while it is rebuilding the database. Using this method, you can check progress by either playing with your TV or media-player, or by using the Web Browser (right-click mouse menu). The BookMark Toolbar contains buttons for each Media Server. Each button reports the number of files for the media-server. For now, we have only created and started [MiniDLNA] Simply press refresh to watch the status.

Once you have created the media-server database, miniDLNA(ReadyDLNA) will automatically start in "pooled mode".
During startup, the media-server will report the mode it is operating in and show the location of the database being used.
The following command will always report the mode and status: foxydlna status

You can create a media-server specifically for audio (music) by building the database for the foxymusic media-server.
Use the same principle above, if you followed the direction given above, you already have a directory \mediapool\audio, so you just need to create the database.
You can do this by entering the command: foxymusic rebuild

To create a media-sever specifically for photos (images) we just need to create the database for the foxyphoto media-server.
Use the same principle above, if you followed the direction given above, you already have a directory \mediapool\image, so you just need to create the database.
You can do this by entering the command: foxyphoto rebuild

Status Buttons for FoxyMusic and FoxyPhoto are already on the Bookmark toolbar of the Web Browser

Note: FoxyMusic and FoxyPhoto share directories/folders with the "pooled" mode of operation. However, the configuration files are subtlety different in that these media-servers only look for and report the correct media content to the client. This means that if you are browsing images or music, that is all your client will see. This has been done to stop some TV's and media-players always asking you what content you wish to view or use. It can often eliminate 3 keypresses and also stops the media-player getting confused and trying to play a movie when it is is music mode.

The remaining 4 media-servers are specifically for video files. The video file layout and mappings match my defaults in the drive-pool such that each dedicated video media-server has it's own unique drive pool. This has been done to assist people with large video libraries as I hope it will allow you to build your libraries and manage, maintain and view them in a group fashion. I don't know how you will decide to categorise or break-up/divide your library, but one thought is something along the lines of: General Programs, Documentaries, Lite Entertainment, Home Made - You can categorise them however you wish, I'm sure you get the idea. I will add a little more about the other 4 video media-servers shortly, but the basics are the same.
In the end, there will be:
foxymedia1 = foxyvideo1 rebuild
foxymedia2 = foxyvideo2 rebuild
foxymedia3 = foxyvideo3 rebuild
foxymedia4 = foxyvideo4 rebuild

At startup, FoxyRoxy will only start active and live media-servers, Any unused or servers that do not have valid databases will not be started and there won't be any errors.
Once you plug in the drives, label them correctly to place them in a drive-pool and then build the appropriate database, everything is auto-magic :)

I hope that now, the various volume labels, drive pool names, and disk channel assignments all make sense.
I think it all hangs together rather well :)

In a fully loaded and configured system you could have the following media-servers running:
FoxyDLNA (mixed video/audio/image - in pooled mode)
FoxyMusic (music only)
FoxyPhoto (images only)
FoxyVideo1 (Video Library 1)
FoxyVideo2 (Video Library 2)
FoxyVideo3 (Video Library 3)
FoxyVideo4 (Video Library 4)

To save you hunting everywhere, a "configuration" directory has been created that contains links to the various configuration files.
\config contains soft links to the configuration files for drive pooling, snapRAID, media servers, and system startup.
Clicking any link from File Manager opens the appropriate file in edit mode.
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Next

Return to Using FoxyRoxyLinux



Who is online

Users browsing this forum: No registered users and 1 guest

cron