2015 ... Where Has The Time Gone ... Beta-4

Tips and Techniques on day to day stuff

2015 ... Where Has The Time Gone ... Beta-4

Postby jbv » Sun Jan 04, 2015 7:54 am

Hi,

Wow. What a hectic two years. Since moving North, my feet have hardly touched the ground.
Professionally things have just been insane. As my responsibilities grew, my free time shrank.

I have had very little time to spend on this project, although I did play with it from time to time.
Over the last two weeks, I have done quite a bit of major surgery and I am now almost happy.

I found quite a few little glitches with the identification of various disc partitions and file-systems and have sorted all of that, with the one exception that if you have an exFAT partition on a drive it won't always mount at start-up. If you hot-plug it, it will mount every time, but for some reason it won't always mount at power-up. I think this is a glitch in the exFAT driver, and as I can't imagine anyone having their primary drives in exFAT format, I will need to ignore this one.

Drive Pooling is now working like a charm.
The Drive Pool is built dynamically at power-up and you can hot swap drives in and out of the pool without any problems.
Everything is working on the partition label so it doesn't matter what device the drive is attached as, it always drops into the right spot. You can even add/delete USB Hard Drives to the pool.
I have verified that you can use 4TB drives with NTFS filesystems for both read and write (Even though the driver says it only supports 2TB volumes - it does work). The system supports SATA and SATA Port Multipliers with any speed, which means 1.5Gb, 3.0Gb, or 6.0Gb SATA drives all work.

I found a few things with the standard Debian Squeeze Kernel that weren't right, so I have now had to rebuild the Kernel and tweaked some options. While doing this, it really got to be to hard to support all of the various builds, so I have decided to only support/build the 686-Bigmem kernel. It took me 4 days (with one 36-hour straight stint) just to get this right. I had never delved into this space before, so it took a little bit of learning and a heck of a lot of regression testing. I used the original Debian Source (with patches), so all of the code is the same, I just changed some of the options to do with timers, iNotify and aufs. However, this meant that I then had to rebuild the entire sound sub-system, so all in all it took some learning and testing to ensure it was right. While I was in this space, I also increased the number of loop devices, which means that FoxyRoxyLinux can now load 32 squashfs files (the standard is/was 8).

I also found a really neat tool that allows you to have only 1 keyboard and mouse on your desk and us it between multiple machines. This means that you still have a monitor on the FoxyRoxy machine, but you don't need a keyboard or mouse. As you move the mouse off of one screen it auto-magically pops onto the other screen. You can also copy stuff between machines using the clipboards, which was really handy while I was building kernels etc. The other machines can be either Linux, Windows, or Mac's. I have put the Linux client into our base package.

Quite a few things got tweaked, upgraded or otherwise fixed. The short list is:

I fixed a few things in Dbus that got/were broken
ntfs-3g (2014.2.15) - Latest Version (built from source)
exfat (1.1.1) - Latest Version (built from Source)
ffmpeg - Latest Debian Version
libavformat - Latest Debian Version
mplayer - Latest Debian Version
smplayer - Latest Debian Version
audacious - Latest Debian Version
minidlna - Latest Version (built from Source)

As far as various media-files are concerned, FoxyRoxy now plays files that it wouldn't before. They are all a lot smoother too.

I have cleaned up a lot of the various file associations, so when you select a file in XFE (our File Explorer) it is opened by the proper tool. I've added a few hot-keys to OpenBox which just make things neat. Example the Win-D key now takes you to the Desktop Win-E key now opens XFE (explorer) Win-C now opens x-term-console (LXconsole) print-screen (now does a screen Capture) Alt+Print-Screen (now does a active window Capture).

I have integrated all of the extended filesystem drivers such as HFS, HFS+, eXFAT and NTFS into the core package.
I have pulled out pretty much everything that is configurable or a saved configuration and put it into it's own small squashfs file.

There are now 3 core squashfs files.
01-FoxyRoxy
02-FoxyDesktop
05-FoxyConfig

All of our extensions/Add-ons such as Firewall, Skype, Office, Thunderbird, Java, Vuze, Xchat, LMS, etc are fully functional.
I have now started to use FoxyRoxy as one my key Media Servers.

Conky/CPUinfo still show the base RAM usage to be less than 100Mb with everything loaded and running.
While playing an MP3 audio file locally and also watching an MP4 movie and having 2 x WDLiveTV boxes pulling (large high-def, high-bitrate) files from the Media Server, this increases a little to a whopping 130Mb :) If you then also try to watch a 1080p DTS Audio movie, it all works. The sound is a little crazy (but perfect - you can hear all of them properly), and memory usage jumps to about 170Mb :) :)

I am considering putting the Firewall and Logitech Media Server (SlimServer) in the core package.
I am also playing with Deluge (which is working well) and am considering embedding this into the Desktop (core package) also.

I will play with it for a little longer to make sure it is all solid, then I will sort out the last part - SnapShot RAID
I will build one more release before I go to the next step of adding the RAID.

Testing and fixing every little glitch has been the real killer.
If things return to a somewhat normal scenario professionally, I should have the next release of FoxyRoxy up within a few weeks.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: 2015 - Where Has The Time Gone

Postby saintless » Sun Jan 04, 2015 9:31 am

Hi, Brenton!
I will check for new upload to test it.
I thought it might be already available for download but I get the old iso.

Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: 2015 - Where Has The Time Gone

Postby jbv » Mon Jan 05, 2015 11:29 am

Hi Toni,

Thanks. I thought you might be to busy with Debian Dog. For those who are not aware of this, Toni has started his own Debian (Wheezy) based distro. It is quite different to FoxyRoxy in many ways. It looks quite incredible from what I have worked out while perusing the forum. It also looks as though he has garnered the support of a couple of really competent technical assistants. To check it out, follow: < This Link - click me >

If you wouldn't mind doing some testing, I will try to prepare a preliminary ISO this coming weekend.
I will also try to prepare some notes on how it all works and how you can configure bits and pieces.

To be honest with you Toni, I don't think anyone is watching here. Other than yourself, Kazza and Steen, no-one else really seemed interested. With my work commitments over the last 18/24 months, I can't say that I'd blame them either. I had hoped for a little more technical assistance when I started the Forum thing. I'm happy that you were able to get it.

For me personally, I'm really happy with where it is at, and where it is going.
I'm pretty sure I will have something (finished) for myself, and my family/friends within the next month or two.
For a few reasons, I don't think I can get the same thing with DD or the newer Wheezy based distros.

I've also looked at some (all that I know of/have found) developments including new Home/Domestic-NAS distros that have popped-up over the last 18 months. while some are very polished on the surface, and look all whizz-bang, from what I have seen, they haven't got the base right - even if they have used some of the same core components. Basically, I can't see anything other than pain in looking to use them, so I will continue with what I started.

Anyway, if you are keen to see and test what I've got so far; as mentioned, I will try to do some notes/docs during the week and try to get a preliminary ISO built for you this weekend.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: 2015 - Where Has The Time Gone

Postby saintless » Mon Jan 05, 2015 3:25 pm

Hi, Brenton :)
jbv wrote:Thanks. I thought you might be to busy with Debian Dog.

Never to busy to test FoxyRoxy on my old machines :) I couldn't help you much with the scripting part but I still love this distro and it will be a pleasure testing and helping to improve it.
If you wouldn't mind doing some testing, I will try to prepare a preliminary ISO this coming weekend.

Yes, please, but only when you have time to do it. I'm checking the forum almost every day for new posts.

For me personally, I'm really happy with where it is at, and where it is going.
I'm pretty sure I will have something (finished) for myself, and my family/friends within the next month or two.
For a few reasons, I don't think I can get the same thing with DD or the newer Wheezy based distros.

Same for me with DD, Brenton. Wheezy version is the main one but I'm working at the same time on Squeeze version mostly for my self. It is important to have FoxyRoxy working for your needs. I'm sure someday I will buy a TV like yours and FoxyRoxy will be the best I can find :)
BTW some of your ideas for FoxyRoxy are included in DD like the zerosizing in the remaster script.
I can suggest few small alternatives for wifi support and others from puppy linux already working in Squeeze for me. Maybe if you post a list of what else is needed for FoxyRoxy I will be able to help you better now than before.

Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: 2015 - Where Has The Time Gone

Postby SBP » Sun Jan 11, 2015 9:44 am

Hi all.

Good to see you all here again. I have been checking the forum from time to time so it is good to have you back and even having done some work on Foxyroxy again.

My main interest in Foxyroxy is to use it as a server for my audio files which then will be served to Squeezebox audio players via Logitech Media Server (LMS), so I was happy to see that you still have plans for LMS in Foxyroxy.

As you might remember I have developed a small Squeezebox emulator the "piCorePlayer" running on a raspberry ARM board.
It has matured over time and if you need a small audio player I really think you should give it a try. The audio out from the raspberry has improved by dedicated I2S-DACs attached to the raspberry like this one: http://www.geekroo.com.au/products/1482
Recently, another Ausie has joined me in developing the player (Greg Erskine) from down under.

Steen
SBP
 
Posts: 74
Joined: Thu Aug 16, 2012 5:27 am
Location: Denmark

Re: 2015 - Where Has The Time Gone

Postby jbv » Sun Jan 18, 2015 7:22 am

Hi Steen (good to see you), Hi Toni,

I have placed Beta-4 along with an updated USB Installer in our Main Download Area.

Beta-4 contains the items described above along with fresh/updated Build and Source sqf's.

The file 91-kernelsource.squashfs.no is quite large at >200Mb
The reason for this is that it contains the untouched "master" .tar/bz2 file as downloaded from Debian, in addition to an unpacked and untouched copy of the source and kernel-patches. These files reside in /usr/src. The working copies of these are in /source/kernel which also contains the source for alsa sound.

This "beta" version of FoxyRoxy has almost all of the configuration stuff inside 05-FoxyConfig.squashfs.
I say almost, because I still need to pull out the init.d start-up stuff from /etc
I haven't created scripts for updating FoxyConfig yet, so this is a manual process using 05-load and 05-save

Having said that, 99-snap should still work for most things, but I really need to spend a few days crawling over it to make sure that it is perfect and picks up everything, and working out if I need to create any more 99-snap extensions.

Some of the new scripts include:
02-CleanFoxyDesktop - this one is neat. It removes packages from 02-FoxyDesktop.squashfs and fixes up dpkg, but you need to be very careful with it. You can only use this after you have removed and purged packages. You can't use it if you have added or updated any packages.

runminidlna - is a souped-up minidlna manager. In addition to start/stop/status, it lets you restart, rebuild, debug, debug-rebuild
The rebuild options, rebuild the minidlna database.
The script looks to see if drive-pooling is enabled. If it is, it uses a different configuration.

fullnetinfo - shows your internal and external IP address details

MountStora is an example of how to mount a Windows share using samba

Okay, a few notes on drive pooling:
The drive pool is managed and controlled the following file:
/etc/udev/FoxyDrives.conf

If you 05-load the "default" config into /tmp this file will be /tmp/sqf-config/etc/udev/FoxyDrives.conf.no
Simply rename it by removing the .no then edit it to suit your requirements.

I think the contents of the file and what does what are pretty self-explanatory, but basically.
standby is what the HDD standby time will be set to.
logfile is where FoxyRoxy will create and update it's log files related to drive mounting etc.
mapfile is where FoxyRoxy will keep the "map" of the current drive pool. This is used when adding and/or removing drives to/from the pool.
It is important that the map file be updated correctly, so just leave these two alone.

pool allows you to create up to 4 independent drive pools.
Each drive pool must have a unique name and a unique mount point.
My standard arrangement has pools drvpool1 through drvpool4
You will see that each pool has a dedicated assignment/intended-use through its mount point in /mediapool

newdrive and newmount are not used yet. The intent is that you can configure a "default" drive pool arrangement for drives that aren't specified in detail.

You assign a HDD (or a partition on a HDD) to a pool with the last section.
label= The partition or volume label, The pool this drive/partition should be added into, The permissions, and the point in the drive/partition to mount from

It is important that you do not have any spaces in the volume/partition label or the path to the mount point

The spindown script will unhook all drive pooling before spinning down a hard disc.
Un-mounting a partition should also remove the partition from the drive pool.
The drive pool mapfile /var/log/foxypool.map should be automatically updated with all drive pool changes.

If you make any direct file add/change/delete operations using /media/device these will automatically be reflected in /mediapool/xxxx although minidlna won't see them and won't update it's database automatically. I know how to fix this and I will do this in a future revision.

If you add/change/delete any files using the /mediapool/xxx path, these are automatically reflected in the appropriate /media/device path and minidlna will update it's database automatically.

Files added to the mediapool are stored on the hard drive with the most available space.

Comments above regarding minidlna will also hold true for other packages that maintain their own database such as SqueezeServer and similar. This will be somewhat dependent upon how you setup their configuration files, although I expect you would have these programs also use /mediapool to read/write their data as that is the whole reason for having drive pooling :) My "fix" for this with minidlna will also hold true for similar packages (when I get around to it).

If you look at the tail end of /etc/rc.local you will see how the drive pool is built at startup if the file /etc/udev/FoxyDrive.conf exists.
Remember the easiest way to find/tweak/rename these files is by loading the config file into /tmp using 05-load

The reason for restarting minidlna after the drive pool is built is because the minidlna service starts before the drive pool is built.
Without a drive pool, there is a fun little video and a few pictures and one sound file that are embedded into 02-FoxyDesktop so you can test the system without needing to have an entire film/sound/image library. Once you have created your drive pool, you simply restart FoxyRoxy, then rebuild the database using the runminidlna script. From then on, FoxyRoxy will automagically "flip" the system during start-up so you will get your live a/v library :)

As minidlna and similar programs require databases that I don't want to place on the USB stick, I have decided to keep one drive/partition for these databases.
This drive/partition can always be rebuilt if it fails or crashes and it should not be part of the primary drive pools that hold any media.
When I get the snapshot raid working (next), I will use the same volume/disk/partition to hold my raid snapshots.
This way all of the data that will be changing and is somewhat dynamic and can be easily rebuilt, will be stored on the same drive.
If anything happens to that drive, it will be a simple process to rebuild it, so nothing will get lost or messed up.
All of our configuration stuff will still be on the USB stick, which will be easily backed up anywhere.
I intend to leave a bit of space on each of my media drives by an extra partition, that won't be added to the mediapool.
This area will be used for backups which I will mirror onto every drive.

If you want to have a look at the keyboard/mouse thing I am now using, you can do so by having a look <here synergy-project>
You can download packages for various Operating Systems from <here for downloads>

To load/start the daemon in FoxyRoxy, look at the end of /etc/xdg/openbox/autostart.sh where I have also put some notes about a "trick" you may need to use to load the Synergy Master on a Windows-7 machine :)

Remember, it is easier to 05-load to find all of these configuration files.

Note, when you are looking at various configuration files and some of the sample scripts; internally I use static IP addresses on my machines. Internally, my network uses 192.168.5.xxx addressing, so you will most likely need to change the IP addresses and/or names you see in my scripts, but it should be straight forward to do so.

Oh, this beta also makes changes to the default SAMBA "shares" .
In addition to /home/share/public, /media and /mediapool are also shared without security and with full read-write access.
To change this, you would edit /etc/samba/smb.conf

I'd be interested to hear what you think of it all, and of course, if you see anything wrong, let me know.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: 2015 ... Where Has The Time Gone ... Beta-4

Postby jbv » Sun Jan 18, 2015 8:04 am

Hi Toni,

Could I please ask you to take a moment and look at two things for me, that I'm not 100% certain about.

I noticed that gnome-disks was not working in previous releases.
It can be found in the [System Tools] menu by selecting [Disk Utility]

This seemed to be because the udisks-daemon either wasn't loading, or wasn't loading properly. I'm really not sure which of the two it was.
I found that starting the udisks-daemon service as part of the openbox startup resolved the issue.
Therefore FoxyRoxy now starts this in the /etc/xdg/openbox/autostart.sh script
Is this the right place?

The other thing that isn't quite right is that while the Disk Utility now starts and works, some functions don't.
If you try to Unmount a Volume, or Check a Filesystem from the Disk Utility, it errors with something related to "PolicyKit".
I've tried to stay away from this and while these functions aren't critical (as in I can ignore them), I was wondering if you might have any idea as to what I may have not got right, that is causing these errors.

If you know, can you steer me in the right direction.
If you don't, then don't waste any time with it, as I know you've got your own project keeping you busy :)

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: 2015 ... Where Has The Time Gone ... Beta-4

Postby saintless » Sun Jan 18, 2015 12:20 pm

Hi, Brenton.
I see many changes and it will take me some time to understand the instruction and test properly :) I will post the results here.
I need firmware-linux-nonfree package for my hardware and after installing I get inetrnet connection on boot.
Installing firmware-linux-nonfree shows messages about not updating initramfs for kerenl bigmem. I think you can safely remove 2.6.32-5-486, 2.6.32-5-686, 2.6.32-5-686-bigmem from /var/lib/initramfs-tools/ but it is minor issue.
Sound is working from boot.

I found that starting the udisks-daemon service as part of the openbox startup resolved the issue.
Therefore FoxyRoxy now starts this in the /etc/xdg/openbox/autostart.sh script
Is this the right place?

Yes, FoxyRoxy boots to root so it is OK to keep it there. You can do it also from $HOME/.config/openbox/autostart.sh with different autostart.sh content for each user account.

The other thing that isn't quite right is that while the Disk Utility now starts and works, some functions don't.
If you try to Unmount a Volume, or Check a Filesystem from the Disk Utility, it errors with something related to "PolicyKit".
I've tried to stay away from this and while these functions aren't critical (as in I can ignore them), I was wondering if you might have any idea as to what I may have not got right, that is causing these errors.

After some googling here is what works for me:
Code: Select all
apt-get install policykit-1
apt-get install dbus

Create file /etc/polkit-1/localauthority/50-local.d/55-storage.pkla with content:
Code: Select all
[Storage Permissions]
Identity=unix-group:plugdev
Action=org.freedesktop.udisks.filesystem-mount;org.freedesktop.udisks.drive-eject;org.freedesktop.udisks.drive-detach;org.freedesktop.udisks.luks-unlock;org.freedesktop.udisks.inhibit-polling;org.freedesktop.udisks.drive-set-spindown
ResultAny=yes
ResultActive=yes
ResultInactive=no


Edit: Note all this is in one line (attached zip with the file):
Code: Select all
Action=org.freedesktop.udisks.filesystem-mount;org.freedesktop.udisks.drive-eject;org.freedesktop.udisks.drive-detach;org.freedesktop.udisks.luks-unlock;org.freedesktop.udisks.inhibit-polling;org.freedesktop.udisks.drive-set-spindown

Reboot (or exit X -> login as user and password live -> sudo login to root again). Then mount, unmount external usb drive works and ejecting also works without errors (eject works after unmounting all existing mounted partition on the usb).

Cheers, Toni
Attachments
55-storage.pkla.zip
(345 Bytes) Downloaded 335 times
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: 2015 ... Where Has The Time Gone ... Beta-4

Postby saintless » Sun Jan 18, 2015 12:36 pm

Just to complete the information from previous post for user account one more step will be needed:
adding user to group plugdev:
Code: Select all
usermod -a -G plugdev <your username>

The information is taken from here:
http://unix.stackexchange.com/questions/66654/how-to-mount-usb-stick-on-debian-sid
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: 2015 ... Where Has The Time Gone ... Beta-4

Postby jbv » Sun Jan 18, 2015 8:05 pm

Hi Toni,

Thanks for that. Sorry for missing the few initramfs things.
I will test and confirm the PolicyKit info in the next day or two.
Sincerely appreciated.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Next

Return to Using FoxyRoxyLinux



Who is online

Users browsing this forum: No registered users and 1 guest

cron