dpkg = the unsung hero while apt-get receives all the glory

Here's a radical concept. Let's leave notes and mini-how-to's for each other

dpkg = the unsung hero while apt-get receives all the glory

Postby jbv » Sun Sep 16, 2012 12:11 am

Howdy,

I am about to embark on something that will be critical as we move forward. In many ways it is probably more critical than getting out system services start-up stuff where I would like it to be. Therefore, I would like to get this sorted before moving on to our start-up thing.

First some Background: While everyone loves the thought of just being able to apt-get stuff, very few if any people have taken a look under the hood of apt-get, and think of it as being a piece of magic that just works. For the large part, and to a certain degree, this is true. But, have you ever wondered how it does it's magic and why?

Have you ever had a problem where things did not "just work", or stuff just went screwy on you? If you have, chances are that the reason underlying the problem was because what I will refer to as the dpkg database got messed up. The first thing you need to understand is that while apt-get does some pretty tricky stuff, the lions share of the work is actually being done in the background by dpkg.

It is dpkg that actually installs, updates and removes stuff.
It is dpkg that keeps track of what is where, what is installed, what version something is and a whole lot more.
While apt does have some smarts, it is pretty brain-dead by comparison to dpkg and without dpkg, apt wouldn't know what to do or what to get. It is dpkg that tells apt what is installed and what isn't. Once again it is dpkg that does the hard work.

dpkg keeps track of things with two very important files that I will refer to as "the database".

These files are:
/var/lib/dpkg/available
/var/lib/dpkg/status

Now from my limited poking, it seems that available is the list of packages that your system has "available" to it, in so far as it appears to be a list of packages you have tried to install. Whereas status contains the same core information with the actual currently known state of the package. If everything went well during the installation phase, then this will usually be "install ok installed".

If you open the file /var/lib/dpkg/available, you will see the details of the first package in the database:
Code: Select all
Package: tcpd
Priority: standard
Section: net
Installed-Size: 128
Maintainer: Marco d'Itri <md@linux.it>
Architecture: i386
Source: tcp-wrappers
Version: 7.6.q-19
Replaces: libwrap0 (<< 7.6-8)
Depends: libc6 (>= 2.3), libwrap0 (>= 7.6-4~)
Size: 36094
Description: Wietse Venema's TCP wrapper utilities
 Wietse Venema's network logger, also known as TCPD or LOG_TCP.
 .
 These programs log the client host name of incoming telnet,
 ftp, rsh, rlogin, finger etc. requests.
 .
 Security options are:
  - access control per host, domain and/or service;
  - detection of host name spoofing or host address spoofing;
  - booby traps to implement an early-warning system.

This is a veritable gold-mine of information.

Now open the file /var/lib/dpkg/status.
Did you notice that it is basically the same file, yet it has an additional line that has been inserted below the package name?

That's it. That is all we need. With just a little more poking we can learn everything we need to about what packages are installed, what files are in any package, where they are, what dependencies they have, and a whole lot more.

Lets quickly take a small detour to discuss some of the other files in /var/lib/dpkg
available-old and status-old are simply backups and we can probably remove these from our primary sqf at some point of time, but they will be compressed to be much smaller so for now, lets just forget about them. lock will almost always be 0 bytes in size and is used by dpkg/apt/synaptic/whatever when they are doing their thing. It is being used as a simple mechanism to "lock" the database while any of these programs are doing their thing and can not risk another program from messing with stuff while it is about to. I'm not sure what the other files here do.

There are two other key directories that contain data that is critical to dpkg These are:

/var/lib/dpkg/alternatives
This directory contains files required by dpkg to update/patch the /etc/alternatives if a newly installed package needs to or wants/tries to. Alternatives, is a whole different ball of wax that allows you to easily change your default web-browser, text editor, whatever. Alternatives work in Foxy and to the best of my knowledge, none of this is broken.

/var/lib/dpkg/info
This directory contains what I will refer to as supplementary details that dpkg needs to function properly. It really should be considered as another and important part of the core we need to maintain and keep healthy. For the main part and provided we keep our database clean and healthy, the files here will look after themselves.

As a brief introduction, the files in /var/lib/dpkg/info are the files that dpkg/apt use to verify the .deb xxx.md5sums and work out what to put where, or what to remove from where (if un-installing) xxx.list, in addition to what to do to install the package xxx.postinst and what to do when un-installing xxx.postrm
These files are normal text files, so you can open them and have a look at the detailed information in each of them.

I hope you are starting to "see the picture" developing here.
Once it has been installed, the .deb file is of no use.
We can always download it again if needed, and the apt databases can and are always rebuilt whenever you type apt-get update In fact, everything else is pretty much irrelevant, as provided we keep our primary database files and the info directory clean, we can rebuild anything. In a typical hard-disk Linux system, most people run into huge problems when their database or info files get mangled, weather it be due to an install that didn't "just work", or a system failure, or some other glitch. In many if not most situations, once this occurs people either limp along with a broken system and just live with it, or start rebuilding their system from scratch. As our core files are all physically located inside of the squashfs container file and can never be written to, we are somewhat protected from system errors and glitches as in a total worst case scenario we just restart and we will always have what we had before things got messed up. For hard-drive installs a glitch or error can quickly become a nightmare with no end.

Having said the above, we still have our own nightmare to be concerned about and our system files may be protected from system glitches, but they are not protected from our own foolish actions performed by us though lack of knowledge or understanding.

Let me paint a scenario for you
We decide that we want to install a new package and clean it up.
No problem, we just do the following:
Code: Select all
apt-get update
apt-get install this_is_a_neat_package
95-create
95-load
... we now might try running ...
95-clean (after setting the appropriate options in that script)
... and finally we use...
99-zapmans /tmp/sqf-snap
... then the final step ...
95-save

As we are now proud of what we have just done, we do a quick 95-load again.
We have a quick look at what we loaded, make any final little clean-ups, check that our dpkg files all appear to be there, because we are now paranoid about them ;) and all looks good in the world. If we made any changes, we do a final 95-save and we have a really neat sqf

I do hope there were a few other simple little things you did before doing that final 95-save
What I do once I am confidant that everything is looking good ... I :

1) Decide on the filename I will use for my new sqf and think about where I want it to load

2) Decide on the name I want to use as a temp working area in the future

Let's say, I decide to physically name the file 25-FoxysNeatPackage.squashfs with a working directory name of sqf-neato. I trust that if you've come this far with us, you have seen that I have thought about the load-point and numbered the physical file accordingly and also given the name a meaning, and that I have also maintained the "standard" whereby the working directory starts with sqf- and is related to the package. The reasons for this is so that I not only maintain the standards we are developing, but I can also easily find things where I expect them to be.

3) I now copy create a scripts directory in the working sqf-snap directory
4) I copy /scripts/parked/85-load into the new sqf-snap/scripts directory
5) I copy /scripts/parked/85-save into the new sqf-snap/scripts directory
6) I rename the 85-load/save files in my working directory so they have the correct load and save names.
- in this instance I would rename them to be 25-load and 25-save
7) Next we edit our new 25-load and 25-save files and change the names on lines 3 and 4 to be what we want
8) I then rename my working directory from sqf-snap to be the new name I chose
- in this, I would rename it to be sqf-neato
9) I then copy my new save and load scripts into /scripts
10) I then run my new save script
- in this instance 25-save
11) I then look in /live/image/live to see that the file was created and looks good
12) I then make sure that my temp working directory has been cleaned
13) Next, we make sure by using our load script
- in this instance 25-load
14) Another look in our working directory and if all looks good...
15) I remove the [i]now[/b] old 95-snap.squashfs file from /live/image/live
- this is to make sure it doesn't load when we reboot the system to do our final test.

About now you are probably thinking "Okay Brenton, so you've given me some guidance and instructions on how to create a proper sqf. Thanks for that, but what on earth has this got to do with dkpg, and how in the heck did you wander so far from where you started. You were doing okay at the beginning, but seriously mate, you lost the plot somewhere along the way."

"Au contraire, mon frere" You take a moment to think about this for a little while.
I'm going to have a nice cup of coffee before I return to finish this tale, and bring you to where I want you to be.
Stay tuned folks, this one is a doozey :)

Cheers
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby KazzaMozz » Sun Sep 16, 2012 7:34 am

Hi Brenton
seriously your are killing me just when I thought I was up with it all :lol: :lol: :lol: :lol:
Au contraire, mon frere" You take a moment to think about this for a little while.
I'm going to have a nice cup of coffee before I return to finish this tale, and bring you to where I want you to be.
Stay tuned folks, this one is a doozey :)


Magic I can't wait for the next post
Cheers
Kazza
User avatar
KazzaMozz
 
Posts: 332
Joined: Tue Aug 21, 2012 9:59 pm
Location: Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby jbv » Sun Sep 16, 2012 10:48 am

Hi Kazza,

Sorry, I got distracted while having my morning coffee. It's been a very long and tiring day. I had a mental to-do list today, that had about 14 or 15 things on it. Sadly I only got about 4 or 5 of them done. Some related to FoxyRoxy and while part-2 was on the list, it did not get done. A large part of the day was swallowed-up with breaking all of the WiFi here while trying to re-configure the network to integrate a new wireless bridge. I'm really starting to hate wireless - cable is so much easier and a heck of a lot more secure too. I promise this will get finished. In the meantime, just give the whole scenario a little more thought yourself.

Trust me, in part-2; things will get real-ugly, real-quick.
We have a plan though, but it would be nice if we can also get some help to implement the plan.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby KazzaMozz » Sun Sep 16, 2012 9:19 pm

Hi Brenton
I have been reading up on dpkg very interesting, you are right I had no idea of it's power. I have so much to learn but I'm getting <there>

Cheers
Kazza
User avatar
KazzaMozz
 
Posts: 332
Joined: Tue Aug 21, 2012 9:59 pm
Location: Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby jbv » Tue Sep 18, 2012 9:22 am

Sorry this has not been completed yet. There are few reasons for this.

I've had a hectic time at work this week while getting something ready for manufacture/production. By the time I get home, I'm pretty beat and just need to turn-off for a while, before shutting down. Finally, I figure that giving you a little longer to ponder the various scenarios can't hurt :)

Finishing this is my next and highest priority for FoxyRoxyLinux at the moment as it is very important. Rest assured, I have not brought you this far to leave you dangling. I just need to be a little more relaxed and unwound to complete it properly.

On a positive note, to help unwind and relax, I have started doing something that will prove to me that what I think will work, will in fact work, and that is always a good thing :)

I will be back to this very soon, as it is somewhat important (critical) that we nail this before racing further ahead with our project.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby KazzaMozz » Tue Sep 18, 2012 12:29 pm

Looking forward to the next chapter.
Cheers
Kazza
User avatar
KazzaMozz
 
Posts: 332
Joined: Tue Aug 21, 2012 9:59 pm
Location: Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby jbv » Sat Sep 22, 2012 3:55 am

The looming problem is that if we are not very careful with what we do, we can easily run into a scenario where the dpkb.db [dpkg database files] may be messed up, or not match the current system.

About now, I can hear you telling me that this won’t occur, because you were very careful to make sure that the files in /var/lib/dpkg were included in your .sqf, and you also made sure that the dpkg.info [/var/lib/dpkg/info/*] files were included also.

Okay, fair-enough. But let’s take a moment to consider some scenarios:

You now have 25-FoxyNeatPackage.sqashfs
Your package has the dpkg.db and dpkg.info files in it, and they are Golden – Great

Scenario 1) You now see another neat Application that you want to install.
Easy, just fire-up apt-get and add the new application. You clean it, and everything is sweet (let’s call this package 26-NextApp.squashfs) . A few days later, you decide that you don’t want the 25-FoxyNeatPackage to load every time, so you rename it to be a .noload Then you restart your system and suddenly 26-NextApp starts giving you errors or won’t run. After some debugging, you discover that it is missing a library.

ke? How can that happen?
It was working fine yesterday…. You can try running apt-get install which may fix things. Now, you’ve got 26-NextApp and another 95-snap that you’ve got to manually inject.

Scenario 2) You park 25-FoxyNeatPackage for a while by making it a .noload
Then create 24-NextApp. It works great until you load 25-FoxyNeatPackage and use something like Synaptic to check on the status of the installed files, which shows you that half or all of the installed files for 24-NextApp are missing, even though it may still appear in the [Taskbar Menu] and it even works, so what is wrong?

Scenario 3) You send a friend a copy of 24-NextApp and they really like it. They thank you for sending it to them and saving them a lot of time as they wouldn’t have been able to have made such a nice AddOn themselves. They use it for a few months, make their own AddOns and other changes. Then one day, they find a new package that is just like 24-NextApp, but better. They get it working and delete 24-NextApp because they don’t need it any more. The moment they do this, half of the packages they made (including the new one) don’t work …. eek!

All of these scenarios are actually being caused by the dpkg.db not being a proper and accurate reflection of what is _really_ on the system. Take a moment to consider that the dpkg.db and dpkg.info files are all overlayed when the next .sqf is loaded during the system start-up. At the end of the start-up sequence, dpkg.db will always be the dpkg.db from the last .sqf, while dpkg.info will contain everything that is in all of the .sqfs So, now we have a situation where we really don’t know what the real state of dpkg.db is or should be and we can’t really just rebuild it from the dpkg.info stuff because it may and probably will contain stuff that isn’t or shouldn’t be on the system. We will have libs missing because when you apt-got something apt-get didn’t bother getting a lib file because dpkg told it it was on the system already, which it may very well have been when the sqf was created or on the machine that the sqf was created on, but it may not have been when the newer package was created or it may have vanished when another .sqf was removed. Can you see how things can get real ugly, real quick? If people have been injecting these sqfs and dpkg-files into their systems, then it can only get worse.

Dropping the dpkg.db and/or dpkg.info stuff into our config file doesn’t help here either.

What I propose to do is as follows:

I will endeavor to create a clear and concise guide that will help people create their own .sqf’s
A key part of this will be to create a sqf that will be what will now be called an “orphan app”.

An orphan app is an Application that contains all of the libraries required to let it run and it will also create the changes required to update the dpkg.db files, but it won’t actually touch the dpkg.db itself. The dpkg.info files can and should stay in the appropriate place. The dpkg.db files (in the orphan app) will only contain the details of the packages installed by dpkg for the application, so they will not be a complete dpkg.db, they will be a set of difference files, and of course they will have a unique filename as they will also live in /var/lib/dpkg.

Assuming the new package is going to be called 24-NextApp, I propose to use the physical
filenames /var/lib/dpkg/24-available.new and /var/lib/dpkg/24-status.new

I think I can create scripts that you will run as part of the sqf building/cleaning process that
will parse the /var/lib/dpkg/info directory and create these files.

Preliminary testing has shown that I can do it manually; now all that remains is to script it. This will not be a trivial or easy task, and it will take quite a bit of research, but I’m confident that I can do it.

I will also try to create a generic script that will allow you to inject a properly created Orphan App into your system and while doing so, it will also fixup your dpkg.db and dpkg.info files so that they will properly reflect the true state of dpkg on your system.

Now, for this to work, we must all use it and adhere to the rules that are still being defined and won’t be cast in stone until the scripts have been developed and it works.

From where we are with DevRel_03, I am confidant that I can make this auto-magically integrate itself into DevRel_03 without needing to go to another DevRel

I can already foresee one scenario that we will need to be somewhat careful of. This will only impact people who wish to make a generic orphan app. What will be required will be to ensure that the “base” dpkg.db is the “active” dpkg.db whenever a new orphan app is being made. The base dpkg.db will be the original dpkg.db that DevRel_03 (or whatever the distro is) was using. This should be a simple matter of keeping a “base” copy and “flipping” it before doing the install.

All up, it shouldn’t be to hard to do, but it won’t be an overnight thing.

Comments/Suggestions ?

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby saintless » Sat Sep 22, 2012 4:58 am

Hi, Brenton,

I think the only disadvantage of this type "orphan" squash files is the limited number of squash files Debian live can use. Even if you increase the number of loop devices and make them load on the fly in one point the /live folder could become real mess holding 50 - 60 squash files with smaller and bigger applications like Dillo, XBMC etc.

Do you think it will be possible and practical to create a script which will merge lets say 5 or 10 or even more "orphan" applications into single "orphan" squash file whit his own unique name of dpkg files?

Cheers, Toni
User avatar
saintless
 
Posts: 246
Joined: Sat Jul 14, 2012 7:01 am
Location: Bulgaria

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby KazzaMozz » Sat Sep 22, 2012 6:01 am

Hi Brenton & Toni

About now, I can hear you telling me that this won’t occur, because you were very careful to make sure that the files in /var/lib/dpkg were included in your .sqf, and you also made sure that the dpkg.info [/var/lib/dpkg/info/*] files were included also.

I know I did do this in one of th XBMC builds, swapping across files here and there I have missed 2 library files libgls The keep coming up as broken links. I'll post what the are. It still works fine as they aren't required for it to run but I know it's because I was a bit cocky and deleted them at one stage and forgot I had.
When I do the rebuild I now know exactly what steps I need and so the final will not have this issue.

Anyway I can see where it can potentially can get very ugly very quickly, so it does require some very strict guidelines in place to avoid people creating a nightmare for themselves.

Scenario 2) You park 25-FoxyNeatPackage for a while by making it a .noload
Then create 24-NextApp. It works great until you load 25-FoxyNeatPackage and use something like Synaptic to check on the status of the installed files, which shows you that half or all of the installed files for 24-NextApp are missing, even though it may still appear in the [Taskbar Menu] and it even works, so what is wrong?


This is exactly one scenario I managed to achieve re the SMPlayer. Mind you I only managed to do it the once :lol: :lol: and I think it was when I was messing around with a variety of sticks. I haven't been able to replicate it as yet (was very late one night) :lol: :lol:
Not showing in the Synaptic and blow me down it was in the menu and worked. So I agree we will have to be very concise.
The old Do's and DON'TS clauses so to speak. If people adhere to the rules (when they are in place) then all should be well. (hopefully)

Cheers
Kazza
User avatar
KazzaMozz
 
Posts: 332
Joined: Tue Aug 21, 2012 9:59 pm
Location: Australia

Re: dpkg = the unsung hero while apt-get receives all the gl

Postby jbv » Sat Sep 22, 2012 8:22 am

Hi Toni,

saintless wrote:I think the only disadvantage of this type "orphan" squash files is the limited number of squash files Debian live can use. Even if you increase the number of loop devices and make them load on the fly in one point the /live folder could become real mess holding 50 - 60 squash files with smaller and bigger applications ..[]..


Even if we ignore the Debian-Live limit on 8 squash-files, which I have a partial solution to for much later.
While it would be one disadvantage, I don't think if would be he only one.

FoxyRoxyLinux will not be a replacement for a real Linux HDD based OS.
That is not my goal. My goal is for a good, fast, stable, 100% Debian compliant OS that does not write back to the main OS media. This means that ideally, you would test a package to see that it works for you and does what you want, and then inject it into your particular build of FoxyRoxyLinux. This should help avoid fragmentation, derivatives and a lot of other things that seem to plague "live" systems. It may not be perfect, although It think it might come close. Worst case, in my opinion it stands a darn good chance of working better than anything else that is out there.

saintless wrote:Do you think it will be possible and practical to create a script which will merge lets say 5 or 10 or even more "orphan" applications into single "orphan" squash file whit his own unique name of dpkg files?


I can't see why you couldn't do this manually without any real issue.
A script would not really be required. It wouldn't take much to actually take a bunch of orphan packages and wrap them up into a single sqf. If you wanted to, you could then also merge all of the dpkg.db.new files for them into a single set of dpkg.db.new files.

Once I've got some of these scripts working and post them, I hope you will see what I mean.

My hope and plan is that the scripts I am working to create, will also contain template scripts for you to add into your own sqfs to ensure uniformity and compliance with what I hope will become the standard.

Cheers, Brenton
jbv
 
Posts: 600
Joined: Sat Jul 14, 2012 2:02 am
Location: Sydney, Australia

Next

Return to Dev Notes



Who is online

Users browsing this forum: No registered users and 1 guest

cron