snorp.net

One of the reasons I wanted to revive rcd was so that I could use it to play with some package management ideas I’ve been kicking around. One of these ideas is a way to reliably rollback changes made during an RPM transaction. That is, actually make RPM transactions transactional.

Recently, a colleague introduced me to device-mapper, a kernel system used for block device redirection. There is a really cool thing that uses it called dm-snapshot, which allows you to redirect all writes to a device into a separate device. What I would like to do is use this to store all of the changes made during an rpm transaction. I think it would just need a bit of patching so that it only stores the changes made by the rcd/rpm process (and children). If anything goes wrong, you can just trash the snapshot data and things are exactly as they were in the beginning. Of course, if it succeeds without problelms, you need to merge the snapshot changes into the original device. This is where things get fuzzy, as dm-snapshot does not have this ability. However, Mark McLoughlin has created a set of patches that add this feature as part of the Stateless Linux project. Sadly, the patches do not appear to be a high priority for the kernel guys right now, so I guess this approach will have to be put on hold.

In any case, a system for performing this rollback stuff would be ridiculously useful in general — not just for package management. It looks like it will be a little more than I can do by myself in a weekend hack, though, so hopefully someone else will carry the torch?

Red Carpet, that is. Yes, that Red Carpet. I’ve taken some time lately to give some love to our old friends rcd and rug (the original rug, not the rewritten one). First I got everything building on a modern distro (openSUSE 10.2). This took more effort than I thought it would, but eventually things worked well. After that, I set out to make rcd more usable with yum services. Here is a list of the main changes:

• Add native yum support. I removed the ‘helix’ service support and replaced it with something that understands yum metadata. This means you can just do ‘rug sa repo_url name’ for any yum service. I used the excellent yum parser that Tambet wrote for the libredcarpet backend of zmd to accomplish this.
• Remove channel subscriptions. Since yum services don’t provide multiple channels, subscriptions aren’t really necessary. They have been replaced with the ability to disable a service.
• Add sleep ability. One of the main complaints against rcd was that it used too much memory. This was mostly because over time the heap would become fragmented. The ’sleep’ feature avoids this by running the main rcd daemon only when necessary. After a period of inactivity (3 minutes by default), the main daemon replaces itself with a smaller daemon. This smaller daemon simply waits until a request comes in and launches the full daemon to respond.

With the above changes, rcd is once again a joy to use. I would like to get the GUI working again, but there is some kind of threading problem preventing it from running. I would also like to add ftp support, but that is not a top priority.

I know there are probably SUSE users reading this asking “Ok, sounds fine, but is it fast?”. While it may not be the fastest thing out there, I think you will be surprised at the results (I was). Here are a few simple benchmarks from normal usage scenarios:

First, lets look at the number of services I currently have added:

% rug sl            

#  | Service URI                                           | Name
---+-------------------------------------------------------+------------
1  | http://go-mono.com/download-stable/suse-102-i586?r... | mono
2  | http://software.opensuse.org/download/FATE/openSUS... | fate
3  | http://ftp.suse.com/pub/suse/update/10.2/?name=upd... | updates
4  | http://download.opensuse.org/distribution/10.2/rep... | suse-nonoss
5  | http://download.opensuse.org/distribution/10.2/rep... | suse-oss
6  | http://packman.inode.at/suse/10.2?remote_only=1;na... | packman
7  | http://software.opensuse.org/download/home:/cybero... | cyborg
8  | http://software.opensuse.org/download/X11:/XGL/ope... | xgl
9  | http://software.opensuse.org/download/Beagle/openS... | beagle
10 | http://software.opensuse.org/download/games:/actio... | games
11 | http://software.opensuse.org/download/Virtualizati... | virt
12 | http://software.opensuse.org/download/home:/kraxel... | kvm

So 12 services, and the package count is almost 21000. 22500 if you also count the ones in the rpm database. How long does it take to load all of those?

Cold filesystem cache, daemon is sleeping:

% time rug ping > /dev/null
rug ping  0.17s user 0.02s system 1% cpu 13.735 total

14 seconds to respond isn’t terrible, considering the cold filesystem cache. Now that the kernel has it cached, though, how long does it take?

Warm filesystem cache, daemon is sleeping:

I just tried out Avant Window Navigator for the first time, after seeing Neil’s latest blog entry. It’s pretty slick, and worth trying if you have xgl/aiglx/whatever. I also wrote a Banshee plugin which makes the current track cover appear in the dock.

Banshee playing “Stadium Arcadium”

You can get the plugin here. Just drop the dll into ~/.gnome2/banshee/plugins.

Monster Truck Lloyd

Originally uploaded by snorp.

Saw this while getting some food this morning. Only in Kansas…

Hi Steve,

Earlier today I read your “Thoughts on Music” post. Afterwards, my initial reaction was “That’s great! You get ‘em Steve!”. It’s no secret that DRM is broken by design, but it’s nice to see one of the biggest users of it say so. However, I was quickly reminded by a colleague that Apple hardly seems interested in the “everything works with everything” utopia you describe. One specific example is the iTunes music sharing feature. Soon after it was released, some smart people figured out how it worked and developed software to be compatible with it. This let people access their iTunes-shared music from devices or operating systems that didn’t have iTunes. Soon after, Apple implemented a mechanism which prevented non-iTunes clients from doing this. Why? It wasn’t because of the record companies. The music purchased on the iTunes Music Store was still protected by FairPlay, so they had no reason to complain. The only conceivable reason you’d have for doing this is to force consumers into vendor lock in. Well, it didn’t work. Some more smart people defeated your mechanism and once again people were playing their iTunes-shared music using whatever software they liked best (be it iTunes or otherwise). But that didn’t stop Apple from re-implementing a similar protection scheme again in iTunes 7. This time you even knifed some business partners in the process. This kind of behavior isn’t at all congruent with what you’re saying in your post. Have you changed your mind now? Will the next iTunes release remove this restriction? If your “Thoughts on Music” was sincere, I’d expect so.

I’m still working hard on vmx-manager, and I think it’s coming along pretty well. I spent a good chunk of last week writing my own code for creating VMware Virtual Disks. The VMDK specification is pretty straightforward, but implementing it proved to be more tedious than I expected (”the devil is in the details”). The end result seems to work pretty well, so now vmx-manager can create disks without relying on qemu or vmware-vdiskmanager. Soon I might try to add a couple more features in this area, such as the ability to grow an existing disk (which in theory should be easy — just add extents). I’ve made a screencast of the app as I put it through its paces, and you can find up-to-date screenshots here.

When I was writing the flat extent support (used for pre-allocated disks), I wanted to do something different than just writing a bunch of zeros out to a file (which is slow). It seemed to me that it should be possible to ask the filesystem to quickly give me a file of a specific size. I didn’t care what was in it, so it should be able to just find a bunch of unused sectors (or whatever) and mark them as mine, right? I was able to find no such feature in ext3 or Linux in general, and I guess the reason is probably due to security concerns. You obviously don’t want to give people a way to read deleted data. It would be nice, though, if the fs could mark the data in such a way that it would be zeros until you write to it. Maybe that’s just too expensive, I don’t know. Anyway, if anyone knows how I could accomplish such a thing, please let me know.

I’ve been working on an application recently which lets you create, configure, and run VMware virtual machines (it just forks out to VMware Player for the running part). It’s nearing usefulness now, so I thought I’d post some screenshots and stuff.

The main window

The configuration window

You need either qemu or vmware-vdiskmanager installed in order to create new hard disks, and it probably fails pretty badly currently if you don’t. If I get a chance, I may write my own stuff to create the hard disks (the format is a public spec, woo!). Anyway, you can get it from the vmx-manager module in gnome CVS.

I did a jack-o-lantern this week for the first time in probably 15 years. I am happy with the result.

Last weekend I spent some time porting Tangerine to Mac OS X. The actual music sharing daemon worked fine with no changes, except I had to modify the path to the xml database for the iTunes plugin. With that out of the way, I set out to create a native configuration GUI for it. I had never used Obj-C or Cocoa or anything like that before, so I thought it would be fun to learn all of that stuff. The Apple developer tools are pretty nice, and it wasn’t long before I had a semi-working preference pane. The last couple of nights I polished it up to the point where I think it’s releasable, so here we go.

Tangerine’s Preference Pane on Mac OS X

I’m not an expert on Apple interfaces, so I’d welcome input from someone who has experience with this. It looks ok to me, though. The “automatic” selection uses Spotlight to find all of your music and share it. This is the same thing it does on Linux and Windows with Beagle and Google Desktop, respectively.

Download Now!

My next post will be about my experience porting this app to Windows and Mac OS X, what they have that’s better/worse than Linux, etc.

Update: To use Tangerine you’ll need the Mono framework installed. Get it here.

The iPod situation is now mostly fixed. I was able to figure out how iTunes gets the iPod serial number (and other info) with the help of a USB monitoring tool and Matt Dharm (the usb-storage guy). After that, Aaron used the metric ton of SysInfo samples you guys sent in to get a serial number -> model number mapping.

To get info about (recent) device, iTunes requests an xml document from it over USB. I saw this in the USB trace I had, but the data was nowhere on the disk exposed by USB Mass Storage. Puzzling. Not really knowing much about USB MC, I enlisted the help of Matt, who obviously knows a lot more. He informed me that it was requesting the data using a special SCSI INQUIRY command, and I could probably use SG_IO to get it out. I ran sg_inq on my iPod with the right parameters, and out came the xml. Success! After that I wrote a hal method for libipoddevice to pull the info out (since you need to be root), and the rest was just parsing the xml and doing the serial -> model mapping.

Even though it was a fairly painful couple of days trying to fix this, I’m pretty happy with the result. The xml from the device includes some really tasty stuff, including information on the image formats for cover art and photos. We are using this data now in ipod-sharp instead of the static table we had before, which will be really great for maintenance. New iPods with previously unknown image formats will Just Work! There is also data on the video formats, which I will soon use to add video support to Dopi