I wanted to move a few of the programs I run into SMF, one of the aforementioned awesome features.  SMF is Sun’s replacement init process, which takes a more active approach to services than the traditional “tell it to start and hope it does.” It, and all the other new init replacements borrow a good bit from djb’s daemontools, which everyone seems to be realizing, ten years later, is actually a good idea.

SMF provides two primary benefits over traditional init script methods, in my mind, the first of which is process supervision. Process supervision means that when your app crashes, you don’t have to rely on monitoring scripts (or other people) to notice it’s gone, SMF notices immediately and attempts to restart the process. If it won’t stay started after a few tries, it’ll stop and switch the process to maintenance mode, meaning you get to fix it.

The second is its ability to break out instances of an application by essentially subclassing the main “default” instance and redefining configuration variables, such as which config file to read, where to keep the database, etc. This means you can, with a few commands, bring up another instance of apache for testing, or maybe a number of mongrels serving different apps, from the generic service definition.

Sun has a pretty good how-to online dealing with postgresql, which is helpful, but leaves out a few chunks. SMF expects you to provide it with start and stop methods, which is helpful if you just want to move a legacy service over to SMF, but some applications are simpler than that and work fine by just running them and sending them a kill signal when they need to go away. In this case, you can just give the command to launch the daemon as the start method, and fill the stop method with :kill. This will instruct SMF to send a kill signal to the process and children executed by the start method. Incidentially, if the app reloads configuration when sent -HUP, you can fill in :kill -HUP as the refresh method. This isn’t explicitly spelled out in any documentation I’ve read, but it’s used in a few places in the manifests that ship with Solaris.

Below is a manifest I whipped up for mt-daapd, which scans and shares out all of my spiffy tunes so itunes can play ‘em.

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<service_bundle type="manifest" name="mt-daapd">
    <service name="network/mt-daapd" type="service" version="1">
        <dependency name="network" grouping="require_all" restart_on="none" type="service">
            <service_fmri value="svc:/milestone/network:default"/>
        </dependency>
        <exec_method type="method" name="start" exec="/export/home/pjjw/opt/pkgs/mt-daapd/sbin/mt-daapd -y -c %{mt-daapd/conffile}" timeout_seconds="10" />
        <exec_method type="method" name="refresh" exec=":kill -HUP" timeout_seconds="10" />
        <exec_method type="method" name="stop" exec=":kill" timeout_seconds="10" />
        <instance name="default" enabled="false">
            <property_group name='config' type='application'>
                <propval name='conffile' type='astring' value='/export/home/pjjw/opt/pkgs/mt-daapd/etc/mt-daapd.conf' />
            </property_group>
        </instance>
        <stability value="Evolving"/>
        <template>
            <common_name>
                <loctext xml:lang="C">Firefly Media Server</loctext>
            </common_name>
            <documentation>
                <doc_link name="Firefly Media Server homepage" uri="http://fireflymediaserver.org"/>
            </documentation>
        </template>
    </service>
</service_bundle>

I run two instances of this app- one which shares out the music that i’ve sorted and tagged properly, and then another one that shares out stuff that’s just arrived into torrent directories or the like. Ignore the ridiculous location I’ve installed this app to and note the property group and properties. I’ve got two config files, one describing one configuration, and another describing the other. I can make the second instance from the first with the following commands:

svccfg -s network/mt-daapd add incoming
svccfg -s network/mt-daapd:incoming setprop config/conffile = astring: /export/home/pjjw/opt/pkgs/mt-daapd/etc/incoming.conf

..and there it is, a new instance of the server.

[00:28:05][pjjw@push:~]$ svcs mt-daapd
STATE          STIME    FMRI
online         Oct_17   svc:/network/mt-daapd:incoming
online         Oct_17   svc:/network/mt-daapd:default

I’d actually just been running this in screen for about 6 months and restarting it when it crashed. This is much nicer now.

This isn’t impossible to recover from- the only thing that got overwritten was, for the most part, LVM data, and it so happens that the boundaries of the volumes is more or less known (25% each), so recovery should be possible just by re-writing correct (or near-correct) headers setting the boundaries in the right spot. Problem with this approach is that you get one shot, and it’s not like we have another 12TB sitting around that we can copy the data to. Here’s where dm-mapper came in to save my ass.

Using dm-mapper’s snapshot target, you can create a working copy of a large device that you want to test a few smallish changes to- like, for example, rewriting the partition table and fscking.

The snapshot target actually uses two separate device-mapper types, snapshot-origin, which is the device that you want to create the snapshot of, and snapshot, which becomes the device that you’ll be able to make your ephemeral changes to. The snapshot device requires a snapshot-origin device as well as a device to hold the written changes. For this, it’s easy to use a file-backed loopback device.

First, we set up the snapshot-origin device:

DEVSIZE=`sudo /sbin/blockdev --getsize /dev/sdb`
sudo /sbin/dmsetup create rescue-base-real --table \
        "0 $DEVSIZE linear /dev/loop0 0"
sudo /sbin/dmsetup create rescue-base --table \
        "0 $DEVSIZE snapshot-origin "

Next, we make a file as large as we expect the changes to be and create a loopback device for it:

dd if=/dev/zero of=/tmp/backing-store bs=1048576 count=1024 # 1GB
# get name of first available loopback
BACKINGDEV=`sudo /sbin/losetup -f`
sudo /sbin/losetup -f /tmp/backing-store

Now, finally, create the snapshot device:

sudo /sbin/dmsetup create rescue-snap --table \
        "0 $DEVSIZE snapshot /dev/mapper/rescue-base $BACKINGDEV n 8"

The last two arguments to the above command instruct dm-mapper that this is a non-persistent device, and to use a chunk size of 8 for copy-on-write operations.

Now you’ve got yourself a device you can play with that won’t cause any permanent changes if you guess the extents incorrectly, or if a fsck does the wrong thing. As an example, it took me about three tries to get the extents set up correctly for maximum data recovery. If I’d been working with the bare device, this wouldn’t have ended as well.

INTERACTIVE INPUT EDITING AND HISTORY SUBSTITUTION
       The Python inteterpreter supports editing of the current input line and
       history substitution, similar to facilities found in the Korn shell and
       the  GNU  Bash shell.  However, rather than being implemented using the
       GNU Readline library, this Python interpreter  uses  the BSD  EditLine
       library editline(3) with a GNU Readline emulation layer.

       The  readline  module  provides the access to the EditLine library, but
       there are a few major differences compared to a traditional implementa-
       tion  using  the  Readline  library.   The command language used in the
       preference files is that of EditLine, as described in editrc(5) and not
       that   used  by  the  Readline  library. This  also  means that  the
       parse_and_bind() routines uses EditLine commands.  And  the preference
       file itself is ~/.editrc instead of ~/.inputrc.

       For  example,  the rlcompleter module, which defines a completion func-
       tion for the  readline  modules,  works correctly  with the  EditLine
       libraries, but needs to be initialized somewhat differently:

              import rlcompleter
              import readline
              readline.parse_and_bind ("bind ^I rl_complete")

       For vi mode, one needs:

              readline.parse_and_bind ("bind -v")

Well, that explains why Python doesn’t play nice with my readline stuff.

I’ve settled on naming my network “608lovesyou”, changing all my passwords on my machines, and opening up my LAN to the building. Here’s hoping there’s nobody malicious around (ha). Hopefully either Kamikaze will be finished soon or the OpenWRT people will fix the WDS problem (WDS is way out of my depth).

Also, Airtunes is totally rad.