how to un-shoot your foot 9

Posted by peter on September 29, 2008

Recently I shot myself in the foot pretty bad. We have a ~12TB data array that was set up as a raw LVM2 device- no partition table. There were some issues with one of the cluster members, so I went to rescue boot to attempt to correct it. Redhat’s rescue boot, in the name of spreading democracy, proactively offers to put a partition table on anything that doesn’t have one already, and I fat-fingered the key that made this happen right over the top of our array. Whoops.

This isn’t impossible to recover from- the only thing that got overwritten was, for the most part, LVM data, and it so happens that the boundaries of the volumes is more or less known (25% each), so recovery should be possible just by re-writing correct (or near-correct) headers setting the boundaries in the right spot. Problem with this approach is that you get one shot, and it’s not like we have another 12TB sitting around that we can copy the data to. Here’s where dm-mapper came in to save my ass.

Using dm-mapper’s snapshot target, you can create a working copy of a large device that you want to test a few smallish changes to- like, for example, rewriting the partition table and fscking.

The snapshot target actually uses two separate device-mapper types, snapshot-origin, which is the device that you want to create the snapshot of, and snapshot, which becomes the device that you’ll be able to make your ephemeral changes to. The snapshot device requires a snapshot-origin device as well as a device to hold the written changes. For this, it’s easy to use a file-backed loopback device.

First, we set up the snapshot-origin device:

DEVSIZE=`sudo /sbin/blockdev --getsize /dev/sdb`
sudo /sbin/dmsetup create rescue-base-real --table \
        "0 $DEVSIZE linear /dev/loop0 0"
sudo /sbin/dmsetup create rescue-base --table \
        "0 $DEVSIZE snapshot-origin "

Next, we make a file as large as we expect the changes to be and create a loopback device for it:

dd if=/dev/zero of=/tmp/backing-store bs=1048576 count=1024 # 1GB
# get name of first available loopback
BACKINGDEV=`sudo /sbin/losetup -f`
sudo /sbin/losetup -f /tmp/backing-store

Now, finally, create the snapshot device:

sudo /sbin/dmsetup create rescue-snap --table \
        "0 $DEVSIZE snapshot /dev/mapper/rescue-base $BACKINGDEV n 8"

The last two arguments to the above command instruct dm-mapper that this is a non-persistent device, and to use a chunk size of 8 for copy-on-write operations.

Now you’ve got yourself a device you can play with that won’t cause any permanent changes if you guess the extents incorrectly, or if a fsck does the wrong thing. As an example, it took me about three tries to get the extents set up correctly for maximum data recovery. If I’d been working with the bare device, this wouldn’t have ended as well.


Use this link to trackback from your own site.


Leave a response

  1. Anselmo Sun, 13 Dec 2015 19:09:26 UTC

    Could you publish a tutraiol detailing the technical aspects of this installation? I wonder what kind of controller you used? How would you implement a similar solution with a wireless LED controller?

  2. Thu, 28 Apr 2016 21:08:52 UTC

    Un exercice maîtrisé avec brio! Chaque pochette est comme une mini page. Tu as su brillamment exploiter les possibilités offertes par ce système pour mettre en page le quotidien.

  3. Sat, 30 Apr 2016 21:08:47 UTC

    Parabéns! Monica, é claro que o rebento vai poder escolher, tem é de ser blogger.De resto escolhe sobre o que é que escreve

  4. Sat, 30 Apr 2016 21:24:08 UTC

    Very good website you have here but I was wanting to know if you knew of any message boards that cover the same topics talked about in this article? I’d really like to be a part of online community where I can get opinions from other knowledgeable individuals that share the same interest. If you have any suggestions, please let me know. Thank you!

  5. Fri, 08 Jul 2016 10:31:08 UTC

    Buon pranzo caro Ludo. Come!!, dici sempre che abbiamo una grande squadra e poi scrivi ” con il materiale che avevano a disposizione”.Continuo a non ritenerlo un grande allenatore….giusto riconoscer che ha fatto un buon lavoro ma stop. Per me colpa della sua presunzione ha frenato tante volte il Napoli.

  6. Sun, 10 Jul 2016 10:50:30 UTC

    اولا اخي اشكرك كثيرا على هذا الشرح الوافي ولكن لدي بعض الملاحظات على المرحلة الاخيرة لأنني لم افهم ماحصل لماذا قام اتيسو بقتل صديقته وايضا هنالك تكملة للعبة شاهدتة على موقع يوتيوب وهو عند اكمال جميع مراحل اللعبة لم افهمه اطلاقا ارجو منك التوضيح ولك جزيل الشكر والتقدير

  7. Tue, 09 Aug 2016 07:24:06 UTC

    It seems too bad, but you've been trying to figure this all out for awhile now, haven't you? I do think it's going to be a loss for the crime-writing and reading community, but that's not to make you feel bad. I think your own writing must come first. I hope you'll log in on one of the other blogs we all love and let us now know how it's all going. And don't hesitate to start it all up again if you find yourself missing it. It could happen.

  8. karstadt kostenlos kreditkarte Wed, 19 Oct 2016 11:37:23 UTC

    que barbaridad y que irresponsables esperemos que haya justicia que horror lo que estarán pasando estas familias ahora lo siento mucho un gran abrazo para todas las familias

  9. http://www./ Fri, 04 Nov 2016 11:52:07 UTC

    Lecker Kirschen, ich hab leider nur noch ein paar am Baum hängen, der Rest schlummert auch schon in Gläsern vor sich hin!LG Andrea